SparkSQLCompare fails with Spark Connect sessions (pyspark.sql.connect.session.SparkSession)

[fdosani]

Summary

datacompy.spark.SparkSQLCompare raises an AssertionError when used with a Spark Connect session (pyspark.sql.connect.session.SparkSession). It only works with a classic JVM-backed pyspark.sql.SparkSession.

Steps to Reproduce

# spark is a pyspark.sql.connect.session.SparkSession (e.g. from Spark Connect)  
import datacompy  
  
comp = datacompy.spark.SparkSQLCompare(  
    spark, df1, df2, join_columns="id"  
)  

Error

AssertionError:  
  File ".../datacompy/spark.py", line 440, in _dataframe_merge  
    df1 = df1.withColumn("_merge_left", F.lit(True))  
  File ".../pyspark/sql/functions.py", line 95, in _invoke_function  
    assert SparkContext._active_spark_context is not None 

Root Cause

datacompy/spark.py imports from pyspark.sql.functions (the classic JVM-backed module), which internally asserts that a local SparkContext is active. Spark Connect sessions have no local SparkContext — they communicate with the Spark driver over a remote gRPC connection — so this assertion always fails.

Expected Behavior

SparkSQLCompare should work with both classic pyspark.sql.SparkSession and Spark Connect pyspark.sql.connect.session.SparkSession.