org.apache.spark.sql.catalyst.expressions.BoundReference cannot be cast to class org.apache.spark.sql.ColumnarExpression

[kazuyukitanimura]

Describe the bug

In SparkSessionExtensionSuite, there are 4 test failures with org.apache.spark.sql.catalyst.expressions.BoundReference cannot be cast to class org.apache.spark.sql.ColumnarExpression
when spark.comet.enabled=true is set.

inject columnar AQE on
inject columnar AQE off
SPARK-39991: AQE should retain column statistics from completed query stages
reset column vectors

Steps to reproduce

SparkSessionExtensionSuite is not using SharedSparkSessionBase, so add spark.comet.enabled=true by changing code

Expected behavior

No response

Additional context

No response

[andygrove]

@kazuyukitanimura Is this issue still valid?

[kazuyukitanimura]

@andygrove
Yes, I think so

datafusion-comet/dev/diffs/3.4.3.diff

Line 981 in e7a3214

+ // https://github.com/apache/datafusion-comet/issues/1197