test: teach CometCastSuite about the codegen dispatch path - #5187
Draft
andygrove wants to merge 1 commit into
Draft
test: teach CometCastSuite about the codegen dispatch path#5187andygrove wants to merge 1 commit into
andygrove wants to merge 1 commit into
Conversation
`CometCast` mixes in `CodegenDispatchFallback`, so an `Incompatible` or `Unsupported` support level no longer takes the plan out of Comet. The suite still assumed the old compatible-or-fallback binary. - Replace the `hasIncompatibleType` boolean with a `CastExecution` expectation (`Native`, `Dispatched`, `SparkFallback`) and assert the declared path from the verbose extended explain, not just that the plan stayed in Comet. - Rewrite `all valid cast combinations covered` around those three outcomes: it now pins that no pair in the matrix leaves Comet, and rejects ignored tests instead of requiring them for every non-`Compatible` pair. - Un-ignore the eight stale tests that the dispatcher now handles.
andygrove
marked this pull request as draft
August 1, 2026 02:19
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
Closes #5186.
Rationale for this change
CometCastSuitewas written when a cast was either implemented natively or fell back to Sparkfor the whole plan.
CometCastnow mixes inCodegenDispatchFallback, so anIncompatibleorUnsupportedsupport level no longer means the plan leaves Comet -- the JVM codegen dispatcherruns Spark's own generated code for the cast inside the Comet pipeline, and the results are
Spark-compatible by construction.
Three assumptions in the suite were stale:
all valid cast combinations coveredrequired anignored test for every pair that is notCompatible, on the grounds that the test would otherwise fail. For a codegen-dispatched castthe test now passes, so the rule forced us to ignore working coverage.
ignored tests carried comments about wrong results or Arrow errors from the nativepath. Those casts are dispatched rather than executed natively now, and all of them produce
correct results today.
castTest'shasIncompatibleTypeflag meant "do not assert native execution", which cannotdistinguish "no Rust kernel but the plan stays in Comet" from "the plan falls back to Spark".
What changes are included in this PR?
A three-way
CastExecutionexpectation replaceshasIncompatibleType.Nativemeans Comet'sown kernel evaluates the cast,
Dispatchedmeans the JVM codegen dispatcher compiles Spark'sCast.doGenCodeinto the pipeline, andSparkFallbackmeans the plan leaves Comet. The first twoboth assert a fully native plan;
SparkFallbackonly compares answers, as before.castTestnow also pins which of the first two happened, rather than only that the plan stayedin Comet. It enables
explain.codegen.enabledplus the verbose explain format soCometExecRule'sJVM codegen dispatcher: <names>roll-up is visible, then asserts the cast is(or is not) named there. That assertion has teeth in both directions -- flipping a
Nativetestto
Dispatchedlocally fails with "Expected the cast to be routed through the JVM codegendispatcher, but the plan does not report it".
Only the non-ANSI
cast()is checked that precisely. The ANSI block enablesallowIncompatible,which pushes an
Incompatiblecast back onto the native path, so there the assertion stays at"the plan is fully Comet".
The
SparkFallbackcases in this suite are all about the input type rather than the cast: thebyte/short tests when
COMET_PARQUET_UNSIGNED_SMALL_INT_CHECKis on, and thecast ArrayType to StringTypeelement types thatCometScanTypeCheckerrejects. Both now say so by name(
smallIntScanExecution,scanExecution) instead of via a boolean calledhasIncompatibleType.all valid cast combinations coveredis rebuilt around the three outcomes. It computes theexpected path per pair from
CometCast.isSupportedplusCometBatchKernelCodegen .isSupportedDataType, and now asserts a genuinely new invariant: no pair in the matrix resolves toSparkFallback-- every cast betweenCometCast.supportedTypesstays inside Comet, eithernatively or through the dispatcher. Ignored tests are then rejected outright, with an
outcome-specific explanation, instead of being required for every non-
Compatiblepair.Eight stale
ignores are removed. All eight pass:cast FloatType to DecimalType(10,2)andcast DoubleType to DecimalType(10,2)--Incompatiblefor rounding differences (Make cast from float/double to decimal compatible with Spark #1371), so the non-ANSI cast dispatches. The existing
- allow incompatvariants still cover the native kernel, and are now the demonstration of theNative/Dispatcheddistinction rather than a workaround for an ignored test.cast TimestampType to BooleanType / ByteType / ShortType / IntegerType / FloatType / DoubleType / DecimalType(10,2)--Unsupported(Implement Spark-compatible CAST from timestamp to numeric types #352, Cast timestamp to decimal is unsupported #1280), so they dispatch. Spark's ANSI and TRY modesreject timestamp -> boolean at analysis time (
Cast.canAnsiCast), so that one keepstestAnsi = false, testTry = false; the numeric targets are allowed in ANSI/TRY and exerciseboth, including the overflow errors Spark raises, which now come from Spark's own code running
in-pipeline.
The stale issue links and "expected X, actual null" notes are replaced with a description of why
there is no native kernel.
How are these changes tested?
170/171 and 171/172 pass respectively. The one failure,
cast StringType to TimestampType, ispre-existing and unrelated: it fails identically on
mainin this environment because Spark'srow-at-a-time and Comet's vectorized execution report different offending values from the fuzz
data in the ANSI error message. Verified by stashing this change and re-running that test alone.
Note on ordering: #5185 ignores
cast BooleanType to DecimalType(10,2)purely to satisfy the oldmatrix rule. If that PR lands first, the ignore has to become
expectedExecution = CastExecution.Dispatchedon rebase; the matrix test here will say so.