feat: remove native cast from boolean to decimal - #5185
Open
andygrove wants to merge 1 commit into
Open
Conversation
Boolean -> Decimal is an edge case that nobody uses in practice, and the native implementation has to reproduce Spark's precision/scale and overflow semantics for it. That is not worth the complexity, so mark the cast unsupported in `CometCast` and let the `CodegenDispatchFallback` mixin route it through Spark's own generated code inside the Comet pipeline. The projection still runs natively; only the cast itself is evaluated by Spark's codegen. Adds SQL file tests covering non-ANSI and ANSI behavior, including the value-dependent overflow edge cases.
comphead
reviewed
Jul 31, 2026
| // `CometCast.isSupported` reports it as unsupported and the matrix-consistency test above | ||
| // requires this one to be ignored. Execution coverage, including the ANSI overflow edge cases, | ||
| // lives in `sql-tests/expressions/cast/cast_boolean_to_decimal{,_ansi}.sql`. | ||
| ignore("cast BooleanType to DecimalType(10,2)") { |
Contributor
There was a problem hiding this comment.
do we need this ignored test? are we planning to switch it back after future changes?
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
N/A
Rationale for this change
Comet has a hand-written native kernel for casting
booleantodecimal. This is not worthcarrying:
realistic query shape where a user needs
1/0as a scaled decimal rather than as aninteger or a double.
Spark's full
Decimal.toPrecisionsemantics: pick the unscaled value for the target scale,decide whether it fits the target precision, and then distinguish non-ANSI (return NULL),
ANSI (throw a Spark-compatible
NUMERIC_VALUE_OUT_OF_RANGEerror), andtry_cast(returnNULL) behavior. Overflow is value-dependent --
falsescales to0, which fits everydecimal type, while
trueoverflows anything that cannot hold1(decimal(1,1),decimal(2,2),decimal(38,38)). That is a lot of surface area, and a lot of ways tosilently diverge from Spark, for a cast with no real users.
Since
CometCastmixes inCodegenDispatchFallback, marking the cast unsupported does notpush the plan back to Spark. The JVM codegen dispatcher runs Spark's own generated code for
the cast inside the Comet pipeline, so the enclosing projection still executes natively and
the results are Spark-compatible by construction.
What changes are included in this PR?
CometCast.canCastFromBooleanno longer reportsDecimalTypeasCompatible, soboolean -> decimal is routed through the codegen dispatcher.
cast_boolean_to_decimalkernel, its dispatch arm incast_array, andits Criterion benchmark.
fact that the planner must not send it to the native path.
CometCastSuite: thecast BooleanType to DecimalType(10,2)test is ignored to satisfy thematrix-consistency assertion in
all valid cast combinations covered, and a new test pinsthe
Unsupportedsupport level. TheDecimalType(14,4)andDecimalType(30,0)tests stayactive and now exercise the dispatch path.
cast_boolean_to_decimal.sqlandcast_boolean_to_decimal_ansi.sqlcovering both modes. Non-ANSI covers the long fast path (
precision <= 18) and theBigDecimalpath (precision > 18), the tightest representable types, overflow returningNULL, literals,
try_cast, predicates, and aggregates. The ANSI file covers the edge cases:overflow throwing
NUMERIC_VALUE_OUT_OF_RANGEfordecimal(1,1),decimal(2,2), anddecimal(38,38); overflow raised from inside an aggregate and from inside a filterpredicate;
falseandNULLinputs not throwing even for those types; andtry_castsuppressing the error. The non-error queries double as sentinels -- they assert fully native
execution, so a silent fallback to Spark would fail the test rather than let the
expect_errorqueries pass vacuously.How are these changes tested?
New SQL file tests:
Both files pass, which also confirms the cast stays inside the Comet pipeline via the codegen
dispatcher rather than falling the plan back to Spark. Rust unit tests in
conversion_funcs::booleanalso pass.