Component Selection
Describe the Bug
In Spark-compatible mode, Bolt returns incorrect results when a
runtime-computed floating-point value is exactly the first value outside the
positive range of the destination integer type:
FLOAT 2^31 cast to INT
DOUBLE 2^63 cast to BIGINT
Java Spark saturates these values to the destination type's maximum value.
Bolt native instead returns the destination type's minimum value.
The expressions intentionally depend on id so Catalyst cannot constant-fold
the casts before native execution.
Reproduction Steps
Run with ANSI mode disabled:
SET spark.sql.ansi.enabled=false;
WITH src AS (
SELECT id FROM range(0, 2)
)
SELECT
id,
CAST(
CAST(
CAST(id AS FLOAT) * CAST('2147483648' AS FLOAT)
AS FLOAT
)
AS INT
) AS float_to_int,
CAST(
CAST(id AS DOUBLE) * CAST('9223372036854775808' AS DOUBLE)
AS BIGINT
) AS double_to_bigint
FROM src
ORDER BY id;
Bolt native result:
id float_to_int double_to_bigint
0 0 0
1 -2147483648 -9223372036854775808
Java Spark result:
id float_to_int double_to_bigint
0 0 0
1 2147483647 9223372036854775807
Bolt Version / Commit ID
8040d32a87d951349544089134a66cf91e86e214
System Configuration
- OS: Linux
- Compiler: Clang 22 with libc++
- Build Type: Release
- CPU Arch: x86_64
- Framework: Spark, comparing Bolt native execution with Java Spark execution
Expected Behavior
Bolt's Spark-compatible cast should match Java Spark:
id float_to_int double_to_bigint
0 0 0
1 2147483647 9223372036854775807
Additional context
The relevant path appears to be the floating-point-to-integral conversion in
bolt/expression/CastExpr-tpl.cpp.
For these destination types, converting
std::numeric_limits<ToType>::max() to the source floating-point type rounds
it up:
INT_MAX becomes FLOAT 2^31
INT64_MAX becomes DOUBLE 2^63
As a result, a strict comparison such as
from > std::numeric_limits<ToType>::max() does not identify the exact 2^N
boundary as overflow after the usual conversion. The subsequent out-of-range
C++ floating-point-to-integer conversion is undefined behavior; on this x86_64
build it produces the integer minimum value.
Component Selection
Describe the Bug
In Spark-compatible mode, Bolt returns incorrect results when a
runtime-computed floating-point value is exactly the first value outside the
positive range of the destination integer type:
FLOAT 2^31cast toINTDOUBLE 2^63cast toBIGINTJava Spark saturates these values to the destination type's maximum value.
Bolt native instead returns the destination type's minimum value.
The expressions intentionally depend on
idso Catalyst cannot constant-foldthe casts before native execution.
Reproduction Steps
Run with ANSI mode disabled:
Bolt native result:
Java Spark result:
Bolt Version / Commit ID
8040d32a87d951349544089134a66cf91e86e214System Configuration
Expected Behavior
Bolt's Spark-compatible cast should match Java Spark:
Additional context
The relevant path appears to be the floating-point-to-integral conversion in
bolt/expression/CastExpr-tpl.cpp.For these destination types, converting
std::numeric_limits<ToType>::max()to the source floating-point type roundsit up:
INT_MAXbecomesFLOAT 2^31INT64_MAXbecomesDOUBLE 2^63As a result, a strict comparison such as
from > std::numeric_limits<ToType>::max()does not identify the exact2^Nboundary as overflow after the usual conversion. The subsequent out-of-range
C++ floating-point-to-integer conversion is undefined behavior; on this x86_64
build it produces the integer minimum value.