Skip to content

[Bug] Spark float-to-integer casts return minimum at positive overflow boundary #809

Description

@yangzhg

Component Selection

  • Core Engine (Expression eval, Memory, Vector)
  • Connectors / File Formats (Hive, Parquet, etc.)
  • API / Bindings (Python, etc.)
  • Build
  • Other

Describe the Bug

In Spark-compatible mode, Bolt returns incorrect results when a
runtime-computed floating-point value is exactly the first value outside the
positive range of the destination integer type:

  • FLOAT 2^31 cast to INT
  • DOUBLE 2^63 cast to BIGINT

Java Spark saturates these values to the destination type's maximum value.
Bolt native instead returns the destination type's minimum value.

The expressions intentionally depend on id so Catalyst cannot constant-fold
the casts before native execution.

Reproduction Steps

Run with ANSI mode disabled:

SET spark.sql.ansi.enabled=false;

WITH src AS (
  SELECT id FROM range(0, 2)
)
SELECT
  id,
  CAST(
    CAST(
      CAST(id AS FLOAT) * CAST('2147483648' AS FLOAT)
      AS FLOAT
    )
    AS INT
  ) AS float_to_int,
  CAST(
    CAST(id AS DOUBLE) * CAST('9223372036854775808' AS DOUBLE)
    AS BIGINT
  ) AS double_to_bigint
FROM src
ORDER BY id;

Bolt native result:

id  float_to_int  double_to_bigint
0   0             0
1   -2147483648   -9223372036854775808

Java Spark result:

id  float_to_int  double_to_bigint
0   0             0
1   2147483647    9223372036854775807

Bolt Version / Commit ID

8040d32a87d951349544089134a66cf91e86e214

System Configuration

  • OS: Linux
  • Compiler: Clang 22 with libc++
  • Build Type: Release
  • CPU Arch: x86_64
  • Framework: Spark, comparing Bolt native execution with Java Spark execution

Expected Behavior

Bolt's Spark-compatible cast should match Java Spark:

id  float_to_int  double_to_bigint
0   0             0
1   2147483647    9223372036854775807

Additional context

The relevant path appears to be the floating-point-to-integral conversion in
bolt/expression/CastExpr-tpl.cpp.

For these destination types, converting
std::numeric_limits<ToType>::max() to the source floating-point type rounds
it up:

  • INT_MAX becomes FLOAT 2^31
  • INT64_MAX becomes DOUBLE 2^63

As a result, a strict comparison such as
from > std::numeric_limits<ToType>::max() does not identify the exact 2^N
boundary as overflow after the usual conversion. The subsequent out-of-range
C++ floating-point-to-integer conversion is undefined behavior; on this x86_64
build it produces the integer minimum value.

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions