Skip to content

[Feature] Support Spark bitmap_bucket_number #759

Description

@zhangxffff

Feature Category

SQL Function / Operator

Problem / Use Case

Apache Spark 3.5 provides bitmap_bucket_number, but Bolt has no native Spark SQL implementation. Gluten sees Spark's BitmapExpressionUtils.bitmapBucketNumber invocation and falls back instead of running the expression in Bolt.

Proposed Solution

Implement Spark-compatible bitmap_bucket_number:

bitmap_bucket_number(value: BIGINT) -> BIGINT

For Spark's 32,768-bit bucket size, behavior is:

value > 0  => 1 + (value - 1) / 32768
value <= 0 => value / 32768

Add the Gluten/Substrait expression mapping and register the function for Spark-compatible Bolt builds. Preserve Spark null behavior and implicit casting to BIGINT.

References / Prior Art

  • Apache Spark 3.5 BitmapBucketNumber
  • Apache Spark 3.5 BitmapExpressionUtils.bitmapBucketNumber
  • Spark BitmapExpressionsQuerySuite case bitmap_bucket_number

Acceptance Criteria

  • Add native Bolt tests for positive, zero, negative, bucket-boundary, null, and cast inputs.
  • Match Spark 3.5 results.
  • Execute the corresponding Gluten query with a native project transformer.

Importance

Medium

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions