Feature Category
SQL Function / Operator
Problem / Use Case
Apache Spark 3.5 provides bitmap_bucket_number, but Bolt has no native Spark SQL implementation. Gluten sees Spark's BitmapExpressionUtils.bitmapBucketNumber invocation and falls back instead of running the expression in Bolt.
Proposed Solution
Implement Spark-compatible bitmap_bucket_number:
bitmap_bucket_number(value: BIGINT) -> BIGINT
For Spark's 32,768-bit bucket size, behavior is:
value > 0 => 1 + (value - 1) / 32768
value <= 0 => value / 32768
Add the Gluten/Substrait expression mapping and register the function for Spark-compatible Bolt builds. Preserve Spark null behavior and implicit casting to BIGINT.
References / Prior Art
- Apache Spark 3.5
BitmapBucketNumber
- Apache Spark 3.5
BitmapExpressionUtils.bitmapBucketNumber
- Spark
BitmapExpressionsQuerySuite case bitmap_bucket_number
Acceptance Criteria
- Add native Bolt tests for positive, zero, negative, bucket-boundary, null, and cast inputs.
- Match Spark 3.5 results.
- Execute the corresponding Gluten query with a native project transformer.
Importance
Medium
Feature Category
SQL Function / Operator
Problem / Use Case
Apache Spark 3.5 provides
bitmap_bucket_number, but Bolt has no native Spark SQL implementation. Gluten sees Spark'sBitmapExpressionUtils.bitmapBucketNumberinvocation and falls back instead of running the expression in Bolt.Proposed Solution
Implement Spark-compatible
bitmap_bucket_number:For Spark's 32,768-bit bucket size, behavior is:
Add the Gluten/Substrait expression mapping and register the function for Spark-compatible Bolt builds. Preserve Spark null behavior and implicit casting to
BIGINT.References / Prior Art
BitmapBucketNumberBitmapExpressionUtils.bitmapBucketNumberBitmapExpressionsQuerySuitecasebitmap_bucket_numberAcceptance Criteria
Importance
Medium