Feature Category
SQL Function / Operator
Problem / Use Case
Apache Spark 3.5 provides bitmap_count, but Bolt has no native Spark SQL implementation. Queries currently retain Spark's BitmapExpressionUtils.bitmapCount invocation and fall back instead of running in Bolt.
Proposed Solution
Implement Spark-compatible bitmap_count:
bitmap_count(bitmap: BINARY) -> BIGINT
The function should return the population count across every input byte and preserve Spark null and binary input-type behavior. It must accept output produced by bitmap_construct_agg and bitmap_or_agg.
Add the Gluten/Substrait expression mapping and register the function for Spark-compatible Bolt builds.
References / Prior Art
- Apache Spark 3.5
BitmapCount
- Apache Spark 3.5
BitmapExpressionUtils.bitmapCount
- Spark
BitmapExpressionsQuerySuite case bitmap_count
Acceptance Criteria
- Add native Bolt tests for empty binary input, all-zero/all-one bytes, mixed bits, null, and invalid input types.
- Match Spark 3.5 results.
- Execute the corresponding Gluten query with a native project transformer.
Importance
Medium
Feature Category
SQL Function / Operator
Problem / Use Case
Apache Spark 3.5 provides
bitmap_count, but Bolt has no native Spark SQL implementation. Queries currently retain Spark'sBitmapExpressionUtils.bitmapCountinvocation and fall back instead of running in Bolt.Proposed Solution
Implement Spark-compatible
bitmap_count:The function should return the population count across every input byte and preserve Spark null and binary input-type behavior. It must accept output produced by
bitmap_construct_aggandbitmap_or_agg.Add the Gluten/Substrait expression mapping and register the function for Spark-compatible Bolt builds.
References / Prior Art
BitmapCountBitmapExpressionUtils.bitmapCountBitmapExpressionsQuerySuitecasebitmap_countAcceptance Criteria
Importance
Medium