Skip to content

[Feature] Support Spark bitmap_bit_position #758

Description

@zhangxffff

Feature Category

SQL Function / Operator

Problem / Use Case

Apache Spark 3.5 provides bitmap_bit_position, but Bolt has no native Spark SQL implementation. Queries currently retain Spark's BitmapExpressionUtils.bitmapBitPosition invocation and fall back instead of running in Bolt.

Proposed Solution

Implement Spark-compatible bitmap_bit_position:

bitmap_bit_position(value: BIGINT) -> BIGINT

For Spark's 32,768-bit bucket size, behavior is:

value > 0  => (value - 1) % 32768
value <= 0 => (-value) % 32768

Add the Gluten/Substrait expression mapping and register the function for Spark-compatible Bolt builds. Preserve Spark null behavior and implicit casting to BIGINT.

References / Prior Art

  • Apache Spark 3.5 BitmapBitPosition
  • Apache Spark 3.5 BitmapExpressionUtils.bitmapBitPosition
  • Spark BitmapExpressionsQuerySuite case bitmap_bit_position

Acceptance Criteria

  • Add native Bolt tests for positive, zero, negative, boundary, null, and cast inputs.
  • Match Spark 3.5 results.
  • Execute the corresponding Gluten query with a native project transformer.

Importance

Medium

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions