Skip to content

[Feature] Support Spark bitmap_count #760

Description

@zhangxffff

Feature Category

SQL Function / Operator

Problem / Use Case

Apache Spark 3.5 provides bitmap_count, but Bolt has no native Spark SQL implementation. Queries currently retain Spark's BitmapExpressionUtils.bitmapCount invocation and fall back instead of running in Bolt.

Proposed Solution

Implement Spark-compatible bitmap_count:

bitmap_count(bitmap: BINARY) -> BIGINT

The function should return the population count across every input byte and preserve Spark null and binary input-type behavior. It must accept output produced by bitmap_construct_agg and bitmap_or_agg.

Add the Gluten/Substrait expression mapping and register the function for Spark-compatible Bolt builds.

References / Prior Art

  • Apache Spark 3.5 BitmapCount
  • Apache Spark 3.5 BitmapExpressionUtils.bitmapCount
  • Spark BitmapExpressionsQuerySuite case bitmap_count

Acceptance Criteria

  • Add native Bolt tests for empty binary input, all-zero/all-one bytes, mixed bits, null, and invalid input types.
  • Match Spark 3.5 results.
  • Execute the corresponding Gluten query with a native project transformer.

Importance

Medium

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions