Skip to content

Fix Arrow narrow integer bloom filter checks - #10467

Open
wenn-id wants to merge 1 commit into
apache:mainfrom
wenn-id:fix/arrow-sbbf-coercion
Open

Fix Arrow narrow integer bloom filter checks#10467
wenn-id wants to merge 1 commit into
apache:mainfrom
wenn-id:fix/arrow-sbbf-coercion

Conversation

@wenn-id

@wenn-id wenn-id commented Jul 29, 2026

Copy link
Copy Markdown

Closes #5550.

Rationale for this change

Arrow Int8, Int16, UInt8, and UInt16 values are widened to Parquet INT32 when written. Checking their original narrow byte representation against the serialized bloom filter therefore produces false negatives.

What changes are included in this PR?

  • Add an Arrow-aware array kernel that widens narrow integer arrays to their Parquet physical representation before checking Sbbf.
  • Share the narrow-integer coercion helper with ArrowWriter, keeping writer and checker semantics aligned.
  • Preserve Arrow nulls in the returned BooleanArray and reject unsupported Arrow types.
  • Add round-trip tests using bloom filters written by ArrowWriter, serialized, reopened, and checked for present, absent, boundary, and null values.

Are there any user-facing changes?

Adds parquet::arrow::bloom_filter::check_bloom_filter for Int8, Int16, UInt8, and UInt16 arrays. Other Arrow-to-Parquet coercions remain unsupported and return an error.

Testing

  • cargo test -p parquet --lib --features arrow (1,237 passed)
  • cargo test -p parquet --lib bloom_filter --features arrow -- --nocapture (38 passed)
  • cargo clippy -p parquet --lib --all-features -- -D warnings
  • RUSTDOCFLAGS="-D warnings" cargo doc -p parquet --all-features --no-deps
  • cargo +stable fmt --all -- --check
  • git diff --check

AI usage

Hermes Agent was used to help analyze the existing code and draft the implementation and tests. I reviewed the complete diff, added independent negative-membership coverage, initialized the official test-data submodules, ran the full Parquet library test suite and quality gates above, and can maintain and debug the change.

@github-actions github-actions Bot added the parquet Changes to the parquet crate label Jul 29, 2026
@Jefffrey

Copy link
Copy Markdown
Contributor

i find it very curious that within 10 minutes of me updating the original issue it was picked up, then 12 minutes after a full PR is prepared and submitted 🤔

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

parquet Changes to the parquet crate

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Bloom filters for i8 and i16 always return false negatives

2 participants