IN LIST: add keyed frozen set for large primitive lists

[geoffreyclaude]

Which issue does this PR close?

• Part of Further improve performance of IN list evaluation #19241.
• Stacked on #23014.
• Next in stack: #23016.
• Extracted from Optimize IN performance with specialized implementations #19390.

Rationale for this change

#23014 handles small primitive IN lists with direct comparisons. Larger lists still use the generic HashSet path, which stores Arrow array indices and invokes a generic comparator during lookup.

This PR adds a keyed immutable FrozenSet for fixed-width values. Each bucket stores two values directly; only buckets with additional collisions consult a native hashbrown::HashTable overflow table. The first real member is also the empty-slot sentinel, so the primary table stores values without Option wrappers.

This keeps the common probe small and predictable while retaining exact collision handling. Existing optimizer thresholds are unchanged; any threshold change belongs in a dedicated follow-up PR.

What changes are included in this PR?

• Adds a reusable keyed FrozenSet with two inline values per bucket and native HashTable overflow handling.
• Routes larger 4- and 8-byte primitive IN lists through PrimitiveFrozenFilter.
• Uses Arrow physical compatibility when accepting primitive input arrays.
• Supports u128 values for later representation-based optimizations without enabling general 128-bit primitive routing here.
• Covers empty sets, duplicates, the sentinel member, forced collisions, slices, dictionaries, nulls, and physical compatibility.

Are these changes tested?

Yes.

• cargo fmt --all
• cargo test -p datafusion-physical-expr --lib expressions::in_list
• cargo clippy --all-targets --all-features -- -D warnings
• ./dev/rust_lint.sh
• RUST_BACKTRACE=1 cargo test --profile ci --exclude datafusion-examples --exclude datafusion-benchmarks --exclude datafusion-cli --workspace --lib --tests --bins --features avro,json,backtrace,extended_tests,recursive_protection,parquet_encryption

Are there any user-facing changes?

No. This is an internal performance optimization only.

Local benchmark snapshot

Built and run with release-nonlto, filtered to the relevant large primitive-list rows:

cargo bench -p datafusion-physical-expr --profile release-nonlto --bench in_list_strategy -- <filter> --save-baseline <baseline>

Filters used: primitive/i32/large_list and primitive/i64/large_list.

Method: directly compared Criterion slope point estimates from isolated saved baselines. The expression is constructed outside the timed loop. Lower is better; changes within +/-5% are treated as noise.

Compared baselines: #23014 -> #23015

Relevant scope: the 12 large primitive-list rows shared with #23014.

Summary: 12 relevant rows, 12 faster, 0 slower, 0 within +/-5%. Geometric-mean time decreased by 51.6%.

A separate extension run covered eight added 1K/10K-list rows. All eight improved, with a 31.7% geometric-mean decrease.

Largest relevant deltas:

Benchmark	Before	After	Change
primitive/i64/large_list/list=128/match=50%	57.18 us	14.94 us	-73.9% (3.83x faster)
primitive/i32/large_list/list=33/match=50%	48.78 us	14.20 us	-70.9% (3.44x faster)
primitive/i64/large_list/list=17/match=50%	42.05 us	13.33 us	-68.3% (3.16x faster)
primitive/i64/large_list/list=32/match=50%	41.50 us	15.29 us	-63.1% (2.71x faster)
primitive/i32/large_list/list=64/match=50%	42.28 us	16.04 us	-62.1% (2.64x faster)
primitive/i32/large_list/list=256/match=50%	39.71 us	18.05 us	-54.6% (2.20x faster)

Full relevant table (12 rows)

Benchmark	Before	After	Change
primitive/i32/large_list/list=33/match=0%	23.82 us	12.38 us	-48.0% (1.92x faster)
primitive/i32/large_list/list=33/match=50%	48.78 us	14.20 us	-70.9% (3.44x faster)
primitive/i32/large_list/list=64/match=0%	23.91 us	13.08 us	-45.3% (1.83x faster)
primitive/i32/large_list/list=64/match=50%	42.28 us	16.04 us	-62.1% (2.64x faster)
primitive/i32/large_list/list=256/match=0%	18.63 us	13.70 us	-26.5% (1.36x faster)
primitive/i32/large_list/list=256/match=50%	39.71 us	18.05 us	-54.6% (2.20x faster)
primitive/i64/large_list/list=17/match=0%	17.59 us	15.79 us	-10.3% (1.11x faster)
primitive/i64/large_list/list=17/match=50%	42.05 us	13.33 us	-68.3% (3.16x faster)
primitive/i64/large_list/list=32/match=0%	19.63 us	17.84 us	-9.1% (1.10x faster)
primitive/i64/large_list/list=32/match=50%	41.50 us	15.29 us	-63.1% (2.71x faster)
primitive/i64/large_list/list=128/match=0%	24.63 us	15.62 us	-36.6% (1.58x faster)
primitive/i64/large_list/list=128/match=50%	57.18 us	14.94 us	-73.9% (3.83x faster)

For context, across the 12 cases shared with the original direct-probe implementation, this layout was 4.7% faster overall geometrically: miss-only cases were 11.6% slower, while 50%-hit cases were 18.7% faster.

[codecov-commenter]

Codecov Report

❌ Patch coverage is 84.16801% with 98 lines in your changes missing coverage. Please review.
⚠️ Please upload report for BASE (main@7ff7278). Learn more about missing BASE report.

Files with missing lines	Patch %	Lines
...l-expr/src/expressions/in_list/primitive_filter.rs	83.48%	14 Missing and 39 partials ⚠️
.../physical-expr/src/expressions/in_list/strategy.rs	75.86%	31 Missing and 11 partials ⚠️
...hysical-expr/src/expressions/in_list/frozen_set.rs	97.58%	1 Missing and 2 partials ⚠️

Additional details and impacted files

@@           Coverage Diff           @@
##             main   #23015   +/-   ##
=======================================
  Coverage        ?   80.70%           
=======================================
  Files           ?     1090           
  Lines           ?   369356           
  Branches        ?   369356           
=======================================
  Hits            ?   298090           
  Misses          ?    53480           
  Partials        ?    17786           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.