IN LIST: add branchless filter for small primitive lists by geoffreyclaude · Pull Request #23014 · apache/datafusion

geoffreyclaude · 2026-06-18T07:31:47Z

Which issue does this PR close?

Part of Further improve performance of IN list evaluation #19241.
Stacked on #23311.
Next in stack: IN LIST: add keyed frozen set for large primitive lists #23015.
Extracted from Optimize IN performance with specialized implementations #19390.

Rationale for this change

For very small IN lists, building or probing a hash table can be more work than just comparing the input value with each constant.

For example, for x IN (10, 20, 30), the fast path can behave like:

x == 10 OR x == 20 OR x == 30

Because the list is tiny, those comparisons are cheap. The implementation stores the constants in a fixed-size array and checks them with a compact comparison chain.

“Branchless” here means the comparisons are combined without stopping at the first match. That can be faster for these small fixed-width lists because the CPU gets a predictable sequence of simple operations instead of hash-table setup and probe logic.

For primitive values that are not already plain unsigned integers, this PR keeps the logical Arrow type explicit and uses a matching same-width comparison representation only inside the branchless filter. For example, Float16 uses UInt16 storage, Float32 uses UInt32 storage, and TimestampNanosecond uses UInt64 storage. Decimal128 and IntervalMonthDayNano use their own 16-byte native representation. This preserves bit-pattern equality while relying on Arrow's native primitive compatibility rules: timestamp timezone metadata and Decimal128 precision/scale metadata may differ, while incompatible primitive representations remain rejected.

What changes are included in this PR?

Adds a const-generic BranchlessFilter for small primitive IN lists.
Adds thresholds for when this path is used:
- up to 16 values for 1-byte types
- up to 8 values for 2-byte types
- up to 32 values for 4-byte types
- up to 16 values for 8-byte types
- up to 4 values for 16-byte types
Keeps dispatch concrete and explicit in strategy.rs.
Maps each optimized logical type to the comparison representation used by the branchless filter:
- Int8 -> UInt8
- Int16, Float16 -> UInt16
- Int32, Float32, Date32, Time32 -> UInt32
- Int64, Float64, Date64, Time64, Timestamp, Duration -> UInt64
- Decimal128, IntervalMonthDayNano -> their native 16-byte representation
Leaves larger 1-byte and 2-byte lists on the existing bitmap filters.
Leaves larger 4-byte and 8-byte lists on the existing hash/generic paths.
Leaves wider primitive types such as Decimal256 and unsupported complex types on the generic path.
Keeps the same IN / NOT IN null behavior as the rest of the stack.
Adds focused coverage for branchless null handling, signed boundary values, slices, Float16/Float32/Float64 bit patterns, compatible timestamp/Decimal128 metadata, incompatible timestamp units, IntervalMonthDayNano values, and same-width wrong-type probe rejection.

Are these changes tested?

Yes.

cargo fmt --all -- --check
cargo test -p datafusion-physical-expr expressions::in_list --lib
cargo test -p datafusion-physical-expr --bench in_list_strategy --no-run
cargo clippy --all-targets --all-features -- -D warnings

Are there any user-facing changes?

No. This is an internal performance optimization only.

Local benchmark snapshot

Built and run with release-nonlto, filtered to the relevant small primitive-list rows:

cargo bench -p datafusion-physical-expr --profile release-nonlto --bench in_list_strategy -- <filter> --save-baseline <baseline>

Filters used: narrow_integer, primitive/i32/small_list, primitive/i64/small_list, f32/small_list, timestamp_ns/small_list, and interval_month_day_nano/small_list.

Method: directly compared Criterion's raw sample minima (min(time / iterations)) from sample.json. Lower is better; changes within +/-5% are treated as noise.

Compared baselines: #23311 -> #23014

Relevant scope: small primitive-list rows.

Summary: 39 relevant rows, 28 faster, 0 slower, 11 within +/-5%.

Largest relevant deltas:

Benchmark	Before	After	Change
`timestamp_ns/small_list/list=4/match=50%`	46.55 us	3.17 us	-93.2% (14.69x faster)
`f32/small_list/list=4/match=50%`	33.93 us	3.04 us	-91.0% (11.15x faster)
`primitive/i32/small_list/list=4/match=50%`	32.63 us	3.08 us	-90.5% (10.58x faster)
`primitive/i64/small_list/list=4/match=50%`	33.55 us	3.18 us	-90.5% (10.54x faster)
`timestamp_ns/small_list/list=4/match=0%`	19.57 us	3.18 us	-83.8% (6.16x faster)
`f32/small_list/list=4/match=0%`	18.14 us	3.05 us	-83.2% (5.95x faster)
`primitive/i32/small_list/list=4/match=0%`	17.00 us	3.04 us	-82.1% (5.59x faster)
`primitive/i64/small_list/list=4/match=0%`	17.12 us	3.22 us	-81.2% (5.31x faster)
`primitive/i32/small_list/list=16/match=50%/NOT_IN`	31.98 us	7.26 us	-77.3% (4.41x faster)
`nulls/primitive/i32/small_list/list=16/match=50%/nulls=20%`	29.35 us	7.32 us	-75.1% (4.01x faster)
`timestamp_ns/small_list/list=16/match=50%`	45.32 us	11.79 us	-74.0% (3.84x faster)
`nulls/primitive/i32/small_list/list=16/match=50%/nulls=50%`	25.89 us	7.31 us	-71.8% (3.54x faster)
`nulls/primitive/i32/small_list/list=16/match=50%/nulls=20%/NOT_IN`	26.05 us	7.42 us	-71.5% (3.51x faster)
`interval_month_day_nano/small_list/list=4/match=50%`	52.94 us	15.52 us	-70.7% (3.41x faster)
`f32/small_list/list=32/match=50%`	38.78 us	13.27 us	-65.8% (2.92x faster)
`primitive/i64/small_list/list=16/match=50%`	29.46 us	11.76 us	-60.1% (2.50x faster)

Full relevant table (39 rows)

Benchmark	Before	After	Change
`narrow_integer/u8/list=4/match=0%`	3.86 us	2.79 us	-27.8% (1.38x faster)
`narrow_integer/u8/list=4/match=50%`	3.84 us	2.78 us	-27.7% (1.38x faster)
`narrow_integer/u8/list=16/match=0%`	3.88 us	3.85 us	-0.8% (within noise)
`narrow_integer/u8/list=16/match=50%`	3.84 us	3.86 us	+0.5% (within noise)
`narrow_integer/i16/list=4/match=0%`	3.93 us	3.18 us	-19.1% (1.24x faster)
`narrow_integer/i16/list=4/match=50%`	3.92 us	3.16 us	-19.5% (1.24x faster)
`narrow_integer/i16/list=64/match=0%`	3.96 us	3.82 us	-3.5% (within noise)
`narrow_integer/i16/list=64/match=50%`	3.91 us	3.80 us	-2.9% (within noise)
`narrow_integer/i16/list=256/match=0%`	3.90 us	3.81 us	-2.5% (within noise)
`narrow_integer/i16/list=256/match=50%`	3.97 us	3.81 us	-4.1% (within noise)
`narrow_integer/f16/list=4/match=0%`	3.87 us	3.16 us	-18.5% (1.23x faster)
`narrow_integer/f16/list=4/match=50%`	3.94 us	3.15 us	-20.2% (1.25x faster)
`narrow_integer/f16/list=64/match=0%`	3.87 us	3.84 us	-0.6% (within noise)
`narrow_integer/f16/list=64/match=50%`	3.93 us	3.85 us	-1.9% (within noise)
`narrow_integer/f16/list=256/match=0%`	3.90 us	3.84 us	-1.5% (within noise)
`narrow_integer/f16/list=256/match=50%`	3.87 us	3.91 us	+1.2% (within noise)
`nulls/narrow_integer/u8/list=16/match=50%/nulls=20%`	3.92 us	4.02 us	+2.5% (within noise)
`primitive/i32/small_list/list=4/match=0%`	17.00 us	3.04 us	-82.1% (5.59x faster)
`primitive/i32/small_list/list=4/match=50%`	32.63 us	3.08 us	-90.5% (10.58x faster)
`primitive/i32/small_list/list=32/match=0%`	16.34 us	13.33 us	-18.5% (1.23x faster)
`primitive/i32/small_list/list=32/match=50%`	31.17 us	13.31 us	-57.3% (2.34x faster)
`primitive/i32/small_list/list=16/match=50%/NOT_IN`	31.98 us	7.26 us	-77.3% (4.41x faster)
`nulls/primitive/i32/small_list/list=16/match=50%/nulls=20%`	29.35 us	7.32 us	-75.1% (4.01x faster)
`nulls/primitive/i32/small_list/list=16/match=50%/nulls=20%/NOT_IN`	26.05 us	7.42 us	-71.5% (3.51x faster)
`nulls/primitive/i32/small_list/list=16/match=50%/nulls=50%`	25.89 us	7.31 us	-71.8% (3.54x faster)
`primitive/i64/small_list/list=4/match=0%`	17.12 us	3.22 us	-81.2% (5.31x faster)
`primitive/i64/small_list/list=4/match=50%`	33.55 us	3.18 us	-90.5% (10.54x faster)
`primitive/i64/small_list/list=16/match=0%`	16.34 us	11.93 us	-27.0% (1.37x faster)
`primitive/i64/small_list/list=16/match=50%`	29.46 us	11.76 us	-60.1% (2.50x faster)
`f32/small_list/list=4/match=0%`	18.14 us	3.05 us	-83.2% (5.95x faster)
`f32/small_list/list=4/match=50%`	33.93 us	3.04 us	-91.0% (11.15x faster)
`f32/small_list/list=32/match=0%`	22.05 us	13.43 us	-39.1% (1.64x faster)
`f32/small_list/list=32/match=50%`	38.78 us	13.27 us	-65.8% (2.92x faster)
`timestamp_ns/small_list/list=4/match=0%`	19.57 us	3.18 us	-83.8% (6.16x faster)
`timestamp_ns/small_list/list=4/match=50%`	46.55 us	3.17 us	-93.2% (14.69x faster)
`timestamp_ns/small_list/list=16/match=0%`	19.73 us	12.07 us	-38.8% (1.63x faster)
`timestamp_ns/small_list/list=16/match=50%`	45.32 us	11.79 us	-74.0% (3.84x faster)
`interval_month_day_nano/small_list/list=4/match=0%`	20.12 us	13.20 us	-34.4% (1.52x faster)
`interval_month_day_nano/small_list/list=4/match=50%`	52.94 us	15.52 us	-70.7% (3.41x faster)

alamb · 2026-07-09T21:28:23Z

For example, for x IN (10, 20, 30), the fast path can behave like:

x == 10 OR x == 20 OR x == 30

in fact DataFuson does this rewrite explicitly today:

DataFusion CLI v54.0.0
> create table t (v int) as values (1),(2), (3), (4);
0 row(s) fetched.
Elapsed 0.034 seconds.

> explain select * from t where v IN (10, 20, 30);
+---------------+-------------------------------+
| plan_type     | plan                          |
+---------------+-------------------------------+
| physical_plan | ┌───────────────────────────┐ |
|               | │         FilterExec        │ |
|               | │    --------------------   │ |
|               | │         predicate:        │ |
|               | │ v = 10 OR v = 20 OR v = 30│ |
|               | └─────────────┬─────────────┘ |
|               | ┌─────────────┴─────────────┐ |
|               | │       DataSourceExec      │ |
|               | │    --------------------   │ |
|               | │         bytes: 112        │ |
|               | │       format: memory      │ |
|               | │          rows: 1          │ |
|               | └───────────────────────────┘ |
|               |                               |
+---------------+-------------------------------+
1 row(s) fetched.
Elapsed 0.014 seconds.

alamb

I am pretty sure this code is unreachable via SQL / the external APIs as small IN LISTS get rewritten to an OR chain (see my other comment)

Thus if we are going to add this specialization, we should also turn off the IN LIST rewrite (and benchmark performance against the OR)

geoffreyclaude · 2026-07-10T09:26:22Z

I am pretty sure this code is unreachable via SQL / the external APIs as small IN LISTS get rewritten to an OR chain (see my other comment)

Thus if we are going to add this specialization, we should also turn off the IN LIST rewrite (and benchmark performance against the OR)

The OR rewrite threshold is currently set to 3 by THRESHOLD_INLINE_INLIST, so this code is reachable starting at size 4. The OR rewrite also doesn't trigger for dynamic filters, which directly create a standard IN filter.
I also ran some micro benchmarks locally to compare OR rewrite to the branchless case, and branchless is faster even for size 2 (1.5x for size 2, 2.5x for size 4.) I'll look into committing these somewhere clean.

  create table t (v int) as values (1),(2),(3),(4);
  explain select * from t where v IN (10, 20, 30, 40);

  Output:

  +---------------+-------------------------------+
  | plan_type     | plan                          |
  +---------------+-------------------------------+
  | physical_plan | ┌───────────────────────────┐ |
  |               | │         FilterExec        │ |
  |               | │    --------------------   │ |
  |               | │         predicate:        │ |
  |               | │   v IN (10, 20, 30, 40)   │ |
  |               | └─────────────┬─────────────┘ |
  |               | ┌─────────────┴─────────────┐ |
  |               | │       DataSourceExec      │ |
  |               | │    --------------------   │ |
  |               | │         bytes: 112        │ |
  |               | │       format: memory      │ |
  |               | │          rows: 1          │ |
  |               | └───────────────────────────┘ |
  |               |                               |
  +---------------+-------------------------------+

  -- Dynamic filters also reach IN-list evaluation directly.
  set datafusion.explain.analyze_level = 'dev';
  set datafusion.execution.target_partitions = 1;

  explain analyze
  with
    build(a,b,c) as (values (1,10,100),(2,20,200),(3,30,300)),
    probe(x,y,z) as (values (5,10,1000),(15,20,2000),(25,30,3000),(35,40,4000))
  select *
  from build
  join probe on build.b = probe.y
  order by x;

  -- Relevant output:
  -- FilterExec: DynamicFilter [ y@1 >= 10 AND y@1 <= 30 AND y@1 IN (SET) ([10, 20, 30]) ]

codecov-commenter · 2026-07-20T15:54:43Z

Codecov Report

❌ Patch coverage is 84.31953% with 53 lines in your changes missing coverage. Please review.
⚠️ Please upload report for BASE (main@f33dcec). Learn more about missing BASE report.
⚠️ Report is 14 commits behind head on main.

Files with missing lines	Patch %	Lines
...l-expr/src/expressions/in_list/primitive_filter.rs	83.67%	9 Missing and 31 partials ⚠️
.../physical-expr/src/expressions/in_list/strategy.rs	86.02%	5 Missing and 8 partials ⚠️

Additional details and impacted files

@@           Coverage Diff           @@
##             main   #23014   +/-   ##
=======================================
  Coverage        ?   80.74%           
=======================================
  Files           ?     1089           
  Lines           ?   369136           
  Branches        ?   369136           
=======================================
  Hits            ?   298058           
  Misses          ?    53292           
  Partials        ?    17786

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

alamb · 2026-07-20T21:16:07Z

I hope to review this very soon (like tomorrow)

geoffreyclaude · 2026-07-21T07:42:26Z

Thus if we are going to add this specialization, we should also turn off the IN LIST rewrite

@alamb I looked into this a bit more (especially the OR rewrite micro-benchmarks), and I think this should actually be done as a dedicated new PR once the full IN LIST optimization stack has landed.

This PR is still valuable by itself starting at lists of size 4 (and for dynamic filters), and removing the OR rewrite is not trivial to do (as the gains depend on the data type, eg utf-8 probably still benefits from OR rewrite...)

alamb · 2026-07-22T15:10:02Z

I will review this today

alamb · 2026-07-22T16:25:40Z

run benchmark in_list_strategy

adriangbot · 2026-07-22T16:27:45Z

🤖 Benchmark running (GKE) | trigger
Instance: c4a-highmem-16 (12 vCPU / 65 GiB) | Linux bench-c5048702402-1228-grpkv 6.12.85+ #1 SMP Mon May 11 08:17:35 UTC 2026 aarch64 GNU/Linux

CPU Details (lscpu)

Architecture:                            aarch64
CPU op-mode(s):                          64-bit
Byte Order:                              Little Endian
CPU(s):                                  16
On-line CPU(s) list:                     0-15
Vendor ID:                               ARM
Model name:                              Neoverse-V2
Model:                                   1
Thread(s) per core:                      1
Core(s) per cluster:                     16
Socket(s):                               -
Cluster(s):                              1
Stepping:                                r0p1
BogoMIPS:                                2000.00
Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache:                               1 MiB (16 instances)
L1i cache:                               1 MiB (16 instances)
L2 cache:                                32 MiB (16 instances)
L3 cache:                                80 MiB (1 instance)
NUMA node(s):                            1
NUMA node0 CPU(s):                       0-15
Vulnerability Gather data sampling:      Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit:             Not affected
Vulnerability L1tf:                      Not affected
Vulnerability Mds:                       Not affected
Vulnerability Meltdown:                  Not affected
Vulnerability Mmio stale data:           Not affected
Vulnerability Reg file data sampling:    Not affected
Vulnerability Retbleed:                  Not affected
Vulnerability Spec rstack overflow:      Not affected
Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
Vulnerability Spectre v2:                Mitigation; CSV2, BHB
Vulnerability Srbds:                     Not affected
Vulnerability Tsa:                       Not affected
Vulnerability Tsx async abort:           Not affected
Vulnerability Vmscape:                   Not affected

Comparing perf/in_list_branchless_filter (06f2b6b) to f33dcec (merge-base) diff using: in_list_strategy
Results will be posted here when complete

File an issue against this benchmark runner

alamb

Thank you @geoffreyclaude and @kosiew

My main concern about this Pr is that it generates a LOT of new code (by instantiating several hundred BranchlessFilter) -- by Claude Code's estiamte it is "~507 distinct BranchlessFilter<T, N> types, e" and by my measurement increases the code size by 4-6.7 percent

I think if we templated on the physical type (e.g. i32) rather than all the logical types we could reduce the number of distinct types substantially, but fundamentally creating many specialized code paths for the different sizes seems like a very expensive way to increase performance

Release binary size is 5MB (6.7%) larger and debug binary is 17MB (4%) larger

One way we could potentially reduce the code size would be to minimize the parts of the code that really need specializations. For example, I think the actual storage values could be a runtime value, and then just optimize the inner compare loop:

Compare the values 8 at a time or something using chunks_exact and then just generate a special comparison for 7,6,5,4,3,2,1 values to catch the last value.

That could also make this optimization more applicable as well, to any size where we wanted implement branchless (brute force comparison)

Details

Release

# release
cargo build --release --bin datafusion-cli
# debug
cargo build --bin datafusion-cli

Size of datafusion-cli on main

 74M	/Users/andrewlamb/Downloads/datafusion-main-base-in-list

Size on this branch

 79M	/Users/andrewlamb/Downloads/datafusion-in-list

Debug size

main

412M	target/debug/datafusion-cli

this branch

429M	target/debug/datafusion-cli

alamb · 2026-07-22T16:34:47Z

+branchless_filter_type!(UInt16Type, UInt16Type, BRANCHLESS_MAX_2B);
+branchless_filter_type!(Float16Type, UInt16Type, BRANCHLESS_MAX_2B);
+
+branchless_filter_type!(Int32Type, UInt32Type, BRANCHLESS_MAX_4B);


const BRANCHLESS_MAX_4B: usize = 32;

do I read the code above that this is creating 32 specializations (just) for UInt32 IN lists

Yes, that was correct for the previous version: UInt32 alone generated 32 copies of the full BranchlessFilter 😬.

The follow-up removes the list length from BranchlessFilter, so there is now only one full filter per type. Only the small comparison function still varies by length. I put the detailed size and performance results in my reply to the overall review.

alamb · 2026-07-22T16:38:34Z

+        };
+    }
+
+    match in_array.data_type() {


i found it somewhat hard to follow by reading the code to understand what cases would use the branchless filter and what would use the bitmap filter for small integer types -- because the cases that are covered by branchless filters are behind some macros and then the bitmap filtering happens only if there isn't a matchess branchless filter

I am not sure there is a concrete action to take from this observation

Good point. The latest refactor removed the per-length dispatch from this module, but it's still not super clear, I admit.

For the small types, Int8/UInt8 use direct comparisons for up to 16 non-null values, and Int16/UInt16/Float16 for up to 8. Longer lists fall back to the bitmap filter. I added a short comment to make that explicit...

alamb · 2026-07-22T16:45:10Z

+    };
+}
+
+branchless_filter_type!(Int8Type, UInt8Type, BRANCHLESS_MAX_1B);


Given we now have fast bitmap filter implementations for 8 and 16 bit integer types, what extra value do branchless filters give over those bitmaps?

UNless the performance win is compelling, I suggest we simply use the bitmap implementation for those types

Branchless is still faster (1.5x) for small lists (eg, 4 values). Since the latest commits resolves the compilation size issue, I think the branchless path is worth keeping for short 8 and 16 bit type lists, with the bitmap filter remaining the fallback for larger ones.

alamb · 2026-07-22T16:49:53Z

+        // The branchless filter reads the same Arrow value buffer as the
+        // comparison type. That is only valid when both native types have the
+        // same width, so catch any bad mapping here at compile time.
+        const _: () = assert!(


my reading of this loop is that it instantiates max_len copies of the branchless filter

Yes, exactly: this loop was the main source of the code-size increase.

After the follow-up it still creates one small comparison function per supported length, but it no longer creates max_len copies of the whole filter. The rest of the filter is shared now; I included the resulting size and benchmark numbers in my reply to the overall review.

adriangbot · 2026-07-22T17:18:48Z

🤖 Benchmark completed (GKE) | trigger

Instance: c4a-highmem-16 (12 vCPU / 65 GiB)

CPU Details (lscpu)

Architecture:                            aarch64
CPU op-mode(s):                          64-bit
Byte Order:                              Little Endian
CPU(s):                                  16
On-line CPU(s) list:                     0-15
Vendor ID:                               ARM
Model name:                              Neoverse-V2
Model:                                   1
Thread(s) per core:                      1
Core(s) per cluster:                     16
Socket(s):                               -
Cluster(s):                              1
Stepping:                                r0p1
BogoMIPS:                                2000.00
Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache:                               1 MiB (16 instances)
L1i cache:                               1 MiB (16 instances)
L2 cache:                                32 MiB (16 instances)
L3 cache:                                80 MiB (1 instance)
NUMA node(s):                            1
NUMA node0 CPU(s):                       0-15
Vulnerability Gather data sampling:      Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit:             Not affected
Vulnerability L1tf:                      Not affected
Vulnerability Mds:                       Not affected
Vulnerability Meltdown:                  Not affected
Vulnerability Mmio stale data:           Not affected
Vulnerability Reg file data sampling:    Not affected
Vulnerability Retbleed:                  Not affected
Vulnerability Spec rstack overflow:      Not affected
Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
Vulnerability Spectre v2:                Mitigation; CSV2, BHB
Vulnerability Srbds:                     Not affected
Vulnerability Tsa:                       Not affected
Vulnerability Tsx async abort:           Not affected
Vulnerability Vmscape:                   Not affected

Details

group                                                                HEAD                                   perf_in_list_branchless_filter
-----                                                                ----                                   ------------------------------
dictionary/i32/dict=10/list=16                                       1.00      7.6±0.01µs        ? ?/sec    1.02      7.7±0.01µs        ? ?/sec
dictionary/i32/dict=100/list=16                                      1.00      7.7±0.01µs        ? ?/sec    1.01      7.8±0.01µs        ? ?/sec
dictionary/i32/dict=100/list=16/NOT_IN                               1.00      7.7±0.02µs        ? ?/sec    1.02      7.9±0.01µs        ? ?/sec
dictionary/i32/dict=100/list=4                                       1.00      7.8±0.01µs        ? ?/sec    1.00      7.8±0.01µs        ? ?/sec
dictionary/i32/dict=100/list=64                                      1.00      7.8±0.01µs        ? ?/sec    1.00      7.7±0.00µs        ? ?/sec
dictionary/i32/dict=1000/list=16                                     1.03      9.1±0.01µs        ? ?/sec    1.00      8.8±0.01µs        ? ?/sec
dictionary/utf8_long/dict=100/list=16                                1.00      8.3±0.01µs        ? ?/sec    1.00      8.3±0.01µs        ? ?/sec
dictionary/utf8_short/dict=50/list=32                                1.00      8.1±0.01µs        ? ?/sec    1.00      8.1±0.01µs        ? ?/sec
dictionary/utf8_short/dict=50/list=8                                 1.00      8.0±0.01µs        ? ?/sec    1.00      8.0±0.01µs        ? ?/sec
dictionary/utf8_short/dict=500/list=20                               1.00      9.6±0.01µs        ? ?/sec    1.00      9.6±0.01µs        ? ?/sec
f32/large_list/list=64/match=0%                                      1.00     15.2±0.01µs        ? ?/sec    1.06     16.2±0.01µs        ? ?/sec
f32/large_list/list=64/match=50%                                     1.03     24.7±0.20µs        ? ?/sec    1.00     24.1±0.42µs        ? ?/sec
f32/small_list/list=32/match=0%                                      1.00     15.4±0.01µs        ? ?/sec    1.00     15.5±0.01µs        ? ?/sec
f32/small_list/list=32/match=50%                                     1.50     23.2±0.38µs        ? ?/sec    1.00     15.5±0.01µs        ? ?/sec
f32/small_list/list=4/match=0%                                       4.38     15.3±0.01µs        ? ?/sec    1.00      3.5±0.00µs        ? ?/sec
f32/small_list/list=4/match=50%                                      7.79     27.2±0.33µs        ? ?/sec    1.00      3.5±0.00µs        ? ?/sec
fixed_size_binary/fsb16/list=10000/match=0%                          1.01     25.6±0.37µs        ? ?/sec    1.00     25.3±0.05µs        ? ?/sec
fixed_size_binary/fsb16/list=10000/match=50%                         1.00     53.1±0.14µs        ? ?/sec    1.02     54.3±0.32µs        ? ?/sec
fixed_size_binary/fsb16/list=256/match=0%                            1.00     23.9±0.21µs        ? ?/sec    1.00     23.9±0.03µs        ? ?/sec
fixed_size_binary/fsb16/list=256/match=50%                           1.00     48.7±0.35µs        ? ?/sec    1.01     49.2±0.32µs        ? ?/sec
fixed_size_binary/fsb16/list=4/match=0%                              1.00     22.8±0.02µs        ? ?/sec    1.00     22.8±0.03µs        ? ?/sec
fixed_size_binary/fsb16/list=4/match=50%                             1.00     53.5±0.47µs        ? ?/sec    1.04     55.4±0.43µs        ? ?/sec
fixed_size_binary/fsb16/list=64/match=0%                             1.00     22.8±0.02µs        ? ?/sec    1.00     22.8±0.02µs        ? ?/sec
fixed_size_binary/fsb16/list=64/match=50%                            1.00     54.2±0.51µs        ? ?/sec    1.04     56.1±0.27µs        ? ?/sec
interval_month_day_nano/small_list/list=4/match=0%                                                          1.00     10.4±0.01µs        ? ?/sec
interval_month_day_nano/small_list/list=4/match=50%                                                         1.00     10.4±0.01µs        ? ?/sec
narrow_integer/f16/list=256/match=0%                                 1.00      5.2±0.00µs        ? ?/sec    1.00      5.2±0.01µs        ? ?/sec
narrow_integer/f16/list=256/match=50%                                1.00      5.2±0.01µs        ? ?/sec    1.00      5.2±0.00µs        ? ?/sec
narrow_integer/f16/list=4/match=0%                                   1.43      5.2±0.01µs        ? ?/sec    1.00      3.7±0.00µs        ? ?/sec
narrow_integer/f16/list=4/match=50%                                  1.43      5.2±0.01µs        ? ?/sec    1.00      3.7±0.00µs        ? ?/sec
narrow_integer/f16/list=64/match=0%                                  1.00      5.2±0.02µs        ? ?/sec    1.00      5.2±0.00µs        ? ?/sec
narrow_integer/f16/list=64/match=50%                                 1.00      5.2±0.01µs        ? ?/sec    1.00      5.2±0.00µs        ? ?/sec
narrow_integer/i16/list=256/match=0%                                 1.00      5.2±0.00µs        ? ?/sec    1.00      5.2±0.00µs        ? ?/sec
narrow_integer/i16/list=256/match=50%                                1.00      5.2±0.01µs        ? ?/sec    1.00      5.2±0.01µs        ? ?/sec
narrow_integer/i16/list=4/match=0%                                   1.43      5.2±0.00µs        ? ?/sec    1.00      3.7±0.00µs        ? ?/sec
narrow_integer/i16/list=4/match=50%                                  1.43      5.2±0.00µs        ? ?/sec    1.00      3.7±0.00µs        ? ?/sec
narrow_integer/i16/list=64/match=0%                                  1.00      5.2±0.00µs        ? ?/sec    1.00      5.2±0.00µs        ? ?/sec
narrow_integer/i16/list=64/match=50%                                 1.00      5.2±0.00µs        ? ?/sec    1.00      5.2±0.00µs        ? ?/sec
narrow_integer/u8/list=16/match=0%                                   1.12      5.2±0.00µs        ? ?/sec    1.00      4.7±0.00µs        ? ?/sec
narrow_integer/u8/list=16/match=50%                                  1.11      5.2±0.00µs        ? ?/sec    1.00      4.7±0.00µs        ? ?/sec
narrow_integer/u8/list=4/match=0%                                    1.56      5.2±0.00µs        ? ?/sec    1.00      3.3±0.01µs        ? ?/sec
narrow_integer/u8/list=4/match=50%                                   1.56      5.2±0.00µs        ? ?/sec    1.00      3.3±0.01µs        ? ?/sec
nulls/narrow_integer/u8/list=16/match=50%/nulls=20%                  1.11      5.3±0.00µs        ? ?/sec    1.00      4.8±0.00µs        ? ?/sec
nulls/primitive/i32/large_list/list=64/match=50%/nulls=20%           1.00     22.0±0.30µs        ? ?/sec    1.10     24.3±0.31µs        ? ?/sec
nulls/primitive/i32/small_list/list=16/match=50%/nulls=20%           2.54     24.0±0.10µs        ? ?/sec    1.00      9.4±0.01µs        ? ?/sec
nulls/primitive/i32/small_list/list=16/match=50%/nulls=20%/NOT_IN    2.37     22.6±0.10µs        ? ?/sec    1.00      9.5±0.01µs        ? ?/sec
nulls/primitive/i32/small_list/list=16/match=50%/nulls=50%           1.92     18.1±0.06µs        ? ?/sec    1.00      9.5±0.01µs        ? ?/sec
nulls/utf8/long_24b/list=16/match=50%/nulls=20%                      1.00     69.4±0.22µs        ? ?/sec    1.02     70.7±0.29µs        ? ?/sec
nulls/utf8/short_8b/list=16/match=50%/nulls=20%                      1.00     58.2±0.45µs        ? ?/sec    1.02     59.3±0.53µs        ? ?/sec
nulls/utf8view/long_24b/list=16/match=50%/nulls=20%                  1.00     78.9±0.67µs        ? ?/sec    1.10     86.7±0.18µs        ? ?/sec
nulls/utf8view/short_8b/list=16/match=50%/nulls=20%                  1.13     47.6±0.20µs        ? ?/sec    1.00     42.1±0.18µs        ? ?/sec
nulls/utf8view/short_8b/list=16/match=50%/nulls=20%/NOT_IN           1.14     47.7±0.21µs        ? ?/sec    1.00     41.7±0.15µs        ? ?/sec
nulls/utf8view/short_8b/list=16/match=50%/nulls=50%                  1.15     34.3±0.11µs        ? ?/sec    1.00     29.7±0.12µs        ? ?/sec
primitive/i32/large_list/list=256/match=0%                           1.00     12.2±0.11µs        ? ?/sec    1.04     12.6±0.02µs        ? ?/sec
primitive/i32/large_list/list=256/match=50%                          1.06     20.4±0.17µs        ? ?/sec    1.00     19.2±0.14µs        ? ?/sec
primitive/i32/large_list/list=64/match=0%                            1.06     12.4±0.05µs        ? ?/sec    1.00     11.8±0.01µs        ? ?/sec
primitive/i32/large_list/list=64/match=50%                           1.00     19.6±0.13µs        ? ?/sec    1.22     23.9±0.36µs        ? ?/sec
primitive/i32/small_list/list=16/match=50%/NOT_IN                    2.48     23.4±0.24µs        ? ?/sec    1.00      9.4±0.00µs        ? ?/sec
primitive/i32/small_list/list=32/match=0%                            1.00     12.3±0.01µs        ? ?/sec    1.25     15.5±0.01µs        ? ?/sec
primitive/i32/small_list/list=32/match=50%                           1.24     19.1±0.18µs        ? ?/sec    1.00     15.5±0.01µs        ? ?/sec
primitive/i32/small_list/list=4/match=0%                             3.56     12.4±0.01µs        ? ?/sec    1.00      3.5±0.00µs        ? ?/sec
primitive/i32/small_list/list=4/match=50%                            6.76     23.6±0.40µs        ? ?/sec    1.00      3.5±0.00µs        ? ?/sec
primitive/i64/large_list/list=128/match=0%                           1.05     12.7±0.02µs        ? ?/sec    1.00     12.1±0.06µs        ? ?/sec
primitive/i64/large_list/list=128/match=50%                          1.01     17.8±0.14µs        ? ?/sec    1.00     17.7±0.17µs        ? ?/sec
primitive/i64/large_list/list=32/match=0%                            1.00     12.2±0.06µs        ? ?/sec    1.00     12.2±0.03µs        ? ?/sec
primitive/i64/large_list/list=32/match=50%                           1.00     18.9±0.20µs        ? ?/sec    1.23     23.2±0.46µs        ? ?/sec
primitive/i64/small_list/list=16/match=0%                            1.00     12.2±0.02µs        ? ?/sec    1.32     16.0±0.01µs        ? ?/sec
primitive/i64/small_list/list=16/match=50%                           1.37     22.0±0.88µs        ? ?/sec    1.00     16.0±0.01µs        ? ?/sec
primitive/i64/small_list/list=4/match=0%                             2.81     12.2±0.01µs        ? ?/sec    1.00      4.4±0.01µs        ? ?/sec
primitive/i64/small_list/list=4/match=50%                            4.94     21.5±0.28µs        ? ?/sec    1.00      4.4±0.01µs        ? ?/sec
timestamp_ns/large_list/list=32/match=0%                             1.03     18.0±0.56µs        ? ?/sec    1.00     17.4±0.02µs        ? ?/sec
timestamp_ns/large_list/list=32/match=50%                            1.00     29.2±0.31µs        ? ?/sec    1.05     30.6±0.68µs        ? ?/sec
timestamp_ns/small_list/list=16/match=0%                             1.08     17.4±0.01µs        ? ?/sec    1.00     16.0±0.01µs        ? ?/sec
timestamp_ns/small_list/list=16/match=50%                            1.81     29.1±0.59µs        ? ?/sec    1.00     16.0±0.02µs        ? ?/sec
timestamp_ns/small_list/list=4/match=0%                              4.01     17.4±0.39µs        ? ?/sec    1.00      4.3±0.01µs        ? ?/sec
timestamp_ns/small_list/list=4/match=50%                             6.97     30.2±0.35µs        ? ?/sec    1.00      4.3±0.00µs        ? ?/sec
utf8/long_24b/list=256/match=0%                                      1.01     34.0±0.02µs        ? ?/sec    1.00     33.6±0.03µs        ? ?/sec
utf8/long_24b/list=256/match=50%                                     1.00     71.3±0.31µs        ? ?/sec    1.01     72.2±0.43µs        ? ?/sec
utf8/long_24b/list=4/match=0%                                        1.00     34.1±0.04µs        ? ?/sec    1.00     33.9±0.02µs        ? ?/sec
utf8/long_24b/list=4/match=50%                                       1.00     72.2±0.37µs        ? ?/sec    1.01     72.7±0.44µs        ? ?/sec
utf8/long_24b/list=64/match=0%                                       1.00     33.6±0.03µs        ? ?/sec    1.03     34.7±0.28µs        ? ?/sec
utf8/long_24b/list=64/match=50%                                      1.00     71.3±0.30µs        ? ?/sec    1.00     71.6±0.69µs        ? ?/sec
utf8/mixed_len/list=16/match=0%                                      1.00     36.6±0.10µs        ? ?/sec    1.02     37.2±0.10µs        ? ?/sec
utf8/mixed_len/list=16/match=50%                                     1.00    102.2±0.65µs        ? ?/sec    1.02    104.5±0.79µs        ? ?/sec
utf8/mixed_len/list=64/match=0%                                      1.00     37.4±0.09µs        ? ?/sec    1.03     38.5±0.12µs        ? ?/sec
utf8/mixed_len/list=64/match=50%                                     1.00    115.1±0.53µs        ? ?/sec    1.00    115.3±0.43µs        ? ?/sec
utf8/shared_prefix/pfx=12/list=32/match=50%                          1.00     71.6±0.49µs        ? ?/sec    1.01     72.5±0.43µs        ? ?/sec
utf8/short_8b/list=16/match=50%/NOT_IN                               1.00     61.2±0.46µs        ? ?/sec    1.03     62.9±0.22µs        ? ?/sec
utf8/short_8b/list=256/match=0%                                      1.00     26.2±0.04µs        ? ?/sec    1.00     26.2±0.02µs        ? ?/sec
utf8/short_8b/list=256/match=50%                                     1.00     61.1±0.33µs        ? ?/sec    1.05     64.3±0.39µs        ? ?/sec
utf8/short_8b/list=4/match=0%                                        1.00     26.3±0.07µs        ? ?/sec    1.00     26.4±0.05µs        ? ?/sec
utf8/short_8b/list=4/match=50%                                       1.00     62.1±0.34µs        ? ?/sec    1.04     64.5±0.17µs        ? ?/sec
utf8/short_8b/list=64/match=0%                                       1.00     26.3±0.02µs        ? ?/sec    1.00     26.3±0.03µs        ? ?/sec
utf8/short_8b/list=64/match=50%                                      1.00     61.3±0.52µs        ? ?/sec    1.03     63.3±0.86µs        ? ?/sec
utf8view/len_12b/list=16/match=0%                                    1.00     17.8±0.02µs        ? ?/sec    1.00     17.8±0.02µs        ? ?/sec
utf8view/len_12b/list=16/match=50%                                   1.00     44.5±0.16µs        ? ?/sec    1.03     46.0±0.31µs        ? ?/sec
utf8view/len_12b/list=64/match=0%                                    1.00     17.8±0.02µs        ? ?/sec    1.00     17.9±0.02µs        ? ?/sec
utf8view/len_12b/list=64/match=50%                                   1.00     44.0±0.28µs        ? ?/sec    1.02     45.1±0.23µs        ? ?/sec
utf8view/long_24b/list=16/match=0%                                   1.00     40.2±0.03µs        ? ?/sec    1.01     40.7±0.23µs        ? ?/sec
utf8view/long_24b/list=16/match=50%                                  1.02     88.2±0.36µs        ? ?/sec    1.00     86.6±0.17µs        ? ?/sec
utf8view/long_24b/list=256/match=0%                                  1.00     40.2±0.03µs        ? ?/sec    1.00     40.1±0.04µs        ? ?/sec
utf8view/long_24b/list=256/match=50%                                 1.01     85.2±0.18µs        ? ?/sec    1.00     84.6±0.18µs        ? ?/sec
utf8view/long_24b/list=4/match=0%                                    1.01     40.6±0.10µs        ? ?/sec    1.00     40.3±0.04µs        ? ?/sec
utf8view/long_24b/list=4/match=50%                                   1.00     86.2±0.19µs        ? ?/sec    1.01     86.8±0.23µs        ? ?/sec
utf8view/long_24b/list=64/match=0%                                   1.00     40.3±0.05µs        ? ?/sec    1.00     40.2±0.04µs        ? ?/sec
utf8view/long_24b/list=64/match=50%                                  1.01     84.5±0.13µs        ? ?/sec    1.00     83.7±0.18µs        ? ?/sec
utf8view/mixed_len/list=16/match=0%                                  1.09     32.3±0.15µs        ? ?/sec    1.00     29.7±0.08µs        ? ?/sec
utf8view/mixed_len/list=16/match=50%                                 1.13     82.0±0.63µs        ? ?/sec    1.00     72.8±0.52µs        ? ?/sec
utf8view/mixed_len/list=64/match=0%                                  1.00     32.6±0.15µs        ? ?/sec    1.03     33.4±0.07µs        ? ?/sec
utf8view/mixed_len/list=64/match=50%                                 1.08     91.1±0.50µs        ? ?/sec    1.00     84.3±0.52µs        ? ?/sec
utf8view/shared_prefix/pfx=12/list=32/match=0%                       1.00     42.3±0.07µs        ? ?/sec    1.02     42.9±0.14µs        ? ?/sec
utf8view/shared_prefix/pfx=12/list=32/match=50%                      1.03     85.5±0.40µs        ? ?/sec    1.00     82.7±0.27µs        ? ?/sec
utf8view/shared_prefix/pfx=16/list=64/match=0%                       1.00     40.5±0.03µs        ? ?/sec    1.00     40.4±0.22µs        ? ?/sec
utf8view/shared_prefix/pfx=16/list=64/match=50%                      1.00     83.4±0.18µs        ? ?/sec    1.01     84.0±0.28µs        ? ?/sec
utf8view/shared_prefix/pfx=8/list=16/match=0%                        1.00     30.0±0.06µs        ? ?/sec    1.00     29.9±0.05µs        ? ?/sec
utf8view/shared_prefix/pfx=8/list=16/match=50%                       1.03     71.6±0.32µs        ? ?/sec    1.00     69.8±0.12µs        ? ?/sec
utf8view/short_8b/list=16/match=0%                                   1.00     17.8±0.02µs        ? ?/sec    1.00     17.8±0.03µs        ? ?/sec
utf8view/short_8b/list=16/match=50%                                  1.00     41.9±0.17µs        ? ?/sec    1.01     42.3±0.28µs        ? ?/sec
utf8view/short_8b/list=256/match=0%                                  1.07     19.0±0.03µs        ? ?/sec    1.00     17.8±0.08µs        ? ?/sec
utf8view/short_8b/list=256/match=50%                                 1.00     40.6±0.18µs        ? ?/sec    1.02     41.5±0.18µs        ? ?/sec
utf8view/short_8b/list=4/match=0%                                    1.00     17.8±0.02µs        ? ?/sec    1.00     17.8±0.07µs        ? ?/sec
utf8view/short_8b/list=4/match=50%                                   1.00     44.0±0.25µs        ? ?/sec    1.02     45.0±0.35µs        ? ?/sec
utf8view/short_8b/list=64/match=0%                                   1.00     17.9±0.01µs        ? ?/sec    1.00     17.9±0.02µs        ? ?/sec
utf8view/short_8b/list=64/match=50%                                  1.00     39.6±0.19µs        ? ?/sec    1.02     40.4±0.38µs        ? ?/sec

Resource Usage

in_list_strategy — base (merge-base)

Metric	Value
Wall time	1485.3s
Peak memory	40.8 MiB
Avg memory	26.7 MiB
CPU user	1495.3s
CPU sys	1.3s
Peak spill	0 B

in_list_strategy — branch

Metric	Value
Wall time	1565.4s
Peak memory	42.0 MiB
Avg memory	27.2 MiB
CPU user	1531.0s
CPU sys	1.3s
Peak spill	0 B

File an issue against this benchmark runner

geoffreyclaude · 2026-07-23T08:44:02Z

One way we could potentially reduce the code size would be to minimize the parts of the code that really need specializations.

@alamb I agree, and I pushed a new remediation commit along those lines.

The IN-list values are now stored at runtime, and the filter chooses a comparison function for that list length when it is built. That function is called once per input array, while the rest of the filter is shared.

I didn't use chunks_exact exactly: keeping one small comparison function per list length was the simplest option that preserved performance. Looking at the release artifacts, the number of copies of BranchlessFilter::try_new and contains drops from 507 to 27. About 86 small comparison functions remain.

I also rebuilt release datafusion-cli for the previous and current versions on the same base, toolchain and arm64 machine:

executable: 82,316,496 → 77,773,840 bytes (-5.5%)
Mach-O __TEXT: 78,479,360 → 73,990,144 bytes (-5.7%)
Mach-O __text: 65,119,604 → 60,957,044 bytes (-6.4%)

That brings the release binary back to roughly the 74M main baseline from your measurement.

For performance, I compared the two versions across 28 focused Criterion cases. The worst median regression was 2.45%; the next two were 1.58% and 1.57%. None reached 3%, and many cases were flat or faster.

Thanks for pushing on this! I think this follow-up addresses the main concern without giving up the performance gains. WDYT?

alamb

Thank you @geoffreyclaude -- I went through this PR again and I think it looks great.

I double checked this the code size and it seems like it is much smaller (a few kb I think)

I personally think it would help to have some more slt coverage, but I will propose that as a follow on PR

Code size measurements

(venv) andrewlamb@Andrews-MacBook-Pro-3:/tmp/branchless_in$ du -s -h target/release/datafusion-cli target/debug/datafusion-cli
 74M	target/release/datafusion-cli
414M	target/debug/datafusion-cli


andrewlamb@Andrews-MacBook-Pro-3:/tmp/branchless_in_merge_base$ du -s -h target/release/datafusion-cli target/debug/datafusion-cli
 74M	target/release/datafusion-cli
413M	target/debug/datafusion-cli

alamb · 2026-07-24T20:17:13Z

+        let non_null_count = in_array.len() - in_array.null_count();
+        // `try_new` can be called on its own, so check the limit here too.
+        if non_null_count > T::MAX_LIST_LEN {
+            return Err(exec_datafusion_err!(


This is probably more correctly an internal error rather than an execution error as it isn't really based on user input (it would signal a bug in the code I think)

alamb · 2026-07-24T20:21:28Z

+        4 => choose!(0, 1, 2, 3, 4),
+        8 => choose!(0, 1, 2, 3, 4, 5, 6, 7, 8),
+        16 => choose!(0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16),
+        32 => choose!(


this is cool -- it makes much more sense to isolate the code repetition to just the comparison function (which is where it is most helpful)

alamb · 2026-07-26T01:29:56Z

🚀

alamb · 2026-07-26T01:30:21Z

Thanks again @geoffreyclaude and @kosiew for the iteration -- IN lists are getting quite cool

geoffreyclaude mentioned this pull request Jun 18, 2026

IN LIST: add keyed frozen set for large primitive lists #23015

Draft

github-actions Bot added the physical-expr Changes to the physical-expr crates label Jun 18, 2026

geoffreyclaude mentioned this pull request Jun 18, 2026

Further improve performance of IN list evaluation #19241

Open

11 tasks

geoffreyclaude force-pushed the perf/in_list_branchless_filter branch 3 times, most recently from 428e3cd to eae4046 Compare June 18, 2026 09:03

This was referenced Jun 18, 2026

IN LIST: reuse primitive filters for FixedSizeBinary #23018

Draft

Optimize IN performance with specialized implementations #19390

Closed

geoffreyclaude changed the title ~~Implement Branchless Filter for small primitive lists~~ IN LIST: add branchless filter for small primitive lists Jun 18, 2026

geoffreyclaude force-pushed the perf/in_list_branchless_filter branch from eae4046 to 3e3651b Compare June 19, 2026 05:11

github-actions Bot added the auto detected api change Auto detected API change label Jun 19, 2026

geoffreyclaude force-pushed the perf/in_list_branchless_filter branch from 3e3651b to 6a1869f Compare June 19, 2026 05:35

github-actions Bot removed the auto detected api change Auto detected API change label Jun 19, 2026

geoffreyclaude force-pushed the perf/in_list_branchless_filter branch 5 times, most recently from bf69ed2 to e040466 Compare June 24, 2026 20:49

geoffreyclaude mentioned this pull request Jun 25, 2026

IN LIST: reinterpret small-width types for bitmap filters #23013

Closed

geoffreyclaude force-pushed the perf/in_list_branchless_filter branch 8 times, most recently from 04ca744 to f89d3b6 Compare June 30, 2026 07:08

geoffreyclaude force-pushed the perf/in_list_branchless_filter branch from f89d3b6 to 409b0cb Compare July 2, 2026 08:54

geoffreyclaude mentioned this pull request Jul 3, 2026

IN LIST: add Float16 bitmap filter #23311

Merged

geoffreyclaude force-pushed the perf/in_list_branchless_filter branch from 409b0cb to 78e3631 Compare July 6, 2026 11:53

geoffreyclaude force-pushed the perf/in_list_branchless_filter branch from 6795d52 to 3abc50d Compare July 9, 2026 16:50

alamb added the performance Make DataFusion faster label Jul 9, 2026

alamb reviewed Jul 9, 2026

View reviewed changes

Use Arrow physical compatibility for branchless filters

d409e3e

Merge branch 'main' into perf/in_list_branchless_filter

06f2b6b

alamb reviewed Jul 22, 2026

View reviewed changes

Reduce branchless IN-list filter code size

5af3108

geoffreyclaude force-pushed the perf/in_list_branchless_filter branch 5 times, most recently from 5af3108 to 08277d2 Compare July 23, 2026 08:54

alamb approved these changes Jul 24, 2026

View reviewed changes

alamb mentioned this pull request Jul 24, 2026

test: add IN list slt coverage for temporal, Decimal128 and Interval types #23875

Open

alamb added this pull request to the merge queue Jul 26, 2026

Merged via the queue into apache:main with commit e3e2cb2 Jul 26, 2026
40 checks passed

geoffreyclaude mentioned this pull request Jul 26, 2026

IN LIST: isolate branchless filter implementation #23907

Draft

Uh oh!

Conversation

geoffreyclaude commented Jun 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Which issue does this PR close?

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

Local benchmark snapshot

Uh oh!

alamb commented Jul 9, 2026

Uh oh!

alamb left a comment

Choose a reason for hiding this comment

Uh oh!

geoffreyclaude commented Jul 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codecov-commenter commented Jul 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

alamb commented Jul 20, 2026

Uh oh!

geoffreyclaude commented Jul 21, 2026

Uh oh!

alamb commented Jul 22, 2026

Uh oh!

alamb commented Jul 22, 2026

Uh oh!

adriangbot commented Jul 22, 2026

Uh oh!

alamb left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

adriangbot commented Jul 22, 2026

Uh oh!

geoffreyclaude commented Jul 23, 2026

Uh oh!

alamb left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

alamb commented Jul 26, 2026

Uh oh!

alamb commented Jul 26, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

geoffreyclaude commented Jun 18, 2026 •

edited

Loading

geoffreyclaude commented Jul 10, 2026 •

edited

Loading

codecov-commenter commented Jul 20, 2026 •

edited

Loading