perf: Replace SipHash with foldhash in BoundedWindowAggExec - #23984
Conversation
Covers Linear mode (all partitions stay live until input end, state maps probed for every live partition on every batch) at two partition cardinalities, plus Sorted mode as a control where finished partitions are pruned eagerly.
PartitionWindowAggStates and PartitionBatches were using IndexMap's default SipHash hasher. Switching to foldhash yields a small but measurable performance improvement. This improves performance in linear mode for 10k partitions by ~6%.
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #23984 +/- ##
==========================================
- Coverage 80.75% 80.75% -0.01%
==========================================
Files 1096 1096
Lines 373588 373588
Branches 373588 373588
==========================================
- Hits 301687 301685 -2
+ Misses 53898 53897 -1
- Partials 18003 18006 +3 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
|
@viirya Let me know if you have a chance to take a look :) |
viirya
left a comment
There was a problem hiding this comment.
LGTM. Swapping the two IndexMap aliases (PartitionWindowAggStates, PartitionBatches) to foldhash RandomState is purely a hasher change — IndexMap preserves insertion order and partition-key equality is decided by PartitionKey's Eq, neither of which the hasher affects, so results are unchanged. Same idea as #23956.
One thing I checked since it's the easy way to break a hasher swap: the two construction sites in bounded_window_agg_exec.rs are correctly updated from IndexMap::new() (only valid for the default hasher) to IndexMap::default(), which works for any Default hasher and builds the same empty map. Good.
Benchmark is honest about the modest win and the linear-mode caveat (#23982). LGTM.
|
@viirya Thanks for the review! |
Which issue does this PR close?
BoundedWindowAggExecto use foldhash instead of SipHash #23983Rationale for this change
PartitionWindowAggStatesandPartitionBatcheswere using IndexMap's default SipHash hasher. Switching to foldhash yields a small but measurable performance improvement.Benchmark below. Note that the improvement is modest in part because linear mode is very inefficient when there are many partitions (#23982); I will send a PR for that separately.
Benchmarks:
What changes are included in this PR?
Are these changes tested?
Yes, covered by existing tests.
Are there any user-facing changes?
No functional change at all.