Commit e8a65f2
feat: add BuildHasher variants for hash_utils (#21820)
## Which issue does this PR close?
- Closes #21428.
## Rationale for this change
This PR adds `BuildHasher`-based variants for `hash_utils` so callers
can compute row hashes with a caller-provided hash builder instead of
always using DataFusion's default `RandomState`.
The main constraint is performance: `with_hashes` is a hot path,
especially for string, dictionary, and nested array hashing. A previous
version in #21429 caused measurable regressions in the default
`RandomState` path, for example `large_utf8: single, no nulls` regressed
from roughly `26.7us` to `36.3us`, and `large_utf8: multiple, no nulls`
from roughly `112us` to `127us`.
This version keeps the default path performance-oriented by avoiding a
fully generic `BuildHasher` rewrite of the existing hot loops.
## What changes are included in this PR?
This PR adds:
- `with_hashes_with_hasher`
- `create_hashes_with_hasher`
- custom-hasher implementations for primitive, string, binary,
byte-view, dictionary, and nested arrays
- tests covering custom hashers, multi-column hashing, and dictionary
equivalence
The implementation intentionally uses a hybrid design:
- Default `RandomState` leaf hot paths remain specialized.
- Custom `BuildHasher` leaf paths live separately in
`hash_utils/build_hasher.rs`.
- Nested/structural logic is shared through an internal child-hashing
adapter, so struct/list/map/union/run/dictionary behavior does not need
to be broadly duplicated.
The trade-off is that there is still some duplication for
primitive/string/binary leaf loops. That duplication is intentional:
those are the hottest loops, and keeping them separate prevents the
existing `RandomState` path from becoming generic over `BuildHasher` or
being perturbed by the custom-hasher implementation.
## Are these changes tested?
Yes.
---------
Co-authored-by: Dmitrii Blaginin <dmitrii@blaginin.me>
Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>1 parent 77b172e commit e8a65f2
2 files changed
Lines changed: 911 additions & 77 deletions
0 commit comments