Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
73 commits
Select commit Hold shift + click to select a range
b70bcd2
GH-17211: refresh history for scalar_hash kernel
drin Jan 12, 2024
57e3ab8
GH-17211: refresh history for updates to key_hash
drin Jan 12, 2024
424049f
GH-17211: refresh history for new compute fn infra
drin Jan 12, 2024
e2b40ba
GH-17211: refresh history of scalar hash benchmark
drin Jan 12, 2024
64a8652
added KeyColumnArray alias
drin Oct 7, 2024
b9628b3
task: revert style and other minor changes
drin Oct 11, 2024
9afaf2a
task: reducing support to some nested types
drin Oct 11, 2024
6b7f309
minor: added type check for ListArray
drin Oct 11, 2024
0a787d6
Update the implementation to use column oriented hash-combine.
kszucs Dec 10, 2024
8762468
add hypothesis test case and fix list hashing
kszucs Dec 11, 2024
c017dd3
ensure to keep newly allocated arrays alive until hash gets computed
kszucs Dec 11, 2024
13b7c20
minor cleanup
kszucs Dec 11, 2024
d9ea389
wrong include
kszucs Dec 11, 2024
a9dd90e
add hash32 function
kszucs Dec 13, 2024
5d5ed28
add hash32 function and address review comments
kszucs Dec 13, 2024
88296ae
update testing suite
kszucs Dec 13, 2024
0384b79
generated struct tests
kszucs Dec 13, 2024
1c0ed44
prepare adding hash32 tests
kszucs Dec 13, 2024
14bed4e
add tests for hash32, dictionary encoded type, extension type
kszucs Dec 16, 2024
fed11a9
ensure that NULL and 0 hashes to different numbers
kszucs Dec 16, 2024
be6167e
fix the benchmarks
kszucs Dec 16, 2024
a69c258
update docs to highlight 0 return for NULL values
kszucs Dec 16, 2024
1609946
use resize instead of reserve
kszucs Dec 17, 2024
bd4698f
fix R compute function filtering test
kszucs Dec 17, 2024
b1c11c7
address review comments
kszucs Dec 18, 2024
3b3ab63
update boolean test case
kszucs Dec 18, 2024
97a6124
update exec checks
kszucs Dec 18, 2024
42ce3b3
use Hasher::kHashBatchTempStackUsage
kszucs Dec 18, 2024
53ceaca
ensure that slices properly work
kszucs Dec 18, 2024
54c1eeb
add tests for fixed_size_binary()
kszucs Dec 18, 2024
530c643
add rudimentary quality tests
kszucs Dec 18, 2024
387d9ad
test temporal types too
kszucs Dec 19, 2024
b4898e2
test temporal and decimal types
kszucs Dec 19, 2024
1fbe38a
add type matcher to explicitly raise NotImplemented for unsupported t…
kszucs Dec 19, 2024
a5883e8
update error message
kszucs Dec 19, 2024
03e1e43
update docs
kszucs Dec 19, 2024
624d6f8
set a tolerance for the randomly generated numbers
kszucs Dec 19, 2024
3ae5e7d
make msvc happy
kszucs Dec 20, 2024
6f13aa2
Rename DCHECK macros to ARROW_DCHECK
kszucs May 13, 2025
3ec331b
Missing blank lines
kszucs May 14, 2025
9c7a785
chore: move key_hash benchmarks to key_hash_benchmark.cc under compute
kszucs Jun 20, 2025
e814a04
Apply suggestions from code review
kszucs Jun 25, 2025
b8186da
use type singleton
kszucs Jun 25, 2025
34dbe42
run clang format on scalar_hash_benchmark.cc
kszucs Jun 25, 2025
5c9f3aa
run cmake format
kszucs Jun 25, 2025
05275f9
Address review comments and fix missing-declarations build failure
kszucs Jul 23, 2026
8e2797f
Fix unreachable slicing check and remove unused hashing_rng in benchm…
kszucs Jul 23, 2026
2d9d13b
Fix nested-type null and hash-corruption bugs in hash32/hash64
kszucs Jul 23, 2026
b99acbd
Remove dead list-like branches from ToColumnArray
kszucs Jul 23, 2026
f52d9a9
Fix independently-null nested fields hashing as non-zero garbage
kszucs Jul 23, 2026
46e8e87
Size key_hash_benchmark's TempVectorStack from kHashBatchTempStackUsage
kszucs Jul 23, 2026
4b2667e
Hoist KeyColumnArray construction out of key_hash_benchmark's timed loop
kszucs Jul 23, 2026
05a5f04
Fix GCC -Werror=range-loop-construct in scalar_hash_test.cc
kszucs Jul 23, 2026
c999521
Guard against a null values buffer in ToColumnArray's binary path
kszucs Jul 23, 2026
b07cb4f
Add R test asserting hash32/hash64 are discoverable via list_compute_…
kszucs Jul 23, 2026
108d14c
Avoid hashing unreferenced child data for sliced list/map arrays, fix…
kszucs Jul 24, 2026
6145f69
Add hypothesis tests asserting hash32/hash64 respect Array.slice()
kszucs Jul 24, 2026
4ad3ee1
Revert kAddend change to the shared HashIntImp; scope the null/0 fix …
kszucs Jul 24, 2026
82efa8b
Unwrap extension types in HashableMatcher so unsupported storage type…
kszucs Jul 24, 2026
0c0a7c1
Add test coverage for zero/null collision across byte widths and FIXE…
kszucs Jul 24, 2026
bb04597
Fix HashChild reusing a nested field's unshifted validity buffer
kszucs Jul 24, 2026
9821238
Avoid the bitmap copy in HashChild by omitting the validity buffer en…
kszucs Jul 24, 2026
6f0b569
Fix double blank lines around the new Hash Functions doc section
kszucs Jul 24, 2026
6fed285
Avoid hashing unreferenced struct nested-field data, avoid gtest macr…
kszucs Jul 24, 2026
04b6655
Hoist the per-column vector out of KeyHashIntegers benchmark loops
kszucs Jul 24, 2026
413a602
Fix Hash32/Hash64 API docstrings
kszucs Jul 24, 2026
42166a3
Fix values-child offset cancellation bug in list/map hashing
kszucs Jul 24, 2026
e64e0cd
Fix valid struct rows coincidentally colliding with the null hash sen…
kszucs Jul 24, 2026
a0cfdcb
Loosen RandomPrimitive's hash-quality tolerance for cross-platform RN…
kszucs Jul 24, 2026
7c66c36
Fix independently-null fields of multi-field structs not hashing to 0
kszucs Jul 27, 2026
eaa6101
Dedupe Hash64Struct{Small,Medium,Large}Strings benchmarks
kszucs Jul 27, 2026
7f07ae2
Make hash32/hash64 null-aware instead of reserving 0 for nulls
kszucs Jul 27, 2026
8a4a50c
Fix zero-width fixed_size_binary hashing, and simplify
kszucs Jul 28, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions cpp/src/arrow/CMakeLists.txt
Original file line number Diff line number Diff line change
Expand Up @@ -819,6 +819,7 @@ if(ARROW_COMPUTE)
compute/kernels/scalar_arithmetic.cc
compute/kernels/scalar_boolean.cc
compute/kernels/scalar_compare.cc
compute/kernels/scalar_hash.cc
compute/kernels/scalar_if_else.cc
compute/kernels/scalar_nested.cc
compute/kernels/scalar_random.cc
Expand Down
1 change: 1 addition & 0 deletions cpp/src/arrow/compute/CMakeLists.txt
Original file line number Diff line number Diff line change
Expand Up @@ -185,6 +185,7 @@ add_arrow_compute_test(row_test
arrow_compute_testing)

add_arrow_compute_benchmark(function_benchmark)
add_arrow_compute_benchmark(key_hash_benchmark)

add_subdirectory(kernels)

Expand Down
10 changes: 10 additions & 0 deletions cpp/src/arrow/compute/api_scalar.cc
Original file line number Diff line number Diff line change
Expand Up @@ -957,6 +957,16 @@ Result<Datum> MapLookup(const Datum& arg, MapLookupOptions options, ExecContext*
return CallFunction("map_lookup", {arg}, &options, ctx);
}

// ----------------------------------------------------------------------
// Hash functions
Result<Datum> Hash32(const Datum& input_array, ExecContext* ctx) {
return CallFunction("hash32", {input_array}, ctx);
}

Result<Datum> Hash64(const Datum& input_array, ExecContext* ctx) {
return CallFunction("hash64", {input_array}, ctx);
}

// ----------------------------------------------------------------------

} // namespace compute
Expand Down
41 changes: 41 additions & 0 deletions cpp/src/arrow/compute/api_scalar.h
Original file line number Diff line number Diff line change
Expand Up @@ -1807,5 +1807,46 @@ ARROW_EXPORT Result<Datum> NanosecondsBetween(const Datum& left, const Datum& ri
/// \note API not yet finalized
ARROW_EXPORT Result<Datum> MapLookup(const Datum& map, MapLookupOptions options,
ExecContext* ctx = NULLPTR);

/// \brief Construct a hash value for each row of the input.
///
/// The result has the same length and shape as the input (Array in, Array out;
/// ChunkedArray in, ChunkedArray out), but with element type UInt32. For a nested
/// input type (struct, list, map, etc.), each row's child values are combined into a
/// single hash for that row, recursively. A null input row produces a null output row;
/// within a struct, a null field makes that whole row null, while within a list or map a
/// null element does not (only the row's own validity matters there). Hash values are not
/// guaranteed to be stable across different versions of the library, and this function
/// does not currently take options, though these may be added in the future.
///
/// \param[in] input_array input data to hash
/// \param[in] ctx function execution context, optional
/// \return elementwise hash values
///
/// \since 26.0.0
/// \note API not yet finalized
ARROW_EXPORT
Result<Datum> Hash32(const Datum& input_array, ExecContext* ctx = NULLPTR);

/// \brief Construct a hash value for each row of the input.
///
/// The result has the same length and shape as the input (Array in, Array out;
/// ChunkedArray in, ChunkedArray out), but with element type UInt64. For a nested
/// input type (struct, list, map, etc.), each row's child values are combined into a
/// single hash for that row, recursively. A null input row produces a null output row;
/// within a struct, a null field makes that whole row null, while within a list or map a
/// null element does not (only the row's own validity matters there). Hash values are not
/// guaranteed to be stable across different versions of the library, and this function
/// does not currently take options, though these may be added in the future.
///
/// \param[in] input_array input data to hash
/// \param[in] ctx function execution context, optional
/// \return elementwise hash values
///
/// \since 26.0.0
/// \note API not yet finalized
ARROW_EXPORT
Result<Datum> Hash64(const Datum& input_array, ExecContext* ctx = NULLPTR);

} // namespace compute
} // namespace arrow
1 change: 1 addition & 0 deletions cpp/src/arrow/compute/initialize.cc
Original file line number Diff line number Diff line change
Expand Up @@ -31,6 +31,7 @@ Status RegisterComputeKernels() {
internal::RegisterScalarArithmetic(registry);
internal::RegisterScalarBoolean(registry);
internal::RegisterScalarComparison(registry);
internal::RegisterScalarHash(registry);
internal::RegisterScalarIfElse(registry);
internal::RegisterScalarNested(registry);
internal::RegisterScalarRandom(registry); // Nullary
Expand Down
2 changes: 2 additions & 0 deletions cpp/src/arrow/compute/kernels/CMakeLists.txt
Original file line number Diff line number Diff line change
Expand Up @@ -77,6 +77,7 @@ add_arrow_compute_test(scalar_math_test

add_arrow_compute_test(scalar_utility_test
SOURCES
scalar_hash_test.cc
scalar_random_test.cc
scalar_set_lookup_test.cc
scalar_validity_test.cc
Expand All @@ -89,6 +90,7 @@ add_arrow_benchmark(scalar_cast_benchmark PREFIX "arrow-compute")
add_arrow_compute_benchmark(scalar_arithmetic_benchmark)
add_arrow_compute_benchmark(scalar_boolean_benchmark)
add_arrow_compute_benchmark(scalar_compare_benchmark)
add_arrow_compute_benchmark(scalar_hash_benchmark)
add_arrow_compute_benchmark(scalar_if_else_benchmark)
add_arrow_compute_benchmark(scalar_list_benchmark)
add_arrow_compute_benchmark(scalar_random_benchmark)
Expand Down
Loading
Loading