Describe the bug
Nested floating-point elements use total-order comparison in Comet, where -0.0 and 0.0 are distinct. Spark uses ordering.equiv for nested elements, which checks numeric equality first and therefore treats both signed zeros as equal.
This affects:
arrays_overlap when the array elements are themselves arrays or structs containing floats.
- The nested fallback in
array_position.
The flat arrays_overlap path is intentionally different: Spark uses Double.equals there and distinguishes signed zeros.
Steps to reproduce
CREATE TABLE t(
a ARRAY<ARRAY<DOUBLE>>,
b ARRAY<ARRAY<DOUBLE>>,
v ARRAY<DOUBLE>
) USING parquet;
INSERT INTO t VALUES (
array(array(0.0D)),
array(array(-0.0D)),
array(-0.0D)
);
SELECT arrays_overlap(a, b), array_position(a, v) FROM t;
Spark returns true and 1; Comet returns false and 0.
Expected behavior
Nested arrays_overlap and array_position should treat -0.0 as equal to 0.0, matching Spark, while continuing to treat NaN as equal to NaN.
Additional context
Tracked separately from #5176 because the mismatch predates that refactor. Arrow's make_comparator uses total ordering; a potential fix is to normalize negative zero in nested float leaves before constructing the comparator.
Describe the bug
Nested floating-point elements use total-order comparison in Comet, where
-0.0and0.0are distinct. Spark usesordering.equivfor nested elements, which checks numeric equality first and therefore treats both signed zeros as equal.This affects:
arrays_overlapwhen the array elements are themselves arrays or structs containing floats.array_position.The flat
arrays_overlappath is intentionally different: Spark usesDouble.equalsthere and distinguishes signed zeros.Steps to reproduce
Spark returns
trueand1; Comet returnsfalseand0.Expected behavior
Nested
arrays_overlapandarray_positionshould treat-0.0as equal to0.0, matching Spark, while continuing to treat NaN as equal to NaN.Additional context
Tracked separately from #5176 because the mismatch predates that refactor. Arrow's
make_comparatoruses total ordering; a potential fix is to normalize negative zero in nested float leaves before constructing the comparator.