Skip to content

Nested array comparison does not match Spark for signed zero #5191

Description

@peterxcli

Describe the bug

Nested floating-point elements use total-order comparison in Comet, where -0.0 and 0.0 are distinct. Spark uses ordering.equiv for nested elements, which checks numeric equality first and therefore treats both signed zeros as equal.

This affects:

  • arrays_overlap when the array elements are themselves arrays or structs containing floats.
  • The nested fallback in array_position.

The flat arrays_overlap path is intentionally different: Spark uses Double.equals there and distinguishes signed zeros.

Steps to reproduce

CREATE TABLE t(
  a ARRAY<ARRAY<DOUBLE>>,
  b ARRAY<ARRAY<DOUBLE>>,
  v ARRAY<DOUBLE>
) USING parquet;

INSERT INTO t VALUES (
  array(array(0.0D)),
  array(array(-0.0D)),
  array(-0.0D)
);

SELECT arrays_overlap(a, b), array_position(a, v) FROM t;

Spark returns true and 1; Comet returns false and 0.

Expected behavior

Nested arrays_overlap and array_position should treat -0.0 as equal to 0.0, matching Spark, while continuing to treat NaN as equal to NaN.

Additional context

Tracked separately from #5176 because the mismatch predates that refactor. Arrow's make_comparator uses total ordering; a potential fix is to normalize negative zero in nested float leaves before constructing the comparator.

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions