fix(test): tolerate boundary-tie in IVF distance_range assertions

[xuanyu-z]

Problem

test_build_ivf_pq::case_3 (DistanceType::Dot) fails on linux-arm with:

assertion `left == right` failed
  left: 4
  right: 5
at rust/lance/src/index/vector/ivf/v2.rs:4321

The test partitions top-k results around part_dist = dists[part_idx] using:

• left filter: strict d < part_dist, limit part_idx
• right filter: inclusive d >= part_dist, limit k - part_idx

It then asserts left_res.num_rows() == part_idx and that the row-id ordering matches the original top-k.

Root cause

When dists[part_idx - 1] == dists[part_idx], two vectors tie at the partition boundary:

• the strict-less left filter excludes both tied vectors → left_res has part_idx - 1 rows
• the inclusive right filter includes both tied vectors → one row shifts from left to right, and the trailing original row drops off right_res's limit
• the original row-id ordering of the tied vectors is determined by an internal tiebreaker, so the row-id assertions can fail even when the count matches

The failure is deterministic per architecture but architecture-dependent because ARM NEON uses fused multiply-add (FMA) for SIMD dot products by default, while x86 (without -mfma) does separate mul+add. FMA produces a single rounding for a*b + c; separate ops produce two. For Dot distance specifically — which has fewer transformations than L2 or Cosine — this ULP-level difference is enough to make two distances tie on ARM that don't tie on x86. Random vectors generated from StdRng::seed_from_u64([13; 32]) happen to hit such a tie at the partition boundary on ARM.

Fix

Detect the tie explicitly (dists[part_idx - 1] == part_dist) and weaken the count and row-id assertions for the affected positions only. The unconditional distance-value assertions immediately below (left_dists < part_dist, right_dists >= part_dist) still verify partition correctness in both branches.

Related

Similar test relaxations on this code area:

• fix: use relaxed recall assertion for HNSW-based indexes #6620 — relaxed recall assertion for HNSW-based indexes
• fix: increase nprobes in test_query_delta_indices for deterministic recall #6953 — increased nprobes for deterministic recall in test_query_delta_indices

Test plan

• cargo test -p lance --lib --release test_build_ivf_pq passes locally on x86 macOS (exercises the no-tie branch, all 12 cases including case_3)
• CI: linux-arm should now pass on test_build_ivf_pq::case_3 (the failing case from PR feat: support Utf8View and BinaryView in encoding and filter coercion #6985)

[claude]

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

[BubbleCal]

LGTM!

[codecov]

Codecov Report

❌ Patch coverage is 60.00000% with 6 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
rust/lance/src/index/vector/ivf/v2.rs	60.00%	6 Missing ⚠️

📢 Thoughts on this report? Let us know!