Skip to content

Make edge-harvest consistency test deterministic#169

Draft
leynos wants to merge 1 commit into
mainfrom
fix-mutation-baseline
Draft

Make edge-harvest consistency test deterministic#169
leynos wants to merge 1 commit into
mainfrom
fix-mutation-baseline

Conversation

@leynos

@leynos leynos commented Jul 10, 2026

Copy link
Copy Markdown
Owner

Closes #155.

The first scheduled mutation-testing run failed its unmutated baseline
because hnsw::tests::edge_harvest::build_with_edges_has_consistent_count::case_1
passes under CI's cargo nextest (one process per test) but fails under
the plain cargo test that cargo-mutants uses. This PR removes the
test's dependence on process isolation so the baseline is stable.

Diagnosis

The test built the same seeded index twice on Rayon's shared global
pool and asserted the two harvested edge counts fell within a ±50%
tolerance. Insertion order — and therefore the HNSW graph structure and
the edges harvested — depends on Rayon scheduling, and the variance
grows with contention on the pool from concurrently running tests.
Under nextest each test owns its process, so the pool is uncontended
and the spread stays small. Under plain cargo test, tests share one
process; even the sibling rstest case alone was enough to breach the
tolerance:

  • Run alone, single-threaded: 10/10 passes.
  • Run with only its sibling case in the same process: 2 failures in 10
    runs, e.g. edge counts should be similar: 24 vs 10 (tolerance: 5).

So the failure is contention-dependent rather than seed- or
RNG-related, and the full 527-test binary that cargo-mutants runs makes
it far more likely still.

Fix

chutoro-core/src/hnsw/tests/edge_harvest/mod.rs
— each build now runs inside its own single-threaded Rayon pool
(ThreadPoolBuilder::num_threads(1) + pool.install). Insertions then
proceed in node order on one worker with a deterministic per-worker
RNG, so two same-seed builds must produce identical harvests. The
tolerance heuristic (including its coverage-instrumentation widening)
is replaced with exact equality of both the edge count and the full
EdgeHarvest, which is strictly stronger and independent of the test
runner's process model, scheduler behaviour, and machine load.

The genuinely parallel construction path remains exercised by
build_with_edges_returns_valid_edges, whose assertions are
scheduling-independent invariants, and by the coverage tests in
chutoro-core/src/hnsw/tests/edge_harvest/coverage.rs.

No caller-configuration change is needed: the workflow's
extra-args: "--all-features --test-workspace=true" is correct; only
the test was at fault.

Validation

  • cargo test --all-features --workspace (the cargo-mutants baseline
    configuration): three consecutive runs, all green, no failures.
  • make test (nextest, --all-features): 993 passed, 1 skipped.
  • make typecheck, make check-fmt, make markdownlint: green.
  • make lint-clippy: green.
  • make lint-whitaker with the locally installed Whitaker suite fails
    on no_unwrap_or_else_panic in
    chutoro-core/src/hnsw/tests/property/graph_topology_tests/tests.rs:273,
    a file untouched by this diff and unchanged since Implement CPU HNSW candidate-edge harvest topologies, fixtures, and tests #71. This appears
    to be a newer local suite than CI's pinned whitaker-installer
    0.2.5; reported here rather than fixed, as it is pre-existing.

`build_with_edges_has_consistent_count` compared two same-seed
parallel builds and asserted their harvested edge counts fell within a
tolerance. Insertion order on Rayon's shared global pool is perturbed
by whatever else the process is running, so the variance grows with
concurrently executing tests. Under `cargo nextest` (one process per
test) the pool is uncontended and the test passed; under plain
`cargo test`, as run by cargo-mutants for its unmutated baseline, the
sibling `rstest` case alone was enough to push the spread past the
tolerance (observed 24 vs 10 against a tolerance of 5, failing roughly
two runs in ten), blocking every scheduled mutation run.

Run each build inside its own single-threaded Rayon pool instead.
Insertions then proceed in node order on one worker with a
deterministic per-worker RNG, so two same-seed builds must produce
identical harvests. This replaces the tolerance heuristic (including
its coverage-instrumentation widening) with an exact equality
assertion that is independent of test-runner process model, scheduler
behaviour, and machine load. The parallel construction path remains
exercised by `build_with_edges_returns_valid_edges`, whose assertions
are scheduling-independent invariants.

Verified with three consecutive green `cargo test --all-features
--workspace` runs (the cargo-mutants baseline configuration) and a
green `make test` (nextest) run.

Closes #155.

@sourcery-ai sourcery-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry @leynos, you have reached your weekly rate limit of 500000 diff characters.

Please try again later or upgrade to continue using Sourcery

@coderabbitai

coderabbitai Bot commented Jul 10, 2026

Copy link
Copy Markdown
Contributor

Important

Review skipped

Draft detected.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Organization UI

Review profile: ASSERTIVE

Plan: Pro Plus

Run ID: b00c1a12-ed4e-46e7-acc9-623a3bfe0ed2

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch fix-mutation-baseline

Comment @coderabbitai help to get the list of available commands.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Test fails under plain cargo test, blocking the mutation-testing baseline

1 participant