Skip to content

test(network): prevent probe-cap wedge in the dropping-carrier blocksync fuzz#419

Draft
p0mvn wants to merge 1 commit into
ironwood-mainfrom
fix/blocksync-fuzz-probe-wedge
Draft

test(network): prevent probe-cap wedge in the dropping-carrier blocksync fuzz#419
p0mvn wants to merge 1 commit into
ironwood-mainfrom
fix/blocksync-fuzz-probe-wedge

Conversation

@p0mvn

@p0mvn p0mvn commented Jul 2, 2026

Copy link
Copy Markdown

Motivation

The ironwood-main coverage job fails on zakura::testkit::blocksync_fuzz::tests::fuzz_reliability_discounts_dropping_carrier (introduced in #361), e.g. run 28624563853: sync stalled at 60 of 120 (state_samples=99, max_outstanding=8, rejects=1). It fails on CI coverage runs while passing locally (plain and under cargo llvm-cov).

Diagnosis

The stall is a probe-cap wedge, not slowness:

  1. The scenario's only source for blocks 61–120 is a carrier that drops 30% of requests; an unproven peer has a probe budget of initial_block_probe_requests = 1 (default).
  2. On schedulings where the carrier's first probe request is dropped, requests_without_block_progress hits the cap and the fill loop stops issuing to it (FillStop::InitialBlockProbeRequestCap) — nothing resets the streak (no view reset, no late body, the request was dropped).
  3. The liveness deadline then parks and disconnects the carrier — the rejects=1 in the panic message is exactly this trace_protocol_reject_liveness.
  4. Fuzz-harness peers never reconnect, so the sync wedges permanently at the fast peer's ceiling (height 60) until the 90s deadline expires.

Which scheduling class a run falls into depends on request interleaving (the per-peer RNG is deterministic, but which draw each request consumes is ordering-dependent) — CI's slow shared runners hit the wedge ordering; local runs don't (0/12 failures under CPU stress). Repro: raising drop_probability to 0.9 with the default one-probe budget reproduces the exact CI signature locally — sync stalled at 60 of 120 (..., rejects=1).

Solution

Give the test's unproven carrier a probe budget of 8 (P(8 consecutive drops at 30%) ≈ 7e-5), with a comment documenting the wedge. The probe-shed policy itself is covered by fuzz_silent_peer_request_cap; this test's purpose is the BBR reliability discount, and its assertion (min_reliability_permille < 1000) is unchanged and still engages.

Product-level observation (not changed here, flagging for @evan-forbes): a freshly connected peer whose single probe response is lost gets no further requests until liveness parks and disconnects it. In production this self-heals via cooldown + redial, but a default probe budget of 1 is aggressive against transient loss; a slightly larger default (2–3) may be worth considering.

Tests

  • fuzz_reliability_discounts_dropping_carrier passes plain and under cargo llvm-cov nextest (the CI coverage invocation); the full blocksync_fuzz group passes (21 tests); cargo fmt --all -- --check and cargo clippy -p zebra-network --lib --tests -- -D warnings pass.

AI Disclosure

  • AI tools were used: Claude for the diagnosis, repro, test fix, and PR description.

PR Checklist

  • The PR title follows conventional commits format: type(scope): description
  • The PR follows the contribution guidelines.
  • This change was discussed in an issue or with the team beforehand.
  • The solution is tested.
  • The documentation and changelogs are up to date.

@v12-auditor

v12-auditor Bot commented Jul 2, 2026

Copy link
Copy Markdown

Note

Complete: Audit complete. V12 did not find any issues that need review.

Open the full results here.

Analyzed one file, diff 1a11b74...b78a468.

@p0mvn
p0mvn marked this pull request as draft July 2, 2026 22:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant