test(network): prevent probe-cap wedge in the dropping-carrier blocksync fuzz#419
Draft
p0mvn wants to merge 1 commit into
Draft
test(network): prevent probe-cap wedge in the dropping-carrier blocksync fuzz#419p0mvn wants to merge 1 commit into
p0mvn wants to merge 1 commit into
Conversation
|
Note Complete: Audit complete. V12 did not find any issues that need review. Open the full results here. Analyzed one file, diff |
p0mvn
marked this pull request as draft
July 2, 2026 22:50
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
The
ironwood-maincoverage job fails onzakura::testkit::blocksync_fuzz::tests::fuzz_reliability_discounts_dropping_carrier(introduced in #361), e.g. run 28624563853:sync stalled at 60 of 120 (state_samples=99, max_outstanding=8, rejects=1). It fails on CI coverage runs while passing locally (plain and undercargo llvm-cov).Diagnosis
The stall is a probe-cap wedge, not slowness:
initial_block_probe_requests = 1(default).requests_without_block_progresshits the cap and the fill loop stops issuing to it (FillStop::InitialBlockProbeRequestCap) — nothing resets the streak (no view reset, no late body, the request was dropped).rejects=1in the panic message is exactly thistrace_protocol_reject_liveness.Which scheduling class a run falls into depends on request interleaving (the per-peer RNG is deterministic, but which draw each request consumes is ordering-dependent) — CI's slow shared runners hit the wedge ordering; local runs don't (0/12 failures under CPU stress). Repro: raising
drop_probabilityto 0.9 with the default one-probe budget reproduces the exact CI signature locally —sync stalled at 60 of 120 (..., rejects=1).Solution
Give the test's unproven carrier a probe budget of 8 (P(8 consecutive drops at 30%) ≈ 7e-5), with a comment documenting the wedge. The probe-shed policy itself is covered by
fuzz_silent_peer_request_cap; this test's purpose is the BBR reliability discount, and its assertion (min_reliability_permille < 1000) is unchanged and still engages.Product-level observation (not changed here, flagging for @evan-forbes): a freshly connected peer whose single probe response is lost gets no further requests until liveness parks and disconnects it. In production this self-heals via cooldown + redial, but a default probe budget of 1 is aggressive against transient loss; a slightly larger default (2–3) may be worth considering.
Tests
fuzz_reliability_discounts_dropping_carrierpasses plain and undercargo llvm-cov nextest(the CI coverage invocation); the fullblocksync_fuzzgroup passes (21 tests);cargo fmt --all -- --checkandcargo clippy -p zebra-network --lib --tests -- -D warningspass.AI Disclosure
PR Checklist
type(scope): description