test(zakura): make block-sync fuzzers deterministic#438
Draft
evan-forbes wants to merge 5 commits into
Draft
Conversation
The main-lane block-sync fuzz suite carried two CI flake sources:
- fuzz_reorg was an inverted xfail: it asserted the mock reorg scenario
*panics* ("should keep failing until the high-fidelity committer tier
lands"). Because the underlying scenario is multi_thread + real-clock, it
flaked in both directions and reddened ironwood-main whenever timing let
the mock reach the target anyway. Reduce it to a documented #[ignore]
no-op; keep its VerifiedReset/slow scaffolding under #[allow(dead_code)]
for the future Committer<MockVerifier> tier.
- Prune redundant timing-sensitive scenarios whose properties are already
covered deterministically in block_sync/tests.rs: delete
fuzz_silent_peer_request_cap (covered by the block_liveness_* tests) and
drop the flaky protocol_rejects==0 reaper assertion in
fuzz_one_slow_peer_hol (covered by
block_liveness_progress_before_deadline_keeps_peer_alive).
Swap std::time::Instant -> tokio::time::Instant module-wide in the block-sync subsystem (peer routine, registry, admission, BBR, sequencer, service, bench). Duration stays std. Behavior-identical in production: tokio::time::Instant::now() reads the same real monotonic clock as std in a non-paused runtime, so elapsed(), comparisons, and arithmetic are unchanged. Instant never crosses the module's public API (no pub fn takes or returns it), so this is not a breaking change and does not touch any other zakura module. The point is tests: under #[tokio::test(start_paused = true)] the reactor's timers now share the harness's virtual clock, so the in-process block-sync fuzzers can be made deterministic (the reactor previously slept on the tokio clock but measured elapsed with the frozen std clock, causing deadline re-spins and garbage BBR windows under paused time). 220 block_sync tests pass; clippy -D warnings clean.
Flip 18 of the 21 block-sync fuzz scenarios to `#[tokio::test(start_paused = true)]`. With the reactor now on tokio::time::Instant (previous commit), the reactor timers and the in-process harness sleeps share one virtual clock that auto-advances only when every task is parked, so each scenario runs to its seeded outcome in deterministic virtual time instead of racing real-clock deadlines on a loaded CI runner. Verified: the two worst prior flakes (fuzz_mixed_block_sizes, fuzz_reliability_discounts_dropping_carrier) are 20/20 identical; all 18 pass. Three sustained-backpressure scenarios (fuzz_commit_stall, fuzz_multi_peer_tight_budget, fuzz_peer_that_stops_reading_is_disconnected) stay on real-time multi-thread: they drive a reactor<->sequencer<->routine feedback loop that stays perpetually runnable while the byte budget is pinned, so tokio's virtual clock never auto-advances and the scenario livelocks under paused time. Under real time a commit on another worker frees the budget and it converges. This is the same latent busy-spin that makes these three flaky under CI load; a proper fix belongs in the reactor, tracked separately.
6 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
Split out of #435.
The remaining block-sync fuzz changes are determinism and cleanup work. This PR is stacked on #437 because
fuzz_commit_stallonly converges understart_pausedonce the reactor/sequencer/routine busy-spin is fixed.Solution
tokio::time::Instant.#[tokio::test(start_paused = true)].fuzz_commit_stalltostart_paused.Final annotation count in
blocksync_fuzz/tests.rs: 19start_paused, 2 real-time multi-thread.Validation
cargo fmt --all -- --checkcargo test -p zebra-network fuzz_commit_stallThe test filter matched and passed:
fuzz_commit_stallfuzz_commit_stall_resident_plateaufuzz_commit_stall_resident_plateau_multiblockAI Disclosure
Used Codex to split #435 into focused branches/PRs and run the targeted validation above.