test(zakura): restore the dropped test suite and cover the reorg stack end-to-end

[p0mvn]

Motivation

The reorg fix stack (#490–#496) shipped with strong store-level coverage but three integration gaps, one of which the Pillar-1a crash table explicitly commits to: no automated kill-between-batches test, no end-to-end reorging CommitHeaderRange through the write worker, and no reactor-layer reorg scenarios. Worse, the canary cherry-picks dropped every test file, so the entire header_store_coherence harness did not exist on the fleet lineage where the 1a switch orchestration lives.

Stacks on #496 (zakura/writeguards-canary-v3).

Solution

Three commits, reviewable in order:

1. Harness restore — cherry-picks the test content of test(state): add zakura header-store coherence harness with corruption repros #490/fix(state): verify zakura header-store invariants in consensus reads #493/fix(state): audit and self-repair the zakura header store at startup #494/fix(state): own zakura header-store mutations with a single suffix-replacement primitive #495 onto the fleet lineage (~57 tests: coherence suite, shared block-test helpers, proptest regression seeds). The only adaptation is the fleet range-writer API: helpers unwrap .tip_hash from HeaderRangeCommitOutcome.

2. Switch orchestration end-to-end (service/write/switch_orchestration_tests.rs) — drives reorging header-range commits through the real worker function against a real header store and non-finalized state:

   • reorging commit: store switches, stranded body suffix rolls back to the fork, truncated chain published, response reports the reorg; post-reorg audit asserted clean, and a planted-stray-row variant asserts the audit hook repairs and emits the metric;
   • kill-between-batches (the crash-table commitment): body rollback ran, prepared batch dropped unwritten — the intermediate is coherent, a reopen passes the startup audit with no repair, and re-running the same commit converges (recovery is the ordinary flow);
   • whole-suffix switch: a reorg at the nf root empties the chain set without panicking and publishes the emptied state;
   • losing-then-winning branch: lower-work conflict rejected with byte-identical stores, higher-work extension switches, the original branch switches back, and its rolled-back bodies recommit through the full validation path — rollback-not-ban end to end.

   Fixture design (spike results recorded in the test module docs): fabricated bodies need V4 coinbases, and the tests run on a network whose upgrades activate above the test heights, so nf commits take the pre-Heartwood regime like the existing fake-mainnet fixtures.

3. Reactor and driver scenarios — following the fix(zakura): recover header sync from a frontier stranded on an abandoned branch #476 walk-back test patterns:

   • lower-work conflict at the peer layer: the serving peer is never scored or disconnected, keeps receiving requests, and a later successful commit completes the switch (plus the zebrad classification test pinning LowerWorkConflict → Local);
   • the feat(state): sequence branch switches as evaluate, body rollback, then header rewrite #496 flow at the driver: a CommittedHeaderRange outcome with reorged_at set must not drive any follow-up state request — the mock state panics on anything but the commit itself;
   • store-incoherence walk-back to convergence: ContextMismatch failures from independent peers start a walk-back with no one scored, the frontier re-anchors from the store's reanchor target, and the re-delivered range re-commits through the fork point.

Tests

This PR is tests only (no production code changes beyond pub(crate) visibility on test-module paths and a doc comment). All gates green locally:

• cargo test -p zebra-state --lib (352 passed), cargo test -p zebra-network --lib (901 passed), cargo test -p zebrad --lib -- zakura_header_sync_driver_tests (45 passed)
• cargo fmt --all -- --check, cargo clippy -p zebra-state -p zebra-network -p zebrad --all-targets -- -D warnings
• PROPTEST_CASES=1024 cargo test -p zebra-state --release --lib prop_random_sequences_uphold_invariants (discovery sweep)

Follow-up Work

• Pruning coverage for the coherence harness lands on the main stack (stacked on fix(state): own zakura header-store mutations with a single suffix-replacement primitive #495) since it extends fix(state): audit and self-repair the zakura header store at startup #494/fix(state): own zakura header-store mutations with a single suffix-replacement primitive #495's suite.
• Layer-3 controlled-mining net is spec'd separately; implementation deferred until after the canary-v3 soak verdict.
• The fleet↔main reconciliation must carry both this PR's test tree and the main-stack prune coverage.

AI Disclosure

• AI tools were used: Claude Code wrote the tests and fixtures under review; the author reviewed every line and ran all gates locally.

PR Checklist

• The PR title follows conventional commits format: type(scope): description
• The solution is tested.

[v12-auditor]

Note

Complete: Audit complete. V12 found four issues worth reviewing.

Open the full results here.

Finding	Severity	Details
F-96658	🟠 High	Root rollback leaves stale forks A higher-work Zakura header range can reorg the stored header chain at the non-finalized root while the non-finalized state contains multiple forks sharing that old root. commit_header_range() invokes invalidate_stranded_body_suffix() before writing the header suffix, but that rollback is keyed only by the best-chain hash at reorged_at. When the hash is the non-finalized root, remove_block_and_descendants() removes only the selected chain whose tip matches chain_tip_hash; other chains that contain the same old root hash remain in chain_set. The rollback publisher then accepts any remaining best chain, and the header batch is written to the new branch. This leaves the durable header store naming reorged_to_hash at the reorg height while a surviving non-finalized body chain can still contain old_hash at that same height.
F-96659	🟠 High	Empty rollback preserves stale tip When a header reorg rolls back the entire non-finalized body suffix, invalidate_stranded_body_suffix() sends the emptied NonFinalizedState watch value but deliberately does not update ChainTipSender. ChainTipSender permanently switches to non-finalized tips after the first non-None tip and ignores later None values. The LatestChainTip mirror can therefore remain on the pre-reorg body tip even though the non-finalized state is empty. Zakura block-sync frontier discovery merges the read-state tip with this stale latest tip and selects the higher height. Missing-body discovery then begins after that stale height, skipping the new branch bodies at and below it.
F-96661	🟠 High	Context errors force reanchors A peer-supplied Headers range receives only context-free checks in the reactor before it is sent to state. State contextual checks can reject the same range for consensus-context failures such as median-time or expected-difficulty mismatch. The production driver maps every CommitHeaderRangeError::ValidateContextError to HeaderSyncCommitFailureKind::ContextMismatch. The reactor handles ContextMismatch by recording stale-anchor evidence and never reporting peer misbehavior. The stale-anchor counter is global and is not keyed by range, anchor, response hash, or specific validation error, so Sybil peers can turn invalid contextual data into unscored reanchor or walk-back signals.
F-96662	🟠 High	Lower-work retries duplicate ranges The production driver classifies CommitHeaderRangeError::LowerWorkConflict as HeaderSyncCommitFailureKind::Local. The reactor reports misbehavior only for InvalidPeerRange; for Local, it clears the range assignment and retries. RangeScheduler::retry blindly pushes the same RangeRequest to the front of the queue without deduplication. Because pop_assignable leaves a queued range in place until the fanout threshold is filled, sub-fanout connectivity lets the original queued item remain while each Local failure adds another copy. The same lower-work peer becomes eligible again after assignment clearing, so it can repeatedly receive and fail the same range.

And one more auto-invalidated finding.

Analyzed seven files, diff c4345ff...65d0f41.