feat(state): sequence branch switches as evaluate, body rollback, then header rewrite

[p0mvn]

Motivation

Pillar 1a of REORG_PLAN.md (§2): sequence every branch switch as evaluate → body rollback → header rewrite, with every crash intermediate coherent. On this branch the walk-back (PR #476) already recovers from forks, but its sequencing is inverted relative to the plan's crash-point table: the header suffix is rewritten on disk first, and the stranded non-finalized body suffix is invalidated afterwards, from the driver, as a separate request — leaving a crash window where the header store names branch B while the body chain still extends branch A past the fork. Two further gaps surfaced while pinning the plan's requirements in tests: InvalidateBlock is a ban, not a rollback (BlockPreviouslyInvalidated refuses re-delivery — exactly the stranding hazard §2 warns about if the winning branch evaporates), and invalidating the non-finalized root panics in update_latest_chain_channels (expect on an empty chain set) — a latent crash reachable by the existing driver flow for forks right above the finalized tip.

Base/stack note: this PR targets the fleet branch because the machinery it orchestrates (walk-back, HeaderRangeCommitOutcome.reorged_at) exists only here. The head branch also carries cherry-picks of the main-stacked PRs #491/#493/#494/#495 (tests dropped, per canary convention); those are under review in their own PRs. The reviewable delta of this PR is the last three commits: the StoreIncoherent → ContextMismatch routing (the fleet-branch decision documented in #493), the #495 pick conflict resolution, and the Pillar-1a commit.

Solution

The switch orchestration moves inside the state write worker (commit_header_range in write.rs), which owns both stores, so the sequencing is a straight-line code path rather than a driver round-trip:

1. Evaluate: batch preparation validates the whole candidate in memory (linkage, checkpoints, contextual difficulty, strictly-greater work gate) and decides the fork point; disk writes are staged only through set_canonical_suffix (fix(state): own zakura header-store mutations with a single suffix-replacement primitive #495). Rejection leaves every store untouched.
2. Body rollback before the header rewrite reaches disk: if the commit reorgs at height h and the best body chain holds a different block there, the stranded suffix is rolled back via the new NonFinalizedState::rollback_block and the truncated chain is published. The commit outcome now carries reorged_to_hash (the new branch's hash at h) so the worker recognizes the stranded suffix without re-reading state.
3. Header rewrite: the prepared batch is written (one atomic suffix replacement), then the post-reorg store audit runs (fix(state): own zakura header-store mutations with a single suffix-replacement primitive #495's hook).

Supporting changes:

• NonFinalizedState::rollback_block — same chain surgery as invalidate_block (extracted into a shared remove_block_and_descendants), but without recording the blocks in the invalidated list. A reorg rollback is a chain-selection outcome, not a verdict: the rolled-back branch revalidates and recommits normally the moment it wins the work race again. Request::InvalidateBlock (operator-facing) keeps its ban-until-reconsidered semantics unchanged.
• Empty-chain-set guard: rolling back from the non-finalized root empties the chain set; the orchestration publishes the emptied state on the watch channel and skips the tip publish instead of hitting update_latest_chain_channels' expect. (The tip watch keeps the stale tip until the new branch's bodies commit; block sync follows the header store, so re-download reanchors correctly — same behavior as today, minus the panic.)
• Driver simplification: the post-commit invalidate_reorged_body_suffix call is removed from the commit path (replaced by a reorg-commit log + sync.header.reorg_commits counter). The helper itself stays for the startup/periodic body-suffix reconciliation sweep, which repairs stranding left behind by older binaries — that path is recovery, not switch orchestration.
• Crash safety / idempotency (§2 crash-point table): the rollback mutates the in-memory non-finalized state; the header rewrite is one atomic batch. A crash between them restarts the node "behind" on a coherent store — the backup restores or resync re-delivers whichever branch currently wins, and the ordinary flow re-runs the switch. Recovery is never a special code path; the startup audit (fix(state): audit and self-repair the zakura header store at startup #494) remains the backstop for any torn state.

Tests

• New orchestration tests in write.rs (fake-chain fixture, three-block nf chain):
  • stranded_body_suffix_rolls_back_from_the_fork — suffix removed from the reorged height up, shared prefix kept, truncated chain published on the watch channel;
  • stranded_suffix_rollback_is_not_a_ban — caught the ban live: written against invalidate_block first, it failed with BlockPreviouslyInvalidated; green with rollback_block. Pins the §2 requirement that a rolled-back block recommits when its branch wins again;
  • whole_suffix_rollback_skips_publish_without_panicking — regression for the empty-chain-set panic;
  • rollback_is_a_noop_when_bodies_match_or_are_absent — matching and absent heights are untouched.
• cargo test -p zebra-state --lib — 305 passed, 0 failed. cargo test -p zebrad --lib header_sync — 43 passed (includes the fix(zakura): recover header sync from a frontier stranded on an abandoned branch #476 walk-back reactor tests, unchanged, and the classification test re-pinned to ContextMismatch).
• cargo fmt --all -- --check, cargo clippy -p zebra-state -p zebrad --all-targets -- -D warnings — clean.
• Soak: deployed to testnet-4 as canary-v3 (binary 731b7610…) with the full stack (fix(state): validate linkage in zakura header-store writers #491+fix(state): verify zakura header-store invariants in consensus reads #493+fix(state): audit and self-repair the zakura header store at startup #494+fix(state): own zakura header-store mutations with a single suffix-replacement primitive #495+1a); startup audit passed on the real store, node at the fleet tip. The soak watcher now tracks BODY-ROLLBACK events so natural fork events during the soak exercise the new sequencing.

Specifications & References

• REORG_PLAN.md §2 Pillar 1a (steps 1–5, the crash-point table, and the "rollback, not ban" requirement), §3 row 3.
• Stacked conceptually on fix(state): own zakura header-store mutations with a single suffix-replacement primitive #495 (set_canonical_suffix) ← fix(state): audit and self-repair the zakura header store at startup #494 (startup audit) ← fix(state): verify zakura header-store invariants in consensus reads #493 (verified reads) ← fix(state): validate linkage in zakura header-store writers #491 (write guards); this head branch carries them as cherry-picks because they are main-stacked while the walk-back lives here.

Follow-up Work

• Frontier notification (step 5) currently rides the existing chain-tip mirror; a dedicated reanchor event could tighten the window where the tip watch is stale after a whole-suffix rollback.
• When the fleet branch merges main-ward, the fix(state): validate linkage in zakura header-store writers #491–fix(state): own zakura header-store mutations with a single suffix-replacement primitive #495 cherry-picks dissolve and this PR's delta rebases onto the true stack.

AI Disclosure

• AI tools were used: Claude Code wrote the orchestration, the rollback variant, tests, and this PR description, directed and reviewed by the author.

PR Checklist

• The PR title follows conventional commits format
• This change was discussed with the team beforehand (Pillar 1a of the agreed REORG_PLAN.md)
• The solution is tested.
• The documentation and changelogs are up to date (CHANGELOG entry rides the main-stacked PRs; this branch is the fleet canary lineage).

[v12-auditor]

Note

Complete: Audit complete. V12 found one issue worth reviewing.

Open the full results here.

Finding	Severity	Details
F-96651	🟡 Medium	Missing provisional headers can strand the body suffix Zakura can rewrite the stored header suffix onto a different branch without invalidating the existing non-finalized body suffix when the divergence begins at heights that are merely absent from the provisional header store. In the commit path, replace_from treats missing rows as sufficient to replace headers, but HeaderRangeCommitOutcome only reports a reorg for explicit stored-header conflicts, so commit_header_range skips invalidate_stranded_body_suffix on this path. The underlying view used for that decision, header_by_height, only sees finalized and provisional header rows, not the in-memory NonFinalizedState, so a real body-chain fork can be misclassified as a harmless extension if the Zakura header store lags. The fallback reconciliation path repeats the same assumption by querying zakura_header_hash directly and returning early when the body-tip height has no provisional header row, even though suffix replacement intentionally deletes those rows above the fork. As a result, both the primary rollback path and the intended repair path can miss the same stranded body suffix, leaving header and body state divergent.

Analyzed 13 files, diff 6f8b4ad...c4345ff.