fix(state): audit and self-repair the zakura header store at startup

[p0mvn]

Motivation

Pillar 3 (startup half) of REORG_PLAN.md, stacked on #493 — pulled ahead of Pillars 1/1a because it gates fleet-wide #491 rollout. The Phase-1.5 write guards (#491) prevent new corruption and the Pillar-2 verified reads (#493) refuse to consume old corruption, but a store poisoned by a pre-#491 binary stays poisoned on disk: every read that crosses the bad rows fails, header sync cannot progress past the window, and the only remedy is a manual resync per node — which doesn't scale to the fleet. This closes that gap: a corrupted store heals itself at startup, converting any residual bug in this class from a permanent on-disk wedge into a self-healing, observable transient.

Solution

New startup_audit module in zebra-state's finalized-state DB layer. ZebraDb::new (skipped for read-only instances, which cannot repair and already surface corruption as StoreIncoherentError) runs an audit that ports the Phase-1 harness semantics (A1 bijection, A2 linkage, A3 tip integrity; A4's oracle comparison has no runtime analogue) into the node:

• Linkage walk from the finalized tip up through the zakura frontier, verifying at every height that the hash and header rows agree, the stored header is the block its hash row names, it links to the row below, and the height_by_hash index round-trips. The last height passing every check is the last coherent height.
• Tip integrity: rows in any zakura column family above the last coherent height (stranded suffixes above gaps or broken links) are violations, as are height_by_hash entries whose target row disagrees (orphaned reverse-index rows from pre-fix(state): validate linkage in zakura header-store writers #491 overwrites).
• Committed-height check with the pruning-safe predicate (the fix(state): validate linkage in zakura header-store writers #491 bug-2 residual): stale zakura rows at committed heights are detected with the contains_height semantics — the consensus hash_by_height row, which every committed block writes and pruning retains — not body presence, which under-reports on pruned stores and would miss exactly the pre-fix(state): validate linkage in zakura header-store writers #491 re-delivery shape.
• Repair = one atomic WriteBatch: truncate the zakura column families to the last coherent height, remove stale committed-height rows and orphaned reverse entries, and let header sync re-download the suffix. Headers are re-fetchable; correctness beats preserved rows. Verified commitment roots at committed heights are never scanned or touched — only provisional roots above the finalized tip are in scope.
• On violation: warn! with the fault shapes and the state.zakura.header_store.incoherent metric — the primary §3a soak signal (must be 0 after the first clean startup; a single early heal is the expected signature of a store poisoned by an old binary).

Pillar-2 interaction: the audit checks are a superset of the read-path checks over the same rows, so any store that would fail the #493 verified reads is healed by this audit — pinned by a test that shows the same range delivery rejected with StoreIncoherent before the heal and committed after it.

Cost: O(header frontier) — the zakura column families only hold rows above the finalized tip (plus the stale rows the audit exists to remove), and the roots history below the tip is never scanned. On non-zakura nodes the column families are empty and the audit is a no-op.

Scope: the startup-audit half only; the after-every-reorg-batch hook lands with Pillar 1 per the plan. No writer or reader changes; no refactors.

[v12-auditor]

Note

Complete: Audit complete. V12 found one issue worth reviewing.

Open the full results here.

Finding	Severity	Details
F-96644	🟠 High	Startup audit materializes peer-fed frontier Writable startup now invokes audit_and_repair_zakura_header_store() before the database finishes opening. That audit eagerly materializes the full Zakura header frontier into BTreeMap and Vec collections, including headers, both hash indexes, body-size hints, and provisional commitment-root heights, then builds another hash index and hashes through the collected chain. Header sync can populate those rows from peer-supplied but valid bounded ranges: forward scheduling targets the peer-advertised best tip, request clamping is per message, accepted responses are forwarded as CommitHeaderRange, and the write worker persists them as header-only state without bodies. The current limits (MAX_HS_RANGE, message bytes, and the state range cap) constrain individual responses, not the aggregate number of durable header-only rows above the finalized tip. A hostile peer that serves many valid contiguous headers and height-matching roots can therefore leave a very large frontier on disk, and every later writable restart must deserialize and walk all of it before the node can continue.

Analyzed three files, diff 08f791a...b354be0.