Skip to content

fix(state): verify zakura header-store invariants in consensus reads#493

Merged
evan-forbes merged 13 commits into
ironwood-mainfrom
zakura-linkage-verified-reads
Jul 7, 2026
Merged

fix(state): verify zakura header-store invariants in consensus reads#493
evan-forbes merged 13 commits into
ironwood-mainfrom
zakura-linkage-verified-reads

Conversation

@p0mvn

@p0mvn p0mvn commented Jul 6, 2026

Copy link
Copy Markdown

Motivation

This PR helps to identify poisoned header-store, and, instead of downscoring a peer for this, silently failing, which prompts a fallback to legacy.

#491 prevents writing/poisoning the store in the first place, and #494 and #496 add auto-repair mechanisms.

Specifically, today, recent_header_context walks the height index blindly, so one stale row inside the 28-height difficulty window makes the node reject every honest header with InvalidDifficultyThreshold.

This has been reproduced and identified via testnet incidents.

This PR catches the poison: corruption found during a read becomes an explicit local error instead of a wrong validation verdict blamed on peers.

Solution

New StoreIncoherentError (exported from zebra-state), returned by readers instead of consuming rows that fail an invariant check:

  • recent_header_context verifies as it walks. At every step: the stored header must be the block its hash row names (HeaderHashMismatch — one header hash per consumed row, negligible next to the validation the window feeds), it must link to the row below (BrokenLinkage), and a missing row below a stored one is a Gap, not the end of history (rows are contiguous from genesis: full-block rows below the body tip — retained under pruning — and zakura rows above). A short window is returned only when the walk genuinely reaches genesis; a silently shortened mid-chain window would shift the difficulty adjustment instead of failing.
  • The header-hash check closes a subtle edge: parent-link verification alone consumes the window's bottom-edge row unverified (its own link is one step below the window), so a foreign header row whose hash row still matches the chain could poison exactly one entry. Hashing each consumed row makes the walk fully self-verifying.
  • Anchor round-trip distinguishes fault domains. A hash the store has never indexed stays UnknownAnchor; a hash→height entry whose height→hash row disagrees is a bijection violation in our own indexes and now reports StoreIncoherent(BijectionMismatch).
  • CommitHeaderRangeError::StoreIncoherent is classified Local (non-scoring) in zebrad's header_range_commit_failure_kind: the range was rejected because our store cannot supply trustworthy context, not because the peer's range was shown invalid. This hard-codes the rule that local store incoherence must never score or disconnect peers.

StoreIncoherent rejections are side-effect free (validation happens before any batch write), so the store is untouched and the fault can only trigger recovery — on the fleet branch the ContextMismatch/walk-back machinery picks it up; Pillar 3's audit-and-repair is the end state.

p0mvn added 4 commits July 6, 2026 16:35
Close the three write-path corruption bugs proven by the Phase-1
header-store coherence suite (PR #490):

1. prepare_header_range_batch_with_roots now rejects ranges whose first
   header does not link to the anchor or whose headers are not
   internally contiguous (new CommitHeaderRangeError::UnlinkedRange,
   classified as a local non-scoring failure in zebrad).
2. The range insert loop skips heights that already have a committed
   block, so re-deliveries can no longer strand provisional zakura rows
   below the body tip.
3. prepare_zakura_header_from_committed_block refuses seeds that do not
   link to the stored row below them as silent no-ops; header-range
   sync converges the store instead (scenario s11).

Flip the four *_upholds_invariants twins to permanent regression gates,
delete the corruption_repro_* tests, remove the discovery masking, and
un-ignore the random invariant sweep (clean at PROPTEST_CASES=4096).
Pillar 2 of the REORG_PLAN: a corrupted header store can no longer poison
difficulty validation — readers surface the storage fault explicitly.

- recent_header_context verifies, at every step of its walk, that the
  stored header is the block its hash row names (HeaderHashMismatch),
  that it links to the row below (BrokenLinkage), and that no row is
  missing below a stored one (Gap), returning the new StoreIncoherentError
  instead of a poisoned or silently shortened difficulty window.
- The range writer's anchor round-trip distinguishes a bijection
  violation in our own indexes (StoreIncoherent::BijectionMismatch) from
  a genuinely unknown anchor (UnknownAnchor).
- CommitHeaderRangeError::StoreIncoherent is classified as a local,
  non-peer-scoring commit failure in zebrad: the range was rejected
  because our store cannot supply trustworthy context, not because the
  peer's range was shown invalid.

New reads.rs suite in header_store_coherence hand-corrupts the column
families and pins the reader/writer behavior for all four fault shapes,
including side-effect freedom of the rejections.
@p0mvn p0mvn added the C-bug Bug fix label Jul 6, 2026
@v12-auditor

v12-auditor Bot commented Jul 6, 2026

Copy link
Copy Markdown

Note

Complete: Audit complete. No review-worthy issues remain after automatic triage. Three findings were auto-invalidated.

Open the full results here.

Analyzed five files, diff 93d3456...08f791a.

@p0mvn
p0mvn marked this pull request as draft July 6, 2026 23:43
@p0mvn

p0mvn commented Jul 7, 2026

Copy link
Copy Markdown
Author

@cursor review

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

Reviewed by Cursor Bugbot for commit 4459656. Configure here.

@evan-forbes
evan-forbes changed the base branch from zakura-header-store-write-guards to ironwood-main July 7, 2026 10:56

@evan-forbes evan-forbes left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

for our future selves, I think we can avoid keeping the zebra mixing of logic for state and reactor/services at least slightly

easy approve tho

@evan-forbes

Copy link
Copy Markdown

there are some minor compile issues and conflicts I'll clean up quickly then merge

# Conflicts:
#	CHANGELOG.md
#	zebra-state/src/service/finalized_state/zebra_db/block/tests/header_store_coherence/README.md
#	zebrad/src/commands/start.rs
#	zebrad/src/commands/start/zakura/header_sync_driver.rs
@evan-forbes
evan-forbes merged commit d529534 into ironwood-main Jul 7, 2026
51 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

C-bug Bug fix

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants