diff --git a/CHANGELOG.md b/CHANGELOG.md index 097d5182555..ca9111b570e 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -12,6 +12,20 @@ and this project adheres to [Semantic Versioning](https://semver.org). - Zebra now tags the coinbase input of every block it mines with a `🌸`. The `mining.extra_coinbase_data` option is now limited to 86 bytes (was 94); Zebra refuses to start if it is exceeded. +- Zakura header sync now verifies peer-supplied commitment (tree-aux) roots + against block header commitments before persisting them, writing only the + header-authenticated confirmed prefix (the range tip's root is confirmed by + the next overlapping range). Root verification, the one-block overlap, and the + in-memory ZIP-221 history tree are bounded to the verified-commitment-trees + fast-sync handoff β€” the last checkpoint, the only region where the roots are + consumed β€” and run through the last checkpoint inclusive (its root is needed by + the handoff block). Above that boundary header sync runs plainly (blocks are + fully re-verified). The header frontier keeps tracking the committed chain + (checkpoint or legacy sync); the tree grows only as header ranges commit and is + rebuilt lazily from durable state on the rare occasion a forward range finds it + behind a non-Zakura commit. Serving is unaffected: `BlockRoots` still returns + roots derived from real note-commitment trees at any height where the node has + them. ### Changed diff --git a/docs/design/verified-commitment-trees.md b/docs/design/verified-commitment-trees.md index 186f7f74cbc..1783c94b1d3 100644 --- a/docs/design/verified-commitment-trees.md +++ b/docs/design/verified-commitment-trees.md @@ -342,14 +342,17 @@ Wire and DoS bounds: the header count (`TreeAuxRootCountMismatch`); and the root vector is preallocated only with the already-bounded header count, never an independent untrusted length. - The reactor additionally checks each root's height is `start_height + offset` - (`TreeAuxRootHeightMismatch` / `validate_tree_aux_root_heights`) before the roots reach - state. State re-checks the count and alignment invariants in `CommitHeaderRange` - (`prepare_header_range_batch_with_roots`) as defense in depth, and never + (`TreeAuxRootHeightMismatch` / `validate_tree_aux_root_heights`) and authenticates the roots + against the header commitments (Β§6.4) before the range reaches state. State re-checks the + alignment invariants in `CommitHeaderRange` (`prepare_header_range_batch_with_roots`) as + defense in depth, persists only the header-authenticated confirmed prefix (one shorter than the + headers; the range tip's root is confirmed by the next overlapping range, Β§6.4), and never writes peer-supplied roots for a height whose body is already committed β€” a re-delivered header range over committed heights cannot overwrite the verified serving-index rows. `BlockCommitmentRoots` still carries no trust: a recipient re-verifies every root against its -own checkpoint-committed headers (Β§6) before folding it in, so a forwarding/serving node is -exactly as trustworthy as an originating one. +own checkpoint-committed headers β€” at the header-sync reactor before persisting (Β§6.4) and again +at block commit (Β§6) β€” before folding it in, so a forwarding/serving node is exactly as +trustworthy as an originating one. ## 6. Verification β€” verify-before-commit @@ -417,6 +420,82 @@ is now locked together: `auth_data_root` is `pub(crate)`, `CheckpointVerifiedBlo `set_deferred_pool_balance_change`, and the semantic verifier builds blocks through `from_semantic_data` (auth-data root left unset). Compile-time enforced (fix in commit #192). +### 6.4 Header-sync-layer authentication and startup reconstruction + +Β§6.1–§6.3 are the consensus-critical gate at block-commit time. Header sync adds an _earlier_ gate +in the network reactor, so the roots it persists to `commitment_roots_by_height` and serves to +other peers are already header-authenticated, and so a restart never trusts an unconfirmed root. + +- **Bounded to the last checkpoint (inclusive).** Root verification, the one-block overlap, and the + in-memory header-frontier history tree exist only through the last checkpoint β€” + `CheckpointList::max_height()`, exactly the VCT fast-sync handoff height (Β§7). That is the only + region a consumer reads the persisted roots: above the last checkpoint blocks go through full + semantic verification and recompute their own trees, so peer roots there have no reader. The regime + is **inclusive** of the last checkpoint, because the handoff block _at_ `last_checkpoint` reads its + own persisted root from the roots CF (`vct_roots_at_height` is inclusive; an empty slot there wedges + the handoff on the frozen-frontier safety check, Β§8.1). A root at `H` is only confirmed by the header + at `H + 1`, so the root-carrying regime runs until the frontier reaches `last_checkpoint + 1` β€” that + successor header confirms and persists the `last_checkpoint` root; its own root is never persisted. + Above `last_checkpoint + 1` header sync runs plainly (no roots, no overlap, no tree). +- **The tree is a function of header-range commits; rebuilt lazily.** The frontier tree moves only as + header ranges commit (folding the confirmed prefix). `best_header_tip` itself keeps tracking the + committed main chain β€” including blocks committed by checkpoint sync or the legacy syncer (surfaced + via the full-block mirror) β€” so it can run ahead of the tree when a non-Zakura path commits below the + checkpoint. The tree is not eagerly kept in sync across those paths; instead the one operation that + reads it, the forward root-verifying request, detects a behind tree (`MissingHeaderHistoryTree`) and + rebuilds it from durable state (`ReadRequest::BestHeaderHistoryTree`) β€” a single, guarded reload that + never fires in the normal header-leading path. This replaces the reorg/gossip/catch-up eager-reload + machinery of earlier increments. Two robustness details: the in-flight guard is armed only once the + reload action is actually queued β€” the reactorβ†’driver action channel is a non-blocking `try_send`, + and arming the guard on a dropped send (full/closed channel) would suppress every future rebuild + permanently, since the guard clears only when a reload completes; a dropped send instead leaves it + clear so the next range retry re-dispatches. And the reconstruction re-bases at the _current_ + committed tip read from the same state snapshot: the finalized history tree is stored only at the tip + (there is no per-height finalized history tree), and a concurrent checkpoint/legacy commit can + advance the finalized tip past the reactor's cached `verified_block_tip` during the round-trip, so + folding must start from the tip actually present rather than the caller's (possibly lagging) value β€” + otherwise the base read returns `None` and the rebuild degrades to a network-paced no-progress loop. + And the reload carries the confirmed frontier `(height, hash)` the fold actually reached, not just the + tree: when durable roots have a gap the fold stops _below_ `best_header_tip - 1`, so the rebuilt tree + never reaches the current tip's parent. The reactor then reanchors its header tip down onto that + frontier β€” resuming one block above it, anchored on the frontier hash, exactly as the startup + reconstruction does (`sync.header.history_tree.reanchor`) β€” instead of keeping the stale higher tip, + where every forward range would re-detect the behind tree and re-trigger the identical deterministic + rebuild forever. +- **Verify before persisting.** When a below-checkpoint header range arrives, the reactor folds its + supplied roots into the running header-frontier history tree and checks each header's commitment + against it (`verify_supplied_roots_from_parts` in `zebra-chain/src/parallel/commitment_aux_verify.rs`, + reusing the same header-commitment check plus the Β§6.1 below-Heartwood/NU5/`Nu6_3` pins β€” no new + crypto). A well-formed but wrong root fails this check and the range is rejected and scored + through header sync's misbehavior path before any root reaches state. +- **Serving is not bounded.** The client-side gate above never restricts what the node will _serve_. + `BlockRoots` (`block_roots_by_height_range`) remains a general API: per height it derives roots + from the real finalized note-commitment trees, else the real non-finalized chain's trees, else the + provisional header-sync roots CF β€” so a node with committed bodies above the checkpoint serves + real-tree roots there; only header-only heights fall back to provisional roots. +- **Persist only the confirmed prefix.** A block's commitment binds the history tree as of its + parent, so a range `[start..=end]` authenticates the roots for `[start..=end-1]`; the tip's own + root is only confirmed once the next range delivers `end+1`. Forward ranges therefore overlap by + one block β€” the next request re-anchors at the tip's parent β€” and `CommitHeaderRange` persists + only the header-authenticated confirmed prefix; the range tip's root is never written. The state + writes exactly the roots it is handed (`prepare_header_range_batch_with_roots` accepts a prefix one + shorter than the headers, or none β€” see the checkpoint-backfill note below), so the "one root per + header" wire invariant (Β§5.4) and the persisted set are deliberately distinct. +- **Checkpoint backfill skips this gate.** Only _forward_ ranges carry a frontier tree that can be + folded and checked. _Backward_ checkpoint-backfill ranges (headers below the sync anchor) are + authenticated by the checkpoint hash and fold onto the previous checkpoint's tree, not the forward + frontier the reactor caches, so they are committed without header-commitment validation and + persist no provisional roots. `prepare_header_range_batch_with_roots` accepts an empty roots vector + for exactly this shape; a full-length (tip-included) vector is still rejected, so the trust + boundary is unchanged β€” nothing unauthenticated is ever written. +- **Reconstruct at startup.** The durable roots CF therefore holds a contiguous run of confirmed + roots above the verified body tip, but never the header tip's own root. On startup + `ReadRequest::BestHeaderHistoryTree` folds the durable confirmed roots onto the verified-tip + history tree up to the highest _contiguous_ frontier and returns that frontier `(height, hash)`. + Header sync resumes from `frontier + 1` with an overlapping range that re-validates the next root, + so a restart never folds an unauthenticated root into the header-frontier tree and never caps back + to the verified tip on a one-block gap in the persisted roots. + ## 7. The fast commit path and checkpoint last checkpoint height The commit-path hook lives in `finalized_state.rs`; everything about _where data comes from_ @@ -607,12 +686,16 @@ commitment before it influences the anchor set or the history MMR.** Consequence - The below-NU5 Orchard pin and below-Heartwood Sapling check (Β§6.1) close the only ranges the MMR cannot vouch for. Skipping either would let an untrusted source inject an anchor the legacy recompute never produces β€” a consensus-equivalence break, not just a slowdown. -- The frozen-frontier fail-closed policy (Β§8) means a hostile root never corrupts state: it is - deleted and refused. A malformed root set is rejected at the header-sync reactor before it - reaches state and is scored through header sync's misbehavior path; a well-formed wrong root - is evicted on verify-before-commit and the commit stays parked (Β§8.1). The trade-off is - availability, not integrity: a settled bad root stalls the fast sync at that height instead - of writing wrong state (Β§8.1). +- Peer-supplied roots are authenticated against header commitments at the header-sync reactor + **before they are persisted** (Β§6.4): a malformed root set is rejected on decode, and a + well-formed wrong root fails the header-commitment fold (confirmed by the next range, one-block + lag) and is rejected and scored through header sync's misbehavior path β€” so only the + header-authenticated confirmed prefix is ever written, and a restart rebuilds the header-frontier + tree from those durable confirmed roots (never an unconfirmed tip root). +- The frozen-frontier fail-closed policy (Β§8) is the second, consensus-critical gate at block + commit: a hostile root that reached state anyway is deleted and refused, and the commit stays + parked (Β§8.1). The trade-off is availability, not integrity: a settled bad root stalls the fast + sync at that height instead of writing wrong state (Β§8.1). - DoS bounds on the header-sync roots fields (Β§5.4) β€” the all-or-nothing count check, the per-height alignment check, the bounded preallocation, and the message byte budget β€” protect the serving and client paths from unbounded memory growth. @@ -630,14 +713,29 @@ commitment before it influences the anchor set or the history MMR.** Consequence network. - **Increment 6b β€” adversarial peer policy.** A `zebrad` driver recorded heightβ†’peer provenance and ran a roots-specific cooldown/demotion/disconnect policy over the `tree_aux` stream. -- **Increment 6c β€” fold roots into header sync (current).** The standalone `tree_aux` stream, +- **Increment 6c β€” fold roots into header sync.** The standalone `tree_aux` stream, its driver, in-memory cache writer, and bespoke peer policy are **removed**. Roots now ride the header-sync `Headers` message as all-or-nothing metadata (Β§4.2, Β§5.4), are - persisted provisionally to `commitment_roots_by_height` ahead of body commit, and + persisted to `commitment_roots_by_height` ahead of body commit, and are read back by a DB-backed `PeerSource`. Recovery from a bad/missing root is an in-place commit retry fed only by an in-flight fanout re-delivery of the same header range β€” roots are not individually re-requested, so a settled hole is a fail-closed stall (Β§8.1); peer accountability rides header sync's existing misbehavior scoring. +- **Increment 6d β€” authenticate roots before persisting; reconstruct on restart.** + Header sync now verifies supplied roots against the header commitments before writing them, + persists only the header-authenticated confirmed prefix (the range tip's root is confirmed by the + next overlapping range), and rebuilds the header-frontier history tree from the durable confirmed + roots on startup β€” resuming from the highest contiguous frontier (Β§6.4). This closes the gap + where roots were persisted ahead of authentication and a restart could fold an unconfirmed tip + root into the header-frontier tree. +- **Increment 6e β€” bound the frontier tree to below the last checkpoint (current).** Root + verification, the one-block overlap, and the in-memory header-frontier tree now run only while the + frontier is below the last checkpoint (the VCT handoff height, the only region the roots are + consumed); at/above it header sync runs plainly (Β§6.4). Because that region is checkpoint-final and + gossip-free, the tree advances only by fold-on-commit and never needs a live reload, so the + reorg/gossip/catch-up tree-reload machinery (a `QueryBestHeaderHistoryTree` action, its + `BestHeaderHistoryTreeLoaded` reply, and the reactor's reload dispatches) is **removed**. The + reanchor/follow-verified-tip scheduling is kept but gated to fire only at/above the checkpoint. - **Increment 7 β€” indexing follower lane (archive only).** Relocate `tx_by_loc` + address indexes and the per-height trees + subtree CFs onto an async follower, so archive mode regains historical RPC without re-adding the frontier recompute to the consensus path. @@ -671,6 +769,8 @@ Live commit-path counters distinguish the fast and legacy paths and the failure | `state.vct.fast_path.hit` | a finalized commit consumed header-carried roots to skip the recompute | | `state.vct.fast_path.miss` | a finalized commit did not take the fast path | | `state.vct.root.stalled.height` (gauge) | a height stuck on a retryable stall past the warn threshold | +| `sync.header.history_tree.rebuild` | header-frontier history tree lazily rebuilt (Β§6.4) because a non-Zakura commit ran ahead of it below the checkpoint; **stays 0 in the normal header-leading path**, so a nonzero value flags fallback/catch-up | +| `sync.header.history_tree.reanchor` | a lazy rebuild (Β§6.4) folded to a frontier _below_ `best_header_tip - 1` (durable roots had a gap), so the reactor reanchored its header tip down onto the rebuilt tree; a subset of `rebuild`, and likewise 0 in the normal path | The header-sync `headers_received` / `headers_served` / commit-state trace rows also carry `want_tree_aux_roots` and `tree_aux_roots_len`, so root delivery is visible per range. The diff --git a/zebra-network/src/zakura/handler.rs b/zebra-network/src/zakura/handler.rs index 405b10456e2..c5a68b24138 100644 --- a/zebra-network/src/zakura/handler.rs +++ b/zebra-network/src/zakura/handler.rs @@ -32,6 +32,7 @@ use tokio::{ use tokio_util::sync::CancellationToken; use zebra_chain::{ block::{self, Block, CountedHeader}, + history_tree::HistoryTree, parameters::{Network, NetworkKind}, serialization::{CompactSizeMessage, ZcashDeserialize, MAX_HEADERS_PER_MESSAGE}, transaction::Transaction, @@ -543,6 +544,13 @@ pub struct ZakuraHeaderSyncDriverStartup { pub frontiers: HeaderSyncFrontiers, /// Durable best header tip loaded from state. pub best_header_tip: Option<(block::Height, block::Hash)>, + /// Hash of the durable best header tip's parent, if available. + pub best_header_parent_hash: Option, + /// History tree positioned at the durable best header tip. + /// + /// Reconstructed from durable state so post-Heartwood header-sync root verification can start + /// immediately; the default (empty) tree is the pre-Heartwood/empty-state value. + pub best_header_history_tree: Arc, /// Hash of `frontiers.verified_block_tip`. pub verified_block_tip_hash: block::Hash, } @@ -2776,6 +2784,13 @@ pub async fn spawn_zakura_endpoint_with_header_sync_driver( let best_header_tip = header_sync_driver_startup .as_ref() .map_or(Some(anchor), |startup| startup.best_header_tip); + let best_header_parent_hash = header_sync_driver_startup + .as_ref() + .and_then(|startup| startup.best_header_parent_hash); + let best_header_history_tree = header_sync_driver_startup + .as_ref() + .map(|startup| startup.best_header_history_tree.clone()) + .unwrap_or_else(|| Arc::new(HistoryTree::default())); let sync_frontier = header_sync_driver_startup.as_ref().map(|driver_startup| { let best_header_tip = driver_startup.best_header_tip.unwrap_or(anchor); let initial = FrontierUpdate { @@ -2799,6 +2814,8 @@ pub async fn spawn_zakura_endpoint_with_header_sync_driver( config.zakura.header_sync.clone(), limits.max_frame_bytes, ); + startup.best_header_parent_hash = best_header_parent_hash; + startup.best_header_history_tree = best_header_history_tree; startup.status_refresh_interval = config.zakura.header_sync.status_refresh_interval; startup.trace = trace.clone(); startup.frontier_updates = sync_frontier diff --git a/zebra-network/src/zakura/header_sync/error.rs b/zebra-network/src/zakura/header_sync/error.rs index dbaabf5461e..a6c25cb4758 100644 --- a/zebra-network/src/zakura/header_sync/error.rs +++ b/zebra-network/src/zakura/header_sync/error.rs @@ -121,6 +121,23 @@ pub enum HeaderSyncWireError { #[error("first Zakura header-sync range header does not link to anchor")] FirstHeaderDoesNotLink, + /// A header commitment did not match its supplied auxiliary data. + #[error("invalid Zakura header-sync header commitment: {0}")] + InvalidHeaderCommitment(#[from] block::CommitmentError), + + /// The supplied auxiliary roots could not extend the history tree. + #[error("invalid Zakura header-sync history tree update: {0}")] + HistoryTree(#[from] Arc), + + /// The parent history tree needed to validate a header range is unavailable. + #[error("missing Zakura header-sync parent history tree at height {height:?}, hash {hash:?}")] + MissingHeaderHistoryTree { + /// Height whose history tree is required. + height: block::Height, + /// Hash of the required parent header. + hash: block::Hash, + }, + /// Equihash solution size did not match the active network. #[error("Zakura header-sync Equihash solution size does not match the active network")] WrongEquihashSolutionSize, diff --git a/zebra-network/src/zakura/header_sync/events.rs b/zebra-network/src/zakura/header_sync/events.rs index ffaa75fcc55..33d624c39b2 100644 --- a/zebra-network/src/zakura/header_sync/events.rs +++ b/zebra-network/src/zakura/header_sync/events.rs @@ -1,3 +1,9 @@ +use std::sync::Arc; + +use zebra_chain::{ + history_tree::HistoryTree, parallel::commitment_aux_verify::VerifiedHeaderCommitmentRoots, +}; + use super::{config::*, error::*, validation::*, wire::*, *}; use crate::zakura::{ FrontierUpdate, HeaderSyncPeerSession, HeaderSyncServiceSummary, ServicePeerSnapshot, @@ -15,6 +21,24 @@ pub struct HeaderSyncFrontiers { pub verified_block_hash: block::Hash, } +/// Where to reanchor the header frontier after a lazy history-tree rebuild whose durable header-root +/// frontier folded *below* `best_header_tip - 1` (a gap left by a non-Zakura commit racing ahead). +/// +/// Mirrors the startup resume: the rebuilt tree sits at `parent_hash` (the confirmed frontier), and +/// header sync resumes one block above it at `tip`, re-fetching that block so its root re-confirms. +/// Without this the header tip would stay above the tree and every forward range would re-trigger the +/// identical rebuild forever. +#[derive(Copy, Clone, Debug, Eq, PartialEq)] +pub struct HeaderFrontierReanchor { + /// Height to resume the header frontier at β€” one above the confirmed frontier (`frontier + 1`). + pub tip: block::Height, + /// Hash of the resume header at `tip`. + pub tip_hash: block::Hash, + /// Hash of the confirmed header-root frontier at `tip - 1`; the rebuilt tree sits here, and it is + /// the overlap anchor for the resumed forward range. + pub parent_hash: block::Hash, +} + /// Startup inputs for the dependency-neutral header-sync reactor. #[derive(Clone, Debug)] pub struct HeaderSyncStartup { @@ -26,6 +50,20 @@ pub struct HeaderSyncStartup { pub frontiers: HeaderSyncFrontiers, /// Durable best header tip loaded from storage at startup. pub best_header_tip: Option<(block::Height, block::Hash)>, + /// Hash of the durable best header tip's parent, if the tip is above genesis. + pub best_header_parent_hash: Option, + /// The last checkpoint height β€” the VCT fast-sync handoff boundary. + /// + /// Header sync requests, verifies, persists, and tree-tracks peer-supplied commitment roots only + /// while the header frontier is below this height (the only region a VCT consumer reads them). + /// At/above it the frontier tree is dropped and header sync runs plain. Defaults to + /// `network.checkpoint_list().max_height()` (the height the embedded VCT frontier is pinned to). + pub last_checkpoint_height: block::Height, + /// History tree positioned at the durable best header tip. + /// + /// Pre-Heartwood or empty state uses the default (empty) tree; post-Heartwood this is the tree + /// reconstructed from durable roots so header-sync root verification can start immediately. + pub best_header_history_tree: Arc, /// Shared sync exchange frontier stream. pub frontier_updates: Option>, /// Local stream-5 advertisement. @@ -56,11 +94,15 @@ impl HeaderSyncStartup { config: ZakuraHeaderSyncConfig, max_frame_bytes: u32, ) -> Self { + let last_checkpoint_height = network.checkpoint_list().max_height(); Self { network, anchor, frontiers, best_header_tip, + best_header_parent_hash: None, + last_checkpoint_height, + best_header_history_tree: Arc::new(HistoryTree::default()), frontier_updates: None, config, max_frame_bytes, @@ -217,6 +259,27 @@ pub enum HeaderSyncEvent { }, /// State finalized or verified-body frontiers changed. StateFrontiersChanged(HeaderSyncFrontiers), + /// State returned the history tree rebuilt for the current best header tip. + /// + /// Answers a [`HeaderSyncAction::QueryBestHeaderHistoryTree`] β€” the single lazy rebuild triggered + /// when a below-checkpoint forward range finds the header-frontier tree behind the committed tip + /// (a non-Zakura path committed ahead). The tree is repositioned from durable state. + /// + /// Always sent once per query β€” including on a state read error β€” so the reactor's in-flight + /// rebuild guard is always cleared. `history_tree` is `None` when the rebuild failed (the guard is + /// cleared and the next forward range re-triggers it), `Some` on success. + BestHeaderHistoryTreeLoaded { + /// Best header tip the reload was requested against, used as a staleness guard: the tree is + /// only installed if the reactor tip has not moved since the query was dispatched. + best_header_tip: block::Height, + /// Where to reanchor the header frontier, or `None` when no reanchor is needed (the tree + /// folded to `best_header_tip - 1`, the common no-gap case) or the rebuild failed. `Some` when + /// the durable header-root frontier folded *below* `best_header_tip - 1` β€” a gap left by a + /// non-Zakura commit β€” so the tip must drop onto the rebuilt tree to make progress. + reanchor: Option, + /// History tree reconstructed by state; `None` on failure. + history_tree: Option>, + }, /// State successfully committed a header range. HeaderRangeCommitted { /// First committed height. @@ -225,6 +288,8 @@ pub enum HeaderSyncEvent { tip_height: block::Height, /// New best header tip hash. tip_hash: block::Hash, + /// Hash of the new best header tip's parent, if known. + tip_parent_hash: Option, }, /// State rejected a previously requested range. HeaderRangeCommitFailed { @@ -313,13 +378,28 @@ pub enum HeaderSyncAction { headers: Vec>, /// Advisory serialized body sizes, parallel to `headers`. body_sizes: Vec, - /// Per-height commitment roots, parallel to `headers`. - tree_aux_roots: Vec, + /// Header-layer verified commitment roots for the confirmed prefix of `headers`. + /// + /// `Some` only for below-checkpoint forward ranges (the confirmed prefix is persisted). + /// `None` for checkpoint-authenticated backward backfill ranges (they fold onto the previous + /// checkpoint's tree, not the forward frontier) and for above-checkpoint forward ranges (past + /// the VCT handoff boundary, where roots are neither requested nor needed). Both persist no + /// roots. + verified_roots: Option>, /// Whether the range is expected to be finalized by checkpoint policy. finalized: bool, }, - /// Ask state for the durable best header tip. - QueryBestHeaderTip, + /// Ask state to rebuild the header-frontier history tree at the current best header tip. + /// + /// The single reload trigger: dispatched when a below-checkpoint forward range finds the tree + /// behind the committed tip. State reconstructs from durable roots; `verified_block_tip` is the + /// reconstruction base. + QueryBestHeaderHistoryTree { + /// Verified block tip that is the reconstruction base. + verified_block_tip: block::Height, + /// Header tip the rebuilt tree must be positioned at. + best_header_tip: block::Height, + }, /// Ask state for a bounded contiguous range of headers. QueryHeadersByHeightRange { /// Peer that requested the range. diff --git a/zebra-network/src/zakura/header_sync/mod.rs b/zebra-network/src/zakura/header_sync/mod.rs index 556b1496f3d..75a0d066654 100644 --- a/zebra-network/src/zakura/header_sync/mod.rs +++ b/zebra-network/src/zakura/header_sync/mod.rs @@ -52,8 +52,9 @@ pub use config::{ }; pub use error::{HeaderSyncStartError, HeaderSyncWireError}; pub use events::{ - ExpectedHeadersResponse, HeaderSyncAction, HeaderSyncCommitFailureKind, HeaderSyncEvent, - HeaderSyncFrontiers, HeaderSyncHandle, HeaderSyncMisbehavior, HeaderSyncStartup, + ExpectedHeadersResponse, HeaderFrontierReanchor, HeaderSyncAction, HeaderSyncCommitFailureKind, + HeaderSyncEvent, HeaderSyncFrontiers, HeaderSyncHandle, HeaderSyncMisbehavior, + HeaderSyncStartup, }; pub use reactor::spawn_header_sync_reactor; pub use service::HeaderSyncPeerSession; @@ -61,8 +62,8 @@ pub(crate) use service::{ drive_header_sync_actions, HeaderSyncPassthroughService, HeaderSyncService, }; pub use validation::{ - validate_header_range_links, validate_headers_stateless, validate_new_block_stateless, - HeaderSyncDecodeContext, HeaderSyncValidationContext, + validate_header_aux_commitments, validate_header_range_links, validate_headers_stateless, + validate_new_block_stateless, HeaderSyncDecodeContext, HeaderSyncValidationContext, }; pub use wire::{ HeaderSyncMessage, DEFAULT_HS_MAX_INFLIGHT, DEFAULT_HS_RANGE, MAX_HS_MESSAGE_BYTES, diff --git a/zebra-network/src/zakura/header_sync/reactor.rs b/zebra-network/src/zakura/header_sync/reactor.rs index cfd22679ca0..c5d2b7ed8fa 100644 --- a/zebra-network/src/zakura/header_sync/reactor.rs +++ b/zebra-network/src/zakura/header_sync/reactor.rs @@ -70,7 +70,6 @@ impl HeaderSyncReactor { let mut frontier_updates = self.startup.frontier_updates.clone(); let mut frontier_updates_open = frontier_updates.is_some(); if self.startup.range_state_actions_enabled { - let _ = self.dispatch_action(HeaderSyncAction::QueryBestHeaderTip); let _ = self.dispatch_action(HeaderSyncAction::QueryMissingBlockBodies { from: next_height(self.state.verified_block_tip) .unwrap_or(self.state.verified_block_tip), @@ -190,13 +189,31 @@ impl HeaderSyncReactor { HeaderSyncEvent::StateFrontiersChanged(frontiers) => { self.handle_state_frontiers_changed(frontiers).await; } + HeaderSyncEvent::BestHeaderHistoryTreeLoaded { + best_header_tip, + reanchor, + history_tree, + } => { + self.handle_best_header_history_tree_loaded( + best_header_tip, + reanchor, + history_tree, + ) + .await; + } HeaderSyncEvent::HeaderRangeCommitted { start_height, tip_height, tip_hash, + tip_parent_hash, } => { - self.handle_header_range_committed(start_height, tip_height, tip_hash) - .await + self.handle_header_range_committed( + start_height, + tip_height, + tip_hash, + tip_parent_hash, + ) + .await } HeaderSyncEvent::HeaderRangeCommitFailed { peer, @@ -466,8 +483,13 @@ impl HeaderSyncReactor { self.update_verified_block_tip(height, hash); self.state.schedule.mark_height_covered(height); self.cancel_covered_outstanding(); + // `best_header_tip` is the header frontier and must keep tracking the committed main chain, + // including blocks committed by checkpoint sync or the legacy syncer (surfaced here via the + // full-block mirror). Advance it whenever the committed tip runs ahead; the in-memory tree is + // left untouched β€” a below-checkpoint forward range that then finds the tree behind rebuilds it + // lazily (the single `MissingHeaderHistoryTree` reload path). if height > self.state.best_header_tip { - self.publish_best_tip(height, hash).await; + self.publish_best_tip(height, hash, None).await; } self.schedule().await; } @@ -491,7 +513,7 @@ impl HeaderSyncReactor { self.state.schedule.mark_height_covered(height); self.cancel_covered_outstanding(); if height > self.state.best_header_tip { - self.publish_best_tip(height, hash).await; + self.publish_best_tip(height, hash, None).await; } let destinations = self.eligible_tip_destinations(&peer, height); @@ -594,7 +616,10 @@ impl HeaderSyncReactor { } async fn handle_state_frontiers_changed(&mut self, frontiers: HeaderSyncFrontiers) { - self.state.finalized_height = frontiers.finalized_height; + // The finalized height is append-only; clamp against an out-of-order/stale frontier update so + // the root-regime floor (which keys off it) can never move backward. The verified block tip is + // left free β€” it can legitimately reorg backward above the checkpoint. + self.state.finalized_height = self.state.finalized_height.max(frontiers.finalized_height); self.state.verified_block_tip = frontiers.verified_block_tip; self.state.verified_block_hash = frontiers.verified_block_hash; if self.state.best_header_tip <= self.state.verified_block_tip { @@ -603,11 +628,63 @@ impl HeaderSyncReactor { self.schedule().await; } + /// Installs a lazily-rebuilt header-frontier history tree (the answer to + /// `QueryBestHeaderHistoryTree`). Always clears the in-flight guard so a failed rebuild + /// (`history_tree == None`) does not wedge future rebuilds; installs only on success and only if + /// the best header tip has not moved since the query. + /// + /// When `reanchor` is set the durable header-root frontier folded *below* `best_header_tip - 1` + /// (a gap left by a non-Zakura commit racing ahead), so the tree does not reach the current tip's + /// parent. Drop the header frontier onto the rebuilt tree β€” mirroring the startup resume β€” so the + /// next forward range's parent matches the tree; otherwise the range would re-trigger the identical + /// deterministic rebuild forever. Reschedules so the triggering range is re-tried (against the + /// repositioned tree, or re-triggering the rebuild if it failed). + async fn handle_best_header_history_tree_loaded( + &mut self, + best_header_tip: block::Height, + reanchor: Option, + history_tree: Option>, + ) { + self.state.rebuild_in_flight = false; + if let Some(history_tree) = history_tree { + if best_header_tip == self.state.best_header_tip { + self.state.best_header_history_tree = history_tree; + if let Some(reanchor) = reanchor { + self.reanchor_header_frontier(reanchor).await; + } + } + } + self.schedule().await; + } + + /// Reanchors the header frontier down onto a just-rebuilt tree that folded below the current tip. + /// + /// Discards the forward schedule/commit state stranded above the frontier and repositions the tip + /// at `reanchor.tip`, anchored on `reanchor.parent_hash` (where the tree sits), so `schedule` + /// re-derives a fresh overlap forward range from the reanchored tip. Analogous to + /// [`Self::reanchor_to_verified_block_tip`], but lands on the confirmed header-root frontier rather + /// than the verified block tip, keeping the header lead already re-validated below the frontier. + async fn reanchor_header_frontier(&mut self, reanchor: HeaderFrontierReanchor) { + metrics::counter!("sync.header.history_tree.reanchor").increment(1); + self.state.schedule.clear_forward(); + self.state + .pending_commits + .retain(|_, commit| commit.requested_range.priority != RangePriority::Forward); + self.cancel_forward_outstanding(); + self.publish_best_tip_reanchored( + reanchor.tip, + reanchor.tip_hash, + Some(reanchor.parent_hash), + ) + .await; + } + async fn handle_header_range_committed( &mut self, start_height: block::Height, tip_height: block::Height, tip_hash: block::Hash, + tip_parent_hash: Option, ) { metrics::counter!("sync.header.range.committed").increment(1); self.trace_range_event( @@ -617,9 +694,18 @@ impl HeaderSyncReactor { None, None, ); - self.state - .pending_commits - .retain(|_, range| !range.is_within(start_height, tip_height)); + let committed_history_tree = self.pending_header_history_tree(start_height, tip_height); + let mut clear_assignments = Vec::new(); + self.state.pending_commits.retain(|_, commit| { + let delivered = commit.delivered_range.is_within(start_height, tip_height); + if delivered { + clear_assignments.push(commit.requested_range); + } + !delivered + }); + for range in clear_assignments { + self.state.schedule.retire_request(range); + } self.state .schedule .mark_range_covered(start_height, tip_height); @@ -627,7 +713,13 @@ impl HeaderSyncReactor { // startup. In that path start==tip, so covered-range side effects are bounded. self.cancel_covered_outstanding(); if tip_height > self.state.best_header_tip { - self.publish_best_tip(tip_height, tip_hash).await; + // A genuine range commit always has a matching pending commit; keep the existing tree + // if none is found (e.g. the startup best-header-tip reload, where tip does not advance). + if let Some(committed_history_tree) = committed_history_tree { + self.state.best_header_history_tree = committed_history_tree; + } + self.publish_best_tip(tip_height, tip_hash, tip_parent_hash) + .await; } self.notify_body_gaps().await; self.schedule().await; @@ -657,11 +749,11 @@ impl HeaderSyncReactor { start_height, count, }; - if let Some(range) = self.state.pending_commits.remove(&key) { + if let Some(commit) = self.state.pending_commits.remove(&key) { if kind == HeaderSyncCommitFailureKind::Local { - self.state.schedule.clear_assignment(range); + self.state.schedule.clear_assignment(commit.requested_range); } - self.state.schedule.retry(range); + self.state.schedule.retry(commit.requested_range); } self.schedule().await; } @@ -1024,8 +1116,44 @@ impl HeaderSyncReactor { peer_max_headers_per_response: u32, in_flight_count: usize, ) { - if validate_body_sizes_len(headers.len(), body_sizes.len()).is_err() - || validate_tree_aux_roots_len(headers.len(), tree_aux_roots.len()).is_err() + if validate_body_sizes_len(headers.len(), body_sizes.len()).is_err() { + self.report_misbehavior(peer, HeaderSyncMisbehavior::MalformedMessage) + .await; + self.state.schedule.retry(outstanding.range); + self.schedule().await; + return; + } + + // Tree-aux roots are an optional serving capability: a peer may legitimately have no + // roots for a root-carrying request (it can never serve the root for its own tip, and + // may not persist roots at all). A completely rootless non-empty response is therefore + // not misbehavior β€” the headers are unusable for this root-carrying range, so drop them, + // release the request, and retry the range without scoring the peer. + if outstanding.range.want_tree_aux_roots && !headers.is_empty() && tree_aux_roots.is_empty() + { + metrics::counter!("sync.header.response.rootless").increment(1); + self.record_advisory_unconfirmed(&peer); + self.trace_headers_received( + &peer, + outstanding.range.start_height, + u32::try_from(headers.len()).unwrap_or(u32::MAX), + outstanding.expected_max_count, + peer_max_headers_per_response, + in_flight_count, + outstanding.range.want_tree_aux_roots, + 0, + ); + self.state.schedule.clear_assignment(outstanding.range); + self.state.schedule.retry(outstanding.range); + self.schedule().await; + return; + } + + // A root-carrying request must otherwise be answered with one root per header; a plain + // (above-checkpoint) request must carry no roots (checked separately just below). Only + // enforce the one-per-header count when roots were actually requested. + if outstanding.range.want_tree_aux_roots + && validate_tree_aux_roots_len(headers.len(), tree_aux_roots.len()).is_err() { self.report_misbehavior(peer, HeaderSyncMisbehavior::MalformedMessage) .await; @@ -1188,13 +1316,104 @@ impl HeaderSyncReactor { } } + // Only below-checkpoint forward ranges carry a verifiable frontier tree and confirmed roots. + // Backward (checkpoint-backfill) ranges are authenticated by the checkpoint hash, not ZIP-221 + // header commitments (they fold onto the previous checkpoint's tree, not the forward frontier + // this reactor caches). Above-checkpoint forward ranges request no roots (`want_tree_aux_roots` + // is false past the VCT handoff boundary) and are fully re-verified at block commit. Both + // commit their headers with no provisional roots. + let verified_roots = if outstanding.range.priority == RangePriority::Forward + && outstanding.range.want_tree_aux_roots + { + match self.validate_forward_header_aux_commitments( + &peer, + outstanding.range, + &headers, + &tree_aux_roots, + ) { + Ok(verified_roots) => Some(verified_roots), + Err(error) + if matches!(error, HeaderSyncWireError::MissingHeaderHistoryTree { .. }) => + { + debug!( + ?peer, + ?error, + start_height = ?outstanding.range.start_height, + count = ?header_count, + "Zakura header-sync skipped header auxiliary validation" + ); + self.trace_range_validation_rejected( + &peer, + outstanding.range, + header_count, + "header_aux", + header_sync_wire_error_kind(&error), + ); + // The tree is behind this range's parent β€” a non-Zakura path (checkpoint/legacy + // sync) committed ahead of the header-commit frontier, so `best_header_tip` was + // bumped past where the tree is folded. This is the *only* reload trigger: rebuild + // the tree at the current frontier from durable state, then retry the range. In the + // normal Zakura path (header sync leading) this never fires. Only arm the in-flight + // guard once the reload action is actually queued β€” `dispatch_action` is a + // non-blocking `try_send` that drops on a full/closed channel, and arming the guard + // on a dropped send would suppress every future rebuild permanently (the guard is + // only cleared when the reload completes). A dropped send just leaves the guard + // clear so the next retry re-dispatches. + if !self.state.rebuild_in_flight + && self.dispatch_action(HeaderSyncAction::QueryBestHeaderHistoryTree { + verified_block_tip: self.state.verified_block_tip, + best_header_tip: self.state.best_header_tip, + }) + { + self.state.rebuild_in_flight = true; + metrics::counter!("sync.header.history_tree.rebuild").increment(1); + } + self.state.schedule.clear_assignment(outstanding.range); + self.state.schedule.retry(outstanding.range); + self.schedule().await; + return; + } + Err(error) => { + debug!( + ?peer, + ?error, + start_height = ?outstanding.range.start_height, + count = ?header_count, + "Zakura header-sync rejected header auxiliary data" + ); + self.trace_range_validation_rejected( + &peer, + outstanding.range, + header_count, + "header_aux", + header_sync_wire_error_kind(&error), + ); + self.report_misbehavior(peer.clone(), HeaderSyncMisbehavior::InvalidRange) + .await; + self.state.schedule.retry(outstanding.range); + self.schedule().await; + return; + } + } + } else { + None + }; + + let delivered_range = RangeRequest { + count: header_count, + ..outstanding.range + }; self.state.pending_commits.insert( PendingCommitKey { peer: peer.clone(), start_height: outstanding.range.start_height, count: header_count, }, - outstanding.range, + PendingHeaderCommit { + requested_range: outstanding.range, + delivered_range, + verified_roots: verified_roots.clone(), + }, ); let _ = self.dispatch_action(HeaderSyncAction::CommitHeaderRange { peer, @@ -1202,11 +1421,91 @@ impl HeaderSyncReactor { start_height: outstanding.range.start_height, headers, body_sizes, - tree_aux_roots, + verified_roots: verified_roots.map(Box::new), finalized: outstanding.range.finalized, }); } + // Validates the header auxiliary commitments for the given range. + // Returns the verified root payload if it is valid, otherwise returns an error. + fn validate_forward_header_aux_commitments( + &self, + peer: &ZakuraPeerId, + range: RangeRequest, + headers: &[Arc], + tree_aux_roots: &[BlockCommitmentRoots], + ) -> Result< + zebra_chain::parallel::commitment_aux_verify::VerifiedHeaderCommitmentRoots, + HeaderSyncWireError, + > { + if !range.want_tree_aux_roots || range.priority != RangePriority::Forward { + return Err(HeaderSyncWireError::MissingHeaderHistoryTree { + height: range.start_height, + hash: range.anchor_hash, + }); + } + + let parent_height = previous_height(range.start_height) + .ok_or(HeaderSyncWireError::HeightOutOfRange(range.start_height.0))?; + if !self.cached_header_history_tree_matches(parent_height, range.anchor_hash) { + debug!( + ?peer, + start_height = ?range.start_height, + anchor_hash = ?range.anchor_hash, + "Zakura header-sync cannot validate header auxiliary data without parent history tree" + ); + return Err(HeaderSyncWireError::MissingHeaderHistoryTree { + height: parent_height, + hash: range.anchor_hash, + }); + } + + validate_header_aux_commitments( + &self.startup.network, + &self.state.best_header_history_tree, + headers, + tree_aux_roots, + ) + } + + // Returns true if the cached history tree matches the given parent height and hash. + fn cached_header_history_tree_matches( + &self, + parent_height: block::Height, + parent_hash: block::Hash, + ) -> bool { + let height_matches = header_history_tree_is_at_height( + &self.state.best_header_history_tree, + parent_height, + &self.startup.network, + ); + let hash_matches = (parent_height == self.state.best_header_tip + && parent_hash == self.state.best_header_hash) + || (previous_height(self.state.best_header_tip) == Some(parent_height) + && self.state.best_header_parent_hash == Some(parent_hash)); + + height_matches && hash_matches + } + + // Returns the history tree for the pending commit that matches the given start and tip heights. + // Backward (checkpoint-authenticated) commits carry no verified roots, so they never install a + // frontier tree; the caller keeps its existing tree in that case. + fn pending_header_history_tree( + &self, + start_height: block::Height, + tip_height: block::Height, + ) -> Option> { + self.state + .pending_commits + .values() + .find(|commit| { + commit.delivered_range.start_height == start_height + && commit.delivered_range.end_height() == tip_height + }) + .and_then(|commit| commit.verified_roots.as_ref()) + .map(|verified_roots| Arc::new(verified_roots.tree().clone())) + } + async fn handle_possible_stale_anchor_link_failure( &mut self, peer: &ZakuraPeerId, @@ -1244,9 +1543,9 @@ impl HeaderSyncReactor { self.state.schedule.clear_forward(); self.state .pending_commits - .retain(|_, range| range.priority != RangePriority::Forward); + .retain(|_, commit| commit.requested_range.priority != RangePriority::Forward); self.cancel_forward_outstanding(); - self.publish_best_tip_reanchored(height, hash).await; + self.publish_best_tip_reanchored(height, hash, None).await; } async fn handle_timeouts(&mut self) { @@ -1451,9 +1750,15 @@ impl HeaderSyncReactor { true } - async fn publish_best_tip(&mut self, height: block::Height, hash: block::Hash) { + async fn publish_best_tip( + &mut self, + height: block::Height, + hash: block::Hash, + parent_hash: Option, + ) { self.state.best_header_tip = height; self.state.best_header_hash = hash; + self.state.best_header_parent_hash = parent_hash; metrics::gauge!("sync.header.best_tip.height").set(height.0 as f64); self.trace_frontier_advanced(height, hash); let _ = self.tip.send((height, hash)); @@ -1462,10 +1767,16 @@ impl HeaderSyncReactor { self.broadcast_status_refresh().await; } - async fn publish_best_tip_reanchored(&mut self, height: block::Height, hash: block::Hash) { + async fn publish_best_tip_reanchored( + &mut self, + height: block::Height, + hash: block::Hash, + parent_hash: Option, + ) { let old = (self.state.best_header_tip, self.state.best_header_hash); self.state.best_header_tip = height; self.state.best_header_hash = hash; + self.state.best_header_parent_hash = parent_hash; metrics::gauge!("sync.header.best_tip.height").set(height.0 as f64); self.trace_frontier_reanchored(height, hash); let _ = self.tip.send((height, hash)); @@ -1676,10 +1987,17 @@ impl HeaderSyncReactor { insert_height(row, "finalized_height", frontiers.finalized_height); insert_height(row, "verified_block_tip", frontiers.verified_block_tip); } + HeaderSyncEvent::BestHeaderHistoryTreeLoaded { + best_header_tip, .. + } => { + insert_optional_str(row, hs_trace::KIND, Some("best_header_history_tree_loaded")); + insert_height(row, hs_trace::BEST_HEADER_TIP, *best_header_tip); + } HeaderSyncEvent::HeaderRangeCommitted { start_height, tip_height, tip_hash, + tip_parent_hash: _, } => { insert_optional_str(row, hs_trace::KIND, Some("header_range_committed")); insert_height(row, hs_trace::RANGE_START, *start_height); @@ -1796,8 +2114,13 @@ impl HeaderSyncReactor { insert_height(row, hs_trace::RANGE_START, *start_height); insert_u64(row, hs_trace::RANGE_COUNT, headers.len() as u64); } - HeaderSyncAction::QueryBestHeaderTip => { - insert_optional_str(row, hs_trace::KIND, Some("query_best_header_tip")); + HeaderSyncAction::QueryBestHeaderHistoryTree { + verified_block_tip, + best_header_tip, + } => { + insert_optional_str(row, hs_trace::KIND, Some("query_best_header_history_tree")); + insert_height(row, hs_trace::VERIFIED_BLOCK_TIP, *verified_block_tip); + insert_height(row, hs_trace::BEST_HEADER_TIP, *best_header_tip); } HeaderSyncAction::QueryMissingBlockBodies { from, limit } => { insert_optional_str(row, hs_trace::KIND, Some("query_missing_block_bodies")); @@ -2173,6 +2496,22 @@ impl HeaderSyncReactor { } } +/// Returns true if the history tree is at the given height. +fn header_history_tree_is_at_height( + tree: &zebra_chain::history_tree::HistoryTree, + height: block::Height, + network: &Network, +) -> bool { + match tree.as_ref().map(|tree| tree.current_height()) { + Some(current_height) => current_height == height, + None => match zebra_chain::parameters::NetworkUpgrade::Heartwood.activation_height(network) + { + Some(heartwood) => height < heartwood, + None => true, + }, + } +} + fn header_sync_wire_error_kind(error: &HeaderSyncWireError) -> &'static str { match error { HeaderSyncWireError::OversizedPayload { .. } => "oversized_payload", @@ -2192,6 +2531,9 @@ fn header_sync_wire_error_kind(error: &HeaderSyncWireError) -> &'static str { HeaderSyncWireError::TrailingBytes => "trailing_bytes", HeaderSyncWireError::NonContiguousHeaders => "non_contiguous_headers", HeaderSyncWireError::FirstHeaderDoesNotLink => "first_header_does_not_link", + HeaderSyncWireError::InvalidHeaderCommitment(_) => "invalid_header_commitment", + HeaderSyncWireError::HistoryTree(_) => "history_tree", + HeaderSyncWireError::MissingHeaderHistoryTree { .. } => "missing_header_history_tree", HeaderSyncWireError::WrongEquihashSolutionSize => "wrong_equihash_solution_size", HeaderSyncWireError::InvalidDifficultyThreshold => "invalid_difficulty_threshold", HeaderSyncWireError::DifficultyFilter { .. } => "difficulty_filter", diff --git a/zebra-network/src/zakura/header_sync/scheduler.rs b/zebra-network/src/zakura/header_sync/scheduler.rs index 583d829e30a..c3dd9080249 100644 --- a/zebra-network/src/zakura/header_sync/scheduler.rs +++ b/zebra-network/src/zakura/header_sync/scheduler.rs @@ -162,6 +162,14 @@ impl RangeScheduler { self.assigned.remove(&range); } + pub(super) fn retire_request(&mut self, range: RangeRequest) { + match range.priority { + RangePriority::Forward => self.forward.retain(|queued| *queued != range), + RangePriority::Backward => self.backward.retain(|queued| *queued != range), + } + self.clear_assignment(range); + } + pub(super) fn clear_forward(&mut self) { self.forward.clear(); self.assigned diff --git a/zebra-network/src/zakura/header_sync/service.rs b/zebra-network/src/zakura/header_sync/service.rs index 385ce9aaeee..f8375a08646 100644 --- a/zebra-network/src/zakura/header_sync/service.rs +++ b/zebra-network/src/zakura/header_sync/service.rs @@ -283,7 +283,7 @@ pub(crate) async fn drive_header_sync_actions( "suppressing Zakura header range commit until state driver is wired" ); } - HeaderSyncAction::QueryBestHeaderTip + HeaderSyncAction::QueryBestHeaderHistoryTree { .. } | HeaderSyncAction::QueryMissingBlockBodies { .. } | HeaderSyncAction::BodyGaps { .. } | HeaderSyncAction::HeaderAdvanced { .. } diff --git a/zebra-network/src/zakura/header_sync/state.rs b/zebra-network/src/zakura/header_sync/state.rs index 7b7826a1c37..7e0b1b38fe2 100644 --- a/zebra-network/src/zakura/header_sync/state.rs +++ b/zebra-network/src/zakura/header_sync/state.rs @@ -1,3 +1,7 @@ +use std::sync::Arc; + +use zebra_chain::history_tree::HistoryTree; + use super::{error::*, events::*, scheduler::*, validation::*, wire::*, *}; use crate::zakura::{ HeaderSyncServiceSummary, ServicePeerDirection, DEFAULT_LIVE_SERVICE_SUMMARY_TTL, @@ -8,6 +12,10 @@ pub(super) const HEADER_SYNC_ADVISORY_BACKOFF: Duration = Duration::from_secs(60 pub(super) const HEADER_SYNC_ADVISORY_TTL: Duration = DEFAULT_LIVE_SERVICE_SUMMARY_TTL; pub(super) const HEADER_SYNC_STALE_ANCHOR_LINK_FAILURES: u32 = 3; pub(super) const HEADER_SYNC_STALE_ANCHOR_DISTINCT_PEERS: usize = 2; +/// Below-anchor checkpoint backfill is unused by the current sync wiring and cannot verify +/// roots (backward ranges fold onto the previous checkpoint's tree, which this reactor never +/// tracks), so it stays disabled. See [`HeaderSyncCore::refresh_backward_range`]. +pub(super) const BACKWARD_CHECKPOINT_BACKFILL_ENABLED: bool = false; #[derive(Clone, Debug)] pub(super) struct HeaderSyncCore { @@ -17,20 +25,31 @@ pub(super) struct HeaderSyncCore { pub(super) verified_block_hash: block::Hash, pub(super) best_header_tip: block::Height, pub(super) best_header_hash: block::Hash, + pub(super) best_header_parent_hash: Option, + /// History tree positioned at the parent of the next forward range. + /// + /// Seeded at startup from durable state and repositioned as ranges commit, so peer-supplied + /// roots can be folded and authenticated against header commitments. The empty tree is the + /// natural pre-Heartwood value. + pub(super) best_header_history_tree: Arc, pub(super) peers: HashMap, pub(super) parked_peers: HashSet, pub(super) seen: HeaderHashDedup, pub(super) pending_new_blocks: HashSet, pub(super) schedule: RangeScheduler, - pub(super) pending_commits: HashMap, + pub(super) pending_commits: HashMap, pub(super) advisory: HashMap, pub(super) stale_anchor: StaleAnchorFailures, + /// True while a `QueryBestHeaderHistoryTree` rebuild is outstanding, so a run of forward ranges + /// that all find the tree stale dispatches only one reload. + pub(super) rebuild_in_flight: bool, } impl HeaderSyncCore { pub(super) fn new(startup: &HeaderSyncStartup) -> Result { validate_anchor(&startup.network, startup.anchor)?; let (best_header_tip, best_header_hash) = startup.best_header_tip.unwrap_or(startup.anchor); + let best_header_history_tree = startup.best_header_history_tree.clone(); Ok(Self { anchor: startup.anchor, @@ -39,6 +58,8 @@ impl HeaderSyncCore { verified_block_hash: startup.frontiers.verified_block_hash, best_header_tip, best_header_hash, + best_header_parent_hash: startup.best_header_parent_hash, + best_header_history_tree, peers: HashMap::new(), parked_peers: HashSet::new(), seen: HeaderHashDedup::default(), @@ -47,6 +68,7 @@ impl HeaderSyncCore { pending_commits: HashMap::new(), advisory: HashMap::new(), stale_anchor: StaleAnchorFailures::default(), + rebuild_in_flight: false, }) } @@ -63,7 +85,28 @@ impl HeaderSyncCore { } let checkpoints = startup.network.checkpoint_list(); - let Some(start) = next_height(self.best_header_tip) else { + // Commitment-root work only runs through the VCT fast-sync handoff boundary. + // The root at `last_checkpoint` is confirmed by the next header, so root-carrying + // ranges stop once the frontier reaches `last_checkpoint + 1`. + let last_checkpoint = startup.last_checkpoint_height; + let root_regime_end = next_height(last_checkpoint).unwrap_or(last_checkpoint); + // Only persist roots for below-checkpoint heights this node forward-syncs but has + // not committed yet. + let root_region_floor = self.anchor.0.max(self.finalized_height); + let below_root_boundary = + root_region_floor < last_checkpoint && self.best_header_tip < root_regime_end; + let want_tree_aux_roots = below_root_boundary; + + // Root-carrying ranges redeliver the tip header so its root can be confirmed. + let overlap_forward_range = below_root_boundary + && self + .best_header_parent_hash + .is_some_and(|_| self.best_header_tip > block::Height(0)); + let Some(start) = (if overlap_forward_range { + Some(self.best_header_tip) + } else { + next_height(self.best_header_tip) + }) else { return; }; let mut end = best_peer_tip; @@ -77,6 +120,10 @@ impl HeaderSyncCore { finalized = true; } } + // Keep root-carrying ranges at or below the confirming header for the last checkpoint. + if below_root_boundary { + end = end.min(root_regime_end); + } let count = count_between(start, end); if count == 0 { @@ -85,14 +132,28 @@ impl HeaderSyncCore { self.schedule.ensure_forward(RangeRequest { start_height: start, count, - anchor_hash: self.best_header_hash, + anchor_hash: if overlap_forward_range { + self.best_header_parent_hash + .expect("overlapped ranges have a parent hash") + } else { + self.best_header_hash + }, finalized, - want_tree_aux_roots: true, + want_tree_aux_roots, priority: RangePriority::Forward, }); } pub(super) fn refresh_backward_range(&mut self, startup: &HeaderSyncStartup) { + // Below-anchor checkpoint backfill is explicitly disabled: backward ranges fold onto the + // previous checkpoint's tree, which this reactor never tracks, so their roots cannot be + // verified, and the current node wiring never consumes backfilled headers. Leaving the + // scheduling live would silently emit unsupported below-anchor `GetHeaders` requests. + // Re-enable once backfill has a consumer and checkpoint-bracket tree tracking. + if !BACKWARD_CHECKPOINT_BACKFILL_ENABLED { + return; + } + if self.anchor.0 == block::Height(0) { return; } @@ -123,6 +184,27 @@ impl HeaderSyncCore { } } +#[derive(Clone, Debug)] +pub(super) struct PendingHeaderCommit { + /// The full range requested from the peer. + /// + /// A peer may legally return a short prefix of the requested range. Keep the + /// requested range so success, local commit failure, and reanchor cleanup can + /// clear or retry the scheduler assignment that was created for the original + /// `GetHeaders` request. + pub(super) requested_range: RangeRequest, + /// The range actually delivered by the peer and handed to the commit path. + /// + /// Successful commit events are reported for this delivered range, so coverage + /// and verified frontier-tree lookup must use this range rather than the + /// original request. + pub(super) delivered_range: RangeRequest, + /// `Some` for aux-validated forward ranges; `None` for checkpoint-authenticated backward + /// backfill ranges, which persist no provisional roots and never install a frontier tree. + pub(super) verified_roots: + Option, +} + #[derive(Clone, Debug, Default)] pub(super) struct StaleAnchorFailures { pub(super) count: u32, diff --git a/zebra-network/src/zakura/header_sync/tests.rs b/zebra-network/src/zakura/header_sync/tests.rs index bfe6257b667..b8697a1aeb9 100644 --- a/zebra-network/src/zakura/header_sync/tests.rs +++ b/zebra-network/src/zakura/header_sync/tests.rs @@ -16,6 +16,7 @@ use std::{ sync::{Mutex, OnceLock}, }; use zebra_chain::{ + history_tree::{HistoryTree, HistoryTreeBlockParts}, orchard, parallel::commitment_aux::BlockCommitmentRoots, parameters::{ @@ -30,7 +31,7 @@ use zebra_chain::{ }; use zebra_test::vectors::{ BLOCK_MAINNET_1_BYTES, BLOCK_MAINNET_2_BYTES, BLOCK_MAINNET_3_BYTES, BLOCK_MAINNET_4_BYTES, - BLOCK_MAINNET_GENESIS_BYTES, BLOCK_TESTNET_GENESIS_BYTES, + BLOCK_MAINNET_5_BYTES, BLOCK_MAINNET_GENESIS_BYTES, BLOCK_TESTNET_GENESIS_BYTES, }; #[derive(Default)] @@ -245,6 +246,48 @@ fn roots_from_height(start_height: block::Height, count: usize) -> Vec], +) -> Vec { + headers + .iter() + .enumerate() + .map(|(offset, header)| { + let offset = u32::try_from(offset).expect("test root count fits in u32"); + let height = block::Height(start_height.0 + offset); + let sapling_root = match header + .commitment(network, height) + .expect("test header commitment parses") + { + zebra_chain::block::Commitment::FinalSaplingRoot(root) => root, + _ => sapling::tree::NoteCommitmentTree::default().root(), + }; + BlockCommitmentRoots { + height, + sapling_root, + ..root_at(height) + } + }) + .collect() +} + +fn roots_message_from( + _start_height: block::Height, + headers: Vec>, + tree_aux_roots: Vec, +) -> HeaderSyncMessage { + HeaderSyncMessage::Headers { + body_sizes: vec![0; headers.len()], + headers, + tree_aux_roots, + } +} + async fn validate_headers_stateless_after_equihash_acceptance( headers: Vec>, context: HeaderSyncValidationContext<'_>, @@ -322,6 +365,30 @@ fn regtest_network() -> Network { Network::new_regtest(Default::default()) } +fn pre_sapling_checkpoint_regtest( + checkpoint_height: block::Height, + checkpoint_hash: block::Hash, +) -> Network { + let default_regtest = regtest_network(); + Network::new_regtest(RegtestParameters { + activation_heights: ConfiguredActivationHeights { + before_overwinter: Some(10), + ..Default::default() + }, + checkpoints: Some(ConfiguredCheckpoints::HeightsAndHashes(vec![ + (block::Height(0), default_regtest.genesis_hash()), + (checkpoint_height, checkpoint_hash), + ])), + ..Default::default() + }) +} + +fn pre_sapling_test_header(bytes: &[u8], previous_hash: block::Hash) -> Arc { + let mut header = *mainnet_header(bytes); + header.previous_block_hash = previous_hash; + Arc::new(header) +} + fn checkpoint_testnet_with_hash( checkpoint_height: block::Height, checkpoint_hash: block::Hash, @@ -393,6 +460,12 @@ fn startup_for( ); startup.range_state_actions_enabled = true; startup.inbound_new_block_acceptance_enabled = true; + // Test networks (regtest/custom) usually carry only a genesis checkpoint, so their real + // `max_height()` is 0 and every height would count as above the VCT handoff boundary. These + // fixtures exercise the below-checkpoint fast-sync machinery (root verify/persist/overlap), so + // push the boundary above all test heights. Tests that specifically exercise the at/above- + // checkpoint behavior override `last_checkpoint_height` back down. + startup.last_checkpoint_height = block::Height::MAX; startup } @@ -774,8 +847,7 @@ async fn next_non_query_action(actions: &mut mpsc::Receiver) - let action = next_action(actions).await; if !matches!( action, - HeaderSyncAction::QueryBestHeaderTip - | HeaderSyncAction::QueryMissingBlockBodies { .. } + HeaderSyncAction::QueryMissingBlockBodies { .. } | HeaderSyncAction::QueryHeadersByHeightRange { .. } | HeaderSyncAction::HeaderAdvanced { .. } ) { @@ -806,7 +878,9 @@ async fn next_outbound_get_headers( HeaderSyncMessage::GetHeaders { start_height, count, - want_tree_aux_roots: true, + // Accept both root-carrying (below-checkpoint) and plain (at/above-checkpoint) + // forward requests; callers that care about the flag assert it separately. + want_tree_aux_roots: _, }, } => return (peer, start_height, count), HeaderSyncAction::Misbehavior { peer, reason } => { @@ -817,6 +891,25 @@ async fn next_outbound_get_headers( } } +async fn next_forward_get_headers( + actions: &mut mpsc::Receiver, +) -> (block::Height, u32, bool) { + loop { + if let HeaderSyncAction::SendMessage { + msg: + HeaderSyncMessage::GetHeaders { + start_height, + count, + want_tree_aux_roots, + }, + .. + } = next_non_query_action(actions).await + { + return (start_height, count, want_tree_aux_roots); + } + } +} + async fn assert_no_commit_or_misbehavior(actions: &mut mpsc::Receiver) { while let Ok(Some(action)) = tokio::time::timeout(std::time::Duration::from_millis(50), actions.recv()).await @@ -1139,18 +1232,54 @@ fn headers_codec_rejects_body_size_mismatch_truncation_and_trailing_bytes() { } #[test] -fn decode_rejects_non_empty_headers_without_tree_aux_roots() { +fn header_aux_validation_rejects_bad_roots_before_state_commit() { + let header = mainnet_header(&BLOCK_MAINNET_1_BYTES); + let mut roots = root_at(block::Height(1)); + roots.orchard_root = + orchard::tree::Root::try_from([0u8; 32]).expect("zero is a valid Pallas base field"); + + assert_ne!( + roots.orchard_root, + orchard::tree::NoteCommitmentTree::default().root(), + "the bad root must differ from the pre-NU5 empty Orchard root" + ); + + assert!(matches!( + validate_header_aux_commitments( + &Network::Mainnet, + &HistoryTree::default(), + &[header], + &[roots] + ), + Err(HeaderSyncWireError::InvalidHeaderCommitment(_)) + )); +} + +/// Tree-aux roots are an optional serving capability: a solicited non-empty `Headers` +/// response without roots must decode (the reactor handles it non-punitively), while the +/// same rootless frame with no correlated request is still rejected as unsolicited. +#[test] +fn decode_accepts_rootless_non_empty_solicited_headers() { let headers = vec![mainnet_header(&BLOCK_MAINNET_1_BYTES)]; let mut encoded = headers_message(headers).encode().unwrap(); encoded[HEADER_SYNC_MESSAGE_TYPE_BYTES + HEADER_SYNC_COUNT_BYTES] = 0; encoded.truncate(encoded.len() - HEADER_SYNC_BLOCK_COMMITMENT_ROOTS_BYTES); + match HeaderSyncMessage::decode(&encoded, finalized_headers_context(1, 1)) { + Ok(HeaderSyncMessage::Headers { + headers, + tree_aux_roots, + .. + }) => { + assert_eq!(headers.len(), 1); + assert!(tree_aux_roots.is_empty()); + } + other => panic!("rootless solicited Headers must decode, got {other:?}"), + } + assert!(matches!( - HeaderSyncMessage::decode(&encoded, finalized_headers_context(1, 1)), - Err(HeaderSyncWireError::TreeAuxRootCountMismatch { - headers: 1, - roots: 0, - }) + HeaderSyncMessage::decode(&encoded, HeaderSyncDecodeContext::control()), + Err(HeaderSyncWireError::UnsolicitedHeaders) )); } @@ -1626,6 +1755,7 @@ async fn scheduler_narrows_large_ranges_before_tracking_fanout() { start_height: start, tip_height: chunk_tip, tip_hash: block::Hash([4; 32]), + tip_parent_hash: Some(block::Hash([3; 32])), }) .await .unwrap(); @@ -1641,10 +1771,7 @@ async fn scheduler_narrows_large_ranges_before_tracking_fanout() { .. } = next_non_query_action(&mut fixture.actions).await { - assert_eq!( - start_height, - next_height(chunk_tip).expect("committed chunk tip has successor") - ); + assert_eq!(start_height, chunk_tip,); assert_eq!(count, clamped_count); break; } @@ -1690,45 +1817,6 @@ async fn scheduler_creates_checkpoint_forward_before_backward_ranges() { } } -#[tokio::test(flavor = "current_thread")] -async fn scheduler_creates_backward_checkpoint_terminating_ranges() { - let (network, checkpoint_hash) = checkpoint_regtest(block::Height(3)); - let mut fixture = spawn_test_reactor(startup_for( - network, - (block::Height(3), checkpoint_hash), - Some((block::Height(3), checkpoint_hash)), - )); - let peer_id = peer(7); - - connect_peer(&fixture, peer_id.clone()).await; - advertise_tip( - &fixture, - peer_id, - block::Height(0), - block::Height(3), - DEFAULT_HS_RANGE, - 10, - ) - .await; - - loop { - if let HeaderSyncAction::SendMessage { - msg: - HeaderSyncMessage::GetHeaders { - start_height, - count, - want_tree_aux_roots: true, - }, - .. - } = next_non_query_action(&mut fixture.actions).await - { - assert_eq!(start_height, block::Height(1)); - assert_eq!(count, 3); - break; - } - } -} - #[tokio::test(flavor = "current_thread")] async fn forward_ranges_below_checkpoint_handoff_request_tree_aux_roots() { let network = Parameters::build() @@ -1870,8 +1958,11 @@ async fn incoming_headers_match_outstanding_before_commit() { } } +/// Tree-aux roots are an optional serving capability, so a rootless non-empty response to a +/// root-carrying request is not misbehavior: the headers are unusable for the root regime and +/// must not commit, but the peer keeps its session and the range is retried. #[tokio::test(flavor = "current_thread")] -async fn rootless_non_empty_response_is_malformed() { +async fn rootless_non_empty_response_retries_without_misbehavior() { let checkpoint_hash = block::Hash::from(mainnet_header(&BLOCK_MAINNET_3_BYTES).as_ref()); let (network, _) = checkpoint_testnet_with_hash(block::Height(3), checkpoint_hash); let first_checkpoint = block::Height(3); @@ -1909,12 +2000,21 @@ async fn rootless_non_empty_response_is_malformed() { .await .unwrap(); + // The range must be retried (a fresh `GetHeaders` for the same start height), with no + // commit and no misbehavior along the way. loop { match next_non_query_action(&mut fixture.actions).await { + HeaderSyncAction::SendMessage { + msg: + HeaderSyncMessage::GetHeaders { + start_height, + want_tree_aux_roots: true, + .. + }, + .. + } if start_height == start => break, HeaderSyncAction::Misbehavior { peer, reason } => { - assert_eq!(peer, peer_id); - assert_eq!(reason, HeaderSyncMisbehavior::MalformedMessage); - break; + panic!("rootless response must not be misbehavior, got {reason:?} from {peer:?}"); } HeaderSyncAction::CommitHeaderRange { .. } => { panic!("a rootless non-empty response must not commit") @@ -1922,6 +2022,7 @@ async fn rootless_non_empty_response_is_malformed() { _ => {} } } + assert_no_commit_or_misbehavior(&mut fixture.actions).await; } #[tokio::test(flavor = "current_thread")] @@ -2243,6 +2344,7 @@ async fn covered_hedged_outstanding_ranges_do_not_commit_twice() { start_height: block::Height(1), tip_height: block::Height(2), tip_hash: block::Hash([2; 32]), + tip_parent_hash: None, }) .await .unwrap(); @@ -2300,6 +2402,7 @@ async fn late_covered_response_does_not_reanchor_newer_outstanding_range() { start_height: block::Height(1), tip_height: block::Height(1), tip_hash: committed_hash, + tip_parent_hash: None, }) .await .unwrap(); @@ -2472,6 +2575,7 @@ async fn material_tip_advance_sends_rate_limited_unsolicited_status() { tip_hash: block::Hash( [u8::try_from(height.0).expect("test heights fit in u8"); 32], ), + tip_parent_hash: None, }) .await .unwrap(); @@ -2713,6 +2817,9 @@ async fn reconnect_clears_session_bound_outstanding_ranges() { #[tokio::test(flavor = "current_thread")] async fn full_block_committed_covers_outstanding_height() { let network = regtest_network(); + // A full-block commit covers the outstanding height and advances the header frontier (which tracks + // the committed chain). The in-memory tree is untouched; a later forward range would rebuild it + // lazily if it needed it. let mut fixture = spawn_test_reactor(startup_for( network.clone(), (block::Height(0), network.genesis_hash()), @@ -3904,6 +4011,7 @@ async fn header_sync_metrics_record_status_range_new_block_dedup_and_violation() start_height: next_height(first_checkpoint).expect("checkpoint has a successor"), tip_height: next_height(first_checkpoint).expect("checkpoint has a successor"), tip_hash: committed_hash, + tip_parent_hash: None, }) .await .unwrap(); @@ -4032,6 +4140,7 @@ async fn committed_range_updates_best_tip_watch_and_does_not_advance_finality() start_height: block::Height(1), tip_height: block::Height(1), tip_hash, + tip_parent_hash: None, }) .await .unwrap(); @@ -4041,468 +4150,653 @@ async fn committed_range_updates_best_tip_watch_and_does_not_advance_finality() assert_ne!(fixture.handle.best_header_tip().0, block::Height(0)); } -#[tokio::test(flavor = "current_thread")] -async fn forward_link_wedge_reanchors_to_verified_tip_without_banning() { - let network = regtest_network(); - let verified = (block::Height(0), network.genesis_hash()); - let stranded_tip = (block::Height(3), block::Hash([3; 32])); - let mut startup = HeaderSyncStartup::new( +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn invalid_header_aux_roots_score_before_commit() { + let default_regtest = regtest_network(); + let block1 = pre_sapling_test_header(&BLOCK_MAINNET_1_BYTES, default_regtest.genesis_hash()); + let block1_hash = block::Hash::from(block1.as_ref()); + let network = pre_sapling_checkpoint_regtest(block::Height(1), block1_hash); + let mut fixture = spawn_test_reactor(startup_for( network.clone(), - verified, - HeaderSyncFrontiers { - finalized_height: verified.0, - verified_block_tip: verified.0, - verified_block_hash: verified.1, - }, - Some(stranded_tip), - ZakuraHeaderSyncConfig::default(), - LOCAL_MAX_MESSAGE_BYTES, - ); - startup.range_state_actions_enabled = true; - let mut fixture = spawn_test_reactor(startup); - let mut tip = fixture.handle.subscribe_tip(); - let peers = [peer(61), peer(62)]; + (block::Height(0), network.genesis_hash()), + None, + )); + let peer_id = peer(81); - for peer_id in peers.iter().cloned() { - connect_peer(&fixture, peer_id.clone()).await; - advertise_tip( - &fixture, - peer_id, - verified.0, - block::Height(4), - DEFAULT_HS_RANGE, - 1, - ) - .await; - } + connect_peer(&fixture, peer_id.clone()).await; + advertise_tip( + &fixture, + peer_id.clone(), + block::Height(0), + block::Height(1), + DEFAULT_HS_RANGE, + 1, + ) + .await; - for _ in 0..3 { - let (served_peer, start_height, count) = - next_outbound_get_headers(&mut fixture.actions).await; - assert_eq!(start_height, block::Height(4)); - assert_eq!(count, 1); - fixture - .handle - .send(HeaderSyncEvent::WireMessage { - peer: served_peer, - msg: headers_message_from( - start_height, - vec![mainnet_header(&BLOCK_MAINNET_1_BYTES)], - ), - }) - .await - .unwrap(); - } + let (served_peer, start_height, count) = next_outbound_get_headers(&mut fixture.actions).await; + assert_eq!(served_peer, peer_id); + assert_eq!(start_height, block::Height(1)); + assert_eq!(count, 1); - tip.changed().await.unwrap(); - assert_eq!(*tip.borrow(), verified); - assert_eq!(fixture.handle.best_header_tip(), verified); + let mut roots = roots_from_height(start_height, 1); + roots[0].sapling_root = + sapling::tree::Root::try_from([1u8; 32]).expect("test Sapling root is valid"); + fixture + .handle + .send(HeaderSyncEvent::WireMessage { + peer: peer_id.clone(), + msg: roots_message_from(start_height, vec![block1], roots), + }) + .await + .unwrap(); - let expected_start = verified.0.next().expect("genesis has a successor"); - let mut saw_reanchor_action = false; - for _ in 0..8 { + loop { match next_non_query_action(&mut fixture.actions).await { - HeaderSyncAction::HeaderReanchored { old, new } => { - assert_eq!(old, stranded_tip); - assert_eq!(new, verified); - saw_reanchor_action = true; - } - HeaderSyncAction::SendMessage { - msg: - HeaderSyncMessage::GetHeaders { - start_height, - count: _, - want_tree_aux_roots: true, - }, - .. - } if saw_reanchor_action && start_height == expected_start => { - assert_no_commit_or_misbehavior(&mut fixture.actions).await; - return; - } HeaderSyncAction::Misbehavior { peer, reason } => { - panic!("unexpected misbehavior from {peer:?}: {reason:?}"); + assert_eq!(peer, peer_id); + assert_eq!(reason, HeaderSyncMisbehavior::InvalidRange); + break; + } + HeaderSyncAction::CommitHeaderRange { .. } => { + panic!("invalid header aux roots must not reach state commit") } _ => {} } } - panic!("after re-anchor, header sync did not emit the reanchor action and request forward from the verified tip"); } -#[tokio::test(flavor = "current_thread")] -async fn single_peer_forward_link_failures_do_not_reanchor_globally() { - let network = regtest_network(); - let verified = (block::Height(0), network.genesis_hash()); - let stranded_tip = (block::Height(3), block::Hash([3; 32])); - let mut startup = HeaderSyncStartup::new( +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn valid_header_aux_roots_reach_commit_before_advancing_tip() { + let default_regtest = regtest_network(); + let block1 = pre_sapling_test_header(&BLOCK_MAINNET_1_BYTES, default_regtest.genesis_hash()); + let block1_hash = block::Hash::from(block1.as_ref()); + let network = pre_sapling_checkpoint_regtest(block::Height(1), block1_hash); + let mut fixture = spawn_test_reactor(startup_for( network.clone(), - verified, - HeaderSyncFrontiers { - finalized_height: verified.0, - verified_block_tip: verified.0, - verified_block_hash: verified.1, - }, - Some(stranded_tip), - ZakuraHeaderSyncConfig::default(), - LOCAL_MAX_MESSAGE_BYTES, - ); - startup.range_state_actions_enabled = true; - let mut fixture = spawn_test_reactor(startup); - let mut tip = fixture.handle.subscribe_tip(); - let peer_id = peer(63); + (block::Height(0), network.genesis_hash()), + None, + )); + let peer_id = peer(82); connect_peer(&fixture, peer_id.clone()).await; advertise_tip( &fixture, - peer_id, - verified.0, - block::Height(4), + peer_id.clone(), + block::Height(0), + block::Height(1), DEFAULT_HS_RANGE, 1, ) .await; - for _ in 0..3 { - let (served_peer, start_height, count) = - next_outbound_get_headers(&mut fixture.actions).await; - assert_eq!(start_height, block::Height(4)); - assert_eq!(count, 1); - fixture - .handle - .send(HeaderSyncEvent::WireMessage { - peer: served_peer, - msg: headers_message_from( - start_height, - vec![mainnet_header(&BLOCK_MAINNET_1_BYTES)], - ), - }) - .await - .unwrap(); - } - - assert!( - tokio::time::timeout(std::time::Duration::from_millis(50), tip.changed()) - .await - .is_err(), - "one peer alone must not lower the global header frontier" - ); - assert_eq!(fixture.handle.best_header_tip(), stranded_tip); - assert_no_commit_or_misbehavior(&mut fixture.actions).await; -} - -#[tokio::test(flavor = "current_thread")] -async fn forward_genesis_backfill_reaches_checkpoint_before_finalized_commit() { - let headers = [ - mainnet_header(&BLOCK_MAINNET_1_BYTES), - mainnet_header(&BLOCK_MAINNET_2_BYTES), - mainnet_header(&BLOCK_MAINNET_3_BYTES), - ]; - let checkpoint_hash = block::Hash::from(headers[2].as_ref()); - let (network, _) = checkpoint_testnet_with_hash(block::Height(3), checkpoint_hash); - let mut fixture = spawn_test_reactor(startup_for( - network.clone(), - (block::Height(0), network.genesis_hash()), - None, - )); - let peer_id = peer(43); - - connect_peer(&fixture, peer_id.clone()).await; - advertise_tip( - &fixture, - peer_id.clone(), - block::Height(0), - block::Height(3), - DEFAULT_HS_RANGE, - 1, - ) - .await; - loop { - if let HeaderSyncAction::SendMessage { - msg: - HeaderSyncMessage::GetHeaders { - start_height, - count, - want_tree_aux_roots: true, - }, - .. - } = next_non_query_action(&mut fixture.actions).await - { - assert_eq!(start_height, block::Height(1)); - assert_eq!(count, 3); - break; - } - } + let (served_peer, start_height, count) = next_outbound_get_headers(&mut fixture.actions).await; + assert_eq!(served_peer, peer_id); + assert_eq!(start_height, block::Height(1)); + assert_eq!(count, 1); fixture .handle .send(HeaderSyncEvent::WireMessage { peer: peer_id.clone(), - msg: finalized_headers_message(headers.to_vec()), + msg: finalized_headers_message_from(start_height, vec![block1]), }) .await .unwrap(); - match next_non_query_action(&mut fixture.actions).await { - HeaderSyncAction::CommitHeaderRange { - peer, - start_height, - headers, - finalized, - .. - } => { - assert_eq!(peer, peer_id); - assert_eq!(start_height, block::Height(1)); - assert_eq!(headers.len(), 3); - assert!(finalized); + loop { + match next_non_query_action(&mut fixture.actions).await { + HeaderSyncAction::CommitHeaderRange { + peer, + start_height, + headers, + verified_roots, + .. + } => { + assert_eq!(peer, peer_id); + assert_eq!(start_height, block::Height(1)); + assert_eq!(headers.len(), 1); + let verified_roots = + verified_roots.expect("forward header range carries verified roots"); + assert!( + verified_roots.confirmed_roots().is_empty(), + "a one-header range verifies but has no confirmed prefix yet" + ); + assert_eq!(fixture.handle.best_header_tip().0, block::Height(0)); + break; + } + HeaderSyncAction::Misbehavior { peer, reason } => { + panic!("unexpected misbehavior from {peer:?}: {reason:?}"); + } + _ => {} } - action => panic!("unexpected action: {action:?}"), } } -#[tokio::test(flavor = "current_thread")] -async fn truncated_finalized_backfill_is_rejected_before_commit() { - let headers = [ - mainnet_header(&BLOCK_MAINNET_1_BYTES), - mainnet_header(&BLOCK_MAINNET_2_BYTES), - mainnet_header(&BLOCK_MAINNET_3_BYTES), - ]; - let checkpoint_hash = block::Hash::from(headers[2].as_ref()); - let (network, _) = checkpoint_testnet_with_hash(block::Height(3), checkpoint_hash); +/// When the in-memory tree is behind the header frontier β€” here a startup with a durable best header +/// tip (4) but an empty tree, standing in for a non-Zakura commit that ran the frontier ahead of the +/// tree β€” the forward range that needs the tree triggers exactly one `QueryBestHeaderHistoryTree` +/// lazy rebuild (guarded), with no commit and no misbehavior. +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn missing_header_aux_tree_triggers_single_lazy_rebuild() { + let block4 = mainnet_header(&BLOCK_MAINNET_4_BYTES); + let block4_hash = block::Hash::from(block4.as_ref()); + let block5 = mainnet_header(&BLOCK_MAINNET_5_BYTES); + let block5_hash = block::Hash::from(block5.as_ref()); + let (network, _) = checkpoint_testnet_with_hash(block::Height(5), block5_hash); let mut fixture = spawn_test_reactor(startup_for( network.clone(), (block::Height(0), network.genesis_hash()), - None, + Some((block::Height(4), block4_hash)), )); - let peer_id = peer(44); + let peer_id = peer(83); connect_peer(&fixture, peer_id.clone()).await; advertise_tip( &fixture, peer_id.clone(), block::Height(0), - block::Height(3), + block::Height(5), DEFAULT_HS_RANGE, 1, ) .await; - loop { - if matches!( - next_non_query_action(&mut fixture.actions).await, - HeaderSyncAction::SendMessage { - msg: HeaderSyncMessage::GetHeaders { .. }, - .. - } - ) { - break; - } - } + + let (served_peer, start_height, count) = next_outbound_get_headers(&mut fixture.actions).await; + assert_eq!(served_peer, peer_id); + assert_eq!(start_height, block::Height(5)); + assert_eq!(count, 1); fixture .handle .send(HeaderSyncEvent::WireMessage { - peer: peer_id.clone(), - msg: headers_message(headers[..2].to_vec()), + peer: peer_id, + msg: finalized_headers_message_from(start_height, vec![block5]), }) .await .unwrap(); - match next_non_query_action(&mut fixture.actions).await { - HeaderSyncAction::Misbehavior { peer, reason } => { - assert_eq!(peer, peer_id); - assert_eq!(reason, HeaderSyncMisbehavior::InvalidRange); + // The stale tree triggers exactly one rebuild at the current frontier; never a commit or a score. + let mut rebuilds = 0; + for _ in 0..16 { + match tokio::time::timeout( + std::time::Duration::from_millis(200), + fixture.actions.recv(), + ) + .await + { + Ok(Some(HeaderSyncAction::QueryBestHeaderHistoryTree { + best_header_tip, .. + })) => { + assert_eq!(best_header_tip, block::Height(4)); + rebuilds += 1; + } + Ok(Some(HeaderSyncAction::CommitHeaderRange { .. })) => { + panic!("a stale tree must not commit the range"); + } + Ok(Some(HeaderSyncAction::Misbehavior { .. })) => { + panic!("a stale tree must not score the peer"); + } + Ok(Some(_)) => {} + _ => break, } - action => panic!("unexpected action: {action:?}"), } - assert_no_commit_or_misbehavior(&mut fixture.actions).await; + assert_eq!( + rebuilds, 1, + "a stale tree triggers exactly one lazy rebuild" + ); } -#[tokio::test(flavor = "current_thread")] -async fn backward_checkpoint_backfill_accepts_linking_run_as_finalized() { - let headers = [ - mainnet_header(&BLOCK_MAINNET_1_BYTES), - mainnet_header(&BLOCK_MAINNET_2_BYTES), - mainnet_header(&BLOCK_MAINNET_3_BYTES), - ]; - let checkpoint_hash = block::Hash::from(headers[2].as_ref()); - let (network, _) = checkpoint_testnet_with_hash(block::Height(3), checkpoint_hash); +/// A failed rebuild (`BestHeaderHistoryTreeLoaded { history_tree: None }`, e.g. a state read error) +/// must clear the in-flight guard so a subsequent stale forward range can re-trigger the rebuild β€” the +/// guard must never wedge future rebuilds. Regression test for the lazy-rebuild robustness path. +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn failed_rebuild_clears_guard_and_retriggers() { + let block4 = mainnet_header(&BLOCK_MAINNET_4_BYTES); + let block4_hash = block::Hash::from(block4.as_ref()); + let block5 = mainnet_header(&BLOCK_MAINNET_5_BYTES); + let block5_hash = block::Hash::from(block5.as_ref()); + let (network, _) = checkpoint_testnet_with_hash(block::Height(5), block5_hash); let mut fixture = spawn_test_reactor(startup_for( - network, - (block::Height(3), checkpoint_hash), - Some((block::Height(3), checkpoint_hash)), + network.clone(), + (block::Height(0), network.genesis_hash()), + Some((block::Height(4), block4_hash)), )); - let peer_id = peer(45); + let peer_id = peer(88); connect_peer(&fixture, peer_id.clone()).await; advertise_tip( &fixture, peer_id.clone(), block::Height(0), - block::Height(3), + block::Height(5), DEFAULT_HS_RANGE, 1, ) .await; - loop { - if matches!( - next_non_query_action(&mut fixture.actions).await, - HeaderSyncAction::SendMessage { - msg: HeaderSyncMessage::GetHeaders { .. }, - .. + + // Drive one stale-tree round, feed a *failed* rebuild, then drive a second stale-tree round. + // A second `QueryBestHeaderHistoryTree` must be dispatched β€” proving the guard was cleared by the + // `None` completion rather than wedged. + let mut rebuilds = 0; + for round in 0..2 { + let (_p, start_height, _c) = next_outbound_get_headers(&mut fixture.actions).await; + fixture + .handle + .send(HeaderSyncEvent::WireMessage { + peer: peer_id.clone(), + msg: finalized_headers_message_from(start_height, vec![block5.clone()]), + }) + .await + .unwrap(); + // Wait for this round's rebuild dispatch. + let mut saw = false; + for _ in 0..16 { + match tokio::time::timeout( + std::time::Duration::from_millis(200), + fixture.actions.recv(), + ) + .await + { + Ok(Some(HeaderSyncAction::QueryBestHeaderHistoryTree { .. })) => { + rebuilds += 1; + saw = true; + break; + } + Ok(Some(HeaderSyncAction::SendMessage { + msg: HeaderSyncMessage::GetHeaders { .. }, + .. + })) => { /* the retry re-request; ignore until the rebuild dispatch */ } + Ok(Some(_)) => {} + _ => break, } - ) { - break; } + assert!(saw, "round {round}: expected a rebuild dispatch"); + // Fail the rebuild β€” the guard must clear so the next round can re-trigger. + fixture + .handle + .send(HeaderSyncEvent::BestHeaderHistoryTreeLoaded { + best_header_tip: block::Height(4), + reanchor: None, + history_tree: None, + }) + .await + .unwrap(); } + assert_eq!( + rebuilds, 2, + "a failed rebuild must clear the guard so the next stale range re-triggers it", + ); +} + +/// A gossiped `NewBlock` accepted ahead of the header frontier advances the header tip β€” the frontier +/// tracks the committed chain. It does not touch the in-memory tree; if this were below the checkpoint +/// and a later forward range needed the tree, the lazy rebuild would reposition it. +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn gossip_tip_advance_advances_tip() { + let network = Network::Mainnet; + let block = mainnet_block(&BLOCK_MAINNET_1_BYTES); + let hash = block.hash(); + let height = block.coinbase_height().expect("test block has height"); + let fixture = spawn_test_reactor(startup_for( + network.clone(), + (block::Height(0), network.genesis_hash()), + None, + )); fixture .handle - .send(HeaderSyncEvent::WireMessage { - peer: peer_id.clone(), - msg: headers_message(headers.to_vec()), + .send(HeaderSyncEvent::NewBlockAccepted { + peer: peer(84), + height, + hash, + block, }) .await .unwrap(); - match next_non_query_action(&mut fixture.actions).await { - HeaderSyncAction::CommitHeaderRange { - peer, - start_height, - headers, - finalized, - .. - } => { - assert_eq!(peer, peer_id); - assert_eq!(start_height, block::Height(1)); - assert_eq!(headers.len(), 3); - assert!(finalized); + // The tip advances to the gossiped height (the tip watch is updated synchronously by the bump). + let mut tip = fixture.handle.subscribe_tip(); + for _ in 0..16 { + if fixture.handle.best_header_tip().0 == height { + break; } - action => panic!("unexpected action: {action:?}"), + let _ = tokio::time::timeout(std::time::Duration::from_millis(100), tip.changed()).await; } + assert_eq!( + fixture.handle.best_header_tip().0, + height, + "a gossip accepted ahead of the frontier must advance the header tip", + ); } +/// The root-carrying regime runs *through* the last checkpoint inclusive β€” the VCT handoff block at +/// `last_checkpoint` reads its own persisted root β€” and stops one block above. Confirming the root at +/// `last_checkpoint` needs the `last_checkpoint + 1` header, so the range is capped there. #[tokio::test(flavor = "current_thread")] -async fn checkpoint_backfill_rejects_non_contiguous_run_before_commit() { - let (network, checkpoint_hash) = checkpoint_regtest(block::Height(3)); - let mut fixture = spawn_test_reactor(startup_for( - network, - (block::Height(3), checkpoint_hash), - Some((block::Height(3), checkpoint_hash)), - )); - let peer_id = peer(10); - - connect_peer(&fixture, peer_id.clone()).await; +async fn forward_ranges_request_roots_through_the_last_checkpoint() { + let network = regtest_network(); + let genesis = (block::Height(0), network.genesis_hash()); + let boundary = block::Height(10); + + // Well below the checkpoint: request roots; capped at `last_checkpoint + 1` (heights 1..=11) so the + // range reaching the checkpoint can confirm the root at `last_checkpoint`. + let mut below = startup_for(network.clone(), genesis, None); + below.last_checkpoint_height = boundary; + let mut fixture = spawn_test_reactor(below); + connect_peer(&fixture, peer(70)).await; advertise_tip( &fixture, - peer_id.clone(), - block::Height(0), - block::Height(3), + peer(70), + genesis.0, + block::Height(20), DEFAULT_HS_RANGE, 1, ) .await; - loop { - if matches!( - next_non_query_action(&mut fixture.actions).await, - HeaderSyncAction::SendMessage { - msg: HeaderSyncMessage::GetHeaders { .. }, - .. - } - ) { - break; - } - } + let (start, count, want) = next_forward_get_headers(&mut fixture.actions).await; + assert_eq!(start, block::Height(1)); + assert!(want, "a below-checkpoint forward range must request roots"); + assert_eq!( + count, 11, + "the range is capped one block above the last checkpoint" + ); - fixture - .handle - .send(HeaderSyncEvent::WireMessage { - peer: peer_id.clone(), - msg: headers_message_from( - block::Height(1), - vec![ - mainnet_header(&BLOCK_MAINNET_GENESIS_BYTES), - mainnet_header(&BLOCK_MAINNET_GENESIS_BYTES), - mainnet_header(&BLOCK_MAINNET_GENESIS_BYTES), - ], - ), - }) - .await - .unwrap(); + // Frontier at the last checkpoint: still in the root regime (this range confirms the root at + // `last_checkpoint` via the successor header). + let mut at = startup_for( + network.clone(), + genesis, + Some((boundary, block::Hash([10; 32]))), + ); + at.last_checkpoint_height = boundary; + let mut fixture = spawn_test_reactor(at); + connect_peer(&fixture, peer(71)).await; + advertise_tip( + &fixture, + peer(71), + boundary, + block::Height(20), + DEFAULT_HS_RANGE, + 1, + ) + .await; + let (_start, _count, want) = next_forward_get_headers(&mut fixture.actions).await; + assert!( + want, + "the range reaching the last checkpoint still requests roots (to confirm its own root)", + ); - match next_non_query_action(&mut fixture.actions).await { - HeaderSyncAction::Misbehavior { peer, reason } => { - assert_eq!(peer, peer_id); - assert_eq!(reason, HeaderSyncMisbehavior::InvalidRange); + // Frontier one block above the last checkpoint: plain, no roots. + let mut above = startup_for( + network.clone(), + genesis, + Some((block::Height(11), block::Hash([11; 32]))), + ); + above.last_checkpoint_height = boundary; + let mut fixture = spawn_test_reactor(above); + connect_peer(&fixture, peer(72)).await; + advertise_tip( + &fixture, + peer(72), + block::Height(11), + block::Height(20), + DEFAULT_HS_RANGE, + 1, + ) + .await; + let (start, _count, want) = next_forward_get_headers(&mut fixture.actions).await; + assert_eq!(start, block::Height(12)); + assert!( + !want, + "above the last checkpoint the forward range is plain" + ); +} + +#[tokio::test(flavor = "current_thread")] +async fn forward_link_wedge_reanchors_to_verified_tip_without_banning() { + let network = regtest_network(); + let verified = (block::Height(0), network.genesis_hash()); + let stranded_tip = (block::Height(3), block::Hash([3; 32])); + let mut startup = HeaderSyncStartup::new( + network.clone(), + verified, + HeaderSyncFrontiers { + finalized_height: verified.0, + verified_block_tip: verified.0, + verified_block_hash: verified.1, + }, + Some(stranded_tip), + ZakuraHeaderSyncConfig::default(), + LOCAL_MAX_MESSAGE_BYTES, + ); + startup.range_state_actions_enabled = true; + let mut fixture = spawn_test_reactor(startup); + let mut tip = fixture.handle.subscribe_tip(); + let peers = [peer(61), peer(62)]; + + for peer_id in peers.iter().cloned() { + connect_peer(&fixture, peer_id.clone()).await; + advertise_tip( + &fixture, + peer_id, + verified.0, + block::Height(4), + DEFAULT_HS_RANGE, + 1, + ) + .await; + } + + for _ in 0..3 { + let (served_peer, start_height, count) = + next_outbound_get_headers(&mut fixture.actions).await; + assert_eq!(start_height, block::Height(4)); + assert_eq!(count, 1); + fixture + .handle + .send(HeaderSyncEvent::WireMessage { + peer: served_peer, + // Above the checkpoint the range carries no roots, so the (non-linking) response must + // be rootless too β€” else it is rejected as malformed before the link check. + msg: roots_message_from( + start_height, + vec![mainnet_header(&BLOCK_MAINNET_1_BYTES)], + Vec::new(), + ), + }) + .await + .unwrap(); + } + + tip.changed().await.unwrap(); + assert_eq!(*tip.borrow(), verified); + assert_eq!(fixture.handle.best_header_tip(), verified); + + let expected_start = verified.0.next().expect("genesis has a successor"); + let mut saw_reanchor_action = false; + for _ in 0..8 { + match next_non_query_action(&mut fixture.actions).await { + HeaderSyncAction::HeaderReanchored { old, new } => { + assert_eq!(old, stranded_tip); + assert_eq!(new, verified); + saw_reanchor_action = true; + } + HeaderSyncAction::SendMessage { + msg: + HeaderSyncMessage::GetHeaders { + start_height, + // This test exercises reanchor-and-resume, not the root regime; accept either + // flag (regtest's genesis-only `max_height` makes the resumed range's root + // flag a degenerate edge). + .. + }, + .. + } if saw_reanchor_action && start_height == expected_start => { + assert_no_commit_or_misbehavior(&mut fixture.actions).await; + return; + } + HeaderSyncAction::Misbehavior { peer, reason } => { + panic!("unexpected misbehavior from {peer:?}: {reason:?}"); + } + _ => {} } - action => panic!("unexpected action: {action:?}"), } + panic!("after re-anchor, header sync did not emit the reanchor action and request forward from the verified tip"); } #[tokio::test(flavor = "current_thread")] -async fn header_response_that_does_not_link_to_anchor_is_misbehavior_before_commit() { - let checkpoint_hash = block::Hash::from(mainnet_header(&BLOCK_MAINNET_3_BYTES).as_ref()); +async fn single_peer_forward_link_failures_do_not_reanchor_globally() { + let network = regtest_network(); + let verified = (block::Height(0), network.genesis_hash()); + let stranded_tip = (block::Height(3), block::Hash([3; 32])); + let mut startup = HeaderSyncStartup::new( + network.clone(), + verified, + HeaderSyncFrontiers { + finalized_height: verified.0, + verified_block_tip: verified.0, + verified_block_hash: verified.1, + }, + Some(stranded_tip), + ZakuraHeaderSyncConfig::default(), + LOCAL_MAX_MESSAGE_BYTES, + ); + startup.range_state_actions_enabled = true; + let mut fixture = spawn_test_reactor(startup); + let mut tip = fixture.handle.subscribe_tip(); + let peer_id = peer(63); + + connect_peer(&fixture, peer_id.clone()).await; + advertise_tip( + &fixture, + peer_id, + verified.0, + block::Height(4), + DEFAULT_HS_RANGE, + 1, + ) + .await; + + for _ in 0..3 { + let (served_peer, start_height, count) = + next_outbound_get_headers(&mut fixture.actions).await; + assert_eq!(start_height, block::Height(4)); + assert_eq!(count, 1); + fixture + .handle + .send(HeaderSyncEvent::WireMessage { + peer: served_peer, + // Above the checkpoint the range carries no roots, so the (non-linking) response must + // be rootless too β€” else it is rejected as malformed before the link check. + msg: roots_message_from( + start_height, + vec![mainnet_header(&BLOCK_MAINNET_1_BYTES)], + Vec::new(), + ), + }) + .await + .unwrap(); + } + + assert!( + tokio::time::timeout(std::time::Duration::from_millis(50), tip.changed()) + .await + .is_err(), + "one peer alone must not lower the global header frontier" + ); + assert_eq!(fixture.handle.best_header_tip(), stranded_tip); + assert_no_commit_or_misbehavior(&mut fixture.actions).await; +} + +#[tokio::test(flavor = "current_thread")] +async fn forward_genesis_backfill_reaches_checkpoint_before_finalized_commit() { + let headers = [ + mainnet_header(&BLOCK_MAINNET_1_BYTES), + mainnet_header(&BLOCK_MAINNET_2_BYTES), + mainnet_header(&BLOCK_MAINNET_3_BYTES), + ]; + let checkpoint_hash = block::Hash::from(headers[2].as_ref()); let (network, _) = checkpoint_testnet_with_hash(block::Height(3), checkpoint_hash); - let anchor = (block::Height(0), network.genesis_hash()); - let mut fixture = spawn_test_reactor(startup_for(network, anchor, Some(anchor))); - let peer_id = peer(46); + let mut fixture = spawn_test_reactor(startup_for( + network.clone(), + (block::Height(0), network.genesis_hash()), + None, + )); + let peer_id = peer(43); connect_peer(&fixture, peer_id.clone()).await; advertise_tip( &fixture, peer_id.clone(), block::Height(0), - block::Height(4), + block::Height(3), DEFAULT_HS_RANGE, 1, ) .await; loop { - if matches!( - next_non_query_action(&mut fixture.actions).await, - HeaderSyncAction::SendMessage { - msg: HeaderSyncMessage::GetHeaders { .. }, - .. - } - ) { + if let HeaderSyncAction::SendMessage { + msg: + HeaderSyncMessage::GetHeaders { + start_height, + count, + want_tree_aux_roots: true, + }, + .. + } = next_non_query_action(&mut fixture.actions).await + { + assert_eq!(start_height, block::Height(1)); + assert_eq!(count, 3); break; } } + // Supply roots matching each header's own Sapling commitment: this multi-block forward range is + // aux-validated (Sapling activates at height 2 on this fixture network), so empty roots would be + // correctly rejected. The commit path is exercised with roots that pass validation. + let matching_roots = header_matching_roots(&network, block::Height(1), &headers); fixture .handle .send(HeaderSyncEvent::WireMessage { peer: peer_id.clone(), - msg: headers_message_from( - block::Height(1), - vec![mainnet_header(&BLOCK_MAINNET_2_BYTES)], - ), + msg: roots_message_from(block::Height(1), headers.to_vec(), matching_roots), }) .await .unwrap(); match next_non_query_action(&mut fixture.actions).await { - HeaderSyncAction::Misbehavior { peer, reason } => { + HeaderSyncAction::CommitHeaderRange { + peer, + start_height, + headers, + finalized, + .. + } => { assert_eq!(peer, peer_id); - assert_eq!(reason, HeaderSyncMisbehavior::InvalidRange); + assert_eq!(start_height, block::Height(1)); + assert_eq!(headers.len(), 3); + assert!(finalized); } action => panic!("unexpected action: {action:?}"), } - assert_no_commit_or_misbehavior(&mut fixture.actions).await; } #[tokio::test(flavor = "current_thread")] -async fn checkpoint_backfill_rejects_checkpoint_hash_mismatch_before_commit() { +async fn truncated_finalized_backfill_is_rejected_before_commit() { let headers = [ mainnet_header(&BLOCK_MAINNET_1_BYTES), mainnet_header(&BLOCK_MAINNET_2_BYTES), mainnet_header(&BLOCK_MAINNET_3_BYTES), ]; - let divergent_checkpoint_hash = block::Hash::from(headers[0].as_ref()); - let (network, _) = checkpoint_testnet_with_hash(block::Height(3), divergent_checkpoint_hash); + let checkpoint_hash = block::Hash::from(headers[2].as_ref()); + let (network, _) = checkpoint_testnet_with_hash(block::Height(3), checkpoint_hash); let mut fixture = spawn_test_reactor(startup_for( - network, - (block::Height(3), divergent_checkpoint_hash), - Some((block::Height(3), divergent_checkpoint_hash)), + network.clone(), + (block::Height(0), network.genesis_hash()), + None, )); - let peer_id = peer(46); + let peer_id = peer(44); connect_peer(&fixture, peer_id.clone()).await; advertise_tip( @@ -4530,7 +4824,7 @@ async fn checkpoint_backfill_rejects_checkpoint_hash_mismatch_before_commit() { .handle .send(HeaderSyncEvent::WireMessage { peer: peer_id.clone(), - msg: headers_message(headers.to_vec()), + msg: headers_message(headers[..2].to_vec()), }) .await .unwrap(); @@ -4545,10 +4839,182 @@ async fn checkpoint_backfill_rejects_checkpoint_hash_mismatch_before_commit() { assert_no_commit_or_misbehavior(&mut fixture.actions).await; } -#[tokio::test(flavor = "multi_thread", worker_threads = 2)] -async fn stateless_validation_accepts_valid_contiguous_headers() { - let headers = vec![mainnet_header(&BLOCK_MAINNET_1_BYTES)]; - let context = HeaderSyncValidationContext { +// Backward (below-anchor) checkpoint backfill is explicitly disabled (see +// `backward_checkpoint_backfill_is_explicitly_disabled`), so finalized checkpoint-range +// validation is exercised through the forward genesis-backfill path below. + +#[tokio::test(flavor = "current_thread")] +async fn checkpoint_backfill_rejects_non_contiguous_run_before_commit() { + let (network, _checkpoint_hash) = checkpoint_regtest(block::Height(3)); + let mut fixture = spawn_test_reactor(startup_for( + network.clone(), + (block::Height(0), network.genesis_hash()), + None, + )); + let peer_id = peer(10); + + connect_peer(&fixture, peer_id.clone()).await; + advertise_tip( + &fixture, + peer_id.clone(), + block::Height(0), + block::Height(3), + DEFAULT_HS_RANGE, + 1, + ) + .await; + loop { + if matches!( + next_non_query_action(&mut fixture.actions).await, + HeaderSyncAction::SendMessage { + msg: HeaderSyncMessage::GetHeaders { .. }, + .. + } + ) { + break; + } + } + + fixture + .handle + .send(HeaderSyncEvent::WireMessage { + peer: peer_id.clone(), + msg: headers_message_from( + block::Height(1), + vec![ + mainnet_header(&BLOCK_MAINNET_GENESIS_BYTES), + mainnet_header(&BLOCK_MAINNET_GENESIS_BYTES), + mainnet_header(&BLOCK_MAINNET_GENESIS_BYTES), + ], + ), + }) + .await + .unwrap(); + + match next_non_query_action(&mut fixture.actions).await { + HeaderSyncAction::Misbehavior { peer, reason } => { + assert_eq!(peer, peer_id); + assert_eq!(reason, HeaderSyncMisbehavior::InvalidRange); + } + action => panic!("unexpected action: {action:?}"), + } +} + +#[tokio::test(flavor = "current_thread")] +async fn header_response_that_does_not_link_to_anchor_is_misbehavior_before_commit() { + let checkpoint_hash = block::Hash::from(mainnet_header(&BLOCK_MAINNET_3_BYTES).as_ref()); + let (network, _) = checkpoint_testnet_with_hash(block::Height(3), checkpoint_hash); + let anchor = (block::Height(0), network.genesis_hash()); + let mut fixture = spawn_test_reactor(startup_for(network, anchor, Some(anchor))); + let peer_id = peer(46); + + connect_peer(&fixture, peer_id.clone()).await; + advertise_tip( + &fixture, + peer_id.clone(), + block::Height(0), + block::Height(4), + DEFAULT_HS_RANGE, + 1, + ) + .await; + loop { + if matches!( + next_non_query_action(&mut fixture.actions).await, + HeaderSyncAction::SendMessage { + msg: HeaderSyncMessage::GetHeaders { .. }, + .. + } + ) { + break; + } + } + + fixture + .handle + .send(HeaderSyncEvent::WireMessage { + peer: peer_id.clone(), + msg: headers_message_from( + block::Height(1), + vec![mainnet_header(&BLOCK_MAINNET_2_BYTES)], + ), + }) + .await + .unwrap(); + + match next_non_query_action(&mut fixture.actions).await { + HeaderSyncAction::Misbehavior { peer, reason } => { + assert_eq!(peer, peer_id); + assert_eq!(reason, HeaderSyncMisbehavior::InvalidRange); + } + action => panic!("unexpected action: {action:?}"), + } + assert_no_commit_or_misbehavior(&mut fixture.actions).await; +} + +#[tokio::test(flavor = "current_thread")] +async fn checkpoint_backfill_rejects_checkpoint_hash_mismatch_before_commit() { + let headers = [ + mainnet_header(&BLOCK_MAINNET_1_BYTES), + mainnet_header(&BLOCK_MAINNET_2_BYTES), + mainnet_header(&BLOCK_MAINNET_3_BYTES), + ]; + let divergent_checkpoint_hash = block::Hash::from(headers[0].as_ref()); + let (network, _) = checkpoint_testnet_with_hash(block::Height(3), divergent_checkpoint_hash); + // Forward genesis backfill toward the (divergent) first checkpoint: the linking run's last + // header must hash-match the checkpoint or the whole finalized range is rejected. + let mut fixture = spawn_test_reactor(startup_for( + network.clone(), + (block::Height(0), network.genesis_hash()), + None, + )); + let peer_id = peer(46); + + connect_peer(&fixture, peer_id.clone()).await; + advertise_tip( + &fixture, + peer_id.clone(), + block::Height(0), + block::Height(3), + DEFAULT_HS_RANGE, + 1, + ) + .await; + loop { + if matches!( + next_non_query_action(&mut fixture.actions).await, + HeaderSyncAction::SendMessage { + msg: HeaderSyncMessage::GetHeaders { .. }, + .. + } + ) { + break; + } + } + + fixture + .handle + .send(HeaderSyncEvent::WireMessage { + peer: peer_id.clone(), + msg: headers_message(headers.to_vec()), + }) + .await + .unwrap(); + + match next_non_query_action(&mut fixture.actions).await { + HeaderSyncAction::Misbehavior { peer, reason } => { + assert_eq!(peer, peer_id); + assert_eq!(reason, HeaderSyncMisbehavior::InvalidRange); + } + action => panic!("unexpected action: {action:?}"), + } + assert_no_commit_or_misbehavior(&mut fixture.actions).await; +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn stateless_validation_accepts_valid_contiguous_headers() { + let headers = vec![mainnet_header(&BLOCK_MAINNET_1_BYTES)]; + let context = HeaderSyncValidationContext { network: &Network::Mainnet, now: Utc::now(), start_height: block::Height(1), @@ -4876,3 +5342,548 @@ async fn misbehavior_is_recorded_without_disconnecting_the_peer() { "misbehavior is record-only: an InvalidStatus peer must NOT be disconnected", ); } + +// --------------------------------------------------------------------------- +// Regression tests for requested-vs-delivered range identity and backfill. +// +// A server cannot serve the tree-aux root for its own header tip (the root is +// only confirmed by the next header), so the zebrad driver truncates +// root-carrying responses to root coverage. A successful-but-short delivery is +// therefore an expected, honest response β€” exactly the two-node bootstrap +// shape where a follower requests up to a header-leading peer's tip. The tests +// below assert the correct requester-side bookkeeping for that shape. +// --------------------------------------------------------------------------- + +/// A Heartwood-at-1 regtest (PoW checks are skipped on regtest) with checkpoints at genesis and +/// `checkpoint_height`, so synthetic post-Heartwood header chains carrying real ZIP-221 +/// commitments can drive the root-verifying forward path. +fn heartwood_regtest_with_checkpoint( + checkpoint_height: block::Height, + checkpoint_hash: block::Hash, +) -> Network { + let default_regtest = regtest_network(); + Network::new_regtest(RegtestParameters { + activation_heights: ConfiguredActivationHeights { + overwinter: Some(1), + sapling: Some(1), + blossom: Some(1), + heartwood: Some(1), + canopy: Some(1), + ..Default::default() + }, + checkpoints: Some(ConfiguredCheckpoints::HeightsAndHashes(vec![ + (block::Height(0), default_regtest.genesis_hash()), + (checkpoint_height, checkpoint_hash), + ])), + ..Default::default() + }) +} + +/// Builds a synthetic post-Heartwood header: real solution bytes from a mainnet vector, relinked +/// to `previous_hash`, with `commitment_bytes` set to the ZIP-221 chain history root of the +/// parent tree (the all-zero reserved value at Heartwood activation, where the tree is empty). +fn chain_history_test_header( + bytes: &[u8], + previous_hash: block::Hash, + parent_tree: &HistoryTree, +) -> Arc { + let mut header = *mainnet_header(bytes); + header.previous_block_hash = previous_hash; + header.commitment_bytes = parent_tree + .hash() + .map(<[u8; 32]>::from) + .unwrap_or(block::CHAIN_HISTORY_ACTIVATION_RESERVED) + .into(); + Arc::new(header) +} + +/// Folds the empty-root parts for `header` at `height` into `tree` β€” the same ZIP-221 leaf the +/// header-sync verifier folds when the peer supplies `root_at(height)` alongside the header. +fn fold_empty_roots( + network: &Network, + tree: &mut HistoryTree, + header: &block::Header, + height: block::Height, +) { + let roots = root_at(height); + tree.push_from_parts( + network, + HistoryTreeBlockParts { + header, + height, + sapling_root: &roots.sapling_root, + orchard_root: &roots.orchard_root, + ironwood_root: &roots.ironwood_root, + sapling_tx: roots.sapling_tx, + orchard_tx: roots.orchard_tx, + ironwood_tx: roots.ironwood_tx, + }, + ) + .expect("synthetic test chain extends the history tree"); +} + +/// Regression test: a legally-short served forward range must not +/// permanently wedge the scheduler. +/// +/// The short prefix commits, but the assignment for the *requested* range is only cleared on +/// disconnect, local commit failure, or timeout β€” never on the success path β€” and `prune_covered` +/// never fires for it because the requested end was never covered. The stale assignment then +/// blocks `RangeScheduler::ensure`'s start-height dedupe from ever queueing the follow-up +/// overlapping range (which shares the requested range's start height), so the follower wedges +/// below the checkpoint and never re-requests the missing height. +#[tokio::test(flavor = "current_thread")] +async fn short_served_forward_range_must_not_wedge_the_scheduler() { + let block3 = mainnet_header(&BLOCK_MAINNET_3_BYTES); + let block3_hash = block::Hash::from(block3.as_ref()); + let block4 = mainnet_header(&BLOCK_MAINNET_4_BYTES); + let block4_hash = block::Hash::from(block4.as_ref()); + let (network, _) = checkpoint_testnet_with_hash(block::Height(3), block3_hash); + let mut fixture = spawn_test_reactor(startup_for( + network.clone(), + (block::Height(0), network.genesis_hash()), + Some((block::Height(3), block3_hash)), + )); + let peer_id = peer(90); + + connect_peer(&fixture, peer_id.clone()).await; + advertise_tip( + &fixture, + peer_id.clone(), + block::Height(0), + block::Height(5), + DEFAULT_HS_RANGE, + 1, + ) + .await; + + // The follower requests up to the peer's tip: heights 4..=5, with roots. + let (served_peer, start_height, count) = next_outbound_get_headers(&mut fixture.actions).await; + assert_eq!(served_peer, peer_id); + assert_eq!(start_height, block::Height(4)); + assert_eq!(count, 2); + + // The peer serves a legally-short prefix: header 4 only, with its matching root (a server + // can never serve the root for its own tip, so it truncates the response to root coverage). + let matching_roots = + header_matching_roots(&network, block::Height(4), std::slice::from_ref(&block4)); + fixture + .handle + .send(HeaderSyncEvent::WireMessage { + peer: peer_id.clone(), + msg: roots_message_from(block::Height(4), vec![block4], matching_roots), + }) + .await + .unwrap(); + + // The short prefix is accepted and committed. + loop { + match next_non_query_action(&mut fixture.actions).await { + HeaderSyncAction::CommitHeaderRange { + peer, + start_height, + headers, + .. + } => { + assert_eq!(peer, peer_id); + assert_eq!(start_height, block::Height(4)); + assert_eq!(headers.len(), 1, "the short prefix is a legal response"); + break; + } + HeaderSyncAction::Misbehavior { peer, reason } => { + panic!("unexpected misbehavior from {peer:?}: {reason:?}"); + } + _ => {} + } + } + fixture + .handle + .send(HeaderSyncEvent::HeaderRangeCommitted { + start_height: block::Height(4), + tip_height: block::Height(4), + tip_hash: block4_hash, + tip_parent_hash: Some(block3_hash), + }) + .await + .unwrap(); + + // The uncovered remainder (height 5) must be re-requested from the peer. If the stale + // requested-range assignment is never cleared, the scheduler dedupes every follow-up range + // sharing its start height and no further `GetHeaders` is ever sent. + let requested_remainder = async { + loop { + match fixture.actions.recv().await { + Some(HeaderSyncAction::SendMessage { + msg: + HeaderSyncMessage::GetHeaders { + start_height, + count, + .. + }, + .. + }) => { + let end = start_height.0.saturating_add(count.saturating_sub(1)); + if start_height.0 <= 5 && end >= 5 { + break; + } + } + Some(HeaderSyncAction::Misbehavior { peer, reason }) => { + panic!("unexpected misbehavior from {peer:?}: {reason:?}"); + } + Some(_) => {} + None => panic!("reactor action channel closed"), + } + } + }; + tokio::time::timeout(std::time::Duration::from_secs(3), requested_remainder) + .await + .expect("scheduler wedged: height 5 was never re-requested after a legally-short delivery"); +} + +/// Regression test: a legally-short delivery must install the frontier +/// history tree it just verified, keyed by the *delivered* range β€” not discard it because the +/// pending-commit lookup matches on the requested range end. +/// +/// `pending_header_history_tree` matches pending commits on the requested `range.end_height()`, +/// so a short (but successful) delivery commits its headers while the verified tree for the +/// delivered prefix is dropped. The next overlapping forward range then finds the cached tree +/// behind its parent and dispatches a `QueryBestHeaderHistoryTree` rebuild β€” an O(frontier) +/// durable-state fold per short range β€” instead of committing directly. +#[tokio::test(flavor = "current_thread")] +async fn short_served_forward_range_must_install_the_delivered_frontier_tree() { + let genesis_hash = regtest_network().genesis_hash(); + + // A synthetic post-Heartwood chain 1..=4 with real ZIP-221 commitments: header H commits to + // the history tree of its parent, so the trees are folded alongside the headers. + let mut tree = HistoryTree::default(); + let h1 = chain_history_test_header(&BLOCK_MAINNET_1_BYTES, genesis_hash, &tree); + let h1_hash = block::Hash::from(h1.as_ref()); + let network = heartwood_regtest_with_checkpoint(block::Height(1), h1_hash); + + fold_empty_roots(&network, &mut tree, &h1, block::Height(1)); + let tree_at_1 = tree.clone(); + let h2 = chain_history_test_header(&BLOCK_MAINNET_2_BYTES, h1_hash, &tree); + let h2_hash = block::Hash::from(h2.as_ref()); + fold_empty_roots(&network, &mut tree, &h2, block::Height(2)); + let h3 = chain_history_test_header(&BLOCK_MAINNET_3_BYTES, h2_hash, &tree); + let h3_hash = block::Hash::from(h3.as_ref()); + fold_empty_roots(&network, &mut tree, &h3, block::Height(3)); + let h4 = chain_history_test_header(&BLOCK_MAINNET_4_BYTES, h3_hash, &tree); + + let mut startup = startup_for( + network.clone(), + (block::Height(0), genesis_hash), + Some((block::Height(1), h1_hash)), + ); + startup.best_header_history_tree = Arc::new(tree_at_1); + let mut fixture = spawn_test_reactor(startup); + let peer_id = peer(91); + + connect_peer(&fixture, peer_id.clone()).await; + advertise_tip( + &fixture, + peer_id.clone(), + block::Height(0), + block::Height(4), + DEFAULT_HS_RANGE, + 1, + ) + .await; + + // Heights 2..=4 are requested with roots. + let (served_peer, start_height, count) = next_outbound_get_headers(&mut fixture.actions).await; + assert_eq!(served_peer, peer_id); + assert_eq!(start_height, block::Height(2)); + assert_eq!(count, 3); + + // The peer legally serves a short prefix (2..=3); verification confirms and folds height 2, + // positioning the just-verified frontier tree at height 2. + fixture + .handle + .send(HeaderSyncEvent::WireMessage { + peer: peer_id.clone(), + msg: headers_message_from(block::Height(2), vec![h2.clone(), h3.clone()]), + }) + .await + .unwrap(); + loop { + match next_non_query_action(&mut fixture.actions).await { + HeaderSyncAction::CommitHeaderRange { + start_height, + headers, + verified_roots, + .. + } => { + assert_eq!(start_height, block::Height(2)); + assert_eq!(headers.len(), 2); + let verified_roots = verified_roots.expect("root-carrying range verifies"); + assert_eq!( + verified_roots.confirmed_tip(), + Some(block::Height(2)), + "the delivered prefix confirms height 2 and folds its root", + ); + break; + } + HeaderSyncAction::Misbehavior { peer, reason } => { + panic!("unexpected misbehavior from {peer:?}: {reason:?}"); + } + _ => {} + } + } + fixture + .handle + .send(HeaderSyncEvent::HeaderRangeCommitted { + start_height: block::Height(2), + tip_height: block::Height(3), + tip_hash: h3_hash, + tip_parent_hash: Some(h2_hash), + }) + .await + .unwrap(); + + // The next forward range overlaps the committed tip: heights 3..=4 anchored on header 2. + let (served_peer, start_height, count) = next_outbound_get_headers(&mut fixture.actions).await; + assert_eq!(served_peer, peer_id); + assert_eq!(start_height, block::Height(3)); + assert_eq!(count, 2); + + fixture + .handle + .send(HeaderSyncEvent::WireMessage { + peer: peer_id.clone(), + msg: headers_message_from(block::Height(3), vec![h3, h4]), + }) + .await + .unwrap(); + + // With the delivered tree installed at height 2, this range verifies against it and commits + // directly. Discarding the just-verified tree instead forces a `QueryBestHeaderHistoryTree` + // rebuild from durable state β€” the recurring-cost path this test guards against. + loop { + match next_non_query_action(&mut fixture.actions).await { + HeaderSyncAction::CommitHeaderRange { + start_height, + headers, + .. + } => { + assert_eq!(start_height, block::Height(3)); + assert_eq!(headers.len(), 2); + break; + } + HeaderSyncAction::QueryBestHeaderHistoryTree { .. } => { + panic!( + "the verified frontier tree for the delivered prefix was discarded: the \ + overlapping follow-up range should verify against the installed tree, not \ + trigger a durable-state rebuild" + ); + } + HeaderSyncAction::Misbehavior { peer, reason } => { + panic!("unexpected misbehavior from {peer:?}: {reason:?}"); + } + _ => {} + } + } +} + +/// Regression test for the runtime lazy-rebuild reanchor. A non-Zakura path (checkpoint/legacy sync) +/// runs the header tip ahead of the folded tree, so the forward range that needs the tree triggers a +/// `QueryBestHeaderHistoryTree` rebuild. When durable roots have a gap the rebuilt tree lands *below* +/// `best_header_tip - 1`, so the answer carries a [`HeaderFrontierReanchor`]. The reactor must drop its +/// tip onto the rebuilt tree (emitting `HeaderReanchored`) and then verify the resumed forward range +/// against it β€” instead of keeping the stale higher tip and re-triggering the identical deterministic +/// rebuild forever (the pre-fix loop, where the frontier was discarded before reaching the reactor). +#[tokio::test(flavor = "current_thread")] +async fn lazy_rebuild_reanchors_tip_onto_lower_frontier_tree() { + let genesis_hash = regtest_network().genesis_hash(); + + // A synthetic post-Heartwood chain 1..=4 with real ZIP-221 commitments, so the resumed range can + // fold and verify against the reinstalled tree. + let mut tree = HistoryTree::default(); + let h1 = chain_history_test_header(&BLOCK_MAINNET_1_BYTES, genesis_hash, &tree); + let h1_hash = block::Hash::from(h1.as_ref()); + let network = heartwood_regtest_with_checkpoint(block::Height(1), h1_hash); + + fold_empty_roots(&network, &mut tree, &h1, block::Height(1)); + let tree_at_1 = tree.clone(); + let h2 = chain_history_test_header(&BLOCK_MAINNET_2_BYTES, h1_hash, &tree); + let h2_hash = block::Hash::from(h2.as_ref()); + fold_empty_roots(&network, &mut tree, &h2, block::Height(2)); + let h3 = chain_history_test_header(&BLOCK_MAINNET_3_BYTES, h2_hash, &tree); + let h3_hash = block::Hash::from(h3.as_ref()); + fold_empty_roots(&network, &mut tree, &h3, block::Height(3)); + let h4 = chain_history_test_header(&BLOCK_MAINNET_4_BYTES, h3_hash, &tree); + + // The header tip sits at 3 (bumped by a non-Zakura commit) but the header-frontier tree is still + // the empty startup default β€” the frontier folded nothing above the verified tip at 1. + let mut fixture = spawn_test_reactor(startup_for( + network.clone(), + (block::Height(1), h1_hash), + Some((block::Height(3), h3_hash)), + )); + let peer_id = peer(94); + + connect_peer(&fixture, peer_id.clone()).await; + advertise_tip( + &fixture, + peer_id.clone(), + block::Height(0), + block::Height(4), + DEFAULT_HS_RANGE, + 1, + ) + .await; + + // The forward range above the stale tip is requested and served; aux validation finds the tree + // behind the range's parent and dispatches exactly one rebuild against the current tip (3). + let (served_peer, start_height, _count) = next_outbound_get_headers(&mut fixture.actions).await; + assert_eq!(served_peer, peer_id); + assert_eq!(start_height, block::Height(4)); + fixture + .handle + .send(HeaderSyncEvent::WireMessage { + peer: peer_id.clone(), + msg: headers_message_from(block::Height(4), vec![h4.clone()]), + }) + .await + .unwrap(); + loop { + match next_non_query_action(&mut fixture.actions).await { + HeaderSyncAction::QueryBestHeaderHistoryTree { + best_header_tip, .. + } => { + assert_eq!(best_header_tip, block::Height(3)); + break; + } + HeaderSyncAction::CommitHeaderRange { .. } => { + panic!("a stale tree must not commit the range") + } + HeaderSyncAction::Misbehavior { peer, reason } => { + panic!("unexpected misbehavior from {peer:?}: {reason:?}") + } + _ => {} + } + } + + // Answer the rebuild with a tree that folded only to the confirmed frontier at height 1 β€” below + // `best_header_tip - 1` (2), a durable-root gap β€” plus the reanchor onto it. + fixture + .handle + .send(HeaderSyncEvent::BestHeaderHistoryTreeLoaded { + best_header_tip: block::Height(3), + reanchor: Some(HeaderFrontierReanchor { + tip: block::Height(2), + tip_hash: h2_hash, + parent_hash: h1_hash, + }), + history_tree: Some(Arc::new(tree_at_1)), + }) + .await + .unwrap(); + + // The reactor drops the tip onto the rebuilt tree and resumes the forward range from the reanchor, + // verifying it against the reinstalled tree β€” never a second rebuild. + let mut saw_reanchor = false; + loop { + match next_non_query_action(&mut fixture.actions).await { + HeaderSyncAction::HeaderReanchored { new, .. } => { + assert_eq!(new.0, block::Height(2), "tip reanchored onto the frontier"); + saw_reanchor = true; + } + HeaderSyncAction::SendMessage { + msg: + HeaderSyncMessage::GetHeaders { + start_height, + count, + .. + }, + .. + } => { + // Ignore the pre-reanchor retry of the stale [4..] range; only serve the range the + // reactor resumes from the reanchor at height 2. + if start_height != block::Height(2) { + continue; + } + let headers = [h2.clone(), h3.clone(), h4.clone()]; + let take = (count as usize).min(headers.len()); + fixture + .handle + .send(HeaderSyncEvent::WireMessage { + peer: peer_id.clone(), + msg: headers_message_from(block::Height(2), headers[..take].to_vec()), + }) + .await + .unwrap(); + } + HeaderSyncAction::CommitHeaderRange { start_height, .. } => { + assert_eq!(start_height, block::Height(2)); + break; + } + HeaderSyncAction::QueryBestHeaderHistoryTree { .. } => panic!( + "the reanchored tree was ignored: the resumed range re-triggered a durable-state \ + rebuild instead of verifying against the reinstalled tree" + ), + HeaderSyncAction::Misbehavior { peer, reason } => { + panic!("unexpected misbehavior from {peer:?}: {reason:?}") + } + _ => {} + } + } + assert!( + saw_reanchor, + "the tip must be reanchored down onto the rebuilt frontier tree" + ); +} + +/// Regression test: backward (below-anchor) checkpoint backfill is unused by the current sync +/// wiring and does not work with root verification (backward ranges fold onto the previous +/// checkpoint's tree, which this reactor never tracks), so it must be explicitly disabled rather +/// than left scheduling silently. +/// +/// A node anchored at a checkpoint above genesis whose forward frontier already matches the +/// peer's tip has no forward work; any `GetHeaders` it emits is the backward backfill bracket. +/// It must emit none. +#[tokio::test(flavor = "current_thread")] +async fn backward_checkpoint_backfill_is_explicitly_disabled() { + let checkpoint_hash = block::Hash::from(mainnet_header(&BLOCK_MAINNET_3_BYTES).as_ref()); + let (network, _) = checkpoint_testnet_with_hash(block::Height(3), checkpoint_hash); + let mut fixture = spawn_test_reactor(startup_for( + network, + (block::Height(3), checkpoint_hash), + Some((block::Height(3), checkpoint_hash)), + )); + let peer_id = peer(92); + + connect_peer(&fixture, peer_id.clone()).await; + advertise_tip( + &fixture, + peer_id.clone(), + block::Height(0), + block::Height(3), + DEFAULT_HS_RANGE, + 1, + ) + .await; + + // Drain actions for a bounded window: no backward `GetHeaders` may be scheduled. + while let Ok(Some(action)) = tokio::time::timeout( + std::time::Duration::from_millis(250), + fixture.actions.recv(), + ) + .await + { + if let HeaderSyncAction::SendMessage { + msg: + HeaderSyncMessage::GetHeaders { + start_height, + count, + .. + }, + .. + } = action + { + panic!( + "backward checkpoint backfill must be explicitly disabled, but a below-anchor \ + range was requested: start {start_height:?} count {count}" + ); + } + } +} diff --git a/zebra-network/src/zakura/header_sync/validation.rs b/zebra-network/src/zakura/header_sync/validation.rs index 6e16519e27a..b1b83aa477a 100644 --- a/zebra-network/src/zakura/header_sync/validation.rs +++ b/zebra-network/src/zakura/header_sync/validation.rs @@ -1,4 +1,8 @@ use super::{error::*, events::*, wire::*, *}; +use zebra_chain::{ + history_tree::HistoryTree, + parallel::commitment_aux_verify::{self, SuppliedRootsError, VerifiedHeaderCommitmentRoots}, +}; pub(super) fn validate_anchor( network: &Network, @@ -193,6 +197,37 @@ pub fn validate_header_range_links( validate_internal_continuity(headers) } +/// Verify supplied roots against the header commitments using `parent_history_tree`. +pub fn validate_header_aux_commitments( + network: &Network, + parent_history_tree: &HistoryTree, + headers: &[Arc], + tree_aux_roots: &[BlockCommitmentRoots], +) -> Result { + validate_tree_aux_roots_len(headers.len(), tree_aux_roots.len())?; + + let items = headers + .iter() + .zip(tree_aux_roots.iter()) + .map(|(header, roots)| (header.as_ref(), roots)); + + commitment_aux_verify::verify_supplied_roots_from_parts( + network, + parent_history_tree.clone(), + items, + ) + .map_err(|(_height, error)| supplied_roots_error_into_wire(error)) +} + +fn supplied_roots_error_into_wire(error: SuppliedRootsError) -> HeaderSyncWireError { + match error { + SuppliedRootsError::InvalidHeaderCommitment(error) => { + HeaderSyncWireError::InvalidHeaderCommitment(error) + } + SuppliedRootsError::HistoryTree(error) => HeaderSyncWireError::HistoryTree(error), + } +} + /// Run all context-free validation checks for an inbound full-block tip flood. #[tracing::instrument(skip(block, network))] pub async fn validate_new_block_stateless( diff --git a/zebra-network/src/zakura/testkit/cluster.rs b/zebra-network/src/zakura/testkit/cluster.rs index ab8539787ad..c572636381d 100644 --- a/zebra-network/src/zakura/testkit/cluster.rs +++ b/zebra-network/src/zakura/testkit/cluster.rs @@ -203,6 +203,19 @@ mod tests { } } + /// A `Headers` message carrying no tree-aux roots β€” the correct shape of a response to an + /// above-checkpoint (plain) range. Used by hostile-peer tests whose victims are past the VCT + /// handoff boundary, so a root-carrying response would be rejected as malformed before the + /// intended violation (out-of-range, too-long, bad-PoW, …) is even checked. + fn rootless_headers_message(headers: Vec>) -> HeaderSyncMessage { + let body_sizes = vec![0; headers.len()]; + HeaderSyncMessage::Headers { + headers, + body_sizes, + tree_aux_roots: Vec::new(), + } + } + fn root_at(height: block::Height) -> BlockCommitmentRoots { BlockCommitmentRoots { height, @@ -908,7 +921,10 @@ mod tests { } } HeaderSyncAction::QueryHeadersByHeightRange { - peer, start, count, .. + peer, + start, + count, + want_tree_aux_roots, } => { let headers = local .store @@ -917,11 +933,22 @@ mod tests { .headers_by_range(start, count); let returned_count = u32::try_from(headers.len()).unwrap_or(u32::MAX); if let Some(target) = peer_to_index.get(&peer) { + // Respect the request: an above-checkpoint (plain) range must be served + // without roots, or the requester rejects the response as malformed. + let msg = if want_tree_aux_roots { + headers_message(headers) + } else { + HeaderSyncMessage::Headers { + body_sizes: vec![0; headers.len()], + headers, + tree_aux_roots: Vec::new(), + } + }; let _ = nodes[*target] .handle .send(HeaderSyncEvent::WireMessage { peer: local.peer_id.clone(), - msg: headers_message(headers), + msg, }) .await; let _ = local @@ -962,6 +989,7 @@ mod tests { start_height, tip_height, tip_hash, + tip_parent_hash: None, }) .await; let _ = local @@ -982,18 +1010,19 @@ mod tests { } } } - HeaderSyncAction::QueryBestHeaderTip => { - let (tip_height, tip_hash) = local - .store - .lock() - .expect("test store mutex is not poisoned") - .best_header_tip(); + HeaderSyncAction::QueryBestHeaderHistoryTree { + best_header_tip, .. + } => { + // The e2e header store uses pre-Heartwood vectors, so the empty tree is the + // correct reconstruction; a post-Heartwood tree would be rejected on height. let _ = local .handle - .send(HeaderSyncEvent::HeaderRangeCommitted { - start_height: tip_height, - tip_height, - tip_hash, + .send(HeaderSyncEvent::BestHeaderHistoryTreeLoaded { + best_header_tip, + reanchor: None, + history_tree: Some(Arc::new( + zebra_chain::history_tree::HistoryTree::default(), + )), }) .await; } @@ -1192,6 +1221,21 @@ mod tests { } fn e2e_network(checkpoints: impl IntoIterator) -> Network { + e2e_network_with_shielded_activation(checkpoints, 1) + } + + /// Builds an e2e testnet whose Sapling..Canopy upgrades activate at `shielded_activation`. + /// + /// Overwinter stays at height 1 and NU5+ stay inactive. Setting `shielded_activation` above the + /// synced range keeps those heights in the pre-Sapling regime, where headers present unverified + /// `PreSaplingReserved` commitments β€” the only regime the empty placeholder tree-aux roots + /// served by the test peer can satisfy. `Canopy` must be present and + /// `max_checkpoint >= Canopy - 1` for the checkpoint-coverage check, so the highest usable value + /// is `max_checkpoint + 1`. + fn e2e_network_with_shielded_activation( + checkpoints: impl IntoIterator, + shielded_activation: u32, + ) -> Network { let checkpoints = std::iter::once((block::Height(0), mainnet_genesis_hash())) .chain(checkpoints.into_iter().map(|height| { ( @@ -1207,10 +1251,10 @@ mod tests { .with_activation_heights(ConfiguredActivationHeights { before_overwinter: None, overwinter: Some(1), - sapling: Some(1), - blossom: Some(1), - heartwood: Some(1), - canopy: Some(1), + sapling: Some(shielded_activation), + blossom: Some(shielded_activation), + heartwood: Some(shielded_activation), + canopy: Some(shielded_activation), nu5: None, nu6: None, nu6_1: None, @@ -1220,40 +1264,7 @@ mod tests { #[cfg(zcash_unstable = "zfuture")] zfuture: None, }) - .expect("height-1 activation set is valid") - .with_funding_streams(Vec::new()) - .with_checkpoints(ConfiguredCheckpoints::HeightsAndHashes(checkpoints)) - .expect("e2e checkpoints use valid header hashes") - .to_network() - .expect("e2e network has enough checkpoint coverage") - } - - fn e2e_network_with_checkpoint_hash(height: u32, hash: block::Hash) -> Network { - let checkpoints = vec![ - (block::Height(0), mainnet_genesis_hash()), - (block::Height(height), hash), - ]; - - TestnetParameters::build() - .with_genesis_hash(mainnet_genesis_hash()) - .expect("mainnet genesis vector hash parses") - .with_activation_heights(ConfiguredActivationHeights { - before_overwinter: None, - overwinter: Some(1), - sapling: Some(1), - blossom: Some(1), - heartwood: Some(1), - canopy: Some(1), - nu5: None, - nu6: None, - nu6_1: None, - nu6_2: None, - nu6_3: None, - nu7: None, - #[cfg(zcash_unstable = "zfuture")] - zfuture: None, - }) - .expect("height-1 activation set is valid") + .expect("e2e activation set is valid") .with_funding_streams(Vec::new()) .with_checkpoints(ConfiguredCheckpoints::HeightsAndHashes(checkpoints)) .expect("e2e checkpoints use valid header hashes") @@ -1351,7 +1362,7 @@ mod tests { .send(HeaderSyncEvent::NewBlockDuplicate { peer, height, hash }) .await; } - HeaderSyncAction::QueryBestHeaderTip + HeaderSyncAction::QueryBestHeaderHistoryTree { .. } | HeaderSyncAction::QueryMissingBlockBodies { .. } | HeaderSyncAction::BodyGaps { .. } | HeaderSyncAction::HeaderAdvanced { .. } @@ -2718,7 +2729,12 @@ mod tests { false, )?; let mut cluster = HeaderSyncE2eCluster::new(); - let network = e2e_network([4]); + // Sapling..Canopy activate at height 5, just above the synced 1..=4 range and exactly at the + // checkpoint-coverage bound (Canopy - 1 == the height-4 checkpoint). This keeps heights 1..=4 + // pre-Sapling, so their `PreSaplingReserved` commitments verify against the empty placeholder + // tree-aux roots the test peer serves β€” otherwise a post-Heartwood schedule reads these real + // mainnet vectors as `ChainHistoryRoot` and rejects the placeholder roots. + let network = e2e_network_with_shielded_activation([4], 5); let anchor = (block::Height(0), mainnet_genesis_hash()); let source = cluster.spawn_node( 1, @@ -2811,12 +2827,16 @@ mod tests { Ok(()) } + /// A checkpoint-anchored node syncs forward past its anchor, and below-anchor backward + /// backfill stays explicitly disabled: no `GetHeaders` for the bracket below the anchor is + /// ever sent, and the below-anchor headers stay absent (see + /// `backward_checkpoint_backfill_is_explicitly_disabled` for the reactor-level regression). #[tokio::test(flavor = "multi_thread", worker_threads = 2)] - async fn header_sync_e2e_checkpoint_forward_then_backward_finalizes_backfill( + async fn header_sync_e2e_checkpoint_forward_syncs_without_backward_backfill( ) -> Result<(), BoxError> { let _guard = zebra_test::init(); let mut capture = TraceCapture::for_test_with_keep_override( - "header_sync_e2e_checkpoint_forward_then_backward_finalizes_backfill", + "header_sync_e2e_checkpoint_forward_syncs_without_backward_backfill", false, )?; let (network, checkpoint_hash) = checkpoint_network(3); @@ -2859,18 +2879,35 @@ mod tests { .await; cluster.wait_for_tip(checkpointed, block::Height(4)).await?; - await_until( - "checkpoint backfill headers committed", - Duration::from_secs(5), - || cluster.has_headers(checkpointed, 1..=3), - ) - .await?; + + // Give the scheduler time to (incorrectly) emit a backward bracket if it were still + // enabled, then assert nothing below the anchor was requested or stored. + tokio::time::sleep(Duration::from_millis(500)).await; + assert!( + !cluster.has_headers(checkpointed, 1..=3), + "below-anchor headers must not be backfilled while backward backfill is disabled" + ); capture.flush().await; let reader = capture.reader()?; let target_trace = reader.node("02").table("header_sync"); target_trace.assert_header_range_request(4, 1); - target_trace.assert_header_range_request(1, 3); + let backward_requests = target_trace + .rows() + .iter() + .filter(|row| { + row.get("event").and_then(serde_json::Value::as_str) + == Some(hs_trace::HEADER_GET_HEADERS_SENT) + && row + .get(hs_trace::RANGE_START) + .and_then(serde_json::Value::as_u64) + == Some(1) + }) + .count(); + assert_eq!( + backward_requests, 0, + "backward checkpoint backfill is disabled: no below-anchor GetHeaders may be sent" + ); assert_eq!( cluster.finalized_height(checkpointed).await, block::Height(3) @@ -3034,7 +3071,9 @@ mod tests { .inject( victim, unsolicited, - headers_message(vec![mainnet_block(&BLOCK_MAINNET_1_BYTES).header.clone()]), + rootless_headers_message(vec![mainnet_block(&BLOCK_MAINNET_1_BYTES) + .header + .clone()]), ) .await; await_until( @@ -3070,7 +3109,9 @@ mod tests { .inject( victim, out_of_range, - headers_message(vec![mainnet_block(&BLOCK_MAINNET_2_BYTES).header.clone()]), + rootless_headers_message(vec![mainnet_block(&BLOCK_MAINNET_2_BYTES) + .header + .clone()]), ) .await; cluster @@ -3113,7 +3154,7 @@ mod tests { .inject( victim, response_too_long, - headers_message(vec![ + rootless_headers_message(vec![ mainnet_block(&BLOCK_MAINNET_1_BYTES).header.clone(), mainnet_block(&BLOCK_MAINNET_2_BYTES).header.clone(), ]), @@ -3151,7 +3192,7 @@ mod tests { .inject( bad_continuity_victim, bad_continuity, - headers_message(vec![ + rootless_headers_message(vec![ mainnet_block(&BLOCK_MAINNET_1_BYTES).header.clone(), Arc::new(non_contiguous), ]), @@ -3183,7 +3224,7 @@ mod tests { .inject( bad_pow_victim, bad_pow, - headers_message(vec![Arc::new(bad_pow_header)]), + rootless_headers_message(vec![Arc::new(bad_pow_header)]), ) .await; cluster @@ -3216,7 +3257,7 @@ mod tests { .inject( bad_daa_victim, bad_daa, - headers_message(vec![ + rootless_headers_message(vec![ mainnet_block(&BLOCK_MAINNET_1_BYTES).header.clone(), mainnet_block(&BLOCK_MAINNET_2_BYTES).header.clone(), mainnet_block(&BLOCK_MAINNET_3_BYTES).header.clone(), @@ -3228,44 +3269,10 @@ mod tests { .wait_for_misbehavior_reason(bad_daa_victim, HeaderSyncMisbehavior::InvalidRange) .await?; - let bad_checkpoint_backfill = e2e_peer(101); - let checkpoint_hash = mainnet_block(&BLOCK_MAINNET_1_BYTES).hash(); - let checkpoint_network = e2e_network_with_checkpoint_hash(3, checkpoint_hash); - let checkpointed = cluster.spawn_node( - 6, - checkpoint_network, - (block::Height(3), checkpoint_hash), - E2eHeaderStore::with_checkpoint_anchor(3), - ZakuraTrace::new(capture.tracer_for_node(6), "06"), - )?; - cluster.start_drivers(); - cluster - .connect_peer(checkpointed, bad_checkpoint_backfill.clone()) - .await; - cluster - .inject( - checkpointed, - bad_checkpoint_backfill.clone(), - status_for_tip(3, 4, 1), - ) - .await; - cluster - .wait_for_get_headers(checkpointed, &bad_checkpoint_backfill, block::Height(1), 3) - .await?; - cluster - .inject( - checkpointed, - bad_checkpoint_backfill, - headers_message(vec![ - mainnet_block(&BLOCK_MAINNET_1_BYTES).header.clone(), - mainnet_block(&BLOCK_MAINNET_2_BYTES).header.clone(), - mainnet_block(&BLOCK_MAINNET_3_BYTES).header.clone(), - ]), - ) - .await; - cluster - .wait_for_misbehavior_reason(checkpointed, HeaderSyncMisbehavior::InvalidRange) - .await?; + // Checkpoint-hash-mismatch backfill responses are covered at the reactor level by + // `checkpoint_backfill_rejects_checkpoint_hash_mismatch_before_commit` (via the forward + // genesis-backfill path); the backward below-anchor bracket that used to drive it here + // is explicitly disabled. let over_cap = e2e_peer(91); cluster.connect_peer(victim, over_cap.clone()).await; @@ -3357,7 +3364,7 @@ mod tests { trace.assert_header_violation("status_spam"); trace.assert_header_violation("new_block_spam"); trace.assert_header_violation("malformed_message"); - for node in ["03", "04", "05", "06"] { + for node in ["03", "04", "05"] { reader .node(node) .table("header_sync") diff --git a/zebrad/src/commands/start.rs b/zebrad/src/commands/start.rs index d3a6799870d..c099816cc05 100644 --- a/zebrad/src/commands/start.rs +++ b/zebrad/src/commands/start.rs @@ -2109,7 +2109,7 @@ mod zakura_header_sync_driver_tests { use futures::stream::{FuturesUnordered, StreamExt}; use tokio::sync::mpsc; - use tower::{service_fn, util::BoxService, ServiceExt}; + use tower::{service_fn, util::BoxService, Service, ServiceExt}; use zebra_chain::serialization::ZcashDeserializeInto; use zebra_chain::{block, orchard, parallel::commitment_aux::BlockCommitmentRoots, sapling}; use zebra_network::zakura::testkit::{TraceCapture, TraceValue}; @@ -2120,17 +2120,18 @@ mod zakura_header_sync_driver_tests { Service as ZakuraService, Stream as ZakuraStream, ZakuraHeaderSyncDriverStartup, BLOCK_SYNC_TABLE, COMMIT_STATE_TABLE, DEFAULT_HS_RANGE, }; - use zebra_test::vectors::{BLOCK_MAINNET_1_BYTES, BLOCK_MAINNET_2_BYTES}; + use zebra_test::vectors::{ + BLOCK_MAINNET_1_BYTES, BLOCK_MAINNET_2_BYTES, BLOCK_MAINNET_GENESIS_BYTES, + }; use super::zakura::{ - apply_block_sync_body, block_apply_class, block_roots_cover_range, - block_sync_chain_tip_event, block_sync_missing_body_window, - block_sync_needed_blocks_from_state, block_verify_error_is_duplicate, - body_sizes_for_served_header_range, chain_tip_mirror_frontier_change, - coalesce_ready_needed_block_queries, coalesce_stale_needed_block_queries, - commit_block_sync_body, drive_block_sync_actions, drive_zakura_header_sync_actions, - header_range_commit_failure_kind, notify_block_sync_header_tip, query_block_sync_frontiers, - query_block_sync_needed_blocks, root_covered_query_best_header_tip, + apply_block_sync_body, block_apply_class, block_sync_chain_tip_event, + block_sync_missing_body_window, block_sync_needed_blocks_from_state, + block_verify_error_is_duplicate, body_sizes_for_served_header_range, + chain_tip_mirror_frontier_change, coalesce_ready_needed_block_queries, + coalesce_stale_needed_block_queries, commit_block_sync_body, drive_block_sync_actions, + drive_zakura_header_sync_actions, header_range_commit_failure_kind, + notify_block_sync_header_tip, query_block_sync_frontiers, query_block_sync_needed_blocks, tree_aux_roots_for_served_header_range, verified_block_tip_from_state, BlockApplyClass, BlocksyncThroughputProbe, ZakuraHeaderSyncDriverHandles, ZAKURA_BLOCK_SYNC_CHECKPOINT_FRONTIER_REFRESH_INTERVAL, ZAKURA_BLOCK_SYNC_DRIVER_TIMEOUT, @@ -2290,54 +2291,6 @@ mod zakura_header_sync_driver_tests { ); } - #[test] - fn startup_root_backfill_gate_requires_complete_root_coverage() { - let start = block::Height(10); - let complete_roots = [ - root_at(block::Height(10)), - root_at(block::Height(11)), - root_at(block::Height(12)), - ]; - assert!(block_roots_cover_range(start, 3, &complete_roots)); - assert!(!block_roots_cover_range(start, 3, &complete_roots[..2])); - - let roots_with_gap = [ - root_at(block::Height(10)), - root_at(block::Height(12)), - root_at(block::Height(13)), - ]; - assert!(!block_roots_cover_range(start, 3, &roots_with_gap)); - } - - #[tokio::test] - async fn query_best_header_tip_is_capped_when_roots_are_missing() { - let verified_tip = (block::Height(0), block::Hash([0; 32])); - let durable_header_tip = (block::Height(2), block::Hash([2; 32])); - let read_state = service_fn(move |request: zebra_state::ReadRequest| async move { - match request { - zebra_state::ReadRequest::Tip => Ok::<_, zebra_state::BoxError>( - zebra_state::ReadResponse::Tip(Some(verified_tip)), - ), - zebra_state::ReadRequest::BlockRoots { - start_height, - count, - } => { - assert_eq!(start_height, block::Height(1)); - assert_eq!(count, 2); - Ok(zebra_state::ReadResponse::BlockRoots(Vec::new())) - } - request => panic!("unexpected read request: {request:?}"), - } - }); - - assert_eq!( - root_covered_query_best_header_tip(read_state, durable_header_tip) - .await - .expect("capped query succeeds"), - verified_tip - ); - } - #[test] fn block_verify_error_duplicate_classifier_detects_router_and_block_errors() { let hash = block::Hash([1; 32]); @@ -2670,6 +2623,74 @@ mod zakura_header_sync_driver_tests { reactor_task.abort(); } + /// End-to-end restart-resume: with a persisted header lead (bodies still at genesis), + /// `zakura_header_sync_driver_startup` reconstructs the history tree at the confirmed frontier + /// (`header_tip - 1`), seeds `best_header_parent_hash` with the frontier hash, and resumes header + /// sync from `frontier + 1` so the first forward range overlaps and re-validates the still- + /// unconfirmed tip root β€” the exact posture the running reactor keeps. + #[tokio::test(flavor = "multi_thread")] + async fn header_sync_driver_startup_resumes_at_reconstructed_frontier() { + let network = zebra_chain::parameters::Network::Mainnet; + let (mut state_service, read_state, _, _) = zebra_state::init_test_services(&network).await; + + let genesis = mainnet_block(&BLOCK_MAINNET_GENESIS_BYTES); + let block1 = mainnet_block(&BLOCK_MAINNET_1_BYTES); + let block2 = mainnet_block(&BLOCK_MAINNET_2_BYTES); + + state_service + .ready() + .await + .expect("state service is ready") + .call(zebra_state::Request::CommitCheckpointVerifiedBlock( + zebra_state::CheckpointVerifiedBlock::from(genesis.clone()), + )) + .await + .expect("genesis block commits"); + + // Persist a header lead: headers 1..=2 with the confirmed prefix (height 1 only; the tip's + // root is never persisted). Durable header tip is 2, so the confirmed frontier is 1. + state_service + .ready() + .await + .expect("state service is ready") + .call(zebra_state::Request::CommitHeaderRange { + anchor: genesis.hash(), + headers: vec![block1.header.clone(), block2.header.clone()], + body_sizes: vec![0, 0], + tree_aux_roots: vec![root_at(block::Height(1))], + }) + .await + .expect("header range commits"); + + let startup = super::zakura::zakura_header_sync_driver_startup(read_state, &network) + .await + .expect("driver startup reconstructs from durable state"); + + // Bodies are still at genesis; the header lead advanced to height 2. + assert_eq!(startup.frontiers.verified_block_tip, block::Height(0)); + + // Resume at frontier + 1 == 2 (the durable header tip), anchored at the frontier hash + // (height 1) so the first forward range overlaps and re-validates the tip's root. + assert_eq!( + startup.best_header_tip, + Some((block::Height(2), block2.hash())), + "resume height is frontier + 1, anchored at the durable tip", + ); + assert_eq!( + startup.best_header_parent_hash, + Some(block1.hash()), + "overlap anchor is the confirmed frontier hash (height 1)", + ); + + // The reconstructed tree sits one block behind the header tip, at the frontier. These are + // pre-Heartwood mainnet blocks, so that frontier tree is the empty tree. + assert_eq!( + startup.best_header_history_tree.as_ref(), + &zebra_chain::history_tree::HistoryTree::default(), + "reconstructed frontier tree is the pre-Heartwood empty tree", + ); + } + #[tokio::test] async fn header_sync_driver_header_advanced_updates_exchange_header_only() { let network = zebra_chain::parameters::Network::Mainnet; @@ -2689,6 +2710,10 @@ mod zakura_header_sync_driver_tests { verified_block_hash: genesis_hash, }, best_header_tip: Some((block::Height(0), genesis_hash)), + best_header_parent_hash: None, + best_header_history_tree: Arc::new( + zebra_chain::history_tree::HistoryTree::default(), + ), verified_block_tip_hash: genesis_hash, }), ) diff --git a/zebrad/src/commands/start/zakura/header_sync_driver.rs b/zebrad/src/commands/start/zakura/header_sync_driver.rs index a1d9ca0bec7..b063d6830dd 100644 --- a/zebrad/src/commands/start/zakura/header_sync_driver.rs +++ b/zebrad/src/commands/start/zakura/header_sync_driver.rs @@ -3,7 +3,7 @@ use std::{future::Future, time::Instant}; use color_eyre::eyre::{eyre, Report}; use tokio::{pin, select, sync::mpsc}; use tower::{Service, ServiceExt}; -use tracing::{debug, warn}; +use tracing::{debug, info, warn}; use zebra_chain::{ block::{self}, @@ -11,9 +11,10 @@ use zebra_chain::{ parallel::commitment_aux::BlockCommitmentRoots, }; use zebra_network::zakura::{ - commit_state_trace as cs_trace, BlockSyncFrontiers, Frontier, FrontierChange, HeaderSyncAction, - HeaderSyncCommitFailureKind, HeaderSyncEvent, HeaderSyncFrontiers, ZakuraEndpoint, - ZakuraHeaderSyncDriverStartup, ZakuraTrace, DEFAULT_HS_RANGE, + commit_state_trace as cs_trace, BlockSyncFrontiers, Frontier, FrontierChange, + HeaderFrontierReanchor, HeaderSyncAction, HeaderSyncCommitFailureKind, HeaderSyncEvent, + HeaderSyncFrontiers, ZakuraEndpoint, ZakuraHeaderSyncDriverStartup, ZakuraTrace, + DEFAULT_HS_RANGE, }; #[cfg(test)] @@ -62,12 +63,74 @@ pub(crate) async fn zakura_header_sync_driver_startup( let finalized_height = finalized_tip.map_or(block::Height(0), |(height, _)| height); let verified_block_tip = verified_block_tip_from_state(finalized_tip, verified_block_tip, empty_state_tip); - let best_header_tip = root_covered_best_header_tip_or_verified( - read_state, - best_header_tip.unwrap_or(empty_state_tip), - verified_block_tip, - ) - .await?; + let durable_best_header_tip = best_header_tip.unwrap_or(empty_state_tip); + + // Rebuild the ZIP-221 history tree so post-Heartwood header-sync root verification can resume + // immediately after a restart. The read returns the tree positioned at the highest *contiguous* + // confirmed header-root frontier, and that frontier's `(height, hash)` β€” the single authoritative + // value we use both to anchor overlap and to pick the resume height. + let (best_header_history_tree, (frontier_height, frontier_hash)) = match read_state + .clone() + .oneshot(zebra_state::ReadRequest::BestHeaderHistoryTree { + verified_block_tip: verified_block_tip.0, + best_header_tip: durable_best_header_tip.0, + }) + .await + .map_err(|error| eyre!("{error}"))? + { + zebra_state::ReadResponse::BestHeaderHistoryTree { tree, frontier } => (tree, frontier), + response => Err(eyre!( + "unexpected BestHeaderHistoryTree response: {response:?}" + ))?, + }; + + // Resume one block above the contiguous frontier (where the reconstructed tree sits), anchored at + // it, so the first forward range re-validates from there. If the persisted roots have a one-block + // gap (a header-tip advance that never overlapped), this resumes from the gap instead of capping + // all the way back to the verified tip. With no header lead there is nothing to resume, so the + // durable tip is kept and no overlap anchor is set. + let (best_header_tip, best_header_parent_hash) = + if durable_best_header_tip.0 > verified_block_tip.0 { + let resume_height = frontier_height + .next() + .map_err(|_| eyre!("header frontier height overflow"))?; + // The common (no-gap) frontier is one below the durable tip, so its successor is the durable + // tip and we already hold that hash; only a gap needs a lookup of the durable resume header. + let resume_hash = if resume_height == durable_best_header_tip.0 { + durable_best_header_tip.1 + } else { + match read_state + .oneshot(zebra_state::ReadRequest::HeadersByHeightRange { + start: resume_height, + count: 1, + }) + .await + .map_err(|error| eyre!("{error}"))? + { + zebra_state::ReadResponse::Headers(headers) => headers + .first() + .map(|(_height, hash, _header)| *hash) + .ok_or_else(|| { + eyre!("missing durable header at resume height {resume_height:?}") + })?, + response => Err(eyre!( + "unexpected HeadersByHeightRange response: {response:?}" + ))?, + } + }; + ((resume_height, resume_hash), Some(frontier_hash)) + } else { + (durable_best_header_tip, None) + }; + + info!( + verified = ?verified_block_tip.0, + durable_header_tip = ?durable_best_header_tip.0, + frontier = ?frontier_height, + resume_tip = ?best_header_tip.0, + overlap = best_header_parent_hash.is_some(), + "Zakura header-sync startup: reconstructed history tree at frontier, resuming header sync" + ); Ok(ZakuraHeaderSyncDriverStartup { frontiers: HeaderSyncFrontiers { @@ -76,102 +139,62 @@ pub(crate) async fn zakura_header_sync_driver_startup( verified_block_hash: verified_block_tip.1, }, best_header_tip: Some(best_header_tip), + best_header_parent_hash, + best_header_history_tree, verified_block_tip_hash: verified_block_tip.1, }) } -async fn root_covered_best_header_tip_or_verified( +/// Resolves the header-frontier reanchor a runtime lazy rebuild needs when the durable header-root +/// `frontier` folded *below* `best_header_tip - 1` (a gap left by a non-Zakura commit racing ahead). +/// +/// Mirrors the startup resume in [`zakura_header_sync_driver_startup`]: resume one block above the +/// confirmed frontier, anchored on the frontier hash, looking up the resume header's hash. Returns +/// `Ok(None)` when the frontier already reached `best_header_tip - 1` (the common no-gap rebuild, no +/// reanchor needed). +async fn header_frontier_reanchor( read_state: ReadState, - best_header_tip: (block::Height, block::Hash), - verified_block_tip: (block::Height, block::Hash), -) -> Result<(block::Height, block::Hash), Report> + best_header_tip: block::Height, + frontier: (block::Height, block::Hash), +) -> Result, Report> where ReadState: Service< zebra_state::ReadRequest, Response = zebra_state::ReadResponse, Error = zebra_state::BoxError, - > + Send - + 'static, - ReadState::Future: Send + 'static, + > + Send, + ReadState::Future: Send, { - if best_header_tip.0 <= verified_block_tip.0 { - return Ok(best_header_tip); + let (frontier_height, frontier_hash) = frontier; + let resume_height = frontier_height + .next() + .map_err(|_| eyre!("header frontier height overflow"))?; + // No gap: the tree folded to `best_header_tip - 1`, so its position already matches the parent of + // the next forward range. Nothing to reanchor β€” the plain tree install suffices. + if resume_height >= best_header_tip { + return Ok(None); } - - let Ok(start_height) = verified_block_tip.0.next() else { - return Ok(verified_block_tip); - }; - let best_header_height = best_header_tip.0; - let verified_block_height = verified_block_tip.0; - let count = best_header_height - .0 - .checked_sub(verified_block_height.0) - .ok_or_else(|| eyre!("best header tip is unexpectedly below verified block tip"))?; - let roots = match read_state - .oneshot(zebra_state::ReadRequest::BlockRoots { - start_height, - count, + let tip_hash = match read_state + .oneshot(zebra_state::ReadRequest::HeadersByHeightRange { + start: resume_height, + count: 1, }) .await .map_err(|error| eyre!("{error}"))? { - zebra_state::ReadResponse::BlockRoots(roots) => roots, - response => Err(eyre!("unexpected BlockRoots response: {response:?}"))?, + zebra_state::ReadResponse::Headers(headers) => headers + .first() + .map(|(_height, hash, _header)| *hash) + .ok_or_else(|| eyre!("missing durable header at resume height {resume_height:?}"))?, + response => Err(eyre!( + "unexpected HeadersByHeightRange response: {response:?}" + ))?, }; - - if block_roots_cover_range(start_height, count, &roots) { - Ok(best_header_tip) - } else { - Ok(verified_block_tip) - } -} - -pub(crate) async fn root_covered_query_best_header_tip( - read_state: ReadState, - best_header_tip: (block::Height, block::Hash), -) -> Result<(block::Height, block::Hash), Report> -where - ReadState: Service< - zebra_state::ReadRequest, - Response = zebra_state::ReadResponse, - Error = zebra_state::BoxError, - > + Clone - + Send - + 'static, - ReadState::Future: Send + 'static, -{ - let verified_block_tip = match read_state - .clone() - .oneshot(zebra_state::ReadRequest::Tip) - .await - .map_err(|error| eyre!("{error}"))? - { - zebra_state::ReadResponse::Tip(Some(tip)) => tip, - zebra_state::ReadResponse::Tip(None) => return Ok(best_header_tip), - response => Err(eyre!("unexpected Tip response: {response:?}"))?, - }; - - root_covered_best_header_tip_or_verified(read_state, best_header_tip, verified_block_tip).await -} - -pub(crate) fn block_roots_cover_range( - start_height: block::Height, - count: u32, - roots: &[BlockCommitmentRoots], -) -> bool { - if roots.len() != usize::try_from(count).unwrap_or(usize::MAX) { - return false; - } - - roots.iter().enumerate().all(|(offset, roots)| { - let Ok(offset) = u32::try_from(offset) else { - return false; - }; - start_height - .0 - .checked_add(offset) - .is_some_and(|height| roots.height == block::Height(height)) - }) + Ok(Some(HeaderFrontierReanchor { + tip: resume_height, + tip_hash, + parent_hash: frontier_hash, + })) } #[derive(Clone)] @@ -390,7 +413,7 @@ pub(crate) async fn drive_zakura_header_sync_actions { + Ok(zebra_state::ReadResponse::Headers(mut headers)) => { emit_commit_state( &trace, cs_trace::STATE_READ_SUCCESS, @@ -505,6 +528,16 @@ pub(crate) async fn drive_zakura_header_sync_actions = headers.iter().map(|(height, _, _)| *height).collect(); let tree_aux_roots = if want_tree_aux_roots { @@ -613,11 +646,20 @@ pub(crate) async fn drive_zakura_header_sync_actions { let count = u32::try_from(headers.len()).unwrap_or(u32::MAX); - let tree_aux_roots_len = u32::try_from(tree_aux_roots.len()).unwrap_or(u32::MAX); + // Persist only the header-authenticated confirmed prefix. The range tip's root is + // unconfirmed until the next overlapping range delivers its successor header, so it + // is deliberately excluded here; the state writes exactly what it is given. + let committed_roots = verified_roots + .as_deref() + .map_or_else(Vec::new, |verified_roots| { + verified_roots.confirmed_roots().to_vec() + }); + let tree_aux_roots_len = u32::try_from(committed_roots.len()).unwrap_or(u32::MAX); + let tip_parent_hash = header_range_tip_parent_hash(anchor, start_height, &headers); emit_commit_state( &trace, cs_trace::COMMIT_START, @@ -642,7 +684,7 @@ pub(crate) async fn drive_zakura_header_sync_actions { - emit_commit_state( - &trace, - cs_trace::STATE_READ_START, - "header_sync_driver", - |row| { - insert_cs_str(row, cs_trace::ACTION, "query_best_header_tip"); - }, - ); - let started = Instant::now(); - match read_state + HeaderSyncAction::QueryBestHeaderHistoryTree { + verified_block_tip, + best_header_tip, + } => { + // Reconstruct the header-frontier tree at the current frontier from durable roots. + // Always report completion back (`Some` on success, `None` on failure) so the reactor + // clears its in-flight rebuild guard even when the read errors β€” otherwise the guard + // would wedge and suppress all future rebuilds, stranding the stale tree. + let (history_tree, reanchor) = match read_state .clone() - .oneshot(zebra_state::ReadRequest::BestHeaderTip) + .oneshot(zebra_state::ReadRequest::BestHeaderHistoryTree { + verified_block_tip, + best_header_tip, + }) .await { - Ok(zebra_state::ReadResponse::BestHeaderTip(Some(best_header_tip))) => { - let (tip_height, tip_hash) = match root_covered_query_best_header_tip( + Ok(zebra_state::ReadResponse::BestHeaderHistoryTree { tree, frontier }) => { + // The fold can stop below `best_header_tip - 1` when durable roots have a gap + // (a non-Zakura commit raced ahead). In that case the reactor must drop its tip + // onto the rebuilt tree; resolve that reanchor here, where the read access + // lives, mirroring the startup resume. A resolution failure is reported as a + // failed rebuild (both `None`) so the guard clears and the range re-triggers, + // rather than installing a lower tree the stale tip could never match. + match header_frontier_reanchor( read_state.clone(), best_header_tip, + frontier, ) .await { - Ok(tip) => tip, + Ok(reanchor) => (Some(tree), reanchor), Err(error) => { - trace_state_read_error( - &trace, - "query_best_header_tip_roots", - None, - best_header_tip.0, - 1, - &format!("{error}"), - started, - ); - warn!( - ?error, - "failed to apply Zakura root coverage to best header tip" - ); - continue; + warn!(?error, "failed to resolve Zakura header frontier reanchor"); + (None, None) } - }; - emit_commit_state( - &trace, - cs_trace::STATE_READ_SUCCESS, - "header_sync_driver", - |row| { - insert_cs_str(row, cs_trace::ACTION, "query_best_header_tip"); - insert_cs_height(row, cs_trace::BEST_HEADER_TIP, tip_height); - insert_cs_hash(row, cs_trace::HASH, tip_hash); - }, - ); - let _ = handles - .header_sync - .send(HeaderSyncEvent::HeaderRangeCommitted { - start_height: tip_height, - tip_height, - tip_hash, - }) - .await; - publish_header_frontier( - &handles.endpoint, - tip_height, - tip_hash, - FrontierChange::HeaderAdvanced, - &trace, - ); + } } - Ok(zebra_state::ReadResponse::BestHeaderTip(None)) => {} Ok(response) => { - trace_state_read_error( - &trace, - "query_best_header_tip", - None, - block::Height(0), - 0, - "unexpected_response", - Instant::now(), - ); - warn!(?response, "unexpected BestHeaderTip response") + warn!(?response, "unexpected BestHeaderHistoryTree response"); + (None, None) } Err(error) => { - trace_state_read_error( - &trace, - "query_best_header_tip", - None, - block::Height(0), - 0, - &format!("{error}"), - Instant::now(), - ); - warn!(?error, "failed to query Zakura best header tip") + warn!(?error, "failed to rebuild Zakura best header history tree"); + (None, None) } - } + }; + let _ = handles + .header_sync + .send(HeaderSyncEvent::BestHeaderHistoryTreeLoaded { + best_header_tip, + reanchor, + history_tree, + }) + .await; } HeaderSyncAction::QueryMissingBlockBodies { from, limit } => { log_missing_block_bodies(read_state.clone(), from, limit, &trace).await; @@ -1382,9 +1387,6 @@ fn trace_header_driver_action(trace: &ZakuraTrace, action: &HeaderSyncAction) { insert_cs_height(row, cs_trace::RANGE_START, *start_height); insert_cs_u64(row, cs_trace::RANGE_COUNT, headers.len() as u64); } - HeaderSyncAction::QueryBestHeaderTip => { - insert_cs_str(row, cs_trace::ACTION, "query_best_header_tip"); - } HeaderSyncAction::QueryHeadersByHeightRange { peer, start, count, .. } => { @@ -1393,6 +1395,14 @@ fn trace_header_driver_action(trace: &ZakuraTrace, action: &HeaderSyncAction) { insert_cs_height(row, cs_trace::RANGE_START, *start); insert_cs_u64(row, cs_trace::RANGE_COUNT, u64::from(*count)); } + HeaderSyncAction::QueryBestHeaderHistoryTree { + verified_block_tip, + best_header_tip, + } => { + insert_cs_str(row, cs_trace::ACTION, "query_best_header_history_tree"); + insert_cs_height(row, cs_trace::VERIFIED_BLOCK_TIP, *verified_block_tip); + insert_cs_height(row, cs_trace::BEST_HEADER_TIP, *best_header_tip); + } HeaderSyncAction::QueryMissingBlockBodies { from, limit } => { insert_cs_str(row, cs_trace::ACTION, "query_missing_block_bodies"); insert_cs_height(row, cs_trace::RANGE_START, *from); @@ -1459,6 +1469,24 @@ fn trace_header_commit_finish( ); } +fn header_range_tip_parent_hash( + anchor: block::Hash, + start_height: block::Height, + headers: &[std::sync::Arc], +) -> Option { + if headers.is_empty() { + return None; + } + + if headers.len() == 1 { + return (start_height > block::Height(0)).then_some(anchor); + } + + headers + .get(headers.len().saturating_sub(2)) + .map(|header| block::Hash::from(header.as_ref())) +} + fn trace_header_reactor_event( trace: &ZakuraTrace, action: &'static str, diff --git a/zebrad/src/commands/start/zakura/mod.rs b/zebrad/src/commands/start/zakura/mod.rs index bff8f791fd8..67051364a35 100644 --- a/zebrad/src/commands/start/zakura/mod.rs +++ b/zebrad/src/commands/start/zakura/mod.rs @@ -24,10 +24,9 @@ pub(crate) use block_sync_driver::{ pub(crate) use frontier::{query_block_sync_frontiers, verified_block_tip_from_state}; #[cfg(test)] pub(crate) use header_sync_driver::{ - block_roots_cover_range, block_sync_chain_tip_event, body_sizes_for_served_header_range, + block_sync_chain_tip_event, body_sizes_for_served_header_range, chain_tip_mirror_frontier_change, header_range_commit_failure_kind, - notify_block_sync_header_tip, root_covered_query_best_header_tip, - tree_aux_roots_for_served_header_range, + notify_block_sync_header_tip, tree_aux_roots_for_served_header_range, }; pub(crate) use header_sync_driver::{ drive_zakura_header_sync_actions, mirror_zakura_full_block_commits,