diff --git a/CHANGELOG.md b/CHANGELOG.md index ec7e0fe8eb8..41b6cb6cb1e 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -151,6 +151,24 @@ and this project adheres to [Semantic Versioning](https://semver.org). duplicate-peer handling scaffolding. - Added bounded Zakura header-sync stream-5 wire messages, stateless header validation, and the default `network.zakura.header_sync` config surface. +- Verified-commitment-trees fast checkpoint sync. Below the last checkpoint Zebra + now fetches per-block Sapling/Orchard commitment roots from peers over a new + header-sync-aligned `tree_aux` stream, verifies each root against the node's own + checkpoint-committed block headers (the ZIP-221 ChainHistory MMR plus direct + below-Heartwood/below-NU5 checks), and folds the verified roots into the anchor + set and history tree — skipping the per-block note-commitment frontier recompute + that dominates checkpoint-sync CPU cost. At the checkpoint handoff an embedded + final frontier, verified against that block's proven root, is written as the tip + treestate and normal per-block recompute resumes. The resulting consensus state + is byte-identical to the legacy recompute; a root that cannot be obtained or + verified is rejected rather than recomputed against the stale frozen frontier, so + no untrusted data can influence consensus state. This is the default whenever + `consensus.checkpoint_sync = true` on a network with an embedded handoff frontier + (Mainnet), for both Archive and Pruned storage modes. The new + `consensus.vct_fast_sync` flag (default `true`) selects this fast path; set it to + `false` to keep checkpoint sync enabled while forcing the legacy per-block + recompute. Bumps the state database + format to 27.3.0 (new column families only; no data migration). - Include the `zebra-rollback-state` and `zebra-prune-state` utilities alongside `zebrad` in release Docker images and Docker CI builds. - Use the `5.0.0-rc.3` release identity for this fork's v5 rollback build. @@ -211,6 +229,14 @@ and this project adheres to [Semantic Versioning](https://semver.org). - Use network protocol version 170160 as the NU6.3 minimum on Mainnet, Testnet, and Regtest, matching Zebra's advertised current protocol version. +- Stop the database format-validity check from panicking with "just checked for + genesis block" while a verified-commitment-trees fast sync is in progress. The + check runs on a background thread, concurrently with block commits, and could + read its `is_vct_synced()` guard as `false` and then read an absent genesis + note-commitment tree once a concurrent fast-sync commit set the marker in + between. It now treats an absent genesis tree as a (mid-flight) fast-synced + database — where the genesis-root-caching invariant does not apply — instead of + panicking. - Avoid panics in the block write task when RPC users invalidate a non-finalized root block or reconsider the same invalidated block twice. - Compare RPC authentication cookies in constant time after checking their diff --git a/zebra-chain/src/block/commitment.rs b/zebra-chain/src/block/commitment.rs index 7b1f4e6b23a..e0c3a8141ca 100644 --- a/zebra-chain/src/block/commitment.rs +++ b/zebra-chain/src/block/commitment.rs @@ -396,6 +396,16 @@ pub enum CommitmentError { actual: [u8; 32], }, + #[error( + "invalid pre-NU5 orchard root: expected the empty-tree root {:?}, actual: {:?}", + hex::encode(expected), + hex::encode(actual) + )] + InvalidPreNu5OrchardRoot { + expected: [u8; 32], + actual: [u8; 32], + }, + #[error("missing required block height: block commitments can't be parsed without a block height, block hash: {block_hash:?}")] MissingBlockHeight { block_hash: block::Hash }, diff --git a/zebra-state/CHANGELOG.md b/zebra-state/CHANGELOG.md index 7e523945c3a..45b7d3376f3 100644 --- a/zebra-state/CHANGELOG.md +++ b/zebra-state/CHANGELOG.md @@ -12,8 +12,31 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 - Bumped the state database format to 28 during upgrade, backfilling empty Ironwood tree, value pool, and index data, then rebuilding stored history tree entries so they use the Ironwood-capable entry size. -- Extended value-pool disk serialization with an Ironwood slot after the - deferred pool, and bumped the state database format version to `27.3.0`. +- Extended value-pool disk serialization with an Ironwood slot after the deferred pool, with the + verified-commitment-trees state database format changes preserved as the intermediate + no-migration version `27.3.0`. +- Extended the `commitment_roots_by_height` serving index (and the `tree_aux` header-sync + payload it is served from) with each block's per-height ZIP-244 `auth_data_root`, so a node + can serve the co-input a peer needs to authenticate a block's note-commitment roots against + its successor's NU5+ header commitment. Consolidated under database format version `27.3.0` + (no version bump); serving-index rows from an earlier pre-release build (64 bytes, no + auth-data root) remain readable. Carrying the auth-data root bumps the Zakura header-sync + stream format to version 5, a breaking wire change (all peers must run the new format). +- Extended the same serving index and `tree_aux` payload with each block's Ironwood + note-commitment root and its three per-pool shielded transaction counts + (Sapling/Orchard/Ironwood). Together with the roots and auth-data root, these are the complete + set of ZIP-221 history-leaf inputs a recipient needs to rebuild each leaf and verify the + supplied roots against its own header commitments during header sync, without downloading the + block body. Rows grow to 152 bytes with a backward-compatible `FromDisk`; still consolidated + under `27.3.0`. Carried in the (already breaking, still version 5) Zakura header-sync stream + format, with `auth_data_root` serialized last. This is the data-carrying half only — nothing + consumes the new fields yet. +- Added the `vct_upgrade_metadata` column family, recording the upgrade height `U` (the lowest + height this binary committed). `tree_aux` root serving now stitches the per-height trees below + `U` with the serving index at and above `U`, so a node that upgraded mid-chain serves a range + crossing `U` as one gap-free batch instead of a short prefix that stalled the fetch client. + Historical note-commitment tree RPCs are unavailable only within the band `[U, H)` (where `H` + is the checkpoint handoff), and available below `U` and at or above `H`. ## [8.0.0] - 2026-06-02 diff --git a/zebra-state/src/error.rs b/zebra-state/src/error.rs index 6487c61907d..54adce5f644 100644 --- a/zebra-state/src/error.rs +++ b/zebra-state/src/error.rs @@ -86,6 +86,26 @@ impl CommitBlockError { } } + /// Returns the missing VCT supplied-root height for retryable root-fetch stalls. + pub fn vct_supplied_root_unavailable_height(&self) -> Option { + match self { + CommitBlockError::ValidateContextError(error) => { + error.vct_supplied_root_unavailable_height() + } + _ => None, + } + } + + /// Returns the height for any retryable VCT root stall (absent/evicted root, or one + /// not yet verifiable for lack of a buffered successor). See + /// [`ValidateContextError::vct_retryable_height`]. + pub fn vct_retryable_height(&self) -> Option { + match self { + CommitBlockError::ValidateContextError(error) => error.vct_retryable_height(), + _ => None, + } + } + /// Returns a suggested misbehaviour score increment for a certain error. pub fn misbehavior_score(&self) -> u32 { 0 @@ -149,6 +169,18 @@ impl CommitCheckpointVerifiedError { pub fn duplicate_location(&self) -> Option<&KnownBlock> { self.0.duplicate_location() } + + /// Returns the missing VCT supplied-root height for retryable root-fetch stalls. + pub fn vct_supplied_root_unavailable_height(&self) -> Option { + self.0.vct_supplied_root_unavailable_height() + } + + /// Returns the height for any retryable VCT root stall (absent/evicted root, or one + /// not yet verifiable for lack of a buffered successor). See + /// [`ValidateContextError::vct_retryable_height`]. + pub fn vct_retryable_height(&self) -> Option { + self.0.vct_retryable_height() + } } impl From for CommitCheckpointVerifiedError { @@ -367,6 +399,23 @@ pub enum ValidateContextError { #[non_exhaustive] NotReadyToBeCommitted, + #[error( + "verified-commitment-trees fast path has no valid supplied root for height \ + {height:?}: the note-commitment frontier is frozen, so this block cannot be \ + committed until a verifiable root is fetched from a peer (retryable)" + )] + #[non_exhaustive] + VctSuppliedRootUnavailable { height: block::Height }, + + #[error( + "verified-commitment-trees fast path cannot yet verify the supplied root for height \ + {height:?}: no successor block is buffered to confirm it against the header chain, and \ + committing it unverified would persist a root that is only checked one block later \ + (irreversibly, once on disk). Commit is deferred until the successor arrives (retryable)" + )] + #[non_exhaustive] + VctSuppliedRootAwaitingSuccessor { height: block::Height }, + #[error("block height {candidate_height:?} is lower than the current finalized height {finalized_tip_height:?}")] #[non_exhaustive] OrphanedBlock { @@ -597,6 +646,33 @@ pub enum ValidateContextError { }, } +impl ValidateContextError { + /// Returns the missing VCT supplied-root height for retryable root-fetch stalls. + /// + /// This is the subset of [`Self::vct_retryable_height`] that warrants a peer *refetch*: + /// the supplied root is absent or was evicted after failing verification, so a different + /// peer must supply a replacement. An await-successor stall ([`Self::vct_retryable_height`] + /// but not this) already has its root and only waits for the next block to be downloaded. + pub fn vct_supplied_root_unavailable_height(&self) -> Option { + match self { + ValidateContextError::VctSuppliedRootUnavailable { height } => Some(*height), + _ => None, + } + } + + /// Returns the height for any retryable VCT root stall: either an absent/evicted supplied + /// root ([`Self::VctSuppliedRootUnavailable`]) or one not yet verifiable because no successor + /// is buffered to confirm it ([`Self::VctSuppliedRootAwaitingSuccessor`]). The write loop + /// parks and retries the same block for both; only the former additionally requests a refetch. + pub fn vct_retryable_height(&self) -> Option { + match self { + ValidateContextError::VctSuppliedRootUnavailable { height } + | ValidateContextError::VctSuppliedRootAwaitingSuccessor { height } => Some(*height), + _ => None, + } + } +} + impl From for ValidateContextError { fn from(value: sprout::tree::NoteCommitmentTreeError) -> Self { ValidateContextError::NoteCommitmentTreeError(value.into()) @@ -657,4 +733,65 @@ mod tests { }; assert_eq!(dup_err.misbehavior_score(), 0); } + + #[test] + fn checkpoint_error_exposes_retryable_vct_root_height() { + let height = Height(42); + let retryable: CommitCheckpointVerifiedError = + ValidateContextError::VctSuppliedRootUnavailable { height }.into(); + assert_eq!( + retryable.vct_supplied_root_unavailable_height(), + Some(height), + "checkpoint commit errors expose retryable VCT root misses" + ); + + let non_retryable: CommitCheckpointVerifiedError = + ValidateContextError::NonSequentialBlock { + candidate_height: Height(5), + parent_height: Height(3), + } + .into(); + assert_eq!( + non_retryable.vct_supplied_root_unavailable_height(), + None, + "unrelated validation errors are not treated as VCT root misses" + ); + assert_eq!( + non_retryable.vct_retryable_height(), + None, + "unrelated validation errors are not retryable VCT stalls" + ); + } + + /// An await-successor stall is retryable (the write loop parks and re-commits) but is + /// *not* a refetch case: the root is present, only its successor is missing. So it must + /// surface through `vct_retryable_height` while `vct_supplied_root_unavailable_height` + /// (which gates the peer refetch) stays `None` — otherwise the committer would spam + /// pointless refetches for a root it already holds. + #[test] + fn await_successor_is_retryable_but_not_a_refetch() { + let height = Height(7); + let awaiting: CommitCheckpointVerifiedError = + ValidateContextError::VctSuppliedRootAwaitingSuccessor { height }.into(); + + assert_eq!( + awaiting.vct_retryable_height(), + Some(height), + "an await-successor stall is retryable", + ); + assert_eq!( + awaiting.vct_supplied_root_unavailable_height(), + None, + "an await-successor stall must not trigger a peer refetch (the root is present)", + ); + + // The unavailable case is both retryable and a refetch trigger. + let unavailable: CommitCheckpointVerifiedError = + ValidateContextError::VctSuppliedRootUnavailable { height }.into(); + assert_eq!(unavailable.vct_retryable_height(), Some(height)); + assert_eq!( + unavailable.vct_supplied_root_unavailable_height(), + Some(height) + ); + } } diff --git a/zebra-state/src/lib.rs b/zebra-state/src/lib.rs index 59bc7fbd460..6c555a22b98 100644 --- a/zebra-state/src/lib.rs +++ b/zebra-state/src/lib.rs @@ -63,8 +63,10 @@ pub use service::{ finalized_state::FinalizedState, init, init_read_only, non_finalized_state::NonFinalizedState, + queued_blocks::QueuedCheckpointVerified, spawn_init_read_only, watch_receiver::WatchReceiver, + write::BlockWriteSender, OutputLocation, ReadState, State, TransactionIndex, TransactionLocation, }; @@ -80,6 +82,10 @@ pub use service::finalized_state::{ preview_rollback_finalized_state, rollback_finalized_state, RollbackBackupSummary, RollbackFinalizedStateError, RollbackFinalizedStateOptions, RollbackFinalizedStateSummary, }; +pub use service::finalized_state::{ + produce_final_frontiers_bytes, validate_final_frontiers_bytes, FinalFrontiersGenerationError, + FinalFrontiersValidationError, +}; pub use service::{ finalized_state::{DiskWriteBatch, FromDisk, IntoDisk, WriteDisk, ZebraDb}, ReadStateService, diff --git a/zebra-state/src/request.rs b/zebra-state/src/request.rs index 709aac8b2af..cf204f08c5d 100644 --- a/zebra-state/src/request.rs +++ b/zebra-state/src/request.rs @@ -40,6 +40,27 @@ use crate::{ CommitSemanticallyVerifiedError, }; +/// Times `$body` and records its duration to the named histogram when the +/// `commit-metrics` feature is enabled; otherwise just evaluates `$body` with +/// zero overhead. Used to profile checkpoint prepare phases. +macro_rules! timed_prepare_phase { + ($name:expr, $body:expr) => {{ + #[cfg(feature = "commit-metrics")] + let _start = std::time::Instant::now(); + let result = $body; + #[cfg(feature = "commit-metrics")] + metrics::histogram!($name).record(_start.elapsed().as_secs_f64()); + result + }}; +} + +/// Minimum transaction count before checkpoint prepare uses Rayon for +/// per-transaction digest fanout. +/// +/// Small blocks are faster serially because Rayon scheduling costs dominate the +/// native ZIP-244 digest work. +const MIN_PARALLEL_CHECKPOINT_PREPARE_TRANSACTIONS: usize = 16; + /// Identify a spend by a transparent outpoint or revealed nullifier. /// /// This enum supports [`transparent::OutPoint`], [`sprout::Nullifier`], @@ -268,9 +289,19 @@ pub struct SemanticallyVerifiedBlock { /// The precomputed ZIP-244 authorizing-data commitment root for this block, /// if it was computed during verification. /// - /// The checkpoint verifier can set this ahead of the single-threaded - /// finalized committer. `None` means the committer falls back to computing - /// it from the block's transactions. + /// The checkpoint verifier sets this (it runs with high concurrency, ahead + /// of the single-threaded finalized committer) so the committer does not + /// have to recompute the per-transaction auth digests on its critical path. + /// `None` means "not precomputed"; the committer falls back to computing it. + /// + /// # Security + /// + /// The finalized checkpoint committer **trusts** a `Some` value as the + /// authorizing data for the ZIP-244 `hashBlockCommitments` header check + /// (`check::block_commitment_is_valid_for_chain_history`), so it must always + /// equal `block.auth_data_root()`. The constructors in this module derive it + /// from `block`, so a value set that way can never be desynced from the block + /// it commits. pub auth_data_root: Option, } @@ -541,7 +572,7 @@ impl CheckpointVerifiedBlock { deferred_pool_balance_change: Option, ) -> Self { let mut block = Self::with_hash(block.clone(), hash.unwrap_or(block.hash())); - block.deferred_pool_balance_change = deferred_pool_balance_change; + block.set_deferred_pool_balance_change(deferred_pool_balance_change); block } /// Creates a block that's ready to be committed to the finalized state, @@ -568,14 +599,77 @@ impl CheckpointVerifiedBlock { } } +fn prepare_block_data( + block: &Block, +) -> ( + Arc<[transaction::Hash]>, + AuthDataRoot, + HashMap, +) { + #[cfg(feature = "commit-metrics")] + { + let transaction_count = block.transactions.len(); + let output_count: usize = block + .transactions + .iter() + .map(|transaction| transaction.outputs().len()) + .sum(); + let v5_transaction_count = block + .transactions + .iter() + .filter(|transaction| transaction.version() == 5) + .count(); + + if let Some(height) = block.coinbase_height() { + metrics::gauge!("zebra.state.prepare.block.height").set(height.0 as f64); + } + metrics::histogram!("zebra.state.prepare.block_tx_count").record(transaction_count as f64); + metrics::histogram!("zebra.state.prepare.block_output_count").record(output_count as f64); + metrics::histogram!("zebra.state.prepare.block_v5_tx_count") + .record(v5_transaction_count as f64); + } + + // Compute each transaction's txid and ZIP-244 auth digest together, for efficiency. + let (transaction_hashes, auth_digests): (Vec<_>, Vec<_>) = + timed_prepare_phase!("zebra.state.prepare.txid_auth_digest.duration_seconds", { + if block.transactions.len() < MIN_PARALLEL_CHECKPOINT_PREPARE_TRANSACTIONS { + block + .transactions + .iter() + .map(|tx| tx.txid_and_auth_digest()) + .unzip() + } else { + use rayon::prelude::*; + block + .transactions + .par_iter() + .map(|tx| tx.txid_and_auth_digest()) + .unzip() + } + }); + let transaction_hashes: Arc<[_]> = transaction_hashes.into(); + let auth_data_root = timed_prepare_phase!( + "zebra.state.prepare.auth_data_root.duration_seconds", + auth_digests + .into_iter() + .map(|auth_digest| auth_digest.unwrap_or(AUTH_DIGEST_PLACEHOLDER)) + .collect::() + ); + let new_outputs = timed_prepare_phase!( + "zebra.state.prepare.new_ordered_outputs.duration_seconds", + transparent::new_ordered_outputs(block, &transaction_hashes) + ); + + (transaction_hashes, auth_data_root, new_outputs) +} + impl SemanticallyVerifiedBlock { /// Creates [`SemanticallyVerifiedBlock`] from [`Block`] and [`block::Hash`]. pub fn with_hash(block: Arc, hash: block::Hash) -> Self { let height = block .coinbase_height() .expect("semantically verified block should have a coinbase height"); - let (transaction_hashes, auth_data_root) = transaction_hashes_and_auth_data_root(&block); - let new_outputs = transparent::new_ordered_outputs(&block, &transaction_hashes); + let (transaction_hashes, auth_data_root, new_outputs) = prepare_block_data(&block); Self { block, @@ -588,6 +682,34 @@ impl SemanticallyVerifiedBlock { } } + /// Creates a [`SemanticallyVerifiedBlock`] from data the semantic verifier + /// has already prepared, leaving the authorizing-data root unset. + /// + /// The semantic verifier binds the ZIP-244 auth-data commitment during + /// contextual validation and the committer recomputes it on that path, so it + /// is not precomputed here. This constructor exists so callers outside the + /// crate build the block through a checked entry point rather than a struct + /// literal, leaving the [`auth_data_root`](Self::auth_data_root) cache unset + /// (see its security note). + pub fn from_semantic_data( + block: Arc, + hash: block::Hash, + height: block::Height, + new_outputs: HashMap, + transaction_hashes: Arc<[transaction::Hash]>, + deferred_pool_balance_change: Option, + ) -> Self { + Self { + block, + hash, + height, + new_outputs, + transaction_hashes, + deferred_pool_balance_change, + auth_data_root: None, + } + } + /// Sets the deferred balance in the block. pub fn with_deferred_pool_balance_change( mut self, @@ -610,8 +732,7 @@ impl From> for SemanticallyVerifiedBlock { let height = block .coinbase_height() .expect("semantically verified block should have a coinbase height"); - let (transaction_hashes, auth_data_root) = transaction_hashes_and_auth_data_root(&block); - let new_outputs = transparent::new_ordered_outputs(&block, &transaction_hashes); + let (transaction_hashes, auth_data_root, new_outputs) = prepare_block_data(&block); Self { block, @@ -625,52 +746,6 @@ impl From> for SemanticallyVerifiedBlock { } } -/// Returns the transaction IDs and ZIP-244 authorizing-data root for `block`. -fn transaction_hashes_and_auth_data_root( - block: &Block, -) -> (Arc<[transaction::Hash]>, AuthDataRoot) { - use rayon::prelude::*; - - let (transaction_hashes, auth_digests): (Vec<_>, Vec<_>) = block - .transactions - .par_iter() - .map(|transaction| transaction.txid_and_auth_digest()) - .unzip(); - - let auth_data_root = auth_digests - .into_iter() - .map(|auth_digest| auth_digest.unwrap_or(AUTH_DIGEST_PLACEHOLDER)) - .collect(); - - (transaction_hashes.into(), auth_data_root) -} - -#[cfg(test)] -mod tests { - use super::*; - - use zebra_chain::serialization::ZcashDeserializeInto; - - #[test] - fn transaction_hashes_and_auth_data_root_matches_separate_computation() { - let _init_guard = zebra_test::init(); - - let block = zebra_test::vectors::BLOCK_MAINNET_1687107_BYTES - .zcash_deserialize_into::() - .expect("NU5 mainnet block deserializes"); - - let (transaction_hashes, auth_data_root) = transaction_hashes_and_auth_data_root(&block); - let expected_transaction_hashes: Vec<_> = block - .transactions - .iter() - .map(|transaction| transaction.hash()) - .collect(); - - assert_eq!(transaction_hashes.as_ref(), expected_transaction_hashes); - assert_eq!(auth_data_root, block.auth_data_root()); - } -} - impl From for SemanticallyVerifiedBlock { fn from(valid: ContextuallyVerifiedBlock) -> Self { Self { @@ -714,12 +789,29 @@ impl Deref for CheckpointVerifiedBlock { &self.0 } } + impl DerefMut for CheckpointVerifiedBlock { fn deref_mut(&mut self) -> &mut Self::Target { &mut self.0 } } +impl CheckpointVerifiedBlock { + /// Sets the deferred pool balance change computed by the checkpoint verifier + /// after construction. + /// + /// This is the only post-construction mutation a caller may perform; it does + /// not touch the block or the precomputed authorizing-data root, so the + /// committer's trusted cache stays bound to the block (see + /// [`SemanticallyVerifiedBlock::auth_data_root`]). + pub fn set_deferred_pool_balance_change( + &mut self, + deferred_pool_balance_change: Option, + ) { + self.0.deferred_pool_balance_change = deferred_pool_balance_change; + } +} + /// Helper trait for convenient access to expected response and error types. pub trait MappedRequest: Sized + Send + 'static { /// Expected response type for this state request. @@ -1417,8 +1509,8 @@ pub enum ReadRequest { /// Returns scheduling-only body-size hints for a contiguous height range. /// - /// Confirmed committed block sizes win over untrusted advertised header - /// hints. Unknown advertised sizes are returned as `None`. + /// Confirmed committed block sizes are preferred over advertised header + /// hints. Unknown sizes are returned as `None`. BlockSizeHints { /// First height to read. from: block::Height, @@ -1759,3 +1851,93 @@ impl TimedSpan { .wait_for_panics() } } + +#[cfg(test)] +mod tests { + use zebra_chain::serialization::ZcashDeserializeInto; + + use super::*; + + /// Loads the NU5 mainnet block 1,687,106 (its v5 transactions exercise the + /// ZIP-244 authorizing-data digests). + fn nu5_block() -> Arc { + Arc::new( + zebra_test::vectors::BLOCK_MAINNET_1687106_BYTES + .zcash_deserialize_into::() + .expect("NU5 test vector block deserializes"), + ) + } + + #[test] + fn transaction_hashes_and_auth_data_root_matches_separate_computation() { + let _init_guard = zebra_test::init(); + + let block = zebra_test::vectors::BLOCK_MAINNET_1687107_BYTES + .zcash_deserialize_into::() + .expect("NU5 mainnet block deserializes"); + + let (transaction_hashes, auth_data_root, _new_outputs) = prepare_block_data(&block); + let expected_transaction_hashes: Vec<_> = block + .transactions + .iter() + .map(|transaction| transaction.hash()) + .collect(); + + assert_eq!(transaction_hashes.as_ref(), expected_transaction_hashes); + assert_eq!(auth_data_root, block.auth_data_root()); + } + + /// The committer trusts a `Some` authorizing-data root for the ZIP-244 header + /// commitment check, so any constructor that *does* precompute it must derive + /// it *from its own block*. The [`From>`] conversion precomputes it + /// (via `prepare_block_data`), so its cached value must equal the block's own + /// root. The `with_hash`/`new` constructors leave it unset and the committer + /// recomputes it at commit time (only for NU5+), so the trusted-cache invariant + /// is never violated. + #[test] + fn checkpoint_verified_block_caches_its_own_auth_data_root() { + let block = nu5_block(); + let expected = Some(block.auth_data_root()); + + assert_eq!( + CheckpointVerifiedBlock::from(block.clone()).auth_data_root, + expected, + "From> must cache the block's own auth data root", + ); + assert_eq!( + CheckpointVerifiedBlock::with_hash(block.clone(), block.hash()).auth_data_root, + None, + "with_hash defers the auth data root to commit-time recompute", + ); + assert_eq!( + CheckpointVerifiedBlock::new(block.clone(), None, None).auth_data_root, + None, + "new defers the auth data root to commit-time recompute", + ); + } + + /// The semantic-verifier constructor leaves the cache empty: that path binds + /// the auth-data commitment during contextual validation, and the committer + /// recomputes it, so there is no precomputed value to trust. + #[test] + fn semantic_constructor_leaves_auth_data_root_unset() { + let block = nu5_block(); + let hash = block.hash(); + let height = block.coinbase_height().expect("test block has a height"); + let (transaction_hashes, _auth_data_root, new_outputs) = prepare_block_data(&block); + + let semantic = SemanticallyVerifiedBlock::from_semantic_data( + block, + hash, + height, + new_outputs, + transaction_hashes, + None, + ); + + assert_eq!( + semantic.auth_data_root, None, + "the semantic path must not precompute an auth data root the committer would trust", + ); + } +} diff --git a/zebra-state/src/service.rs b/zebra-state/src/service.rs index 2fcd28c2255..199c2d49f36 100644 --- a/zebra-state/src/service.rs +++ b/zebra-state/src/service.rs @@ -71,10 +71,10 @@ pub mod check; pub(crate) mod finalized_state; pub(crate) mod non_finalized_state; mod pending_utxos; -mod queued_blocks; +pub(crate) mod queued_blocks; pub(crate) mod read; mod traits; -mod write; +pub(crate) mod write; #[cfg(any(test, feature = "proptest-impl"))] pub mod arbitrary; @@ -89,6 +89,13 @@ use self::queued_blocks::{QueuedCheckpointVerified, QueuedSemanticallyVerified, pub use self::traits::{ReadState, State}; +/// Error returned for historical note-commitment tree/subtree read requests on a +/// verified-commitment-trees fast-synced database, where the per-height trees +/// below the checkpoint handoff height were never written. +const FAST_SYNCED_TREE_UNAVAILABLE_ERROR: &str = + "note commitment treestate is unavailable below the checkpoint on a fast-synced node; \ + historical treestate queries require an archive node"; + /// A read-write service for Zebra's cached blockchain state. /// /// This service modifies and provides access to: @@ -1480,6 +1487,7 @@ where headers } +// Returns the block commitment roots for the given height range fn block_roots_by_height_range( chain: Option, db: &ZebraDb, @@ -1502,7 +1510,7 @@ where .finalized_tip_height() .is_some_and(|finalized_tip| height <= finalized_tip) { - db.finalized_commitment_roots_by_height_range(height..=height) + finalized_state::serve_block_roots(db, height..=height) .into_iter() .next() } else if let Some(chain) = chain @@ -1515,6 +1523,11 @@ where chain.orchard_tree(height.into()), ) { (Some(sapling), Some(orchard)) => { + // The non-finalized chain holds the full block, so derive its shielded + // tx-counts and ZIP-244 auth-data root — the ZIP-221 leaf inputs the + // header and roots don't provide — to serve for header-sync verification + // (zero only if the block is unexpectedly absent). The Ironwood tree does + // not exist below Nu7, so its root is the empty-tree root here. let (sapling_tx, orchard_tx, ironwood_tx, auth_data_root) = chain .block(height.into()) .map(|block| { @@ -1566,6 +1579,70 @@ where roots } +// Returns true if the given roots cover the given height range +fn block_roots_cover_range( + start_height: block::Height, + count: u32, + roots: &[BlockCommitmentRoots], +) -> bool { + if roots.len() != usize::try_from(count).unwrap_or(usize::MAX) { + return false; + } + + roots.iter().enumerate().all(|(offset, roots)| { + let Ok(offset) = u32::try_from(offset) else { + return false; + }; + start_height + .0 + .checked_add(offset) + .is_some_and(|height| roots.height == block::Height(height)) + }) +} + +// Return the highest known tip, but cap it to the verified block tip +// if the header-only extension is not root-covered. +fn root_covered_best_header_tip( + chain: Option, + db: &ZebraDb, + best_disk_header_tip: Option<(block::Height, block::Hash)>, + verified_block_tip: Option<(block::Height, block::Hash)>, +) -> Option<(block::Height, block::Hash)> +where + C: AsRef, +{ + // Choose the best candidate between the best disk header tip and the verified block tip + let best_header_tip = match (best_disk_header_tip, verified_block_tip) { + (Some(header_tip), Some(block_tip)) if block_tip.0 > header_tip.0 => Some(block_tip), + (Some(header_tip), _) => Some(header_tip), + (None, block_tip) => block_tip, + }?; + + // Is the chosen candidate already at or below the verified block tip? + // If yes, there no header-only gap. + let Some(verified_block_tip) = verified_block_tip else { + return Some(best_header_tip); + }; + + if best_header_tip.0 <= verified_block_tip.0 { + return Some(best_header_tip); + } + + let Ok(start_height) = verified_block_tip.0.next() else { + return Some(verified_block_tip); + }; + let best_header_height = best_header_tip.0; + let verified_block_height = verified_block_tip.0; + let count = best_header_height.0.checked_sub(verified_block_height.0)?; + let roots = block_roots_by_height_range(chain, db, start_height, count); + + if block_roots_cover_range(start_height, count, &roots) { + Some(best_header_tip) + } else { + Some(verified_block_tip) + } +} + impl Service for ReadStateService { type Response = ReadResponse; type Error = BoxError; @@ -1819,17 +1896,15 @@ impl Service for ReadStateService { ReadRequest::BestHeaderTip => { let best_disk_header_tip = state.db.best_header_tip(); - let verified_block_tip = read::tip(state.latest_best_chain(), &state.db); - - Ok(ReadResponse::BestHeaderTip( - match (best_disk_header_tip, verified_block_tip) { - (Some(header_tip), Some(block_tip)) if block_tip.0 > header_tip.0 => { - Some(block_tip) - } - (Some(header_tip), _) => Some(header_tip), - (None, block_tip) => block_tip, - }, - )) + let best_chain = state.latest_best_chain(); + let verified_block_tip = read::tip(best_chain.clone(), &state.db); + + Ok(ReadResponse::BestHeaderTip(root_covered_best_header_tip( + best_chain, + &state.db, + best_disk_header_tip, + verified_block_tip, + ))) } ReadRequest::MissingBlockBodies { from, limit } => { @@ -1869,19 +1944,38 @@ impl Service for ReadStateService { Ok(ReadResponse::Blocks(blocks)) } - ReadRequest::SaplingTree(hash_or_height) => Ok(ReadResponse::SaplingTree( - read::sapling_tree(state.latest_best_chain(), &state.db, hash_or_height), - )), + ReadRequest::SaplingTree(hash_or_height) => { + if state.db.vct_historical_tree_unavailable(hash_or_height) { + return Err(FAST_SYNCED_TREE_UNAVAILABLE_ERROR.into()); + } + Ok(ReadResponse::SaplingTree(read::sapling_tree( + state.latest_best_chain(), + &state.db, + hash_or_height, + ))) + } - ReadRequest::OrchardTree(hash_or_height) => Ok(ReadResponse::OrchardTree( - read::orchard_tree(state.latest_best_chain(), &state.db, hash_or_height), - )), + ReadRequest::OrchardTree(hash_or_height) => { + if state.db.vct_historical_tree_unavailable(hash_or_height) { + return Err(FAST_SYNCED_TREE_UNAVAILABLE_ERROR.into()); + } + Ok(ReadResponse::OrchardTree(read::orchard_tree( + state.latest_best_chain(), + &state.db, + hash_or_height, + ))) + } ReadRequest::IronwoodTree(hash_or_height) => Ok(ReadResponse::IronwoodTree( read::ironwood_tree(state.latest_best_chain(), &state.db, hash_or_height), )), ReadRequest::SaplingSubtrees { start_index, limit } => { + // On a fast-synced database, subtrees below the checkpoint handoff + // height were never written, so a below-checkpoint range returns an + // empty list (the existing "no subtree at the start index" contract) + // rather than panicking. A typed archive-mode error for subtrees + // unifies with the indexing watermark in a later increment. let end_index = limit .and_then(|limit| start_index.0.checked_add(limit.0)) .map(NoteCommitmentSubtreeIndex); @@ -2120,7 +2214,7 @@ impl Service for ReadStateService { /// Initialize a state service from the provided [`Config`]. /// Returns a boxed state service, a read-only state service, -/// and receivers for state chain tip updates. +/// receivers for state chain tip updates, and a `tree_aux` roots writer if peer mode is active. /// /// Each `network` has its own separate on-disk database. /// @@ -2218,12 +2312,9 @@ pub fn spawn_init_read_only( pub async fn init_test( network: &Network, ) -> Buffer, Request> { - // TODO: pass max_checkpoint_height and checkpoint_verify_concurrency limit - // if we ever need to test final checkpoint sent UTXO queries - let (state_service, _, _, _) = - StateService::new(Config::ephemeral(), network, block::Height::MAX, 0).await; + let (state_service, _, _, _) = init_test_services_inner(network).await; - Buffer::new(BoxService::new(state_service), 1) + state_service } /// Initializes a state service with an ephemeral [`Config`] and a buffer with a single slot, @@ -2238,6 +2329,18 @@ pub async fn init_test_services( ReadStateService, LatestChainTip, ChainTipChange, +) { + init_test_services_inner(network).await +} + +#[cfg(any(test, feature = "proptest-impl"))] +async fn init_test_services_inner( + network: &Network, +) -> ( + Buffer, Request>, + ReadStateService, + LatestChainTip, + ChainTipChange, ) { // TODO: pass max_checkpoint_height and checkpoint_verify_concurrency limit // if we ever need to test final checkpoint sent UTXO queries diff --git a/zebra-state/src/service/check.rs b/zebra-state/src/service/check.rs index d9aa373d805..e5281b3af12 100644 --- a/zebra-state/src/service/check.rs +++ b/zebra-state/src/service/check.rs @@ -235,6 +235,10 @@ pub(crate) fn block_commitment_is_valid_for_chain_history( "the history tree of the previous block must exist \ since the current block has a ChainHistoryBlockTxAuthCommitment", ); + // Use the auth data root precomputed by the verifier when available + // (it is byte-identical to recomputing it here), so the committer + // does not repeat the per-transaction auth-digest work on its + // single-threaded critical path. let auth_data_root = precomputed_auth_data_root.unwrap_or_else(|| block.auth_data_root()); diff --git a/zebra-state/src/service/check/tests/nullifier.rs b/zebra-state/src/service/check/tests/nullifier.rs index e634f485e59..420a4ee2be7 100644 --- a/zebra-state/src/service/check/tests/nullifier.rs +++ b/zebra-state/src/service/check/tests/nullifier.rs @@ -89,7 +89,7 @@ proptest! { // randomly choose to commit the block to the finalized or non-finalized state if use_finalized_state { let block1 = CheckpointVerifiedBlock::from(Arc::new(block1)); - let commit_result = finalized_state.commit_finalized_direct(block1.clone().into(), None, "test"); + let commit_result = finalized_state.commit_finalized_direct(block1.clone().into(), None, None, "test"); // the block was committed prop_assert_eq!(Some((Height(1), block1.hash)), read::best_tip(&non_finalized_state, &finalized_state.db)); @@ -353,7 +353,7 @@ proptest! { // randomly choose to commit the next block to the finalized or non-finalized state if duplicate_in_finalized_state { let block1 = CheckpointVerifiedBlock::from(Arc::new(block1)); - let commit_result = finalized_state.commit_finalized_direct(block1.clone().into(), None, "test"); + let commit_result = finalized_state.commit_finalized_direct(block1.clone().into(), None, None, "test"); prop_assert_eq!(Some((Height(1), block1.hash)), read::best_tip(&non_finalized_state, &finalized_state.db)); prop_assert!(commit_result.is_ok()); @@ -452,7 +452,7 @@ proptest! { // randomly choose to commit the block to the finalized or non-finalized state if use_finalized_state { let block1 = CheckpointVerifiedBlock::from(Arc::new(block1)); - let commit_result = finalized_state.commit_finalized_direct(block1.clone().into(),None, "test"); + let commit_result = finalized_state.commit_finalized_direct(block1.clone().into(), None, None, "test"); prop_assert_eq!(Some((Height(1), block1.hash)), read::best_tip(&non_finalized_state, &finalized_state.db)); prop_assert!(commit_result.is_ok()); @@ -632,7 +632,7 @@ proptest! { // randomly choose to commit the next block to the finalized or non-finalized state if duplicate_in_finalized_state { let block1 = CheckpointVerifiedBlock::from(Arc::new(block1)); - let commit_result = finalized_state.commit_finalized_direct(block1.clone().into(),None, "test"); + let commit_result = finalized_state.commit_finalized_direct(block1.clone().into(), None, None, "test"); prop_assert_eq!(Some((Height(1), block1.hash)), read::best_tip(&non_finalized_state, &finalized_state.db)); prop_assert!(commit_result.is_ok()); @@ -729,7 +729,7 @@ proptest! { // randomly choose to commit the block to the finalized or non-finalized state if use_finalized_state { let block1 = CheckpointVerifiedBlock::from(Arc::new(block1)); - let commit_result = finalized_state.commit_finalized_direct(block1.clone().into(), None, "test"); + let commit_result = finalized_state.commit_finalized_direct(block1.clone().into(), None, None, "test"); prop_assert_eq!(Some((Height(1), block1.hash)), read::best_tip(&non_finalized_state, &finalized_state.db)); prop_assert!(commit_result.is_ok()); @@ -918,7 +918,7 @@ proptest! { // randomly choose to commit the next block to the finalized or non-finalized state if duplicate_in_finalized_state { let block1 = CheckpointVerifiedBlock::from(Arc::new(block1)); - let commit_result = finalized_state.commit_finalized_direct(block1.clone().into(), None, "test"); + let commit_result = finalized_state.commit_finalized_direct(block1.clone().into(), None, None, "test"); prop_assert_eq!(Some((Height(1), block1.hash)), read::best_tip(&non_finalized_state, &finalized_state.db)); prop_assert!(commit_result.is_ok()); @@ -1025,7 +1025,8 @@ proptest! { let block1_hash; if duplicate_in_finalized_state { let block1 = CheckpointVerifiedBlock::from(Arc::new(block1)); - let commit_result = finalized_state.commit_finalized_direct(block1.clone().into(), None, "test"); + let commit_result = + finalized_state.commit_finalized_direct(block1.clone().into(), None, None, "test"); prop_assert_eq!(Some((Height(1), block1.hash)), read::best_tip(&non_finalized_state, &finalized_state.db)); prop_assert!(commit_result.is_ok()); @@ -1118,7 +1119,7 @@ proptest! { finalized_state.populate_with_anchors(&block2); let block1 = CheckpointVerifiedBlock::from(Arc::new(block1)); - let commit_result = finalized_state.commit_finalized_direct(block1.into(), None, "test"); + let commit_result = finalized_state.commit_finalized_direct(block1.into(), None, None, "test"); prop_assert!(commit_result.is_ok()); let block2 = Arc::new(block2).prepare(); @@ -1172,7 +1173,7 @@ proptest! { finalized_state.populate_with_anchors(&block2); let block1 = CheckpointVerifiedBlock::from(Arc::new(block1)); - let commit_result = finalized_state.commit_finalized_direct(block1.into(), None, "test"); + let commit_result = finalized_state.commit_finalized_direct(block1.into(), None, None, "test"); prop_assert!(commit_result.is_ok()); let block2 = Arc::new(block2).prepare(); @@ -1226,7 +1227,7 @@ proptest! { finalized_state.populate_with_anchors(&block2); let block1 = CheckpointVerifiedBlock::from(Arc::new(block1)); - let commit_result = finalized_state.commit_finalized_direct(block1.into(), None, "test"); + let commit_result = finalized_state.commit_finalized_direct(block1.into(), None, None, "test"); prop_assert!(commit_result.is_ok()); let block2 = Arc::new(block2).prepare(); diff --git a/zebra-state/src/service/check/tests/utxo.rs b/zebra-state/src/service/check/tests/utxo.rs index dd9017bea20..69bfe446f69 100644 --- a/zebra-state/src/service/check/tests/utxo.rs +++ b/zebra-state/src/service/check/tests/utxo.rs @@ -185,7 +185,7 @@ proptest! { // randomly choose to commit the block to the finalized or non-finalized state if use_finalized_state { let block1 = CheckpointVerifiedBlock::from(Arc::new(block1)); - let commit_result = finalized_state.commit_finalized_direct(block1.clone().into(), None, "test"); + let commit_result = finalized_state.commit_finalized_direct(block1.clone().into(), None, None, "test"); // the block was committed prop_assert_eq!(Some((Height(1), block1.hash)), read::best_tip(&non_finalized_state, &finalized_state.db)); @@ -273,7 +273,7 @@ proptest! { if use_finalized_state_spend { let block2 = CheckpointVerifiedBlock::from(Arc::new(block2)); - let commit_result = finalized_state.commit_finalized_direct(block2.clone().into(),None, "test"); + let commit_result = finalized_state.commit_finalized_direct(block2.clone().into(), None, None, "test"); // the block was committed prop_assert_eq!(Some((Height(2), block2.hash)), read::best_tip(&non_finalized_state, &finalized_state.db)); @@ -609,7 +609,7 @@ proptest! { if use_finalized_state_spend { let block2 = CheckpointVerifiedBlock::from(block2.clone()); - let commit_result = finalized_state.commit_finalized_direct(block2.clone().into(), None, "test"); + let commit_result = finalized_state.commit_finalized_direct(block2.clone().into(), None, None, "test"); // the block was committed prop_assert_eq!(Some((Height(2), block2.hash)), read::best_tip(&non_finalized_state, &finalized_state.db)); @@ -878,7 +878,7 @@ fn new_state_with_mainnet_transparent_data( if use_finalized_state { let block1 = CheckpointVerifiedBlock::from(block1.clone()); let commit_result = - finalized_state.commit_finalized_direct(block1.clone().into(), None, "test"); + finalized_state.commit_finalized_direct(block1.clone().into(), None, None, "test"); // the block was committed assert_eq!( diff --git a/zebra-state/src/service/finalized_state.rs b/zebra-state/src/service/finalized_state.rs index 947d1731710..56cdf28e21f 100644 --- a/zebra-state/src/service/finalized_state.rs +++ b/zebra-state/src/service/finalized_state.rs @@ -22,11 +22,15 @@ use std::{ }, }; -use zebra_chain::{block, parallel::tree::NoteCommitmentTrees, parameters::Network}; +use zebra_chain::{ + block::{self, merkle::AuthDataRoot, Block}, + ironwood, orchard, + parallel::tree::NoteCommitmentTrees, + parameters::Network, + sapling, +}; use zebra_db::{ - block::{ - RetentionPlan, ZAKURA_HEADER_BODY_SIZE_BY_HEIGHT, ZAKURA_HEADER_COMMITMENT_ROOTS_BY_HEIGHT, - }, + block::{RetentionPlan, VctData, ZAKURA_HEADER_BODY_SIZE_BY_HEIGHT}, chain::BLOCK_INFO, transparent::{BALANCE_BY_TRANSPARENT_ADDR, TX_LOC_BY_SPENT_OUT_LOC}, }; @@ -74,10 +78,22 @@ static COMMIT_COMPUTE_POOL: LazyLock = LazyLock::new(|| { pub mod column_family; +mod commitment_aux; +mod commitment_aux_verify; mod disk_db; mod disk_format; +mod vct; mod zebra_db; +use vct::VctState; + +/// The verified-commitment-trees `tree_aux` serving read path (design §9): the per-block +/// commitment roots for a height range, derived from the per-height trees. +pub(crate) use commitment_aux::serve_block_roots; + +pub use commitment_aux::{produce_final_frontiers_bytes, FinalFrontiersGenerationError}; +pub use vct::{validate_final_frontiers_bytes, FinalFrontiersValidationError}; + #[cfg(any(test, feature = "proptest-impl"))] mod arbitrary; @@ -174,22 +190,62 @@ pub const PRUNING_METADATA: &str = "pruning_metadata"; /// The name of the column family that marks a verified-commitment-trees /// (vct) synced database. /// -/// A vct-synced database skips historical per-height note-commitment tree -/// writes below the checkpoint handoff height. This column family holds a -/// single entry with that handoff height. +/// A vct-synced database is built by folding verified commitment roots into the +/// anchor set and history tree below the last checkpoint, skipping the per-height +/// note-commitment trees entirely. This column family holds a single entry, keyed +/// by the unit value `()`, mapping to the checkpoint handoff height: the lowest +/// height at which a per-height note-commitment tree is present. Per-height trees +/// are absent for every non-genesis height strictly below it. +/// +/// The presence of this entry marks the database as vct-synced: the historical +/// per-height trees were never written, so the database cannot answer historical +/// tree/subtree RPCs below the handoff height (the RPC handlers return a typed +/// archive-mode error there, §9). Vct sync is the default under checkpoint sync +/// for both Archive and Pruned storage modes, so a vct-synced database reopens in +/// either; the missing-history limitation is enforced at the RPC boundary, not at +/// reopen. This is orthogonal to pruning (which drops raw transactions but keeps +/// the trees); a database can be both. pub const VCT_SYNC_METADATA: &str = "vct_sync_metadata"; -/// The name of the column family that records the verified-commitment-trees -/// upgrade height. +/// The name of the column family that records the verified-commitment-trees upgrade height. +/// +/// This holds a single entry, keyed by the unit value `()`, mapping to `U`: the lowest height +/// this (vct-aware) binary committed, which is also the lowest height present in the +/// [`COMMITMENT_ROOTS_BY_HEIGHT`] serving index. It is written once — on the first committed +/// block — and never moved, so it is a stable boundary as the chain grows. /// -/// This height is the first block committed by code that writes the -/// [`COMMITMENT_ROOTS_BY_HEIGHT`] serving index. +/// `U` is what lets the two root sources be stitched without a gap: heights below `U` predate +/// this binary, so they carry per-height trees but no index entry and are served from the trees; +/// heights at or above `U` carry an index entry and are served from it. Combined with the +/// checkpoint handoff `H` in [`VCT_SYNC_METADATA`], it also bounds the band `[U, H)` in which a +/// vct-synced node holds no per-height tree, so historical tree/subtree RPCs are unavailable +/// there but available below `U` (pre-upgrade trees) and at/above `H` (semantic-sync trees). pub const VCT_UPGRADE_METADATA: &str = "vct_upgrade_metadata"; -/// The name of the column family holding per-height Sapling/Orchard -/// note-commitment roots, keyed by [`block::Height`]. +/// The name of the column family holding the per-height Sapling/Orchard note-commitment +/// roots, keyed by [`block::Height`]. +/// +/// This is the verified-commitment-trees serving index (design §4): a compact +/// `height -> (sapling_root, orchard_root)` map (64 bytes/height) that **every** node +/// persists for each committed block, on both the vct and legacy commit paths. Its purpose +/// is to let a vct-synced node — which folds verified roots in but writes no per-height +/// note-commitment trees — still answer the `tree_aux` `BlockRoots` read, so the +/// root-serving fleet does not collapse as nodes adopt vct sync. The roots are the same +/// values a legacy node derives from its per-height trees via `produce_block_roots`; serving +/// reads this index first and falls back to the trees only for databases written before the +/// index existed. pub const COMMITMENT_ROOTS_BY_HEIGHT: &str = "commitment_roots_by_height"; +/// Provisional peer-supplied per-height Sapling/Orchard roots attached to Zakura +/// header-sync responses. +/// +/// These roots are advisory metadata for header-ahead blocks. They are persisted +/// with `zakura_header_*` so VCT fast sync can read them before full block bodies +/// arrive, but they remain untrusted until block commit verifies them against the +/// header commitments. +pub const ZAKURA_HEADER_COMMITMENT_ROOTS_BY_HEIGHT: &str = + "zakura_header_commitment_roots_by_height"; + /// The finalized part of the chain state, stored in the db. /// /// `rocksdb` allows concurrent writes through a shared reference, @@ -241,6 +297,39 @@ pub struct FinalizedState { #[cfg(feature = "elasticsearch")] /// A collection of blocks to be sent to elasticsearch as a bulk. pub elastic_blocks: Vec, + + /// Verified-commitment-trees state (peer/fixture/capture mode), or `None` + /// when legacy recompute is selected. Shared across clones. + vct: Option>, + + /// Verify-before-commit dedup. Holds the `(height, hash)` of the next + /// block whose commitment was already validated by the previous fast + /// commit's look-ahead (`C(next, candidate)`). When the next block to commit + /// matches, its own commitment check is the identical computation, so it is + /// skipped — making each header commitment check run once instead of twice. + /// Guarded by hash identity (and height monotonicity), so a stale or cloned + /// value can never cause an incorrect skip. + vct_prevalidated_next: Option<(block::Height, block::Hash)>, + + /// `true` while a verified-commitment-trees (vct) fast sync has frozen the + /// note-commitment frontier — i.e. a verified commitment tree block has committed but the + /// checkpoint handoff (which replaces the frontier with the real one) has not. + /// + /// While frozen, the running frontier is no longer the real frontier for the + /// heights being committed, so a legacy recompute would fold a wrong root into + /// the history MMR and corrupt consensus state. The committer therefore refuses + /// to recompute for a height with no valid supplied root in this window, + /// returning a retryable error instead (see `commit_finalized_direct`). Reset to + /// `false` at the handoff, after which legacy recompute resumes from the real + /// frontier. + /// + /// Seeded from durable state on open (not just within a session): a vct sync + /// interrupted by a restart leaves the frozen frontier persisted but the tip + /// below the handoff, so [`FinalizedState::new`] re-derives this flag from the + /// vct-sync marker. Without that, the first post-restart height with no supplied + /// root would legacy-recompute against the stale on-disk frontier and corrupt the + /// MMR — the exact hazard this flag exists to prevent. + vct_frontier_frozen: bool, } impl FinalizedState { @@ -261,6 +350,40 @@ impl FinalizedState { ) } + /// Opens (or creates) the on-disk finalized state database read-write, for + /// offline tooling (e.g. the replay benchmark). + /// + /// Equivalent to [`FinalizedState::new`] but without the `elasticsearch` + /// feature's `enable_elastic_db` parameter, so callers compile unchanged + /// regardless of feature flags (elasticsearch is never enabled here). + pub fn new_writable(config: &Config, network: &Network) -> Self { + Self::new_with_debug( + config, + network, + false, + #[cfg(feature = "elasticsearch")] + false, + false, + ) + } + + /// Opens an existing on-disk finalized state database in **read-only** mode. + /// + /// Intended for offline tooling (e.g. the replay benchmark) that reads + /// committed blocks from a snapshot without triggering format upgrades or any + /// writes to the source database. Read-only opens skip format upgrades, so the + /// pristine snapshot is never mutated. + pub fn new_read_only(config: &Config, network: &Network) -> Self { + Self::new_with_debug( + config, + network, + false, + #[cfg(feature = "elasticsearch")] + false, + true, + ) + } + /// Returns an on-disk database instance with the supplied production and debug settings. /// If there is no existing database, creates a new database on disk. /// @@ -280,6 +403,7 @@ impl FinalizedState { enable_elastic_db, read_only, true, + true, ) } @@ -303,6 +427,35 @@ impl FinalizedState { enable_elastic_db, read_only, false, + true, + ) + } + + /// Reopens an on-disk database for VCT reopen tests without enforcing the + /// interrupted-fast-sync resume guard. + /// + /// Production always enforces the guard: a fast sync interrupted below the + /// checkpoint handoff can only resume through a configured VCT root source, so + /// reopening one without a source (`vct.is_none()`) refuses to open. Tests model a + /// (Mainnet) node restart on a configured network that has no embedded frontiers, + /// then attach a fixture root source after construction the way a real node's + /// configured source would already be present at open time, so they need to skip + /// the constructor-time guard. + #[cfg(test)] + pub(crate) fn new_without_resume_guard( + config: &Config, + network: &Network, + #[cfg(feature = "elasticsearch")] enable_elastic_db: bool, + ) -> Self { + Self::new_with_debug_and_storage_validation( + config, + network, + false, + #[cfg(feature = "elasticsearch")] + enable_elastic_db, + false, + true, + false, ) } @@ -313,6 +466,7 @@ impl FinalizedState { #[cfg(feature = "elasticsearch")] enable_elastic_db: bool, read_only: bool, validate_storage_mode: bool, + enforce_resume_guard: bool, ) -> Self { // Fail fast on an invalid storage configuration, before opening the database. if validate_storage_mode { @@ -361,6 +515,26 @@ impl FinalizedState { read_only, ); + let vct = VctState::from_config( + config.checkpoint_sync, + config.vct_fast_sync, + network, + db.clone(), + ); + + // Re-derive the frozen-frontier flag from durable state: a fast sync + // interrupted before the checkpoint handoff leaves the stale frozen frontier + // on disk (fast commits never write per-height trees) with the tip still below + // the handoff. Reopening in that window must keep the committer frozen so a + // height with no supplied root refuses instead of legacy-recomputing against + // the stale frontier. The handoff height itself carries the real frontier, so + // `tip < handoff` (exclusive) is exactly the frozen region. Read from the + // fast-sync marker, not `vct`, so it holds even if VCT is disabled this run. + let vct_frontier_frozen = db + .vct_synced_below() + .zip(db.finalized_tip_height()) + .is_some_and(|(handoff, tip)| tip < handoff); + #[cfg(feature = "elasticsearch")] let new_state = Self { debug_stop_at_height: config.debug_stop_at_height.map(block::Height), @@ -369,6 +543,9 @@ impl FinalizedState { db, elastic_db, elastic_blocks: vec![], + vct, + vct_prevalidated_next: None, + vct_frontier_frozen, }; #[cfg(not(feature = "elasticsearch"))] @@ -377,6 +554,9 @@ impl FinalizedState { checkpoint_raw_tx_retention_start: None, checkpoint_raw_tx_archive_backlog: Arc::new(AtomicBool::new(false)), db, + vct, + vct_prevalidated_next: None, + vct_frontier_frozen, }; // Pruning is a one-way storage mode. Refuse to open a database that has @@ -390,6 +570,23 @@ impl FinalizedState { ); } + // An *interrupted* fast sync — frozen frontier, tip still below the handoff — can + // only be safely resumed by the fast path (which supplies the verified roots). The + // on-disk frontier is stale, so the committer fails closed on every below-handoff + // height with no supplied root (§8). Reopening without a VCT root source selects the + // legacy committer, which can never supply those roots, so the node would refuse every + // block forever. Refuse to open instead, with a clear recovery path, rather than + // stalling silently. + if enforce_resume_guard && new_state.vct_frontier_frozen && new_state.vct.is_none() { + panic!( + "this database was previously synced in verified commitment tree mode that was \ + interrupted below the checkpoint handoff height. the fast path that supplies \ + the verified roots needed to resume it is disabled. Set \ + `consensus.checkpoint_sync = true` and `consensus.vct_fast_sync = true` to \ + finish the fast sync, or delete the cache directory and re-sync from genesis" + ); + } + // TODO: move debug_stop_at_height into a task in the start command (#3442) if let Some(tip_height) = new_state.db.finalized_tip_height() { if new_state.is_at_stop_height(tip_height) { @@ -551,11 +748,16 @@ impl FinalizedState { &mut self, ordered_block: QueuedCheckpointVerified, prev_note_commitment_trees: Option, - ) -> Result<(CheckpointVerifiedBlock, NoteCommitmentTrees), CommitCheckpointVerifiedError> { + next_checkpoint: Option<(Arc, Option)>, + ) -> Result< + (CheckpointVerifiedBlock, NoteCommitmentTrees), + (QueuedCheckpointVerified, CommitCheckpointVerifiedError), + > { let (checkpoint_verified, rsp_tx) = ordered_block; let result = self.commit_finalized_direct( checkpoint_verified.clone().into(), prev_note_commitment_trees, + next_checkpoint, "commit checkpoint-verified request", ); @@ -576,9 +778,13 @@ impl FinalizedState { .set(checkpoint_verified.height.0 as f64); }; - let _ = rsp_tx.send(result.clone().map(|(hash, _)| hash)); - - result.map(|(_hash, note_commitment_trees)| (checkpoint_verified, note_commitment_trees)) + match result { + Ok((hash, note_commitment_trees)) => { + let _ = rsp_tx.send(Ok(hash)); + Ok((checkpoint_verified, note_commitment_trees)) + } + Err(error) => Err(((checkpoint_verified, rsp_tx), error)), + } } /// Immediately commit a `finalized` block to the finalized state. @@ -599,43 +805,241 @@ impl FinalizedState { &mut self, finalizable_block: FinalizableBlock, prev_note_commitment_trees: Option, + // The next checkpoint block (and its precomputed + // auth data root), used to verify this block's fixture roots before the fast + // path trusts them. `None` is only valid for fast blocks at the checkpoint + // handoff, where the embedded final frontiers independently authenticate + // this height's roots, or outside the checkpoint commit path. + next_checkpoint: Option<(Arc, Option)>, source: &str, ) -> Result<(block::Hash, NoteCommitmentTrees), CommitCheckpointVerifiedError> { - let (height, hash, finalized, prev_note_commitment_trees, retention) = - match finalizable_block { - FinalizableBlock::Checkpoint { - checkpoint_verified, - } => { - // Checkpoint-verified blocks don't have an associated treestate, so we retrieve the - // treestate of the finalized tip from the database and update it for the block - // being committed, assuming the retrieved treestate is the parent block's - // treestate. Later on, this function proves this assumption by asserting that the - // finalized tip is the parent block of the block being committed. - - let block = checkpoint_verified.block.clone(); - let precomputed_auth_data_root = checkpoint_verified.auth_data_root; - let mut history_tree = self.db.history_tree(); - let prev_note_commitment_trees = prev_note_commitment_trees - .unwrap_or_else(|| self.db.note_commitment_trees_for_tip()); - - let mut note_commitment_trees = prev_note_commitment_trees.clone(); - let network = self.network(); + let ( + height, + hash, + finalized, + prev_note_commitment_trees, + retention, + fast_anchor_roots, + fast_sync_below, + ) = match finalizable_block { + FinalizableBlock::Checkpoint { + checkpoint_verified, + } => { + // Checkpoint-verified blocks don't have an associated treestate, so we retrieve the + // treestate of the finalized tip from the database and update it for the block + // being committed, assuming the retrieved treestate is the parent block's + // treestate. Later on, this function proves this assumption by asserting that the + // finalized tip is the parent block of the block being committed. + + let block = checkpoint_verified.block.clone(); + // Auth data root precomputed by the checkpoint verifier (if any), + // so the commitment check below doesn't recompute it here on the + // single-threaded committer. `AuthDataRoot` is `Copy`. + let precomputed_auth_data_root = checkpoint_verified.auth_data_root; + let mut history_tree = self.db.history_tree(); + let prev_note_commitment_trees = prev_note_commitment_trees + .unwrap_or_else(|| self.db.note_commitment_trees_for_tip()); + + let mut note_commitment_trees = prev_note_commitment_trees.clone(); + let network = self.network(); + let height = checkpoint_verified.height; + + // The last checkpoint height (boundary below which the vct + // path skips per-height trees), when final frontiers are loaded. + let vct_last_checkpoint_height = self + .vct + .as_ref() + .and_then(|v| v.vct_sync_last_checkpoint_height()); + + // In vct mode, if the source has this height's roots at or below the + // last checkpoint height, skip the per-block note-commitment frontier recompute + // (`update_trees_parallel`) entirely and fold the supplied roots into the + // anchor set and history leaf instead. The frontier stays the (frozen) + // parent frontier; nothing below the checkpoint reads it for consensus. + // See docs/design/verified-commitment-trees.md. + let vct_roots = self.vct.as_ref().and_then(|v| { + if vct_last_checkpoint_height + .is_some_and(|last_checkpoint_height| height > last_checkpoint_height) + { + None + } else { + v.vct_roots_at_height(height) + } + }); - // Run two independent CPU-intensive crypto operations concurrently - // on the rayon pool (Part 1 of the checkpoint-commit parallelization): - // - // - updating the note commitment trees, and - // - checking this block's commitment against the *parent* history tree. - // - // These are independent: the commitment check reads only the parent - // history tree (not this block's note commitment trees), and the - // history tree push below depends on both, so it runs after the join. + let mut vct_anchor_roots = None; + // `Some(C)` for fast blocks of a persistent fast sync; written + // to the fast-sync marker in the commit batch. + let mut vct_sync_below = None; + + if let Some((sapling_root, orchard_root)) = vct_roots { + // The handoff frontiers are the only non-successor authority that + // can authenticate this block's own supplied roots before they are + // persisted. + let last_checkpoint_frontiers = self + .vct + .as_ref() + .and_then(|v| v.final_frontiers_for_last_checkpoint(height)); + + // This block's own commitment check is identical to the + // previous vct block's look-ahead. When that look-ahead + // already validated this exact header, skip the duplicate. + let block_hash = block.hash(); + let is_prevalidated = self.vct_prevalidated_next == Some((height, block_hash)); + if is_prevalidated { + if let Some(v) = &self.vct { + v.record_prevalidated(); + } + // Observability: the previous fast block's look-ahead already + // validated this header, so its commitment check was skipped (the + // dedup). A subset of `state.vct.fast.block.count`. + metrics::counter!("state.vct.prevalidated.block.count").increment(1); + } + + let mut verification_items = vec![ + commitment_aux_verify::CommitmentRootVerification::with_roots( + block.clone(), + sapling_root, + orchard_root, + precomputed_auth_data_root, + is_prevalidated, + ), + ]; + if let Some((next_block, next_auth)) = &next_checkpoint { + verification_items.push( + commitment_aux_verify::CommitmentRootVerification::header_only( + next_block.clone(), + *next_auth, + ), + ); + } + + // Verifies this block's own header, folds its supplied roots into + // the candidate tree, and when buffered checks the successor header + // against that candidate (the one-block lag). + let candidate = COMMIT_COMPUTE_POOL + .install(|| { + commitment_aux_verify::verify_commitment_roots( + &network, + (*history_tree).clone(), + verification_items, + ) + }) + .map_err(|(_fail_height, error)| { + self.vct_prevalidated_next = None; + self.vct_reject_supplied_root(height, error) + })?; + + if let Some((next_block, _next_auth)) = &next_checkpoint { + self.vct_prevalidated_next = Some(( + (height + 1).expect("checkpoint block heights are valid"), + next_block.hash(), + )); + } else if self + .vct + .as_ref() + .is_some_and(|v| v.vct_root_needs_successor(height, &network)) + { + // Untrusted root at/above Heartwood, no successor to confirm it, + // not the last checkpoint: defer rather than persist it unverified. Leaves + // the database untouched; the block re-commits once the successor + // is buffered. + metrics::counter!("state.vct.root.await_successor.count").increment(1); + return Err(ValidateContextError::VctSuppliedRootAwaitingSuccessor { + height, + } + .into()); + } else { + self.vct_prevalidated_next = None; + } + + history_tree = Arc::new(candidate); + if let Some(v) = &self.vct { + v.record_fast_block(); + } + // Observability: this block folded supplied roots and skipped the + // note-commitment frontier recompute (the verified-commitment-trees + // fast path). Paired with `state.vct.legacy.block.count` below, this + // gives a live fast-vs-legacy ratio. + metrics::counter!("state.vct.fast.block.count").increment(1); + + // When final frontiers are loaded, this is a persistent fast + // sync: mark the database fast-synced (per-height trees absent + // below the handoff height). + vct_sync_below = vct_last_checkpoint_height; + + if let Some((sapling_frontier, orchard_frontier, sprout_frontier)) = + last_checkpoint_frontiers + { + // Checkpoint handoff: verify the supplied frontiers against + // this block's verified roots (collision resistance makes the + // root a binding commitment to the frontier), then write them + // as the real tip treestate via the legacy write path + // (`fast_anchor_roots` left `None`), so post-checkpoint + // semantic verification resumes from a correct frontier. + self.vct_verify_handoff_frontier_roots( + height, + &sapling_frontier, + &orchard_frontier, + &sapling_root, + &orchard_root, + )?; + + // Subtree tips are left `None`: the resuming chain recomputes + // them from the frontier position. + note_commitment_trees = NoteCommitmentTrees { + sprout: sprout_frontier, + sapling: sapling_frontier, + sapling_subtree: None, + orchard: orchard_frontier, + orchard_subtree: None, + ironwood: Arc::::default(), + ironwood_subtree: None, + }; + + // The handoff writes the real final frontier as the tip + // treestate, so the frontier is no longer frozen: heights at and + // above the handoff resume legacy recompute from a correct frontier. + self.vct_frontier_frozen = false; + } else { + vct_anchor_roots = Some((sapling_root, orchard_root)); + + // A non-handoff fast block leaves the note-commitment frontier + // frozen (it folds roots instead of advancing the trees), so a + // later height with no valid supplied root must not legacy-recompute + // against this stale frontier (see the `else` branch below). + self.vct_frontier_frozen = true; + } + } else if self.vct_frontier_frozen { + // Frozen-frontier safety: a fast sync has already frozen the + // note-commitment frontier, but this height has no valid supplied root + // (never fetched, or evicted after failing verification). Recomputing + // here would fold a wrong root into the history MMR and corrupt state, + // so refuse with a retryable error and leave the database untouched — + // the block is committed once a verifiable root is fetched from a peer. + metrics::counter!("state.vct.root.unavailable.count").increment(1); + tracing::warn!( + ?height, + "VCT: no verifiable supplied root for a frozen-frontier height; \ + refusing to recompute (retryable)" + ); + return Err(ValidateContextError::VctSuppliedRootUnavailable { height }.into()); + } else { + // Not a fast block: any cached pre-validation does not apply to + // the next fast block (its parent frontier differs), so clear it. + self.vct_prevalidated_next = None; + + // Observability: this block recomputed the note-commitment frontier + // (the legacy path) — either VCT is off, or the fast path's roots were + // unavailable for this height and it safely fell back. + metrics::counter!("state.vct.legacy.block.count").increment(1); + + // Legacy / capture path: recompute the note-commitment frontier. // - // The commitment check is done here (and not during semantic - // validation) because it needs the history tree root, and the - // checkpoint verifier doesn't run contextual validation. For - // Nu5-onward the block hash commits only to non-authorizing data - // (ZIP-244), so this verifies the authorizing-data commitment. + // Run two independent CPU-intensive crypto operations concurrently + // on the rayon pool: updating the note commitment trees, and + // checking this block's commitment against the *parent* history + // tree. They are independent; the history push below joins them. #[cfg(feature = "commit-metrics")] metrics::histogram!("zebra.state.write.block_tx_count") .record(block.transactions.len() as f64); @@ -669,10 +1073,7 @@ impl FinalizedState { // Surface the tree-update error first, preserving the error // precedence of the previous sequential code. tree_result.map_err(ValidateContextError::from)?; - // `rayon::in_place_scope_fifo` guarantees all spawned tasks - // complete before the scope returns, so `commitment_result` is - // always `Some` here: the spawned closure wrote to it before - // the scope exited. + // `in_place_scope_fifo` joins all spawned tasks, so this is `Some`. commitment_result.expect("scope has already finished")?; // Update the history tree (depends on both operations above). @@ -691,47 +1092,45 @@ impl FinalizedState { .map_err(Arc::new) .map_err(ValidateContextError::from)?; - // Total serial wall time of the checkpoint compute phase (note tree - // update + commitment check, then history push). Compared against the - // summed phase times, this shows the overlap win. #[cfg(feature = "commit-metrics")] metrics::histogram!("zebra.state.write.checkpoint_compute.duration_seconds") .record(_ckpt_compute.elapsed().as_secs_f64()); - - let treestate = Treestate { - note_commitment_trees, - history_tree, - }; - - let height = checkpoint_verified.height; - let hash = checkpoint_verified.hash; - - ( - height, - hash, - FinalizedBlock::from_checkpoint_verified(checkpoint_verified, treestate), - Some(prev_note_commitment_trees), - self.retention_plan(height, true), - ) - } - FinalizableBlock::Contextual { - contextually_verified, - treestate, - } => { - let height = contextually_verified.height; - - ( - height, - contextually_verified.hash, - FinalizedBlock::from_contextually_verified( - contextually_verified, - *treestate, - ), - prev_note_commitment_trees, - self.retention_plan(height, false), - ) } - }; + + let treestate = Treestate { + note_commitment_trees, + history_tree, + }; + + let hash = checkpoint_verified.hash; + + ( + height, + hash, + FinalizedBlock::from_checkpoint_verified(checkpoint_verified, treestate), + Some(prev_note_commitment_trees), + self.retention_plan(height, true), + vct_anchor_roots, + vct_sync_below, + ) + } + FinalizableBlock::Contextual { + contextually_verified, + treestate, + } => { + let height = contextually_verified.height; + + ( + height, + contextually_verified.hash, + FinalizedBlock::from_contextually_verified(contextually_verified, *treestate), + prev_note_commitment_trees, + self.retention_plan(height, false), + None, + None, + ) + } + }; let committed_tip_hash = self.db.finalized_tip_hash(); let committed_tip_height = self.db.finalized_tip_height(); @@ -777,11 +1176,15 @@ impl FinalizedState { &network, source, retention, - None, + VctData::new(fast_anchor_roots, fast_sync_below), ) }); if result.is_ok() { + if let Some(vct) = &self.vct { + vct.evict_committed_roots_through(height); + } + if retention.clears_archive_backlog() { self.checkpoint_raw_tx_archive_backlog .store(false, Ordering::Relaxed); @@ -800,6 +1203,9 @@ impl FinalizedState { "stopping at configured height, flushing database to disk" ); + // POC: emit the equivalence digest + fast-path summary before exit. + self.vct_log_equivalence_digest(); + // We're just about to do a forced exit, so it's ok to do a forced db shutdown self.db.shutdown(true); @@ -815,6 +1221,158 @@ impl FinalizedState { result.map(|hash| (hash, note_commitment_trees)) } + /// POC: `true` when the verified-commitment-trees fast (skip-recompute) path will + /// apply to `height` — i.e. fast mode is active *and* the source already holds this + /// height's roots, so the committer will fold them in and skip the frontier recompute. + /// The write loop uses this to record fast-path hit/miss metrics after a + /// successful checkpoint commit. + pub(crate) fn vct_fast_will_apply(&self, height: block::Height) -> bool { + self.vct + .as_ref() + .is_some_and(|v| v.is_fast() && v.vct_roots_at_height(height).is_some()) + } + + /// Clears any cached successor prevalidation. + /// + /// The finalized write loop calls this when it discards checkpoint queue state, so a + /// look-ahead header that no longer corresponds to the next committed block cannot + /// authorize a later fast-path skip. + pub(crate) fn clear_vct_prevalidated_next(&mut self) { + self.vct_prevalidated_next = None; + } + + /// `true` when committing `height` on the fast path needs a buffered successor before + /// it can safely persist this block's supplied roots. + /// + /// Only untrusted peer-supplied roots at or above Heartwood require this. The + /// checkpoint handoff is exempt because its embedded final frontiers are verified + /// against this block's roots before the real tip treestate is written; trusted + /// local fixtures can commit their tip root on the in-arrears check. + pub(crate) fn vct_fast_needs_successor(&self, height: block::Height) -> bool { + self.vct + .as_ref() + .is_some_and(|v| v.vct_root_needs_successor(height, &self.network())) + } + + /// Verify checkpoint handoff frontiers against this block's supplied roots. + fn vct_verify_handoff_frontier_roots( + &mut self, + height: block::Height, + sapling_frontier: &sapling::tree::NoteCommitmentTree, + orchard_frontier: &orchard::tree::NoteCommitmentTree, + sapling_root: &sapling::tree::Root, + orchard_root: &orchard::tree::Root, + ) -> Result<(), CommitCheckpointVerifiedError> { + if sapling_frontier.root() != *sapling_root || orchard_frontier.root() != *orchard_root { + self.vct_prevalidated_next = None; + return Err(self.vct_reject_supplied_root( + height, + ValidateContextError::VctSuppliedRootUnavailable { height }, + )); + } + + Ok(()) + } + + /// Reject a supplied fast-path root that failed verification for `height`. + /// + /// Evicts the bad root from the source so a re-fetch can replace it with a verifiable + /// one from a different peer, and returns a typed, retryable error. In fast mode the + /// note-commitment frontier is frozen, so the committer cannot recompute the root + /// locally (that would fold a wrong root into the history MMR); it must refuse and + /// leave the database untouched rather than persist or corrupt state. This is what + /// keeps a single malicious peer from halting the sync: the bad root is dropped, not + /// retried forever, and any honest peer's root verifies. + fn vct_reject_supplied_root( + &self, + height: block::Height, + error: ValidateContextError, + ) -> CommitCheckpointVerifiedError { + if let Some(v) = &self.vct { + v.invalidate_fast_root(height); + } + metrics::counter!("state.vct.root.rejected.count").increment(1); + tracing::warn!( + ?height, + ?error, + "VCT: supplied commitment root failed verification; evicted for re-fetch" + ); + ValidateContextError::VctSuppliedRootUnavailable { height }.into() + } + + /// Test-only: enable fast mode reading roots/frontiers from an arbitrary + /// [`commitment_aux::CommitmentRootSource`] (e.g. a payload produced from a + /// database via [`commitment_aux::produce_block_roots`]), so the producer→consumer + /// round-trip can be exercised in-process. `requires_verified_successor` marks + /// whether the installed source is untrusted and must defer tip roots until their + /// successor is buffered. + #[cfg(test)] + pub(in crate::service::finalized_state) fn enable_vct_fast_source( + &mut self, + source: Box, + requires_verified_successor: bool, + ) { + self.vct = Some(VctState::test_with_source( + source, + requires_verified_successor, + )); + } + + /// Test-only: the fast-sync handoff height recorded in the database marker, if any. + #[cfg(test)] + pub(crate) fn vct_fast_synced_below(&self) -> Option { + self.db.vct_synced_below() + } + + /// Test-only: number of blocks that took the fast (skip-recompute) path so far. + #[cfg(test)] + pub(crate) fn vct_fast_count(&self) -> u64 { + self.vct.as_ref().map(|v| v.fast_count()).unwrap_or(0) + } + + /// Test-only: number of fast blocks whose own commitment check was skipped by + /// the dedup (the previous block's look-ahead already validated them). + #[cfg(test)] + pub(crate) fn vct_prevalidated_count(&self) -> u64 { + self.vct + .as_ref() + .map(|v| v.prevalidated_count()) + .unwrap_or(0) + } + + /// POC: log the consensus-equivalence digest (anchor sets + history root) and + /// the fast-path block count at the stop height, so a legacy run and a fast run + /// can be compared. Gated by `VCT_DIGEST` so normal runs pay nothing. + fn vct_log_equivalence_digest(&self) { + if std::env::var_os("VCT_DIGEST").is_none() { + return; + } + + let fast_count = if let Some(v) = &self.vct { + v.fast_count() + } else { + 0 + }; + + let ( + sapling_anchor_count, + sapling_anchor_digest, + orchard_anchor_count, + orchard_anchor_digest, + ) = self.db.vct_anchor_digest(); + let history_root = self.db.history_tree().hash(); + + tracing::info!( + sapling_anchor_count, + sapling_anchor_digest, + orchard_anchor_count, + orchard_anchor_digest, + ?history_root, + vct_fast_blocks = fast_count, + "VCT-DIGEST" + ); + } + #[cfg(feature = "elasticsearch")] /// Store finalized blocks into an elasticsearch database. /// diff --git a/zebra-state/src/service/finalized_state/commitment_aux.rs b/zebra-state/src/service/finalized_state/commitment_aux.rs new file mode 100644 index 00000000000..47fbf583a16 --- /dev/null +++ b/zebra-state/src/service/finalized_state/commitment_aux.rs @@ -0,0 +1,756 @@ +//! Commitment-root source seam and payload types for the verified-commitment-trees +//! fast path (`docs/design/verified-commitment-trees.md` §5, increment 3). +//! +//! The fast path consumes per-block Sapling/Orchard roots and a final frontier at the +//! checkpoint handoff. *Where* that data comes from is abstracted behind +//! [`CommitmentRootSource`], so the committer reads through one seam regardless of +//! source. The production source is the transport-backed [`PeerSource`] over `tree_aux`; +//! tests use a crate-local fixture source over the same `RootMap` shape. +//! +//! It also provides the **producer** half ([`produce_block_roots`] / +//! [`produce_final_frontiers`]): deriving the same payload from an existing database's +//! per-height trees. That is the read path a serving node runs, and tests can feed the +//! DB-produced payload back through the fast path in-process to prove producer and +//! consumer agreement without networking. + +use std::{ + collections::HashMap, + fmt, + sync::{Arc, RwLock}, +}; + +use thiserror::Error; +use zebra_chain::{ + block::{self, merkle::AuthDataRoot}, + orchard, sapling, sprout, +}; + +use super::{FromDisk, IntoDisk, ZebraDb}; + +/// Per-block verified commitment roots — the essential fast-path payload (design §5.1), +/// the wire payload carried over `tree_aux` (increment 6a). Defined in `zebra-chain` so +/// `zebra-network` and `zebra-state` share it without a dependency cycle. +pub(super) use zebra_chain::parallel::commitment_aux::BlockCommitmentRoots; + +/// The verified final note-commitment frontiers at the checkpoint handoff height +/// (design §5.2). +/// +/// Fast mode skips the per-block frontier recompute below the checkpoint, so the +/// running Sapling/Orchard frontiers are never advanced. To let post-checkpoint +/// semantic verification resume, the real frontiers at the checkpoint are supplied +/// here, verified (`frontier.root() == the verified root at the checkpoint`), and +/// written as the tip treestate at the handoff. Subtree tips are not carried: the +/// resuming chain recomputes them from the frontier position. +#[derive(Clone, Debug)] +pub(super) struct FinalFrontiers { + pub(super) height: block::Height, + pub(super) sapling: Arc, + pub(super) orchard: Arc, + pub(super) sprout: Arc, +} + +/// Errors producing [`FinalFrontiers`] from a finalized database. +#[derive(Clone, Debug, Eq, Error, PartialEq)] +pub enum FinalFrontiersGenerationError { + /// The database has no Sapling tree at the requested height. + #[error("missing Sapling final frontier tree at height {height:?}")] + MissingSaplingTree { + /// The requested final frontier height. + height: block::Height, + }, + + /// The database has no Orchard tree at the requested height. + #[error("missing Orchard final frontier tree at height {height:?}")] + MissingOrchardTree { + /// The requested final frontier height. + height: block::Height, + }, +} + +/// Errors parsing [`FinalFrontiers`] from the embedded/frontier-file byte format. +#[derive(Clone, Debug, Eq, PartialEq)] +pub(super) enum FinalFrontiersParseError { + /// The input ended before the 4-byte height field. + MissingHeight { + /// The total number of bytes in the input. + actual_len: usize, + }, + /// The input ended before a tree blob's 4-byte length prefix. + MissingLength { + /// The tree whose length prefix was being read. + tree: &'static str, + /// Byte offset where the length prefix starts. + offset: usize, + /// Bytes remaining from `offset`. + remaining: usize, + }, + /// A tree blob's length prefix points past the end of the input. + TruncatedBlob { + /// The tree whose blob was being read. + tree: &'static str, + /// Byte offset where the blob starts. + offset: usize, + /// Blob length from the prefix. + expected_len: usize, + /// Bytes remaining from `offset`. + remaining: usize, + }, + /// A tree blob's length prefix overflows `usize` arithmetic. + LengthOverflow { + /// The tree whose blob was being read. + tree: &'static str, + /// Byte offset where the blob starts. + offset: usize, + /// Blob length from the prefix. + len: usize, + }, + /// The parser consumed all expected fields, but extra bytes remained. + TrailingBytes { + /// Byte offset where the trailing data starts. + offset: usize, + /// Number of trailing bytes. + trailing_len: usize, + }, +} + +impl fmt::Display for FinalFrontiersParseError { + fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { + match self { + FinalFrontiersParseError::MissingHeight { actual_len } => write!( + f, + "missing final frontier height: expected 4 bytes, got {actual_len}" + ), + FinalFrontiersParseError::MissingLength { + tree, + offset, + remaining, + } => write!( + f, + "missing {tree} frontier length prefix at byte {offset}: expected 4 bytes, got {remaining}" + ), + FinalFrontiersParseError::TruncatedBlob { + tree, + offset, + expected_len, + remaining, + } => write!( + f, + "truncated {tree} frontier blob at byte {offset}: length prefix says {expected_len} bytes, but only {remaining} remain" + ), + FinalFrontiersParseError::LengthOverflow { tree, offset, len } => write!( + f, + "{tree} frontier blob length overflows at byte {offset}: {len} bytes" + ), + FinalFrontiersParseError::TrailingBytes { + offset, + trailing_len, + } => write!( + f, + "unexpected trailing final frontier bytes at byte {offset}: {trailing_len} bytes" + ), + } + } +} + +impl std::error::Error for FinalFrontiersParseError {} + +impl FinalFrontiers { + /// Serialize to the embedded byte format: height (u32 LE), then sapling, orchard, + /// and sprout trees, each as `u32`-LE-length-prefixed `IntoDisk` bytes. Used to + /// create embedded or test final-frontier fixtures. + pub(super) fn to_bytes(&self) -> Vec { + let mut out = Vec::new(); + out.extend_from_slice(&self.height.0.to_le_bytes()); + let blobs: [Vec; 3] = [ + IntoDisk::as_bytes(&*self.sapling), + IntoDisk::as_bytes(&*self.orchard), + IntoDisk::as_bytes(&*self.sprout), + ]; + for blob in blobs { + let len = u32::try_from(blob.len()).expect("note commitment tree fits in u32 bytes"); + out.extend_from_slice(&len.to_le_bytes()); + out.extend_from_slice(&blob); + } + out + } + + /// Parse the embedded byte format written by [`Self::to_bytes`]. + pub(super) fn from_bytes(bytes: &[u8]) -> Result { + let height_bytes = bytes + .get(0..4) + .ok_or(FinalFrontiersParseError::MissingHeight { + actual_len: bytes.len(), + })?; + let height_bytes: [u8; 4] = + height_bytes + .try_into() + .map_err(|_| FinalFrontiersParseError::MissingHeight { + actual_len: bytes.len(), + })?; + let height = block::Height(u32::from_le_bytes(height_bytes)); + + // Read three `u32`-length-prefixed blobs starting after the height. + let mut cursor: usize = 4; + let mut next_blob = |tree: &'static str| -> Result, FinalFrontiersParseError> { + let len_end = + cursor + .checked_add(4) + .ok_or(FinalFrontiersParseError::LengthOverflow { + tree, + offset: cursor, + len: 4, + })?; + let len_bytes = + bytes + .get(cursor..len_end) + .ok_or(FinalFrontiersParseError::MissingLength { + tree, + offset: cursor, + remaining: bytes.len().saturating_sub(cursor), + })?; + let len_bytes: [u8; 4] = + len_bytes + .try_into() + .map_err(|_| FinalFrontiersParseError::MissingLength { + tree, + offset: cursor, + remaining: bytes.len().saturating_sub(cursor), + })?; + // Zebra's supported platforms have at least 32-bit `usize`, so every + // u32 length prefix fits in memory indexes. + let len = u32::from_le_bytes(len_bytes) as usize; + cursor = len_end; + let blob_end = + cursor + .checked_add(len) + .ok_or(FinalFrontiersParseError::LengthOverflow { + tree, + offset: cursor, + len, + })?; + let blob = + bytes + .get(cursor..blob_end) + .ok_or(FinalFrontiersParseError::TruncatedBlob { + tree, + offset: cursor, + expected_len: len, + remaining: bytes.len().saturating_sub(cursor), + })?; + cursor = blob_end; + Ok(blob.to_vec()) + }; + let sapling = next_blob("sapling")?; + let orchard = next_blob("orchard")?; + let sprout = next_blob("sprout")?; + + if cursor != bytes.len() { + return Err(FinalFrontiersParseError::TrailingBytes { + offset: cursor, + trailing_len: bytes.len() - cursor, + }); + } + + Ok(FinalFrontiers { + height, + sapling: Arc::new(::from_bytes( + sapling, + )), + orchard: Arc::new(::from_bytes( + orchard, + )), + sprout: Arc::new(::from_bytes( + sprout, + )), + }) + } +} + +/// Where the fast path's verified per-block roots and handoff frontiers come from. +/// +/// One enduring seam, two enduring data paths: the standard/legacy path rebuilds +/// trees locally and never consults a source; the fast verified path reads roots +/// from *some* source and verifies them against the headers. The production source is +/// [`PeerSource`]; tests may install a trusted local source to isolate committer +/// behavior. The trait carries no trust policy by itself: the owning VCT state decides +/// whether supplied roots must be confirmed by a buffered successor before commit. +pub(super) trait CommitmentRootSource: std::fmt::Debug + Send + Sync { + /// The supplied roots for `height`, if this source has them. + fn vct_root(&self, height: block::Height) + -> Option<(sapling::tree::Root, orchard::tree::Root)>; + + /// The checkpoint handoff height (below which the vct path skips per-height + /// trees), if this source supplies a final frontier. + fn vct_last_checkpoint_height(&self) -> Option; + + /// The verified final frontiers at the handoff height, if supplied. + fn final_frontiers(&self) -> Option<&FinalFrontiers>; + + /// Discard the supplied root for `height` so a later [`fast_root`](Self::fast_root) + /// returns `None` for it. + /// + /// Called by the committer when a supplied root fails verification: dropping the bad + /// root un-poisons the cache so a re-fetch from a different peer can replace it, rather + /// than the committer re-reading the same rejected root forever. The default is a no-op + /// for test-only local sources; the peer source overrides it. + fn invalidate(&self, _height: block::Height) {} + + /// Discard roots for heights that have already been committed. + /// + /// Called after the database write succeeds, so retry paths still keep roots needed + /// for an uncommitted block. The default is a no-op for test-only local sources; the + /// peer source uses this to keep its live fetch-ahead cache bounded during sync. + fn evict_committed_through(&self, _height: block::Height) {} +} + +/// The shared in-memory representation behind the concrete sources: a height→roots +/// map plus the optional handoff frontiers. +#[cfg(test)] +#[derive(Debug, Default)] +struct RootMap { + roots: HashMap, + frontiers: Option, +} + +#[cfg(test)] +impl RootMap { + fn fast_root( + &self, + height: block::Height, + ) -> Option<(sapling::tree::Root, orchard::tree::Root)> { + self.roots.get(&height.0).copied() + } + + fn handoff_height(&self) -> Option { + self.frontiers.as_ref().map(|f| f.height) + } + + fn final_frontiers(&self) -> Option<&FinalFrontiers> { + self.frontiers.as_ref() + } +} + +/// Test-only local source over a height-keyed roots map. +#[cfg(test)] +#[derive(Debug)] +pub(super) struct FixtureSource(RootMap); + +#[cfg(test)] +impl FixtureSource { + pub(super) fn new( + roots: HashMap, + frontiers: Option, + ) -> Self { + FixtureSource(RootMap { roots, frontiers }) + } +} + +#[cfg(test)] +impl CommitmentRootSource for FixtureSource { + fn vct_root( + &self, + height: block::Height, + ) -> Option<(sapling::tree::Root, orchard::tree::Root)> { + self.0.fast_root(height) + } + fn vct_last_checkpoint_height(&self) -> Option { + self.0.handoff_height() + } + fn final_frontiers(&self) -> Option<&FinalFrontiers> { + self.0.final_frontiers() + } +} + +/// A [`CommitmentRootSource`] backed by provisional header-ahead roots in `db`. +/// +/// Header sync persists peer-supplied roots into `db` ahead of body commit; the committer +/// reads them per height through the [`CommitmentRootSource`] seam. The handoff frontier is +/// embedded in the binary (design §5.2), held immutably here and never fetched over the +/// network. The in-memory `cache` is test-only scaffolding for the non-`db` source. +#[derive(Debug)] +pub(super) struct PeerSource { + db: Option, + cache: Arc>, + frontiers: Option, +} + +/// Shared peer-source cache state. +#[derive(Debug, Default)] +struct PeerRootsCache { + roots: HashMap, + committed_through: Option, +} + +impl PeerSource { + /// Create an empty in-memory peer source and a writer sharing its cache. `frontiers` + /// is the embedded handoff frontier (`None` for the bare benchmark, with no checkpoint + /// handoff). The writer lets a test fill roots before and after the source is moved + /// into the committer. + #[cfg(any(test, feature = "proptest-impl"))] + #[allow(dead_code)] + pub(super) fn new(frontiers: Option) -> (Self, PeerSourceWriter) { + let cache = Arc::new(RwLock::new(PeerRootsCache::default())); + let writer = PeerSourceWriter { + cache: Arc::clone(&cache), + }; + ( + PeerSource { + db: None, + cache, + frontiers, + }, + writer, + ) + } + + /// Create a source backed by provisional header-ahead roots in `db`. + pub(super) fn new_with_db(db: ZebraDb, frontiers: Option) -> Self { + PeerSource { + db: Some(db), + cache: Arc::new(RwLock::new(PeerRootsCache::default())), + frontiers, + } + } +} + +/// Test-only writer sharing a [`PeerSource`]'s in-memory cache, so a proptest can fill +/// roots before and after the source is moved into the committer. +#[cfg(any(test, feature = "proptest-impl"))] +#[derive(Clone, Debug)] +pub(super) struct PeerSourceWriter { + cache: Arc>, +} + +#[cfg(any(test, feature = "proptest-impl"))] +impl PeerSourceWriter { + /// Insert roots into the shared in-memory cache. Last write wins per uncommitted + /// height; roots at already-committed heights are ignored. + #[allow(dead_code)] + pub(super) fn insert_roots(&self, roots: impl IntoIterator) { + let mut cache = self.cache.write().expect("peer source roots lock poisoned"); + for r in roots { + if cache + .committed_through + .is_some_and(|height| r.height.0 <= height) + { + continue; + } + + cache + .roots + .insert(r.height.0, (r.sapling_root, r.orchard_root)); + } + } +} + +impl CommitmentRootSource for PeerSource { + fn vct_root( + &self, + height: block::Height, + ) -> Option<(sapling::tree::Root, orchard::tree::Root)> { + if let Some(db) = &self.db { + return db + .zakura_header_commitment_roots_by_height_range(height..=height) + .into_iter() + .next() + .map(|roots| (roots.sapling_root, roots.orchard_root)); + } + + self.cache + .read() + .expect("peer source roots lock poisoned") + .roots + .get(&height.0) + .copied() + } + fn vct_last_checkpoint_height(&self) -> Option { + self.frontiers.as_ref().map(|f| f.height) + } + fn final_frontiers(&self) -> Option<&FinalFrontiers> { + self.frontiers.as_ref() + } + fn invalidate(&self, height: block::Height) { + // Drop the rejected root so the next read misses; header sync can then deliver a + // verifiable replacement for this height from another peer. + if let Some(db) = &self.db { + if let Err(error) = db.delete_zakura_header_commitment_roots([height]) { + tracing::debug!(?error, ?height, "failed to delete rejected VCT root"); + } + return; + } + + self.cache + .write() + .expect("peer source roots lock poisoned") + .roots + .remove(&height.0); + } + + fn evict_committed_through(&self, height: block::Height) { + let mut cache = self.cache.write().expect("peer source roots lock poisoned"); + let start = cache + .committed_through + .map_or(0, |height| height.saturating_add(1)); + + if start <= height.0 { + for cached_height in start..=height.0 { + cache.roots.remove(&cached_height); + } + cache.committed_through = Some(height.0); + } + } +} + +/// Produce the per-block roots payload for `range` from `db`'s per-height trees. +/// +/// This is the serving read path (the future `TreeAuxStatePort::read_block_roots`), +/// minus the network: it derives each root from the stored per-height tree, exactly +/// the value the fast path folds into the anchor set. It requires per-height trees, so +/// the caller restricts it to a non-fast-synced (archive/pre-index) database within the +/// tip, where the trees are present. As defense-in-depth on this peer-triggered read, a +/// height whose tree is unexpectedly absent stops the scan and serves the contiguous +/// prefix collected so far rather than panicking; the wire client validates contiguity +/// and treats a short batch as partial progress. +// The `ReadRequest::BlockRoots` serving read path; also exercised by the round-trip test. +pub(crate) fn produce_block_roots( + db: &ZebraDb, + range: std::ops::RangeInclusive, +) -> Vec { + let (start, end) = (range.start().0, range.end().0); + let mut roots = Vec::new(); + for h in start..=end { + let height = block::Height(h); + let (Some(sapling), Some(orchard)) = ( + db.sapling_tree_by_height(&height), + db.orchard_tree_by_height(&height), + ) else { + break; + }; + // Below the upgrade height the serving index does not exist, so derive the + // auth-data root and the shielded tx-counts from the locally stored block (this + // archival node holds the body for these heights). Zero only if the body is somehow + // absent, in which case the recipient simply re-fetches from a node that has it. + let block = db.block(height.into()); + let (sapling_tx, orchard_tx, ironwood_tx, auth_data_root) = block + .as_ref() + .map(|block| { + ( + block.sapling_transactions_count(), + block.orchard_transactions_count(), + block.ironwood_transactions_count(), + block.auth_data_root(), + ) + }) + .unwrap_or((0, 0, 0, AuthDataRoot::from([0u8; 32]))); + roots.push(BlockCommitmentRoots { + height, + sapling_root: sapling.root(), + orchard_root: orchard.root(), + // The Ironwood tree does not exist below Nu7, so its root is the empty-tree root + // for every currently-servable height (no per-height Ironwood tree store yet). + ironwood_root: zebra_chain::ironwood::tree::NoteCommitmentTree::default().root(), + sapling_tx, + orchard_tx, + ironwood_tx, + auth_data_root, + }); + } + roots +} + +/// Serve the per-block roots for `range`, stitching the two sources at the upgrade height `U`. +/// +/// The `commitment_roots_by_height` serving index only covers heights at and above `U` (the lowest +/// height this binary committed). Heights below `U` predate the index, so they are derived from the +/// per-height trees instead, and the two runs are concatenated. This is what lets a node that +/// upgraded mid-chain serve a request that straddles `U` as one gap-free batch, rather than the +/// short index-only prefix that would stall the client's minimum-progress check. +/// +/// Both sources stop at the first absent height, so the result is always a contiguous run from +/// `range.start()`; a tree gap below `U` is served as the prefix collected so far without reaching +/// into the index. A database that never recorded `U` — a pre-index archive node — derives the +/// whole range from the trees, the original archive fallback. +pub(crate) fn serve_block_roots( + db: &ZebraDb, + range: std::ops::RangeInclusive, +) -> Vec { + let Some(upgrade) = db.vct_upgrade_height() else { + return produce_block_roots(db, range); + }; + + let (start, end) = (*range.start(), *range.end()); + + // Wholly at/above `U`: the index covers it. (`U == 0` for a node that fast-synced from + // genesis takes this path for every request, never touching the absent per-height trees.) + if start >= upgrade { + return db.commitment_roots_by_height_range(range); + } + + // Below `U`: derive the per-height-tree run up to `U - 1` (`start < upgrade` so `upgrade >= 1`). + let trees_end = block::Height(end.0.min(upgrade.0 - 1)); + let mut roots = produce_block_roots(db, start..=trees_end); + + // Continue into the index only if the tree run is contiguous up to `U - 1`; a short run means a + // gap below `U`, so serve it alone and let the client retry the remainder. + if roots.last().map(|root| root.height) == Some(trees_end) && end >= upgrade { + roots.extend(db.commitment_roots_by_height_range(upgrade..=end)); + } + + roots +} + +/// Produce the final frontiers at `height` from `db`'s per-height trees. +/// +/// Sprout is frozen far below any modern checkpoint, so the tip Sprout tree is the frontier at +/// `height`. +pub(super) fn produce_final_frontiers( + db: &ZebraDb, + height: block::Height, +) -> Result { + let sapling = db + .sapling_tree_by_height(&height) + .ok_or(FinalFrontiersGenerationError::MissingSaplingTree { height })?; + let orchard = db + .orchard_tree_by_height(&height) + .ok_or(FinalFrontiersGenerationError::MissingOrchardTree { height })?; + + Ok(FinalFrontiers { + height, + sapling, + orchard, + sprout: db.sprout_tree_for_tip(), + }) +} + +/// Produce serialized final-frontier bytes for the checkpoint handoff at `height`. +/// +/// These bytes use the same format as the embedded `mainnet-frontier.bin` file consumed by +/// [`super::vct`]. +pub fn produce_final_frontiers_bytes( + db: &ZebraDb, + height: block::Height, +) -> Result, FinalFrontiersGenerationError> { + Ok(produce_final_frontiers(db, height)?.to_bytes()) +} + +#[cfg(test)] +mod tests { + use super::*; + + /// The final-frontier serialization round-trips: parsed frontiers carry the same + /// height and tree roots as the originals. + #[test] + fn final_frontiers_bytes_round_trips() { + let frontiers = FinalFrontiers { + height: block::Height(1_687_200), + sapling: Arc::new(Default::default()), + orchard: Arc::new(Default::default()), + sprout: Arc::new(Default::default()), + }; + + let parsed = + FinalFrontiers::from_bytes(&frontiers.to_bytes()).expect("frontiers should parse"); + + assert_eq!(parsed.height, frontiers.height, "height round-trips"); + assert_eq!( + parsed.sapling.root(), + frontiers.sapling.root(), + "sapling frontier round-trips" + ); + assert_eq!( + parsed.orchard.root(), + frontiers.orchard.root(), + "orchard frontier round-trips" + ); + assert_eq!( + parsed.sprout.root(), + frontiers.sprout.root(), + "sprout frontier round-trips" + ); + } + + /// The test fixture source looks up produced roots by height and exposes + /// the handoff frontier — the consumer view of producer output. + #[test] + fn fixture_source_round_trips_payload() { + let roots = vec![ + BlockCommitmentRoots { + height: block::Height(10), + sapling_root: sapling::tree::NoteCommitmentTree::default().root(), + orchard_root: orchard::tree::NoteCommitmentTree::default().root(), + ironwood_root: zebra_chain::ironwood::tree::NoteCommitmentTree::default().root(), + sapling_tx: 0, + orchard_tx: 0, + ironwood_tx: 0, + auth_data_root: AuthDataRoot::from([0u8; 32]), + }, + BlockCommitmentRoots { + height: block::Height(11), + sapling_root: sapling::tree::NoteCommitmentTree::default().root(), + orchard_root: orchard::tree::NoteCommitmentTree::default().root(), + ironwood_root: zebra_chain::ironwood::tree::NoteCommitmentTree::default().root(), + sapling_tx: 0, + orchard_tx: 0, + ironwood_tx: 0, + auth_data_root: AuthDataRoot::from([0u8; 32]), + }, + ]; + let roots = roots + .into_iter() + .map(|root| (root.height.0, (root.sapling_root, root.orchard_root))) + .collect(); + let frontiers = FinalFrontiers { + height: block::Height(11), + sapling: Arc::new(Default::default()), + orchard: Arc::new(Default::default()), + sprout: Arc::new(Default::default()), + }; + + let source = FixtureSource::new(roots, Some(frontiers)); + + assert!( + source.vct_root(block::Height(10)).is_some(), + "produced root is looked up by height" + ); + assert!( + source.vct_root(block::Height(99)).is_none(), + "absent height has no root" + ); + assert_eq!( + source.vct_last_checkpoint_height(), + Some(block::Height(11)), + "handoff height comes from the supplied frontiers" + ); + } + + /// `invalidate` drops a peer-supplied root so a later read misses it, letting the + /// driver re-fetch a verifiable replacement from another peer. This un-poisons the + /// cache after a bad root is rejected by the committer, so one malicious peer cannot + /// wedge the same rejected root in place forever. + #[test] + fn peer_source_invalidate_evicts_a_root() { + let (source, writer) = PeerSource::new(None); + writer.insert_roots([BlockCommitmentRoots { + height: block::Height(42), + sapling_root: sapling::tree::NoteCommitmentTree::default().root(), + orchard_root: orchard::tree::NoteCommitmentTree::default().root(), + ironwood_root: zebra_chain::ironwood::tree::NoteCommitmentTree::default().root(), + sapling_tx: 0, + orchard_tx: 0, + ironwood_tx: 0, + auth_data_root: AuthDataRoot::from([0u8; 32]), + }]); + + assert!( + source.vct_root(block::Height(42)).is_some(), + "the inserted root is present before eviction" + ); + + source.invalidate(block::Height(42)); + + assert!( + source.vct_root(block::Height(42)).is_none(), + "an evicted root is gone, so the next read misses and a re-fetch can replace it" + ); + } +} diff --git a/zebra-state/src/service/finalized_state/commitment_aux_verify.rs b/zebra-state/src/service/finalized_state/commitment_aux_verify.rs new file mode 100644 index 00000000000..5f7be2f1fa2 --- /dev/null +++ b/zebra-state/src/service/finalized_state/commitment_aux_verify.rs @@ -0,0 +1,525 @@ +//! Read-only verification of supplied per-block note-commitment roots against the +//! checkpoint-committed block headers, via the ZIP-221 ChainHistory MMR. +//! +//! This is the "verify" half of the verified-commitment-trees design +//! (`docs/design/verified-commitment-trees.md` §6): given a sequence of per-block +//! Sapling/Orchard roots (from a fixture today, an untrusted peer later), confirm +//! they reconstruct a history tree consistent with the header commitments. The +//! commit path uses this module before persisting supplied roots. +//! +//! It reuses the existing consensus check +//! ([`block_commitment_is_valid_for_chain_history`](crate::service::check::block_commitment_is_valid_for_chain_history)) +//! and [`HistoryTree::push`], which build the V1/V2 leaf from the block body and the +//! supplied roots — so there is no new crypto here. + +use std::sync::Arc; + +use zebra_chain::{ + block::{merkle::AuthDataRoot, Block, Height}, + history_tree::HistoryTree, + orchard, + parameters::{Network, NetworkUpgrade}, + sapling, +}; + +use zebra_chain::block::{Commitment, CommitmentError}; + +use crate::{service::check, ValidateContextError}; + +/// One block-sized step in supplied commitment-root verification. +#[derive(Clone, Debug)] +pub(crate) struct CommitmentRootVerification { + pub(crate) block: Arc, + pub(crate) roots: Option<(sapling::tree::Root, orchard::tree::Root)>, + pub(crate) precomputed_auth_data_root: Option, + pub(crate) skip_parent_check: bool, +} + +impl CommitmentRootVerification { + pub(crate) fn with_roots( + block: Arc, + sapling_root: sapling::tree::Root, + orchard_root: orchard::tree::Root, + precomputed_auth_data_root: Option, + skip_parent_check: bool, + ) -> Self { + CommitmentRootVerification { + block, + roots: Some((sapling_root, orchard_root)), + precomputed_auth_data_root, + skip_parent_check, + } + } + + pub(crate) fn header_only( + block: Arc, + precomputed_auth_data_root: Option, + ) -> Self { + CommitmentRootVerification { + block, + roots: None, + precomputed_auth_data_root, + skip_parent_check: false, + } + } +} + +/// Verifies a supplied Sapling root for a *pre-Heartwood* block directly against the +/// block header (design §6.1). +/// +/// The ZIP-221 history MMR does not exist below Heartwood, so +/// [`block_commitment_is_valid_for_chain_history`](check::block_commitment_is_valid_for_chain_history) +/// is a no-op there and cannot authenticate the supplied roots. This fills that gap: +/// +/// - Sapling..Heartwood: the header's `FinalSaplingRoot` commits the Sapling root +/// directly, so the supplied root must equal it. +/// - Pre-Sapling: the Sapling tree is empty, so the supplied root must be the +/// empty-tree root. +/// +/// Heartwood and later (`ChainHistoryRoot` / `ChainHistoryBlockTxAuthCommitment` / +/// the activation-reserved block) are authenticated by the MMR path and accepted +/// here. The Orchard root below NU5 is pinned separately by +/// [`verify_supplied_orchard_root_below_nu5`]. +pub(crate) fn verify_supplied_sapling_root_below_heartwood( + network: &Network, + block: &Block, + sapling_root: &sapling::tree::Root, +) -> Result<(), ValidateContextError> { + let expected = match block.commitment(network)? { + Commitment::FinalSaplingRoot(header_root) => header_root, + Commitment::PreSaplingReserved(_) => sapling::tree::NoteCommitmentTree::default().root(), + // Heartwood activation and later are authenticated by the MMR path. + _ => return Ok(()), + }; + + if sapling_root != &expected { + return Err(ValidateContextError::InvalidBlockCommitment( + CommitmentError::InvalidFinalSaplingRoot { + expected: <[u8; 32]>::from(expected), + actual: <[u8; 32]>::from(*sapling_root), + }, + )); + } + + Ok(()) +} + +/// Verifies a supplied Orchard root for a *pre-NU5* block (design §6.1). +/// +/// The Orchard tree does not activate until NU5, and no header below NU5 commits to an +/// Orchard root: the ZIP-221 V1 history leaf (Heartwood..Canopy) *ignores* the Orchard +/// root entirely (`zcash_history.rs`, `V1::block_to_history_node`), and below Heartwood +/// there is no MMR at all. So the MMR path that authenticates Orchard roots from NU5 +/// onward cannot vouch for any root below NU5 — yet the fast path folds the supplied +/// Orchard root into the anchor set for every block. Without this check an untrusted +/// source could inject an arbitrary Orchard anchor below NU5 that the legacy recompute +/// path never produces, breaking the §11 trust boundary and consensus equivalence. +/// +/// Below NU5 the Orchard tree is always the empty default, so the supplied root must +/// equal the empty-tree root. At and above NU5 activation the MMR path authenticates +/// the root, so this accepts. +pub(crate) fn verify_supplied_orchard_root_below_nu5( + network: &Network, + height: Height, + orchard_root: &orchard::tree::Root, +) -> Result<(), ValidateContextError> { + // At/above NU5 the ZIP-221 V2 MMR commits to the Orchard root, so it is + // authenticated there, not here. + if let Some(nu5_height) = NetworkUpgrade::Nu5.activation_height(network) { + if height >= nu5_height { + return Ok(()); + } + } + + let expected = orchard::tree::NoteCommitmentTree::default().root(); + if orchard_root != &expected { + return Err(ValidateContextError::InvalidBlockCommitment( + CommitmentError::InvalidPreNu5OrchardRoot { + expected: <[u8; 32]>::from(expected), + actual: <[u8; 32]>::from(*orchard_root), + }, + )); + } + + Ok(()) +} + +/// Verifies that `items` (blocks in ascending height order, with supplied +/// Sapling/Orchard roots when they should be folded in) reconstruct a ZIP-221 +/// history MMR consistent with the block header commitments, starting from `tree` +/// (the parent block's history tree). +/// +/// Returns the final history tree on success, or `(height, error)` for the first +/// block whose header commitment rejects the roots folded in so far. +/// +/// # Lag +/// +/// A block's commitment commits to the history tree as of its *parent*, so the root +/// supplied for height `H` is only confirmed when height `H + 1` is processed. Over a +/// contiguous range `[start..=end]` this therefore confirms the roots at +/// `[start..=end - 1]`; pass the block at `end + 1` to confirm the root at `end`. +pub(crate) fn verify_commitment_roots( + network: &Network, + mut tree: HistoryTree, + items: I, +) -> Result +where + I: IntoIterator, +{ + for item in items { + let CommitmentRootVerification { + block, + roots, + precomputed_auth_data_root, + skip_parent_check, + } = item; + + let height = block + .coinbase_height() + .expect("checkpoint-verified blocks have a coinbase height"); + + // Validate this block's header commitment against the current (parent) tree, + // i.e. against every root already folded in. + if !skip_parent_check { + check::block_commitment_is_valid_for_chain_history( + block.clone(), + network, + &tree, + precomputed_auth_data_root, + ) + .map_err(|error| (height, error))?; + } + + let Some((sapling_root, orchard_root)) = roots else { + continue; + }; + + verify_supplied_sapling_root_below_heartwood(network, &block, &sapling_root) + .map_err(|error| (height, error))?; + verify_supplied_orchard_root_below_nu5(network, height, &orchard_root) + .map_err(|error| (height, error))?; + + // Fold this block's supplied roots into the running MMR (builds the leaf + // from the block body tx-counts + the roots). + tree.push( + network, + block, + &sapling_root, + &orchard_root, + &Default::default(), + ) + .map_err(Arc::new) + .map_err(ValidateContextError::from) + .map_err(|error| (height, error))?; + } + + Ok(tree) +} + +#[cfg(test)] +mod tests { + use super::*; + + use zebra_chain::{ + block::Block, + parameters::{ + testnet::{ConfiguredActivationHeights, RegtestParameters}, + Network::Mainnet, + NetworkUpgrade, + }, + serialization::ZcashDeserializeInto, + }; + + /// Build an empty [`HistoryTree`] (the genesis block is pre-Heartwood). + fn empty_history_tree() -> HistoryTree { + let genesis = Arc::new( + zebra_test::vectors::BLOCK_MAINNET_GENESIS_BYTES + .zcash_deserialize_into::() + .expect("genesis deserializes"), + ); + HistoryTree::from_block( + &Mainnet, + genesis, + &Default::default(), + &Default::default(), + &Default::default(), + ) + .expect("empty history tree for a pre-Heartwood block") + } + + /// A distinct, valid Orchard root that is *not* the empty-tree root, for the + /// negative cases. Zero is a valid Pallas base field element, and the empty + /// Orchard tree root is an uncommitted-leaf hash, so the two differ. + fn non_empty_orchard_root() -> orchard::tree::Root { + let empty = orchard::tree::NoteCommitmentTree::default().root(); + let wrong = orchard::tree::Root::try_from([0u8; 32]) + .expect("zero is a valid pallas base field element"); + assert_ne!( + wrong, empty, + "the negative cases need a root distinct from the empty-tree root" + ); + wrong + } + + fn verification_item( + block: Arc, + sapling_root: sapling::tree::Root, + orchard_root: orchard::tree::Root, + ) -> CommitmentRootVerification { + CommitmentRootVerification::with_roots(block, sapling_root, orchard_root, None, false) + } + + /// Below NU5 the supplied Orchard root must equal the empty-tree root (no header + /// commits to it there), and any other root is rejected. At/above NU5 the MMR + /// authenticates it, so this check accepts unconditionally. + #[test] + fn pins_orchard_root_to_empty_below_nu5_and_defers_above() { + let nu5 = NetworkUpgrade::Nu5 + .activation_height(&Mainnet) + .expect("mainnet has NU5"); + let empty = orchard::tree::NoteCommitmentTree::default().root(); + let wrong = non_empty_orchard_root(); + + // Below NU5: the empty root is accepted, a non-empty root is rejected. + let pre_nu5 = Height(nu5.0 - 1); + verify_supplied_orchard_root_below_nu5(&Mainnet, pre_nu5, &empty) + .expect("the empty-tree root is accepted below NU5"); + let error = verify_supplied_orchard_root_below_nu5(&Mainnet, pre_nu5, &wrong) + .expect_err("a non-empty orchard root must be rejected below NU5"); + assert!( + matches!( + error, + ValidateContextError::InvalidBlockCommitment( + CommitmentError::InvalidPreNu5OrchardRoot { .. } + ) + ), + "rejection uses the dedicated pre-NU5 orchard error, got: {error:?}" + ); + + // Pre-Sapling/Heartwood (well below NU5) is also pinned to empty. + verify_supplied_orchard_root_below_nu5(&Mainnet, Height(1), &empty) + .expect("the empty-tree root is accepted at low heights"); + verify_supplied_orchard_root_below_nu5(&Mainnet, Height(1), &wrong) + .expect_err("a non-empty orchard root must be rejected at low heights"); + + // At and above NU5 the MMR path authenticates the root, so even a non-empty + // root is accepted here (it is checked elsewhere). + verify_supplied_orchard_root_below_nu5(&Mainnet, nu5, &wrong) + .expect("at NU5 the root is authenticated by the MMR, not pinned here"); + verify_supplied_orchard_root_below_nu5(&Mainnet, Height(nu5.0 + 1), &wrong) + .expect("above NU5 the root is authenticated by the MMR, not pinned here"); + } + + #[test] + fn pins_orchard_root_to_empty_when_nu5_is_unconfigured() { + let network = zebra_chain::parameters::Network::new_regtest(RegtestParameters { + activation_heights: ConfiguredActivationHeights { + nu5: None, + ..Default::default() + }, + ..Default::default() + }); + let empty = orchard::tree::NoteCommitmentTree::default().root(); + let wrong = non_empty_orchard_root(); + + verify_supplied_orchard_root_below_nu5(&network, Height(1), &empty) + .expect("the empty-tree root is accepted when NU5 is unconfigured"); + let error = verify_supplied_orchard_root_below_nu5(&network, Height(1), &wrong) + .expect_err("a non-empty orchard root must be rejected when NU5 is unconfigured"); + assert!( + matches!( + error, + ValidateContextError::InvalidBlockCommitment( + CommitmentError::InvalidPreNu5OrchardRoot { .. } + ) + ), + "rejection uses the dedicated pre-NU5 orchard error, got: {error:?}" + ); + } + + /// The verifier confirms real Sapling roots over the Heartwood activation and its + /// next block (the V1 `ChainHistoryRoot` path), and rejects a wrong root at the + /// *next* block (the one-block lag). + #[test] + fn verifies_real_roots_and_rejects_a_wrong_root_at_next_height() { + let (blocks, sapling_roots) = Mainnet.block_sapling_roots_map(); + let activation = NetworkUpgrade::Heartwood + .activation_height(&Mainnet) + .expect("mainnet has Heartwood") + .0; + + let block_at = |height: u32| -> Arc { + Arc::new( + blocks + .get(&height) + .expect("test vector block exists") + .zcash_deserialize_into::() + .expect("block deserializes"), + ) + }; + let root_at = |height: u32| -> sapling::tree::Root { + sapling::tree::Root::try_from(**sapling_roots.get(&height).expect("root vector exists")) + .expect("valid root") + }; + + let act_block = block_at(activation); + let next_block = block_at(activation + 1); + let act_root = root_at(activation); + let next_root = root_at(activation + 1); + let empty_orchard_root = orchard::tree::NoteCommitmentTree::default().root(); + + // Positive: the real roots reconstruct a tree the next block's header commits to. + let ok_items = vec![ + verification_item(act_block.clone(), act_root, empty_orchard_root), + verification_item(next_block.clone(), next_root, empty_orchard_root), + ]; + verify_commitment_roots(&Mainnet, empty_history_tree(), ok_items) + .expect("real roots verify against the headers"); + + // Negative + lag: a wrong root at the activation height (here, the next + // block's root, which is a valid but different root) is only caught when the + // following block's commitment is checked. + assert_ne!(act_root, next_root, "test needs two distinct roots"); + let bad_items = vec![ + verification_item(act_block, next_root, empty_orchard_root), + verification_item(next_block, next_root, empty_orchard_root), + ]; + let (fail_height, _error) = + verify_commitment_roots(&Mainnet, empty_history_tree(), bad_items) + .expect_err("a wrong root must be rejected"); + assert_eq!( + fail_height.0, + activation + 1, + "a wrong root at H is detected at H+1 (the lag)" + ); + } + + /// Real NU5/V2-range verification over the POC range (1,707,211..=1,717,210), + /// exercising the actual [`verify_commitment_roots`] on production data. + /// + /// Gated by env vars so it stays out of normal CI. Requires two read-only forks + /// of the RUNBOOK 1.707M master snapshot: + /// - `VCT_SEED_DB`: an *unsynced* `cp -al` fork (its tip history tree at height + /// 1,707,210 is the seed — mid-NU5-epoch, so no activation boundary to handle). + /// - `VCT_ARCHIVE_DB`: an archive fork synced to >= 1,717,211 (provides the blocks + /// and per-height roots). + /// + /// Run: + /// ```text + /// VCT_SEED_DB= VCT_ARCHIVE_DB= \ + /// cargo test -p zebra-state --lib commitment_aux_verify -- --ignored --nocapture + /// ``` + #[ignore] + #[test] + #[allow(clippy::print_stderr)] // intentional progress output for a manual run + fn verifies_real_nu5_range_over_synced_forks() { + use std::path::PathBuf; + + use crate::{ + constants::{state_database_format_version_in_code, STATE_DATABASE_KIND}, + service::finalized_state::{ZebraDb, STATE_COLUMN_FAMILIES_IN_CODE}, + Config, + }; + + let (Some(seed_dir), Some(archive_dir)) = ( + std::env::var_os("VCT_SEED_DB"), + std::env::var_os("VCT_ARCHIVE_DB"), + ) else { + eprintln!("skipping: set VCT_SEED_DB (unsynced fork) and VCT_ARCHIVE_DB (synced fork)"); + return; + }; + + let open = |dir: PathBuf| -> ZebraDb { + let config = Config { + cache_dir: dir, + ephemeral: false, + ..Default::default() + }; + ZebraDb::new( + &config, + STATE_DATABASE_KIND, + &state_database_format_version_in_code(), + &Mainnet, + true, // skip format upgrades + STATE_COLUMN_FAMILIES_IN_CODE + .iter() + .map(ToString::to_string), + true, // read-only + ) + }; + + let seed_db = open(PathBuf::from(seed_dir)); + let archive_db = open(PathBuf::from(archive_dir)); + + let start = 1_707_211u32; + let end = 1_717_210u32; + + // Seed: the history tree at 1,707,210 (the unsynced fork's tip). + let seed = (*seed_db.history_tree()).clone(); + assert_eq!( + seed_db.finalized_tip_height().map(|h| h.0), + Some(start - 1), + "VCT_SEED_DB must be the unsynced 1,707,210 master fork" + ); + assert!( + archive_db.finalized_tip_height().map(|h| h.0).unwrap_or(0) > end, + "VCT_ARCHIVE_DB must be synced to at least {}", + end + 1 + ); + + // Build (block, sapling_root, orchard_root) for [start..=end+1]; the +1 block + // confirms the in-range root at `end` via the one-block lag. + let item_at = |h: u32| -> CommitmentRootVerification { + let block = archive_db + .block(Height(h).into()) + .expect("archive fork has the block"); + let sapling_root = archive_db + .sapling_tree_by_height(&Height(h)) + .expect("archive fork has the per-height Sapling tree") + .root(); + let orchard_root = archive_db + .orchard_tree_by_height(&Height(h)) + .expect("archive fork has the per-height Orchard tree") + .root(); + verification_item(block, sapling_root, orchard_root) + }; + let items: Vec<_> = (start..=end + 1).map(item_at).collect(); + + // Positive: every supplied root in the range is confirmed by the V2 headers. + verify_commitment_roots(&Mainnet, seed.clone(), items.clone()) + .expect("real NU5 roots verify against the headers"); + eprintln!("VCT NU5 positive: {} blocks verified", items.len()); + + // Negative + lag: corrupt one root mid-range with a distinct valid root (the + // range's first root, certainly different after thousands of sandblast blocks); + // expect rejection at H+1. + let bad_offset = 5_000usize; + let bad_height = start + bad_offset as u32; + let wrong_root = items[0].roots.expect("test verification item has roots").0; + let mut bad_items = items; + assert_ne!( + bad_items[bad_offset] + .roots + .expect("test verification item has roots") + .0, + wrong_root, + "need a distinct wrong root" + ); + bad_items[bad_offset] + .roots + .as_mut() + .expect("test verification item has roots") + .0 = wrong_root; + let (fail_height, _error) = verify_commitment_roots(&Mainnet, seed, bad_items) + .expect_err("a wrong NU5 root must be rejected"); + assert_eq!( + fail_height.0, + bad_height + 1, + "a wrong root at H is detected at H+1 (the lag)" + ); + eprintln!( + "VCT NU5 negative: wrong root at {bad_height} rejected at {}", + fail_height.0 + ); + } +} diff --git a/zebra-state/src/service/finalized_state/disk_format/chain.rs b/zebra-state/src/service/finalized_state/disk_format/chain.rs index ae86b025eb5..e1e490218d2 100644 --- a/zebra-state/src/service/finalized_state/disk_format/chain.rs +++ b/zebra-state/src/service/finalized_state/disk_format/chain.rs @@ -52,6 +52,25 @@ pub struct HistoryTreeParts { } impl HistoryTreeParts { + /// Deserializes history tree parts from raw database bytes. + pub(crate) fn from_bytes_result(bytes: impl AsRef<[u8]>) -> Result { + let bytes = bytes.as_ref(); + let options = bincode::DefaultOptions::new(); + + // Try the current entry width first. Databases written before NU6.3 widened + // `zcash_history::Entry` store narrower entries that fail to parse at the current width, + // so fall back to the legacy width and zero-pad each entry up to the current width. + // + // Legacy-width rows can fail the current-width decoder with errors other than + // `UnexpectedEof`, because the wider entry can read into the next legacy entry and + // interpret arbitrary entry bytes as bincode control bytes. + options.deserialize::(bytes).or_else(|_| { + options + .deserialize::(bytes) + .map(HistoryTreeParts::from) + }) + } + /// Converts [`HistoryTreeParts`] to a [`NonEmptyHistoryTree`]. pub(crate) fn with_network( self, @@ -130,23 +149,7 @@ impl From for HistoryTreeParts { impl FromDisk for HistoryTreeParts { fn from_bytes(bytes: impl AsRef<[u8]>) -> Self { - let bytes = bytes.as_ref(); - let options = bincode::DefaultOptions::new(); - - // Try the current entry width first. Databases written before NU6.3 widened - // `zcash_history::Entry` store narrower entries that fail to parse at the current width, - // so fall back to the legacy width and zero-pad each entry up to the current width. - // - // Legacy-width rows can fail the current-width decoder with errors other than - // `UnexpectedEof`, because the wider entry can read into the next legacy entry and - // interpret arbitrary entry bytes as bincode control bytes. - options - .deserialize::(bytes) - .or_else(|_| { - options - .deserialize::(bytes) - .map(HistoryTreeParts::from) - }) + Self::from_bytes_result(bytes) .expect("deserialization format should match the serialization format used by IntoDisk") } } diff --git a/zebra-state/src/service/finalized_state/disk_format/tests/snapshot.rs b/zebra-state/src/service/finalized_state/disk_format/tests/snapshot.rs index fe1c390a76f..6b3436fc2ef 100644 --- a/zebra-state/src/service/finalized_state/disk_format/tests/snapshot.rs +++ b/zebra-state/src/service/finalized_state/disk_format/tests/snapshot.rs @@ -96,7 +96,7 @@ fn test_raw_rocksdb_column_families_with_network(network: Network) { .expect("test data deserializes"); state - .commit_finalized_direct(block.into(), None, "snapshot tests") + .commit_finalized_direct(block.into(), None, None, "snapshot tests") .expect("test block is valid"); let mut settings = insta::Settings::clone_current(); diff --git a/zebra-state/src/service/finalized_state/tests/prop.rs b/zebra-state/src/service/finalized_state/tests/prop.rs index 74ebd0f5aaa..2f6538cab10 100644 --- a/zebra-state/src/service/finalized_state/tests/prop.rs +++ b/zebra-state/src/service/finalized_state/tests/prop.rs @@ -1,6 +1,9 @@ //! Randomised property tests for the finalized state. -use std::{env, error::Error}; +use std::{collections::HashMap, env, error::Error, fs, sync::Arc}; + +use tempfile::TempDir; +use tokio::sync::oneshot; use zebra_chain::{ block::Height, @@ -13,16 +16,137 @@ use zebra_chain::{ use zebra_test::prelude::*; use crate::{ - config::Config, - service::{ - arbitrary::PreparedChain, - finalized_state::{CheckpointVerifiedBlock, FinalizedState}, - }, - tests::FakeChainHelper, + config::Config, service::arbitrary::PreparedChain, tests::FakeChainHelper, HashOrHeight, +}; + +use super::super::{ + commitment_aux, serve_block_roots, vct::validate_final_frontiers_bytes, + CheckpointVerifiedBlock, DiskWriteBatch, FinalizedState, }; const DEFAULT_PARTIAL_CHAIN_PROPTEST_CASES: u32 = 1; +type TestRootMap = HashMap< + u32, + ( + zebra_chain::sapling::tree::Root, + zebra_chain::orchard::tree::Root, + ), +>; +type SaplingTree = Arc; +type OrchardTree = Arc; +type SproutTree = Arc; + +fn enable_vct_test_fixture_source(state: &mut FinalizedState, roots: TestRootMap) { + state.enable_vct_fast_source( + Box::new(commitment_aux::FixtureSource::new(roots, None)), + false, + ); +} + +fn enable_vct_test_fixture_source_with_handoff( + state: &mut FinalizedState, + roots: TestRootMap, + handoff_height: Height, + sapling: SaplingTree, + orchard: OrchardTree, + sprout: SproutTree, +) { + state.enable_vct_fast_source( + Box::new(commitment_aux::FixtureSource::new( + roots, + Some(commitment_aux::FinalFrontiers { + height: handoff_height, + sapling, + orchard, + sprout, + }), + )), + false, + ); +} + +#[test] +fn vct_generated_final_frontier_bytes_are_node_loader_compatible() -> Result<()> { + let _init_guard = zebra_test::init(); + + let network = ParametersBuilder::default() + .with_activation_heights(ConfiguredActivationHeights { + before_overwinter: Some(1), + overwinter: Some(10), + sapling: Some(15), + blossom: Some(20), + heartwood: Some(25), + canopy: Some(30), + nu5: Some(35), + nu6: Some(40), + nu6_1: Some(45), + nu6_2: Some(47), + nu6_3: Some(48), + nu7: Some(50), + }) + .expect("failed to set activation heights") + .extend_funding_streams() + .to_network() + .expect("failed to build configured network"); + let ledger_strategy = + LedgerState::genesis_strategy(Some(network), None::, None, false); + + proptest!(ProptestConfig::with_cases(1), + |((chain, _count, network, _history_tree) in PreparedChain::default().with_ledger_strategy(ledger_strategy.clone()).with_valid_commitments().no_shrink())| { + let blocks: Vec<_> = chain.iter().collect(); + let nu5 = NetworkUpgrade::Nu5.activation_height(&network).unwrap().0; + let last = (nu5 + 3) as usize; + prop_assert!(blocks.len() > last, "generated chain unexpectedly short"); + let height = Height(last as u32); + + let mut legacy = FinalizedState::new(&Config::ephemeral(), &network, #[cfg(feature = "elasticsearch")] false); + for block in blocks.iter().take(last + 1) { + let cv = CheckpointVerifiedBlock::from(block.block.clone()); + legacy + .commit_finalized_direct(cv.into(), None, None, "vct frontier bytes legacy") + .unwrap(); + } + + let bytes = commitment_aux::produce_final_frontiers_bytes(&legacy.db, height) + .expect("legacy DB has final frontiers at the requested height"); + let temp_dir = TempDir::new().expect("temp dir"); + let path = temp_dir.path().join("frontier.bin"); + fs::write(&path, &bytes).expect("frontier bytes write to temp file"); + + let bytes_from_file = fs::read(&path).expect("frontier bytes read from temp file"); + validate_final_frontiers_bytes(&bytes_from_file, height) + .expect("generated frontier bytes pass node loader validation"); + + let parsed = commitment_aux::FinalFrontiers::from_bytes(&bytes_from_file) + .expect("validated bytes parse as final frontiers"); + prop_assert_eq!(parsed.height, height, "frontier height round-trips"); + prop_assert_eq!( + parsed.sapling.root(), + legacy.db.sapling_tree_by_height(&height).unwrap().root(), + "parsed Sapling frontier matches the DB tree at the requested height" + ); + prop_assert_eq!( + parsed.orchard.root(), + legacy.db.orchard_tree_by_height(&height).unwrap().root(), + "parsed Orchard frontier matches the DB tree at the requested height" + ); + prop_assert_eq!( + parsed.sprout.root(), + legacy.db.sprout_tree_for_tip().root(), + "parsed Sprout frontier matches the DB tip tree" + ); + + let wrong_height = Height(height.0.checked_add(1).expect("test height is in range")); + prop_assert!( + validate_final_frontiers_bytes(&bytes_from_file, wrong_height).is_err(), + "node loader validation rejects a frontier whose height does not match the checkpoint" + ); + }); + + Ok(()) +} + #[test] fn blocks_with_v5_transactions() -> Result<()> { let _init_guard = zebra_test::init(); @@ -39,6 +163,7 @@ fn blocks_with_v5_transactions() -> Result<()> { let (hash, _) = state.commit_finalized_direct( checkpoint_verified.into(), None, + None, "blocks_with_v5_transactions test" ).unwrap(); prop_assert_eq!(Some(height), state.finalized_tip_height()); @@ -115,6 +240,7 @@ fn all_upgrades_and_wrong_commitments_with_fake_activation_heights() -> Result<( state.commit_finalized_direct( checkpoint_verified.into(), None, + None, "all_upgrades test" ).expect_err("Must fail commitment check"); failure_count += 1; @@ -124,10 +250,11 @@ fn all_upgrades_and_wrong_commitments_with_fake_activation_heights() -> Result<( if current_height == nu5_height_plus1 { let mut checkpoint_verified = CheckpointVerifiedBlock::from(block.block.clone()); - checkpoint_verified.auth_data_root = Some([0x42; 32].into()); + checkpoint_verified.0.auth_data_root = Some([0x42; 32].into()); let err = state.commit_finalized_direct( checkpoint_verified.into(), None, + None, "all_upgrades bad auth root test" ).expect_err("Must fail when the supplied auth data root is incorrect"); let commit_error = err @@ -151,6 +278,7 @@ fn all_upgrades_and_wrong_commitments_with_fake_activation_heights() -> Result<( let (hash, _) = state.commit_finalized_direct( checkpoint_verified.into(), None, + None, "all_upgrades test" ).unwrap(); prop_assert_eq!(Some(height), state.finalized_tip_height()); @@ -164,3 +292,1630 @@ fn all_upgrades_and_wrong_commitments_with_fake_activation_heights() -> Result<( Ok(()) } + +/// Verified-commitment-trees fast path (`commit_finalized_direct` Checkpoint arm): +/// committing with correct fixture roots produces the same consensus state (anchor +/// sets + history root) as the legacy recompute path across all upgrade boundaries, +/// and a wrong fixture root is rejected (verify-before-commit) rather than persisted. +/// Exercises: a below-Heartwood seed, history-tree creation at Heartwood, the NU5 +/// V1->V2 transition, verify-ahead against the buffered successor, trusted fixture tip +/// commits without a successor, and rejection of a corrupted root. +#[test] +#[allow(clippy::needless_range_loop)] // the loops index blocks[i+1] and the fixture by height +fn vct_fast_path_matches_legacy_and_rejects_wrong_roots() -> Result<()> { + let _init_guard = zebra_test::init(); + + let network = ParametersBuilder::default() + .with_activation_heights(ConfiguredActivationHeights { + before_overwinter: Some(1), + overwinter: Some(10), + sapling: Some(15), + blossom: Some(20), + heartwood: Some(25), + canopy: Some(30), + nu5: Some(35), + nu6: Some(40), + nu6_1: Some(45), + nu6_2: Some(47), + nu6_3: Some(48), + nu7: Some(50), + }) + .expect("failed to set activation heights") + .extend_funding_streams() + .to_network() + .expect("failed to build configured network"); + let ledger_strategy = + LedgerState::genesis_strategy(Some(network), None::, None, false); + + proptest!(ProptestConfig::with_cases(env::var("PROPTEST_CASES") + .ok() + .and_then(|v| v.parse().ok()) + .unwrap_or(DEFAULT_PARTIAL_CHAIN_PROPTEST_CASES)), + |((chain, _count, network, _history_tree) in PreparedChain::default().with_ledger_strategy(ledger_strategy.clone()).with_valid_commitments().no_shrink())| { + + let blocks: Vec<_> = chain.iter().collect(); + let nu5 = NetworkUpgrade::Nu5.activation_height(&network).unwrap().0; + let heartwood = NetworkUpgrade::Heartwood.activation_height(&network).unwrap().0; + + // Process a bounded prefix [0, last] spanning the Heartwood (history-tree + // creation) and NU5 (V1->V2) boundaries plus a couple of V2 blocks; `last` is + // the tip we compare at. Chains are far longer than this + // (MAX_PARTIAL_CHAIN_BLOCKS), so this is a plain assertion, not a discard. + let last = (nu5 + 3) as usize; + prop_assert!(blocks.len() > last + 1, "generated chain unexpectedly short"); + + // The fast path runs below the checkpoint, seeded from an already-committed + // tip. Seed just before Heartwood so the fast range creates the history tree + // (Heartwood) and crosses NU5 (V1->V2). + let seed = (heartwood - 1) as usize; + + // Legacy pass over [0, last]: record per-block roots for the fast range as + // the fixture, and the golden consensus state at the tip. + let mut legacy = FinalizedState::new(&Config::ephemeral(), &network, #[cfg(feature = "elasticsearch")] false); + let mut fixture = std::collections::HashMap::new(); + for i in 0..=last { + let cv = CheckpointVerifiedBlock::from(blocks[i].block.clone()); + let (_h, trees) = legacy + .commit_finalized_direct(cv.into(), None, None, "vct legacy") + .unwrap(); + if i > seed { + fixture.insert(i as u32, (trees.sapling.root(), trees.orchard.root())); + } + } + let golden_anchors = legacy.db.vct_anchor_digest(); + let golden_history = legacy.db.history_tree().hash(); + + // Fast pass over [0, last] with the correct fixture: genesis..=seed recompute + // (no fixture entry); seed+1..=last verify-ahead against their buffered + // successor. Every fast-eligible block takes the fast path, and the result + // equals legacy. + let mut fast = FinalizedState::new(&Config::ephemeral(), &network, #[cfg(feature = "elasticsearch")] false); + enable_vct_test_fixture_source(&mut fast, fixture.clone()); + for i in 0..=last { + let cv = CheckpointVerifiedBlock::from(blocks[i].block.clone()); + let next = Some((blocks[i + 1].block.clone(), None)); + fast.commit_finalized_direct(cv.into(), None, next, "vct fast") + .expect("verified fast commit succeeds"); + } + prop_assert_eq!(fast.db.vct_anchor_digest(), golden_anchors, "fast anchors must match legacy"); + prop_assert_eq!(fast.db.history_tree().hash(), golden_history, "fast history must match legacy"); + prop_assert_eq!(fast.vct_fast_count(), (last - seed) as u64, "every fast-eligible block took the fast path"); + // The dedup: each header commitment is checked once, not twice. Only the + // first fast block runs its own commitment check; every later fast block + // was already validated by its predecessor's look-ahead, so it is skipped. + prop_assert_eq!(fast.vct_prevalidated_count(), (last - seed - 1) as u64, "every fast block after the first skips its redundant own commitment check"); + + // A trusted local fixture may commit its tip root without a successor: it is + // not adversarial and the root is checked in arrears when a successor arrives. + let mut no_successor = FinalizedState::new(&Config::ephemeral(), &network, #[cfg(feature = "elasticsearch")] false); + enable_vct_test_fixture_source(&mut no_successor, fixture.clone()); + for i in 0..last { + let cv = CheckpointVerifiedBlock::from(blocks[i].block.clone()); + let next = Some((blocks[i + 1].block.clone(), None)); + no_successor + .commit_finalized_direct(cv.into(), None, next, "vct no-successor seed") + .expect("verified fast commit succeeds with successor"); + } + prop_assert!(!no_successor.vct_fast_needs_successor(Height(last as u32)), "a trusted fixture tip can commit without a successor"); + let cv = CheckpointVerifiedBlock::from(blocks[last].block.clone()); + no_successor + .commit_finalized_direct(cv.into(), None, None, "vct trusted fixture no successor") + .expect("trusted fixture tip commits without a successor"); + prop_assert_eq!( + no_successor.db.finalized_tip_height(), + Some(Height(last as u32)), + "the trusted fixture tip committed" + ); + + // Negative: corrupt the fixture Sapling root at a V2 (post-NU5) height with a + // distinct value (the empty root; a V2 block has a non-empty Sapling tree). + // Fast mode cannot recompute a bad root away (the frontier is frozen), so the + // wrong root must be *rejected* by the next block's commitment (verify-before- + // commit) — the commit at that height fails rather than persisting it. + let bad_height = (nu5 + 1) as usize; + let mut bad_fixture = fixture.clone(); + let bad_entry = bad_fixture.get_mut(&(bad_height as u32)).unwrap(); + prop_assert_ne!(bad_entry.0, Default::default(), "a V2 block must have a non-empty Sapling root"); + bad_entry.0 = Default::default(); + + let mut bad = FinalizedState::new(&Config::ephemeral(), &network, #[cfg(feature = "elasticsearch")] false); + enable_vct_test_fixture_source(&mut bad, bad_fixture); + let mut error_height = None; + for i in 0..=last { + let cv = CheckpointVerifiedBlock::from(blocks[i].block.clone()); + let next = Some((blocks[i + 1].block.clone(), None)); + if bad.commit_finalized_direct(cv.into(), None, next, "vct bad").is_err() { + error_height = Some(i); + break; + } + } + prop_assert_eq!(error_height, Some(bad_height), "a wrong fixture root is rejected at its own commit"); + + // Negative (Orchard, below NU5): no header commits to an Orchard root below + // NU5 (V1 history leaves ignore it; no MMR below Heartwood), so the fast path + // pins it to the empty-tree root. Corrupt a below-NU5 fixture Orchard root to + // a non-empty value. Unlike the Sapling MMR path (one-block lag), this is a + // direct check, so it is rejected at the block's *own* commit — closing the + // hole where an untrusted source injects a spurious Orchard anchor. + let bad_orchard_height = (nu5 - 1) as usize; + prop_assert!(bad_orchard_height > seed, "the corrupted height must be in the fast range"); + let empty_orchard = zebra_chain::orchard::tree::NoteCommitmentTree::default().root(); + let wrong_orchard = zebra_chain::orchard::tree::Root::try_from([0u8; 32]) + .expect("zero is a valid pallas base field element"); + prop_assert_ne!(wrong_orchard, empty_orchard, "the wrong root must differ from the empty-tree root"); + + let mut bad_orchard_fixture = fixture.clone(); + let bad_orchard_entry = bad_orchard_fixture.get_mut(&(bad_orchard_height as u32)).unwrap(); + prop_assert_eq!(bad_orchard_entry.1, empty_orchard, "a below-NU5 block has the empty Orchard root"); + bad_orchard_entry.1 = wrong_orchard; + + let mut bad_orchard = FinalizedState::new(&Config::ephemeral(), &network, #[cfg(feature = "elasticsearch")] false); + enable_vct_test_fixture_source(&mut bad_orchard, bad_orchard_fixture); + let mut orchard_error_height = None; + for i in 0..=last { + let cv = CheckpointVerifiedBlock::from(blocks[i].block.clone()); + let next = Some((blocks[i + 1].block.clone(), None)); + if bad_orchard.commit_finalized_direct(cv.into(), None, next, "vct bad orchard").is_err() { + orchard_error_height = Some(i); + break; + } + } + prop_assert_eq!(orchard_error_height, Some(bad_orchard_height), "a wrong below-NU5 orchard root is rejected at its own commit"); + }); + + Ok(()) +} + +/// A verified-commitment-trees fast sync must never legacy-recompute a height whose +/// supplied root is missing once the note-commitment frontier is frozen: the running +/// frontier is no longer the real one, so recomputing would fold a wrong root into the +/// history MMR and silently corrupt consensus state (a peer that omits a height — see the +/// driver's gap handling — could trigger this). Instead the committer must refuse with the +/// retryable `VctSuppliedRootUnavailable` error and leave the database untouched, so the +/// block can be committed later from a fetched root. This guards the liveness/no-corruption +/// half of the peer-source fast path (the bad-root rejection half is covered by +/// `vct_fast_path_matches_legacy_and_rejects_wrong_roots`). +#[test] +#[allow(clippy::needless_range_loop)] // the loop indexes blocks[i+1] and the fixture by height +fn vct_frozen_frontier_hole_refuses_instead_of_recomputing() -> Result<()> { + let _init_guard = zebra_test::init(); + + let network = ParametersBuilder::default() + .with_activation_heights(ConfiguredActivationHeights { + before_overwinter: Some(1), + overwinter: Some(10), + sapling: Some(15), + blossom: Some(20), + heartwood: Some(25), + canopy: Some(30), + nu5: Some(35), + nu6: Some(40), + nu6_1: Some(45), + nu6_2: Some(47), + nu6_3: Some(48), + nu7: Some(50), + }) + .expect("failed to set activation heights") + .extend_funding_streams() + .to_network() + .expect("failed to build configured network"); + let ledger_strategy = + LedgerState::genesis_strategy(Some(network), None::, None, false); + + proptest!(ProptestConfig::with_cases(1), + |((chain, _count, network, _history_tree) in PreparedChain::default().with_ledger_strategy(ledger_strategy.clone()).with_valid_commitments().no_shrink())| { + + let blocks: Vec<_> = chain.iter().collect(); + let nu5 = NetworkUpgrade::Nu5.activation_height(&network).unwrap().0; + let heartwood = NetworkUpgrade::Heartwood.activation_height(&network).unwrap().0; + let last = (nu5 + 3) as usize; + prop_assert!(blocks.len() > last + 1, "generated chain unexpectedly short"); + let seed = (heartwood - 1) as usize; + + // Record the per-block roots for the fast range as the fixture. + let mut legacy = FinalizedState::new(&Config::ephemeral(), &network, #[cfg(feature = "elasticsearch")] false); + let mut fixture = std::collections::HashMap::new(); + for i in 0..=last { + let cv = CheckpointVerifiedBlock::from(blocks[i].block.clone()); + let (_h, trees) = legacy + .commit_finalized_direct(cv.into(), None, None, "vct hole legacy") + .unwrap(); + if i > seed { + fixture.insert(i as u32, (trees.sapling.root(), trees.orchard.root())); + } + } + + // Punch a hole: drop a post-NU5 height's root from the fixture, simulating a + // peer that omitted it (or a root evicted after failing verification). Earlier + // fast blocks freeze the frontier, so this height has no real frontier to + // recompute against. + let hole = (nu5 + 1) as usize; + prop_assert!(hole > seed && hole < last, "the hole must be inside the fast range"); + fixture.remove(&(hole as u32)); + + let mut fast = FinalizedState::new(&Config::ephemeral(), &network, #[cfg(feature = "elasticsearch")] false); + enable_vct_test_fixture_source(&mut fast, fixture); + + let mut error_height = None; + for i in 0..=last { + let cv = CheckpointVerifiedBlock::from(blocks[i].block.clone()); + let next = (i < last).then(|| (blocks[i + 1].block.clone(), None)); + match fast.commit_finalized_direct(cv.into(), None, next, "vct hole fast") { + Ok(_) => {} + Err(error) => { + // The refusal is the typed, retryable error — not a generic + // invalid-block error and not silent corruption. + prop_assert!( + format!("{error:?}").contains("VctSuppliedRootUnavailable"), + "a frozen-frontier hole returns the retryable VctSuppliedRootUnavailable error, got: {error:?}" + ); + error_height = Some(i); + break; + } + } + } + + prop_assert_eq!(error_height, Some(hole), "the commit refuses at the hole height, not before or after"); + // Nothing at or past the hole was persisted: the tip is the last block before + // the hole, so no corrupt MMR leaf was written. + prop_assert_eq!( + fast.db.finalized_tip_height(), + Some(Height((hole - 1) as u32)), + "the database tip stays just below the hole — the refused block left state untouched" + ); + }); + + Ok(()) +} + +/// Retryable VCT root misses must stay internal to the finalized write loop: the +/// public checkpoint commit wrapper returns the queued block and error to the caller +/// that can retry, rather than completing the block's response channel with a +/// transient error. +#[test] +#[allow(clippy::needless_range_loop)] // the loop indexes blocks[i+1] and the fixture by height +fn vct_retryable_root_miss_keeps_checkpoint_response_pending() -> Result<()> { + let _init_guard = zebra_test::init(); + + let network = ParametersBuilder::default() + .with_activation_heights(ConfiguredActivationHeights { + before_overwinter: Some(1), + overwinter: Some(10), + sapling: Some(15), + blossom: Some(20), + heartwood: Some(25), + canopy: Some(30), + nu5: Some(35), + nu6: Some(40), + nu6_1: Some(45), + nu6_2: Some(47), + nu6_3: Some(48), + nu7: Some(50), + }) + .expect("failed to set activation heights") + .extend_funding_streams() + .to_network() + .expect("failed to build configured network"); + let ledger_strategy = + LedgerState::genesis_strategy(Some(network), None::, None, false); + + proptest!(ProptestConfig::with_cases(1), + |((chain, _count, network, _history_tree) in PreparedChain::default().with_ledger_strategy(ledger_strategy.clone()).with_valid_commitments().no_shrink())| { + + let blocks: Vec<_> = chain.iter().collect(); + let nu5 = NetworkUpgrade::Nu5.activation_height(&network).unwrap().0; + let heartwood = NetworkUpgrade::Heartwood.activation_height(&network).unwrap().0; + let last = (nu5 + 3) as usize; + prop_assert!(blocks.len() > last, "generated chain unexpectedly short"); + let seed = (heartwood - 1) as usize; + + let mut legacy = FinalizedState::new(&Config::ephemeral(), &network, #[cfg(feature = "elasticsearch")] false); + let mut fixture = std::collections::HashMap::new(); + for i in 0..=last { + let cv = CheckpointVerifiedBlock::from(blocks[i].block.clone()); + let (_h, trees) = legacy + .commit_finalized_direct(cv.into(), None, None, "vct response legacy") + .unwrap(); + if i > seed { + fixture.insert(i as u32, (trees.sapling.root(), trees.orchard.root())); + } + } + + let hole = (nu5 + 1) as usize; + fixture.remove(&(hole as u32)); + + let mut fast = FinalizedState::new(&Config::ephemeral(), &network, #[cfg(feature = "elasticsearch")] false); + enable_vct_test_fixture_source(&mut fast, fixture); + + for i in 0..hole { + let cv = CheckpointVerifiedBlock::from(blocks[i].block.clone()); + let next = Some((blocks[i + 1].block.clone(), None)); + fast.commit_finalized_direct(cv.into(), None, next, "vct response fast") + .expect("pre-hole fast commits succeed"); + } + + let cv = CheckpointVerifiedBlock::from(blocks[hole].block.clone()); + let (rsp_tx, mut rsp_rx) = oneshot::channel(); + let next = Some((blocks[hole + 1].block.clone(), None)); + let result = fast.commit_finalized((cv, rsp_tx), None, next); + let Err((returned_block, error)) = result else { + panic!("missing frozen-frontier root should return the queued block for retry"); + }; + + prop_assert_eq!(returned_block.0.height, Height(hole as u32)); + prop_assert!( + error.vct_supplied_root_unavailable_height().is_some(), + "the returned error is the typed retryable VCT root miss" + ); + prop_assert!( + matches!(rsp_rx.try_recv(), Err(oneshot::error::TryRecvError::Empty)), + "the checkpoint response stays pending so the write loop can retry internally" + ); + }); + + Ok(()) +} + +/// An *untrusted* (peer) source must never commit a fast block whose own supplied root has +/// no buffered successor to confirm it against the header chain. A block's roots are only +/// committed by the next block's header (the one-block lag), so committing at the sync tip +/// would persist a root checked only one block later — irreversibly, once on disk. A wrong +/// tip root would then wedge the sync with no recovery (the failure surfaces at the next +/// block and is mis-attributed to *its* root). So the committer defers: it refuses the tip +/// block with the retryable `VctSuppliedRootAwaitingSuccessor`, leaves the database +/// untouched, and commits the same height once a successor is buffered. A trusted local +/// fixture is exempt (covered by `vct_fast_path_matches_legacy_and_rejects_wrong_roots`, +/// whose tip commits on the in-arrears check); this guards the peer path specifically. +#[test] +#[allow(clippy::needless_range_loop)] // the loop indexes blocks[i+1] and inserts roots by height +fn vct_peer_source_defers_unverifiable_tip_root_until_successor() -> Result<()> { + use crate::service::finalized_state::commitment_aux::PeerSource; + use zebra_chain::parallel::commitment_aux::BlockCommitmentRoots; + + let _init_guard = zebra_test::init(); + + let network = ParametersBuilder::default() + .with_activation_heights(ConfiguredActivationHeights { + before_overwinter: Some(1), + overwinter: Some(10), + sapling: Some(15), + blossom: Some(20), + heartwood: Some(25), + canopy: Some(30), + nu5: Some(35), + nu6: Some(40), + nu6_1: Some(45), + nu6_2: Some(47), + nu6_3: Some(48), + nu7: Some(50), + }) + .expect("failed to set activation heights") + .extend_funding_streams() + .to_network() + .expect("failed to build configured network"); + let ledger_strategy = + LedgerState::genesis_strategy(Some(network), None::, None, false); + + proptest!(ProptestConfig::with_cases(1), + |((chain, _count, network, _history_tree) in PreparedChain::default().with_ledger_strategy(ledger_strategy.clone()).with_valid_commitments().no_shrink())| { + + let blocks: Vec<_> = chain.iter().collect(); + let nu5 = NetworkUpgrade::Nu5.activation_height(&network).unwrap().0; + let heartwood = NetworkUpgrade::Heartwood.activation_height(&network).unwrap().0; + // The deferral target: a post-NU5 (real MMR root) height, so it sits above + // Heartwood where the root needs a successor to be confirmed. + let tip_target = (nu5 + 1) as usize; + prop_assert!(blocks.len() > tip_target + 1, "generated chain unexpectedly short"); + let seed = (heartwood - 1) as usize; + + // Legacy golden pass to source the correct per-block roots for the fast range. + let mut legacy = FinalizedState::new(&Config::ephemeral(), &network, #[cfg(feature = "elasticsearch")] false); + let mut peer_roots = Vec::new(); + for i in 0..=tip_target { + let cv = CheckpointVerifiedBlock::from(blocks[i].block.clone()); + let (_h, trees) = legacy + .commit_finalized_direct(cv.into(), None, None, "vct defer legacy") + .unwrap(); + if i > seed { + peer_roots.push(BlockCommitmentRoots { + height: Height(i as u32), + sapling_root: trees.sapling.root(), + orchard_root: trees.orchard.root(), + ironwood_root: zebra_chain::ironwood::tree::NoteCommitmentTree::default().root(), + sapling_tx: 0, + orchard_tx: 0, + ironwood_tx: 0, + auth_data_root: blocks[i].block.auth_data_root(), + }); + } + } + + // An untrusted peer source pre-filled with the *correct* roots: the deferral is + // about the missing successor, not a bad root. + let (source, writer) = PeerSource::new(None); + writer.insert_roots(peer_roots); + + let mut fast = FinalizedState::new(&Config::ephemeral(), &network, #[cfg(feature = "elasticsearch")] false); + fast.enable_vct_fast_source(Box::new(source), true); + + // Commit up to (but not including) the tip target, each with its successor. + for i in 0..tip_target { + let cv = CheckpointVerifiedBlock::from(blocks[i].block.clone()); + let next = Some((blocks[i + 1].block.clone(), None)); + fast.commit_finalized_direct(cv.into(), None, next, "vct defer pre-tip") + .expect("pre-tip fast commits succeed"); + } + prop_assert_eq!(fast.db.finalized_tip_height(), Some(Height((tip_target - 1) as u32))); + + // The tip target with no buffered successor must defer, not commit: its own + // (correct) root is not yet confirmed, and the peer source is untrusted. + prop_assert!( + fast.vct_fast_needs_successor(Height(tip_target as u32)), + "an untrusted peer tip root needs successor verification" + ); + let pre_deferral_prevalidated = fast.vct_prevalidated_count(); + let cv = CheckpointVerifiedBlock::from(blocks[tip_target].block.clone()); + let error = fast + .commit_finalized_direct(cv.into(), None, None, "vct defer tip no successor") + .expect_err("an untrusted tip root with no successor must defer, not commit"); + prop_assert!( + error.vct_supplied_root_unavailable_height().is_none(), + "deferral is not a refetch case (the root is present): {error:?}" + ); + prop_assert!( + format!("{error:?}").contains("VctSuppliedRootAwaitingSuccessor"), + "the tip defers with the await-successor error, got: {error:?}" + ); + prop_assert_eq!( + fast.db.finalized_tip_height(), + Some(Height((tip_target - 1) as u32)), + "the deferred block left the database untouched" + ); + let after_deferral_prevalidated = fast.vct_prevalidated_count(); + prop_assert_eq!( + after_deferral_prevalidated, + pre_deferral_prevalidated + 1, + "the deferred attempt uses the predecessor look-ahead" + ); + + // Once a successor is buffered, the very same height commits and the tip advances: + // the deferral was a wait, not a permanent stall. + let cv = CheckpointVerifiedBlock::from(blocks[tip_target].block.clone()); + let next = Some((blocks[tip_target + 1].block.clone(), None)); + fast.commit_finalized_direct(cv.into(), None, next, "vct defer tip with successor") + .expect("the deferred height commits once its successor is buffered"); + prop_assert_eq!( + fast.vct_prevalidated_count(), + after_deferral_prevalidated + 1, + "the retry reuses the preserved predecessor look-ahead" + ); + prop_assert_eq!( + fast.db.finalized_tip_height(), + Some(Height(tip_target as u32)), + "the tip advances once the successor confirms the root" + ); + }); + + Ok(()) +} + +/// A wrong peer-supplied root must be recoverable at the same height: the committer rejects and +/// evicts the bad cached value, leaves the database parked below the height, then commits the +/// same block once the `tree_aux` driver refills that height with a verifiable root. +#[test] +#[allow(clippy::needless_range_loop)] // the loop indexes blocks[i+1] and inserts roots by height +fn vct_peer_source_bad_root_refill_commits_same_height() -> Result<()> { + use crate::service::finalized_state::commitment_aux::PeerSource; + use zebra_chain::parallel::commitment_aux::BlockCommitmentRoots; + + let _init_guard = zebra_test::init(); + + let network = ParametersBuilder::default() + .with_activation_heights(ConfiguredActivationHeights { + before_overwinter: Some(1), + overwinter: Some(10), + sapling: Some(15), + blossom: Some(20), + heartwood: Some(25), + canopy: Some(30), + nu5: Some(35), + nu6: Some(40), + nu6_1: Some(45), + nu6_2: Some(47), + nu6_3: Some(48), + nu7: Some(50), + }) + .expect("failed to set activation heights") + .extend_funding_streams() + .to_network() + .expect("failed to build configured network"); + let ledger_strategy = + LedgerState::genesis_strategy(Some(network), None::, None, false); + + proptest!(ProptestConfig::with_cases(1), + |((chain, _count, network, _history_tree) in PreparedChain::default().with_ledger_strategy(ledger_strategy.clone()).with_valid_commitments().no_shrink())| { + + let blocks: Vec<_> = chain.iter().collect(); + let nu5 = NetworkUpgrade::Nu5.activation_height(&network).unwrap().0; + let heartwood = NetworkUpgrade::Heartwood.activation_height(&network).unwrap().0; + let target = (nu5 + 1) as usize; + prop_assert!(blocks.len() > target + 1, "generated chain unexpectedly short"); + let seed = (heartwood - 1) as usize; + + // Source the true roots from a legacy pass, then poison the target height exactly + // as a malicious peer would. Earlier roots are correct so the frontier freezes + // before the bad root is encountered. + let mut legacy = FinalizedState::new(&Config::ephemeral(), &network, #[cfg(feature = "elasticsearch")] false); + let mut peer_roots = Vec::new(); + let mut correct_target_root = None; + for i in 0..=target { + let cv = CheckpointVerifiedBlock::from(blocks[i].block.clone()); + let (_h, trees) = legacy + .commit_finalized_direct(cv.into(), None, None, "vct refill legacy") + .unwrap(); + if i > seed { + let root = BlockCommitmentRoots { + height: Height(i as u32), + sapling_root: trees.sapling.root(), + orchard_root: trees.orchard.root(), + ironwood_root: zebra_chain::ironwood::tree::NoteCommitmentTree::default().root(), + sapling_tx: 0, + orchard_tx: 0, + ironwood_tx: 0, + auth_data_root: blocks[i].block.auth_data_root(), + }; + if i == target { + correct_target_root = Some(root.clone()); + let mut poisoned = root; + prop_assert_ne!( + poisoned.sapling_root, + Default::default(), + "a V2 target block must have a non-empty Sapling root" + ); + poisoned.sapling_root = Default::default(); + peer_roots.push(poisoned); + } else { + peer_roots.push(root); + } + } + } + let correct_target_root = correct_target_root.expect("target root was produced"); + + let (source, writer) = PeerSource::new(None); + writer.insert_roots(peer_roots); + + let mut fast = FinalizedState::new(&Config::ephemeral(), &network, #[cfg(feature = "elasticsearch")] false); + fast.enable_vct_fast_source(Box::new(source), true); + + for i in 0..target { + let cv = CheckpointVerifiedBlock::from(blocks[i].block.clone()); + let next = Some((blocks[i + 1].block.clone(), None)); + fast.commit_finalized_direct(cv.into(), None, next, "vct refill pre-target") + .expect("pre-target fast commits succeed"); + } + prop_assert_eq!(fast.db.finalized_tip_height(), Some(Height((target - 1) as u32))); + + let cv = CheckpointVerifiedBlock::from(blocks[target].block.clone()); + let next = Some((blocks[target + 1].block.clone(), None)); + let error = fast + .commit_finalized_direct(cv.into(), None, next.clone(), "vct poisoned target") + .expect_err("the poisoned peer root must be rejected before commit"); + prop_assert_eq!( + error.vct_supplied_root_unavailable_height(), + Some(Height(target as u32)), + "the bad root is exposed as a retryable refetch for its own height" + ); + prop_assert_eq!( + fast.db.finalized_tip_height(), + Some(Height((target - 1) as u32)), + "the rejected root left the database parked below the target" + ); + + // Simulate the `tree_aux` driver refilling the evicted height from another peer. + writer.insert_roots([correct_target_root]); + + let cv = CheckpointVerifiedBlock::from(blocks[target].block.clone()); + fast.commit_finalized_direct(cv.into(), None, next, "vct refilled target") + .expect("the same height commits once the peer cache is refilled"); + prop_assert_eq!( + fast.db.finalized_tip_height(), + Some(Height(target as u32)), + "the refilled root unblocks the parked height" + ); + }); + + Ok(()) +} + +/// The frozen-frontier guard must survive a restart. A fast sync interrupted before the +/// checkpoint handoff leaves the stale frozen frontier persisted (fast commits never write +/// per-height trees) with the tip still below the handoff, but the in-memory `frozen` flag +/// is rebuilt from scratch on open. If it came back `false`, the first post-restart height +/// with no supplied root would legacy-recompute against the stale on-disk frontier and +/// corrupt the history MMR — the exact hazard the in-session guard prevents +/// (`vct_frozen_frontier_hole_refuses_instead_of_recomputing`). So `FinalizedState::new` +/// re-derives the flag from the durable fast-sync marker. This reopens the database between +/// freezing and the hole, and asserts that the very first commit of the new session (no +/// prior fast block to re-arm the flag in-session) still refuses with the retryable +/// `VctSuppliedRootUnavailable`, leaves state untouched, and commits once the root arrives. +#[test] +#[allow(clippy::needless_range_loop)] // the loop indexes blocks[i+1] and the fixture by height +fn vct_frozen_frontier_survives_reopen() -> Result<()> { + let _init_guard = zebra_test::init(); + + let network = ParametersBuilder::default() + .with_activation_heights(ConfiguredActivationHeights { + before_overwinter: Some(1), + overwinter: Some(10), + sapling: Some(15), + blossom: Some(20), + heartwood: Some(25), + canopy: Some(30), + nu5: Some(35), + nu6: Some(40), + nu6_1: Some(45), + nu6_2: Some(47), + nu6_3: Some(48), + nu7: Some(50), + }) + .expect("failed to set activation heights") + .extend_funding_streams() + .to_network() + .expect("failed to build configured network"); + let ledger_strategy = + LedgerState::genesis_strategy(Some(network), None::, None, false); + + proptest!(ProptestConfig::with_cases(1), + |((chain, _count, network, _history_tree) in PreparedChain::default().with_ledger_strategy(ledger_strategy.clone()).with_valid_commitments().no_shrink())| { + + let blocks: Vec<_> = chain.iter().collect(); + let nu5 = NetworkUpgrade::Nu5.activation_height(&network).unwrap().0; + let heartwood = NetworkUpgrade::Heartwood.activation_height(&network).unwrap().0; + let handoff_height = nu5 + 3; + let last = handoff_height as usize; + prop_assert!(blocks.len() > last, "generated chain unexpectedly short"); + let seed = (heartwood - 1) as usize; + + // Stop the fast sync two blocks below the handoff, so the tip is inside the + // frozen region and there is room for the hole at `stop + 1` (still below the + // handoff, where the real frontier would have been written). + let stop = (handoff_height - 2) as usize; + let hole = stop + 1; + prop_assert!(seed < stop && hole < last, "the hole must sit inside the frozen fast range"); + + // Legacy golden pass over [0, last]: the per-block fixture for the fast range + // and the real final frontiers at the handoff (needed to configure fast mode). + let mut legacy = FinalizedState::new(&Config::ephemeral(), &network, #[cfg(feature = "elasticsearch")] false); + let mut fixture = std::collections::HashMap::new(); + let mut handoff_trees = None; + for i in 0..=last { + let cv = CheckpointVerifiedBlock::from(blocks[i].block.clone()); + let (_h, trees) = legacy + .commit_finalized_direct(cv.into(), None, None, "vct reopen legacy") + .unwrap(); + if i > seed { + fixture.insert(i as u32, (trees.sapling.root(), trees.orchard.root())); + } + if i == last { + handoff_trees = Some(trees); + } + } + let handoff_trees = handoff_trees.expect("committed the handoff block"); + + // A persistent database so the syncing handle can be dropped and reopened by + // path, modelling a node restart. Archive storage mode (the default): fast sync + // is the default under checkpoint sync, and a fast-synced database reopens fine + // in archive mode, exactly as in production. + let dir = TempDir::new().expect("temp dir"); + let config = Config { + cache_dir: dir.path().to_path_buf(), + ephemeral: false, + ..Config::default() + }; + + // Session 1: a genesis-start fast sync interrupted at `stop`, two blocks below + // the handoff. The fast commits write the fast-sync marker but no per-height + // trees, so the on-disk frontier is frozen and the tip is below the handoff. + { + let mut fast = FinalizedState::new(&config, &network, #[cfg(feature = "elasticsearch")] false); + enable_vct_test_fixture_source_with_handoff( + &mut fast, + fixture.clone(), + Height(handoff_height), + handoff_trees.sapling.clone(), + handoff_trees.orchard.clone(), + handoff_trees.sprout.clone(), + ); + for i in 0..=stop { + let cv = CheckpointVerifiedBlock::from(blocks[i].block.clone()); + let next = Some((blocks[i + 1].block.clone(), None)); + fast.commit_finalized_direct(cv.into(), None, next, "vct reopen fast") + .expect("verified fast commit succeeds"); + } + prop_assert_eq!(fast.vct_fast_synced_below(), Some(Height(handoff_height)), "the interrupted sync left the fast-sync marker"); + prop_assert_eq!(fast.db.finalized_tip_height(), Some(Height(stop as u32)), "the tip is parked below the handoff"); + // Drop releases the database lock for the reopen below. + } + + // Session 2 (restart): reopen the same database, then punch a hole at the next + // height (a peer that omitted it, or a root evicted after failing verification). + // Skip the constructor-time interrupted-fast-sync resume guard: this configured + // network has no embedded frontiers, so `from_config` yields no source, but the + // test attaches a fixture source below the way a real (Mainnet) node's configured + // source is already present at open time. + let mut reopened = FinalizedState::new_without_resume_guard(&config, &network, #[cfg(feature = "elasticsearch")] false); + prop_assert_eq!(reopened.vct_fast_synced_below(), Some(Height(handoff_height)), "the marker is still durable after reopen"); + + let mut holed = fixture.clone(); + holed.remove(&(hole as u32)); + enable_vct_test_fixture_source_with_handoff( + &mut reopened, + holed, + Height(handoff_height), + handoff_trees.sapling.clone(), + handoff_trees.orchard.clone(), + handoff_trees.sprout.clone(), + ); + + // The very first commit of the new session is the hole. No fast block has run + // since the reopen, so the only thing that can arm the guard is the flag seeded + // from the durable marker. Before the fix it came back `false` and this would + // legacy-recompute against the stale frontier; now it refuses. + let cv = CheckpointVerifiedBlock::from(blocks[hole].block.clone()); + let next = Some((blocks[hole + 1].block.clone(), None)); + let error = reopened + .commit_finalized_direct(cv.into(), None, next, "vct reopen hole") + .expect_err("a frozen-frontier hole must refuse after reopen, not recompute"); + prop_assert!( + format!("{error:?}").contains("VctSuppliedRootUnavailable"), + "the reopened committer returns the retryable VctSuppliedRootUnavailable, got: {error:?}" + ); + prop_assert_eq!(reopened.db.finalized_tip_height(), Some(Height(stop as u32)), "the refused block left the reopened state untouched"); + + // Retryable: once a verifiable root for the hole is supplied, the same height + // commits and the tip advances — the refusal was a stall, not a permanent wedge. + enable_vct_test_fixture_source_with_handoff( + &mut reopened, + fixture.clone(), + Height(handoff_height), + handoff_trees.sapling.clone(), + handoff_trees.orchard.clone(), + handoff_trees.sprout.clone(), + ); + let cv = CheckpointVerifiedBlock::from(blocks[hole].block.clone()); + let next = Some((blocks[hole + 1].block.clone(), None)); + reopened + .commit_finalized_direct(cv.into(), None, next, "vct reopen refill") + .expect("the height commits once its root is fetched"); + prop_assert_eq!(reopened.db.finalized_tip_height(), Some(Height(hole as u32)), "the tip advances past the former hole once the root arrives"); + }); + + Ok(()) +} + +/// Verified-commitment-trees checkpoint handoff (merged increments 4+5): a +/// genesis-start fast sync writes the verified final frontier at the handoff +/// height, marks the database fast-synced, guards historical per-height tree reads +/// below the handoff, and leaves the tip treestate (which post-checkpoint semantic +/// verification resumes from) byte-identical to the legacy recompute. +#[test] +#[allow(clippy::needless_range_loop)] // the loops index blocks[i+1] and the fixture by height +fn vct_fast_sync_handoff_marks_database_and_resumes() -> Result<()> { + let _init_guard = zebra_test::init(); + + let network = ParametersBuilder::default() + .with_activation_heights(ConfiguredActivationHeights { + before_overwinter: Some(1), + overwinter: Some(10), + sapling: Some(15), + blossom: Some(20), + heartwood: Some(25), + canopy: Some(30), + nu5: Some(35), + nu6: Some(40), + nu6_1: Some(45), + nu6_2: Some(47), + nu6_3: Some(48), + nu7: Some(50), + }) + .expect("failed to set activation heights") + .extend_funding_streams() + .to_network() + .expect("failed to build configured network"); + let ledger_strategy = + LedgerState::genesis_strategy(Some(network), None::, None, false); + + proptest!(ProptestConfig::with_cases(env::var("PROPTEST_CASES") + .ok() + .and_then(|v| v.parse().ok()) + .unwrap_or(DEFAULT_PARTIAL_CHAIN_PROPTEST_CASES)), + |((chain, _count, network, _history_tree) in PreparedChain::default().with_ledger_strategy(ledger_strategy.clone()).with_valid_commitments().no_shrink())| { + + let blocks: Vec<_> = chain.iter().collect(); + let nu5 = NetworkUpgrade::Nu5.activation_height(&network).unwrap().0; + let heartwood = NetworkUpgrade::Heartwood.activation_height(&network).unwrap().0; + let last = (nu5 + 3) as usize; + prop_assert!(blocks.len() > last, "generated chain unexpectedly short"); + let handoff = Height(last as u32); + + // The fast range is seeded just below Heartwood, so it is authenticated by + // the ZIP-221 MMR (the synthetic chain's pre-Heartwood `FinalSaplingRoot` + // headers are not consistent with the computed trees, so the Sapling-era + // direct-header path can't be exercised here — that rides with the real + // synced node). The handoff is at the tip. + let seed = (heartwood - 1) as usize; + + // Legacy pass over [0, last]: the per-block fixture for the fast range, the + // golden consensus state, and the real final frontiers at the handoff. + let mut legacy = FinalizedState::new(&Config::ephemeral(), &network, #[cfg(feature = "elasticsearch")] false); + let mut fixture = std::collections::HashMap::new(); + let mut handoff_trees = None; + for i in 0..=last { + let cv = CheckpointVerifiedBlock::from(blocks[i].block.clone()); + let (_h, trees) = legacy + .commit_finalized_direct(cv.into(), None, None, "vct legacy") + .unwrap(); + if i > seed { + fixture.insert(i as u32, (trees.sapling.root(), trees.orchard.root())); + } + if i == last { + handoff_trees = Some(trees); + } + } + let golden_anchors = legacy.db.vct_anchor_digest(); + let golden_history = legacy.db.history_tree().hash(); + let golden_tip = legacy.db.note_commitment_trees_for_tip(); + let handoff_trees = handoff_trees.expect("committed the handoff block"); + + // Fast genesis-start pass over [0, last], supplying the verified frontiers + // for the handoff at `last`. + let mut fast = FinalizedState::new(&Config::ephemeral(), &network, #[cfg(feature = "elasticsearch")] false); + enable_vct_test_fixture_source_with_handoff( + &mut fast, + fixture.clone(), + handoff, + handoff_trees.sapling.clone(), + handoff_trees.orchard.clone(), + handoff_trees.sprout.clone(), + ); + prop_assert!(!fast.vct_fast_needs_successor(handoff), "the trusted handoff frontier authenticates the handoff root without a successor"); + for i in 0..=last { + let cv = CheckpointVerifiedBlock::from(blocks[i].block.clone()); + let next = (i < last).then(|| (blocks[i + 1].block.clone(), None)); + fast.commit_finalized_direct(cv.into(), None, next, "vct fast handoff") + .expect("verified fast commit succeeds"); + } + + // The database is marked fast-synced at the handoff height, and the upgrade height is + // genesis: a node that fast-syncs from genesis records `U = 0`, so its whole `[0, H)` + // range is the absent band and every request is served from the index. + prop_assert_eq!(fast.vct_fast_synced_below(), Some(handoff), "fast-sync marker is set to the handoff height"); + prop_assert_eq!(fast.db.vct_upgrade_height(), Some(Height(0)), "genesis fast sync records the upgrade height at genesis"); + + // Consensus state (anchor sets + history root) matches the legacy recompute. + prop_assert_eq!(fast.db.vct_anchor_digest(), golden_anchors, "fast anchors must match legacy"); + prop_assert_eq!(fast.db.history_tree().hash(), golden_history, "fast history must match legacy"); + + // The handoff wrote the real frontier at the checkpoint, so the tip + // treestate that semantic verification resumes from matches legacy. + let fast_tip = fast.db.note_commitment_trees_for_tip(); + prop_assert_eq!(fast_tip.sapling.root(), golden_tip.sapling.root(), "tip sapling frontier must match legacy"); + prop_assert_eq!(fast_tip.orchard.root(), golden_tip.orchard.root(), "tip orchard frontier must match legacy"); + prop_assert_eq!(fast_tip.sprout.root(), golden_tip.sprout.root(), "tip sprout frontier must match legacy"); + + // Historical per-height tree reads below the handoff are unavailable + // (guarded, no panic), while the handoff height itself is present. + prop_assert!(fast.db.sapling_tree_by_height(&Height(last as u32 - 1)).is_none(), "below-handoff sapling tree read is guarded"); + prop_assert!(fast.db.orchard_tree_by_height(&Height(last as u32 - 1)).is_none(), "below-handoff orchard tree read is guarded"); + prop_assert!(fast.db.sapling_tree_by_height(&handoff).is_some(), "handoff sapling tree is present"); + prop_assert!(fast.db.orchard_tree_by_height(&handoff).is_some(), "handoff orchard tree is present"); + + // Root-serving index (design §4): the fast-synced node holds no per-height trees + // below the handoff (asserted just above), yet it must still serve `tree_aux` + // roots for that range so the root-serving fleet does not collapse as nodes + // fast-sync. Those roots come from the compact `commitment_roots_by_height` index + // the fast path persists per block, and they match exactly the roots the + // legacy/archive node derives from its per-height trees. + let below_handoff = Height((seed + 1) as u32)..=Height(last as u32 - 1); + let served = fast.db.commitment_roots_by_height_range(below_handoff.clone()); + let expected = commitment_aux::produce_block_roots(&legacy.db, below_handoff.clone()); + prop_assert!(!served.is_empty(), "a fast-synced node serves below-handoff roots from the index"); + prop_assert_eq!(served, expected.clone(), "index-served roots match the legacy per-height-tree roots"); + + // The same range goes through `serve_block_roots`: with `U = 0` the request starts at + // or above the upgrade height, so it is served entirely from the index — no per-height + // trees (which the fast-synced node lacks below the handoff) are consulted. + prop_assert_eq!(serve_block_roots(&fast.db, below_handoff), expected, "serve_block_roots serves the fast-synced range from the index"); + + // The `z_gettreestate` RPC gate predicate matches the read guard: a + // below-handoff height is unavailable (typed archive-mode error), while the + // handoff height itself is available. + prop_assert!(fast.db.vct_historical_tree_unavailable(HashOrHeight::Height(Height(last as u32 - 1))), "RPC gate: below-handoff treestate is unavailable"); + prop_assert!(!fast.db.vct_historical_tree_unavailable(HashOrHeight::Height(handoff)), "RPC gate: handoff treestate is available"); + + // Negative: a peer can supply a wrong root exactly at the handoff height, + // where there is no buffered checkpoint successor to authenticate it. The + // final embedded frontier still binds the expected root, so the committer + // must reject and retry instead of panicking or writing a bad handoff. + let mut bad_handoff_fixture = fixture.clone(); + let bad_handoff_entry = bad_handoff_fixture + .get_mut(&(last as u32)) + .expect("fixture contains the handoff root"); + prop_assert_ne!(bad_handoff_entry.0, Default::default(), "a post-NU5 handoff block must have a non-empty Sapling root"); + bad_handoff_entry.0 = Default::default(); + + let mut bad_handoff = FinalizedState::new(&Config::ephemeral(), &network, #[cfg(feature = "elasticsearch")] false); + enable_vct_test_fixture_source_with_handoff( + &mut bad_handoff, + bad_handoff_fixture, + handoff, + handoff_trees.sapling.clone(), + handoff_trees.orchard.clone(), + handoff_trees.sprout.clone(), + ); + + let mut error_height = None; + let mut handoff_error = None; + for i in 0..=last { + let cv = CheckpointVerifiedBlock::from(blocks[i].block.clone()); + let next = (i < last).then(|| (blocks[i + 1].block.clone(), None)); + match bad_handoff.commit_finalized_direct(cv.into(), None, next, "vct bad handoff") { + Ok(_) => {} + Err(error) => { + error_height = Some(i); + handoff_error = Some(error); + break; + } + } + } + prop_assert_eq!(error_height, Some(last), "the bad handoff root is rejected at the handoff height"); + let handoff_error = handoff_error.expect("the bad handoff root failed"); + prop_assert!( + format!("{handoff_error:?}").contains("VctSuppliedRootUnavailable"), + "a bad handoff root returns the retryable VctSuppliedRootUnavailable error, got: {handoff_error:?}" + ); + prop_assert_eq!( + bad_handoff.db.finalized_tip_height(), + Some(Height(last as u32 - 1)), + "the refused handoff block left state untouched" + ); + }); + + Ok(()) +} + +/// Switching between the rollout fast path and the manual recompute path is safe at the +/// committed-state boundaries: after the handoff writes the real frontier, legacy recompute can +/// resume from that frontier; before any fast commit has frozen the frontier, a later fast sync +/// can consume verified roots for future heights. +#[test] +#[allow(clippy::needless_range_loop)] // the loops index blocks[i+1] and the fixture by height +fn vct_mode_switches_continue_from_safe_boundaries() -> Result<()> { + let _init_guard = zebra_test::init(); + + let network = ParametersBuilder::default() + .with_activation_heights(ConfiguredActivationHeights { + before_overwinter: Some(1), + overwinter: Some(10), + sapling: Some(15), + blossom: Some(20), + heartwood: Some(25), + canopy: Some(30), + nu5: Some(35), + nu6: Some(40), + nu6_1: Some(45), + nu6_2: Some(47), + nu6_3: Some(48), + nu7: Some(50), + }) + .expect("failed to set activation heights") + .extend_funding_streams() + .to_network() + .expect("failed to build configured network"); + let ledger_strategy = + LedgerState::genesis_strategy(Some(network), None::, None, false); + + proptest!(ProptestConfig::with_cases(1), + |((chain, _count, network, _history_tree) in PreparedChain::default().with_ledger_strategy(ledger_strategy.clone()).with_valid_commitments().no_shrink())| { + let blocks: Vec<_> = chain.iter().collect(); + let nu5 = NetworkUpgrade::Nu5.activation_height(&network).unwrap().0; + let heartwood = NetworkUpgrade::Heartwood.activation_height(&network).unwrap().0; + let handoff_index = (nu5 + 3) as usize; + let post_handoff_tip = handoff_index + 2; + prop_assert!(blocks.len() > post_handoff_tip, "generated chain unexpectedly short"); + let handoff = Height(handoff_index as u32); + let seed = (heartwood - 1) as usize; + + // Legacy golden pass over the full range: source fast roots and final frontiers, then + // compare both switching scenarios against this byte-identical manual recompute. + let mut legacy = FinalizedState::new(&Config::ephemeral(), &network, #[cfg(feature = "elasticsearch")] false); + let mut fixture = std::collections::HashMap::new(); + let mut handoff_trees = None; + let mut post_handoff_roots = None; + for i in 0..=post_handoff_tip { + let cv = CheckpointVerifiedBlock::from(blocks[i].block.clone()); + let (_h, trees) = legacy + .commit_finalized_direct(cv.into(), None, None, "vct switch legacy") + .unwrap(); + if i > seed && i <= handoff_index { + fixture.insert(i as u32, (trees.sapling.root(), trees.orchard.root())); + } + if i == handoff_index { + handoff_trees = Some(trees); + } else if i == handoff_index + 1 { + post_handoff_roots = Some((trees.sapling.root(), trees.orchard.root())); + } + } + let golden_anchors = legacy.db.vct_anchor_digest(); + let golden_history = legacy.db.history_tree().hash(); + let golden_tip = legacy.db.note_commitment_trees_for_tip(); + let handoff_trees = handoff_trees.expect("committed the handoff block"); + let post_handoff_roots = post_handoff_roots.expect("committed a post-handoff block"); + + // Fast -> manual: complete the fast handoff, reopen with the force-disable knob, and + // keep checkpoint sync enabled while post-handoff blocks recompute from the real + // frontier written at the handoff. + let fast_to_manual_dir = TempDir::new().expect("temp dir"); + let fast_config = Config { + cache_dir: fast_to_manual_dir.path().to_path_buf(), + ephemeral: false, + ..Config::default() + }; + { + let mut fast = FinalizedState::new(&fast_config, &network, #[cfg(feature = "elasticsearch")] false); + enable_vct_test_fixture_source_with_handoff( + &mut fast, + fixture.clone(), + handoff, + handoff_trees.sapling.clone(), + handoff_trees.orchard.clone(), + handoff_trees.sprout.clone(), + ); + for i in 0..=handoff_index { + let cv = CheckpointVerifiedBlock::from(blocks[i].block.clone()); + let next = (i < handoff_index).then(|| (blocks[i + 1].block.clone(), None)); + fast.commit_finalized_direct(cv.into(), None, next, "vct switch fast prefix") + .expect("verified fast prefix commits"); + } + prop_assert_eq!(fast.vct_fast_synced_below(), Some(handoff), "fast sync reached the handoff before the switch"); + } + + let manual_config = Config { + vct_fast_sync: false, + ..fast_config + }; + let mut manual = FinalizedState::new(&manual_config, &network, #[cfg(feature = "elasticsearch")] false); + for i in (handoff_index + 1)..=post_handoff_tip { + let cv = CheckpointVerifiedBlock::from(blocks[i].block.clone()); + manual + .commit_finalized_direct(cv.into(), None, None, "vct switch manual suffix") + .expect("manual suffix commits after fast handoff"); + } + let manual_tip = manual.db.note_commitment_trees_for_tip(); + prop_assert_eq!(manual.db.vct_anchor_digest(), golden_anchors, "fast-to-manual anchors match legacy"); + prop_assert_eq!(manual.db.history_tree().hash(), golden_history, "fast-to-manual history matches legacy"); + prop_assert_eq!(manual_tip.sapling.root(), golden_tip.sapling.root(), "fast-to-manual sapling tip matches legacy"); + prop_assert_eq!(manual_tip.orchard.root(), golden_tip.orchard.root(), "fast-to-manual orchard tip matches legacy"); + prop_assert_eq!(manual_tip.sprout.root(), golden_tip.sprout.root(), "fast-to-manual sprout tip matches legacy"); + + // Manual -> fast: commit a prefix with the force-disable knob before any fast block + // can freeze the frontier, then reopen and consume verified roots through the handoff. + let manual_to_fast_dir = TempDir::new().expect("temp dir"); + let manual_prefix_config = Config { + cache_dir: manual_to_fast_dir.path().to_path_buf(), + ephemeral: false, + vct_fast_sync: false, + ..Config::default() + }; + { + let mut manual_prefix = FinalizedState::new(&manual_prefix_config, &network, #[cfg(feature = "elasticsearch")] false); + for i in 0..=seed { + let cv = CheckpointVerifiedBlock::from(blocks[i].block.clone()); + manual_prefix + .commit_finalized_direct(cv.into(), None, None, "vct switch manual prefix") + .expect("manual prefix commits"); + } + } + + let fast_suffix_config = Config { + vct_fast_sync: true, + ..manual_prefix_config + }; + let mut fast_suffix = FinalizedState::new(&fast_suffix_config, &network, #[cfg(feature = "elasticsearch")] false); + let mut guarded_fixture = fixture; + // A stale or over-eager peer cache entry above the handoff must be ignored so + // the committer resumes legacy recompute from the real handoff frontier. + prop_assert_ne!( + post_handoff_roots.0, + Default::default(), + "a post-NU5 post-handoff block must have a non-empty Sapling root", + ); + guarded_fixture.insert( + (handoff_index + 1) as u32, + (Default::default(), post_handoff_roots.1), + ); + enable_vct_test_fixture_source_with_handoff( + &mut fast_suffix, + guarded_fixture, + handoff, + handoff_trees.sapling.clone(), + handoff_trees.orchard.clone(), + handoff_trees.sprout.clone(), + ); + for i in (seed + 1)..=post_handoff_tip { + let cv = CheckpointVerifiedBlock::from(blocks[i].block.clone()); + let next = (i < post_handoff_tip).then(|| (blocks[i + 1].block.clone(), None)); + fast_suffix + .commit_finalized_direct(cv.into(), None, next, "vct switch fast suffix") + .expect("fast suffix commits after manual prefix"); + } + prop_assert_eq!( + fast_suffix.vct_fast_count(), + (handoff_index - seed) as u64, + "an above-handoff cached root must not keep the committer on the fast path", + ); + let fast_suffix_tip = fast_suffix.db.note_commitment_trees_for_tip(); + prop_assert_eq!(fast_suffix.db.vct_anchor_digest(), golden_anchors, "manual-to-fast anchors match legacy"); + prop_assert_eq!(fast_suffix.db.history_tree().hash(), golden_history, "manual-to-fast history matches legacy"); + prop_assert_eq!(fast_suffix_tip.sapling.root(), golden_tip.sapling.root(), "manual-to-fast sapling tip matches legacy"); + prop_assert_eq!(fast_suffix_tip.orchard.root(), golden_tip.orchard.root(), "manual-to-fast orchard tip matches legacy"); + prop_assert_eq!(fast_suffix_tip.sprout.root(), golden_tip.sprout.root(), "manual-to-fast sprout tip matches legacy"); + }); + + Ok(()) +} + +/// Standalone test isolating the verify-before-commit **dedup**: each header +/// commitment is checked once, not twice. +/// +/// - **Skip:** the first fast block runs its own commitment check; the next one +/// is skipped, because the first block's look-ahead already validated it. +/// - **Stale-cache guard:** a cache entry with the right height but the *wrong* +/// hash must not trigger a skip — the guard forces the own check to run, so a +/// stale or mismatched entry can never let an unverified block through. +/// - **Wrapper-hash guard:** a public `CheckpointVerifiedBlock::with_hash` caller +/// cannot replay a stale cached successor hash onto a different block. +#[test] +fn vct_dedup_skips_redundant_check_and_guards_stale_cache() -> Result<()> { + let _init_guard = zebra_test::init(); + + let network = ParametersBuilder::default() + .with_activation_heights(ConfiguredActivationHeights { + before_overwinter: Some(1), + overwinter: Some(10), + sapling: Some(15), + blossom: Some(20), + heartwood: Some(25), + canopy: Some(30), + nu5: Some(35), + nu6: Some(40), + nu6_1: Some(45), + nu6_2: Some(47), + nu6_3: Some(48), + nu7: Some(50), + }) + .expect("failed to set activation heights") + .extend_funding_streams() + .to_network() + .expect("failed to build configured network"); + let ledger_strategy = + LedgerState::genesis_strategy(Some(network), None::, None, false); + + proptest!(ProptestConfig::with_cases(1), + |((chain, _count, network, _history_tree) in PreparedChain::default().with_ledger_strategy(ledger_strategy.clone()).with_valid_commitments().no_shrink())| { + + let blocks: Vec<_> = chain.iter().collect(); + let nu5 = NetworkUpgrade::Nu5.activation_height(&network).unwrap().0 as usize; + + // Seed just before NU5, then operate on four consecutive fast blocks so + // the forged-wrapper regression exercises `hashBlockCommitments`. + let seed = nu5 - 2; + let last = seed + 4; + prop_assert!(blocks.len() > last + 1, "generated chain unexpectedly short"); + + // Legacy pass to record the correct per-block roots as the fixture. + let mut legacy = FinalizedState::new(&Config::ephemeral(), &network, #[cfg(feature = "elasticsearch")] false); + let mut fixture = std::collections::HashMap::new(); + for (i, prepared) in blocks.iter().take(last + 1).enumerate() { + let cv = CheckpointVerifiedBlock::from(prepared.block.clone()); + let (_h, trees) = legacy + .commit_finalized_direct(cv.into(), None, None, "vct dedup legacy") + .unwrap(); + if i > seed { + fixture.insert(i as u32, (trees.sapling.root(), trees.orchard.root())); + } + } + + let mut fast = FinalizedState::new(&Config::ephemeral(), &network, #[cfg(feature = "elasticsearch")] false); + enable_vct_test_fixture_source(&mut fast, fixture); + + // Commit block `i` with its real successor as the one-block look-ahead. + let commit = |fast: &mut FinalizedState, i: usize| { + let cv = CheckpointVerifiedBlock::from(blocks[i].block.clone()); + let next = Some((blocks[i + 1].block.clone(), None)); + fast.commit_finalized_direct(cv.into(), None, next, "vct dedup fast") + .expect("verified fast commit succeeds"); + }; + + // genesis..=seed take the recompute path (no fixture entries), so the dedup + // never engages here. + for i in 0..=seed { + commit(&mut fast, i); + } + prop_assert_eq!(fast.vct_prevalidated_count(), 0, "no fast blocks committed yet"); + + // First fast block: no cached predecessor, so it runs its own check. + commit(&mut fast, seed + 1); + prop_assert_eq!(fast.vct_prevalidated_count(), 0, "the first fast block runs its own commitment check"); + + // Second fast block: its predecessor's look-ahead already validated it, + // so the own check is skipped — the dedup engages. + commit(&mut fast, seed + 2); + prop_assert_eq!(fast.vct_prevalidated_count(), 1, "the second fast block skips its redundant own commitment check"); + + // Stale-cache guard: overwrite the cache with the correct height but the + // hash of a *different* block. The next commit must NOT skip. + let stale_hash = blocks[seed + 1].hash; + prop_assert_ne!(stale_hash, blocks[seed + 3].hash, "stale hash must differ from the real block"); + fast.vct_prevalidated_next = Some((Height((seed + 3) as u32), stale_hash)); + commit(&mut fast, seed + 3); + prop_assert_eq!(fast.vct_prevalidated_count(), 1, "a stale cache entry (wrong hash) must not cause a false skip"); + + // Public wrapper-hash guard: the stale cache records a real look-ahead + // hash, but a caller-controlled checkpoint wrapper tries to replay that + // hash onto a different block whose own NU5 header commitment is invalid. + // The skip must compare the cache against the wrapped block's real hash, + // not the wrapper hash, so the bad commitment is checked and rejected. + let forged_wrapper_hash = blocks[seed + 2].hash; + let bad_block = blocks[seed + 4].block.clone().set_block_commitment([0x42; 32]); + let bad_block_hash = bad_block.hash(); + prop_assert_ne!( + forged_wrapper_hash, + bad_block_hash, + "the forged wrapper hash must differ from the bad block's real hash", + ); + fast.vct_prevalidated_next = + Some((Height((seed + 4) as u32), forged_wrapper_hash)); + let forged = CheckpointVerifiedBlock::with_hash(bad_block, forged_wrapper_hash); + let error = fast + .commit_finalized_direct(forged.into(), None, None, "vct forged wrapper hash") + .expect_err("a forged wrapper hash must not skip the bad block's own commitment check"); + prop_assert!( + format!("{error:?}").contains("VctSuppliedRootUnavailable"), + "the forged wrapper hash path must reject the bad commitment, got: {error:?}", + ); + prop_assert_eq!( + fast.vct_prevalidated_count(), + 1, + "the forged wrapper hash must not increment the prevalidated count", + ); + prop_assert_eq!( + fast.db.finalized_tip_height(), + Some(Height((seed + 3) as u32)), + "the rejected forged block must leave finalized state untouched", + ); + }); + + Ok(()) +} + +/// Clearing a cached VCT successor prevalidation must disarm exactly one possible +/// skip without disabling the normal dedup optimization for future contiguous fast +/// blocks. This covers the write-loop reset/drop behavior indirectly: those paths +/// call `clear_vct_prevalidated_next()` when buffered checkpoint state is discarded. +#[test] +fn vct_clear_prevalidation_cache_disarms_skip_then_dedup_resumes() -> Result<()> { + let _init_guard = zebra_test::init(); + + let network = ParametersBuilder::default() + .with_activation_heights(ConfiguredActivationHeights { + before_overwinter: Some(1), + overwinter: Some(10), + sapling: Some(15), + blossom: Some(20), + heartwood: Some(25), + canopy: Some(30), + nu5: Some(35), + nu6: Some(40), + nu6_1: Some(45), + nu6_2: Some(47), + nu6_3: Some(48), + nu7: Some(50), + }) + .expect("failed to set activation heights") + .extend_funding_streams() + .to_network() + .expect("failed to build configured network"); + let ledger_strategy = + LedgerState::genesis_strategy(Some(network), None::, None, false); + + proptest!(ProptestConfig::with_cases(1), + |((chain, _count, network, _history_tree) in PreparedChain::default().with_ledger_strategy(ledger_strategy.clone()).with_valid_commitments().no_shrink())| { + + let blocks: Vec<_> = chain.iter().collect(); + let nu5 = NetworkUpgrade::Nu5.activation_height(&network).unwrap().0 as usize; + let seed = nu5 - 2; + let last = seed + 5; + prop_assert!(blocks.len() > last + 1, "generated chain unexpectedly short"); + + let mut legacy = FinalizedState::new(&Config::ephemeral(), &network, #[cfg(feature = "elasticsearch")] false); + let mut fixture = std::collections::HashMap::new(); + for (i, prepared) in blocks.iter().take(last + 1).enumerate() { + let cv = CheckpointVerifiedBlock::from(prepared.block.clone()); + let (_h, trees) = legacy + .commit_finalized_direct(cv.into(), None, None, "vct clear legacy") + .unwrap(); + if i > seed { + fixture.insert(i as u32, (trees.sapling.root(), trees.orchard.root())); + } + } + + let mut fast = FinalizedState::new(&Config::ephemeral(), &network, #[cfg(feature = "elasticsearch")] false); + enable_vct_test_fixture_source(&mut fast, fixture); + + let commit = |fast: &mut FinalizedState, i: usize| { + let cv = CheckpointVerifiedBlock::from(blocks[i].block.clone()); + let next = Some((blocks[i + 1].block.clone(), None)); + fast.commit_finalized_direct(cv.into(), None, next, "vct clear fast") + .expect("verified fast commit succeeds"); + }; + + for i in 0..=seed { + commit(&mut fast, i); + } + commit(&mut fast, seed + 1); + prop_assert_eq!(fast.vct_prevalidated_count(), 0, "first fast block runs its own check"); + + commit(&mut fast, seed + 2); + prop_assert_eq!(fast.vct_prevalidated_count(), 1, "second fast block uses predecessor look-ahead"); + + fast.clear_vct_prevalidated_next(); + commit(&mut fast, seed + 3); + prop_assert_eq!( + fast.vct_prevalidated_count(), + 1, + "clearing the cache forces the next fast block to run its own check", + ); + + commit(&mut fast, seed + 4); + prop_assert_eq!( + fast.vct_prevalidated_count(), + 2, + "normal successor dedup resumes after the cleared block commits", + ); + }); + + Ok(()) +} + +/// Increment-3 contract proof: a roots/frontier payload **produced from a database** +/// (the serving read path) can replace the fixture and drives the fast path to +/// byte-identical consensus state. +/// +/// Builds an archive/legacy state over a generated valid-commitment chain (crossing +/// Heartwood and NU5), produces the per-block roots and final frontier from that DB +/// via [`commitment_aux::produce_block_roots`] / [`commitment_aux::produce_final_frontiers`], +/// then drives a fresh fast-sync state that consumes the produced payload through the +/// test-only [`commitment_aux::FixtureSource`]. Asserts the fast anchors + history-tree hash are +/// byte-identical to the legacy build, and that the produced final frontier agrees with +/// the legacy tip frontier and the produced root at the handoff height. +/// +/// This is coverage the existing equivalence test lacks: there the roots are captured +/// from the committer's inline-returned trees, here they come from the **DB read path** +/// a serving node runs. No networking and no DB-format change. +#[test] +#[allow(clippy::needless_range_loop)] // the loops index blocks[i+1] (the look-ahead) and by height +fn vct_db_produced_payload_round_trips_to_byte_identical_state() -> Result<()> { + let _init_guard = zebra_test::init(); + + let network = ParametersBuilder::default() + .with_activation_heights(ConfiguredActivationHeights { + before_overwinter: Some(1), + overwinter: Some(10), + sapling: Some(15), + blossom: Some(20), + heartwood: Some(25), + canopy: Some(30), + nu5: Some(35), + nu6: Some(40), + nu6_1: Some(45), + nu6_2: Some(47), + nu6_3: Some(48), + nu7: Some(50), + }) + .expect("failed to set activation heights") + .extend_funding_streams() + .to_network() + .expect("failed to build configured network"); + let ledger_strategy = + LedgerState::genesis_strategy(Some(network), None::, None, false); + + proptest!(ProptestConfig::with_cases(1), + |((chain, _count, network, _history_tree) in PreparedChain::default().with_ledger_strategy(ledger_strategy.clone()).with_valid_commitments().no_shrink())| { + + let blocks: Vec<_> = chain.iter().collect(); + let nu5 = NetworkUpgrade::Nu5.activation_height(&network).unwrap().0; + let heartwood = NetworkUpgrade::Heartwood.activation_height(&network).unwrap().0; + let last = (nu5 + 3) as usize; + prop_assert!(blocks.len() > last + 1, "generated chain unexpectedly short"); + // Seed below Heartwood so the fast range creates the history tree and + // crosses the NU5 V1->V2 boundary, matching the equivalence test. + let seed = (heartwood - 1) as usize; + + // Legacy/archive pass: a real DB with per-height trees, plus the golden state. + let mut legacy = FinalizedState::new(&Config::ephemeral(), &network, #[cfg(feature = "elasticsearch")] false); + for block in blocks.iter().take(last + 1) { + let cv = CheckpointVerifiedBlock::from(block.block.clone()); + legacy + .commit_finalized_direct(cv.into(), None, None, "vct round-trip legacy") + .unwrap(); + } + let golden_anchors = legacy.db.vct_anchor_digest(); + let golden_history = legacy.db.history_tree().hash(); + + // Produce the payload from the legacy DB's per-height trees (the serving read path). + let last_height = Height(last as u32); + let produced_roots = commitment_aux::produce_block_roots( + &legacy.db, + Height((seed + 1) as u32)..=last_height, + ); + let produced_frontiers = commitment_aux::produce_final_frontiers(&legacy.db, last_height) + .expect("legacy DB has the tip frontier"); + + // The produced final frontier agrees with the legacy tip frontier and with the + // produced root at the handoff height (the two producer outputs are consistent). + let handoff = produced_roots.last().expect("produced a non-empty range"); + prop_assert_eq!(produced_frontiers.sapling.root(), handoff.sapling_root, "produced sapling frontier matches the produced root at handoff"); + prop_assert_eq!(produced_frontiers.orchard.root(), handoff.orchard_root, "produced orchard frontier matches the produced root at handoff"); + prop_assert_eq!(produced_frontiers.sapling.root(), legacy.db.sapling_tree_by_height(&last_height).unwrap().root(), "produced sapling frontier matches legacy tip"); + prop_assert_eq!(produced_frontiers.sprout.root(), legacy.db.sprout_tree_for_tip().root(), "produced sprout frontier matches legacy tip"); + + // Consume the DB-produced roots in a fresh fast-sync state. + let mut fast = FinalizedState::new(&Config::ephemeral(), &network, #[cfg(feature = "elasticsearch")] false); + let produced_roots = produced_roots + .into_iter() + .map(|root| (root.height.0, (root.sapling_root, root.orchard_root))) + .collect(); + fast.enable_vct_fast_source( + Box::new(commitment_aux::FixtureSource::new(produced_roots, None)), + false, + ); + for i in 0..=last { + let cv = CheckpointVerifiedBlock::from(blocks[i].block.clone()); + let next = Some((blocks[i + 1].block.clone(), None)); + fast.commit_finalized_direct(cv.into(), None, next, "vct round-trip fast") + .expect("verified fast commit from DB-produced roots succeeds"); + } + + prop_assert_eq!(fast.db.vct_anchor_digest(), golden_anchors, "fast anchors from DB-produced roots match legacy"); + prop_assert_eq!(fast.db.history_tree().hash(), golden_history, "fast history from DB-produced roots match legacy"); + + // Serving stitch across the upgrade height `U`. Simulate a node that upgraded + // mid-chain: it keeps the full per-height trees (written before the upgrade) but only + // has the serving index from `U` upward. `serve_block_roots` must still return the + // whole requested range as one contiguous run — trees fill `[start, U)`, the index + // fills `[U, end]` — matching the all-trees reference, with no short batch at the + // boundary that would stall the client's minimum-progress check. + let serve_range = Height((seed + 1) as u32)..=last_height; + let all_trees_reference = + commitment_aux::produce_block_roots(&legacy.db, serve_range.clone()); + let upgrade = Height(((seed + 1 + last) / 2) as u32); + prop_assert!( + serve_range.start() < &upgrade && upgrade <= last_height, + "the chosen upgrade height splits the served range" + ); + let mut batch = DiskWriteBatch::new(); + batch.delete_range_commitment_roots_by_height(&legacy.db, &Height(0), &upgrade); + batch.update_vct_upgrade_marker(&legacy.db, upgrade); + legacy + .db + .write_batch(batch) + .expect("simulating a mid-chain upgrade succeeds"); + prop_assert!( + legacy + .db + .commitment_roots_by_height_range(Height(0)..=Height(upgrade.0 - 1)) + .is_empty(), + "the serving index is dropped below the upgrade height" + ); + let stitched = serve_block_roots(&legacy.db, serve_range); + prop_assert_eq!( + stitched, + all_trees_reference, + "serve_block_roots stitches the trees below U with the index at/above U into one gap-free run" + ); + }); + + Ok(()) +} + +/// Verified-commitment-trees consumer half of the `tree_aux` peer source (increment 6a): +/// a [`commitment_aux::PeerSource`] **filled incrementally** by its writer handle (as the +/// driver fills it when root ranges arrive from peers) drives the fast path to +/// byte-identical consensus state. Same harness as the DB-produced round-trip, but the +/// produced roots are inserted into the shared cache in two chunks via +/// [`commitment_aux::PeerSourceWriter`] — proving the fillable, driver-facing source is a +/// drop-in for the fixture. (The network transport that fills it is the rest of 6a.) +#[test] +#[allow(clippy::needless_range_loop)] // the loops index blocks[i+1] (the look-ahead) and by height +fn vct_peer_source_filled_incrementally_drives_byte_identical_state() -> Result<()> { + let _init_guard = zebra_test::init(); + + let network = ParametersBuilder::default() + .with_activation_heights(ConfiguredActivationHeights { + before_overwinter: Some(1), + overwinter: Some(10), + sapling: Some(15), + blossom: Some(20), + heartwood: Some(25), + canopy: Some(30), + nu5: Some(35), + nu6: Some(40), + nu6_1: Some(45), + nu6_2: Some(47), + nu6_3: Some(48), + nu7: Some(50), + }) + .expect("failed to set activation heights") + .extend_funding_streams() + .to_network() + .expect("failed to build configured network"); + let ledger_strategy = + LedgerState::genesis_strategy(Some(network), None::, None, false); + + proptest!(ProptestConfig::with_cases(1), + |((chain, _count, network, _history_tree) in PreparedChain::default().with_ledger_strategy(ledger_strategy.clone()).with_valid_commitments().no_shrink())| { + + let blocks: Vec<_> = chain.iter().collect(); + let nu5 = NetworkUpgrade::Nu5.activation_height(&network).unwrap().0; + let heartwood = NetworkUpgrade::Heartwood.activation_height(&network).unwrap().0; + // The untrusted peer source defers any fast block whose own root has no buffered + // successor, so every committed fast block needs `blocks[i + 1]`. Keep `last` one + // below the chain tip so the deepest commit still has a successor witness. + let last = ((nu5 + 3) as usize).min(blocks.len().saturating_sub(2)); + prop_assert!(last > (nu5 as usize), "generated chain unexpectedly short"); + let seed = (heartwood - 1) as usize; + + // Legacy/archive pass: a real DB with per-height trees, plus the golden state. + let mut legacy = FinalizedState::new(&Config::ephemeral(), &network, #[cfg(feature = "elasticsearch")] false); + for block in blocks.iter().take(last + 1) { + let cv = CheckpointVerifiedBlock::from(block.block.clone()); + legacy + .commit_finalized_direct(cv.into(), None, None, "vct peer-source legacy") + .unwrap(); + } + let golden_anchors = legacy.db.vct_anchor_digest(); + let golden_history = legacy.db.history_tree().hash(); + + // Produce the payload from the legacy DB (the serving read path). + let produced_roots = commitment_aux::produce_block_roots( + &legacy.db, + Height((seed + 1) as u32)..=Height(last as u32), + ); + + // Fill the peer source incrementally via its writer, in two chunks, as the + // driver would when successive root ranges arrive from a peer. + let (peer_source, writer) = commitment_aux::PeerSource::new(None); + let split = produced_roots.len() / 2; + writer.insert_roots(produced_roots[..split].iter().cloned()); + writer.insert_roots(produced_roots[split..].iter().cloned()); + + // Consume the peer-source-supplied roots in a fresh fast-sync state. Each fast + // block is committed with its successor buffered, as the write loop does — the + // untrusted source defers a tip commit with no successor (covered by + // `vct_peer_source_defers_unverifiable_tip_root_until_successor`). + let mut fast = FinalizedState::new(&Config::ephemeral(), &network, #[cfg(feature = "elasticsearch")] false); + fast.enable_vct_fast_source(Box::new(peer_source), true); + for i in 0..=last { + let cv = CheckpointVerifiedBlock::from(blocks[i].block.clone()); + let next = Some((blocks[i + 1].block.clone(), None)); + fast.commit_finalized_direct(cv.into(), None, next, "vct peer-source fast") + .expect("verified fast commit from peer-source roots succeeds"); + } + + prop_assert_eq!(fast.db.vct_anchor_digest(), golden_anchors, "fast anchors from peer-source roots match legacy"); + prop_assert_eq!(fast.db.history_tree().hash(), golden_history, "fast history from peer-source roots match legacy"); + }); + + Ok(()) +} diff --git a/zebra-state/src/service/finalized_state/tests/rollback.rs b/zebra-state/src/service/finalized_state/tests/rollback.rs index 73740a32698..336f05b1d6c 100644 --- a/zebra-state/src/service/finalized_state/tests/rollback.rs +++ b/zebra-state/src/service/finalized_state/tests/rollback.rs @@ -80,7 +80,7 @@ fn sync_to(config: &Config, network: &Network, blocks: &[SemanticallyVerifiedBlo for block in blocks { let checkpoint_verified = CheckpointVerifiedBlock::from(block.block.clone()); state - .commit_finalized_direct(checkpoint_verified.into(), None, "rollback test") + .commit_finalized_direct(checkpoint_verified.into(), None, None, "rollback test") .expect("committing a generated block to a fresh state succeeds"); } } diff --git a/zebra-state/src/service/finalized_state/vct.rs b/zebra-state/src/service/finalized_state/vct.rs new file mode 100644 index 00000000000..a143e67eb3d --- /dev/null +++ b/zebra-state/src/service/finalized_state/vct.rs @@ -0,0 +1,675 @@ +//! Verified-commitment-trees fast-sync experiment state (POC harness). +//! +//! This module holds the embedded-frontier plumbing and run counters for the +//! verified-commitment-trees fast path. On networks with an embedded handoff frontier, +//! the default source is the peer `tree_aux` source. `checkpoint_sync = false` or +//! `consensus.vct_fast_sync = false` selects legacy recompute. +//! +//! [`super`] (`finalized_state.rs`) holds only the commit-path hook (the checkpoint +//! handoff write and the fast-sync marker); everything about *where the data comes +//! from* lives here, behind a small method API so the commit path never touches the +//! experiment's internals. + +use std::sync::{ + atomic::{AtomicU64, Ordering}, + Arc, +}; + +use thiserror::Error; +#[cfg(test)] +use zebra_chain::parallel::tree::NoteCommitmentTrees; +use zebra_chain::{ + block, orchard, + parameters::{Network, NetworkUpgrade}, + sapling, sprout, +}; + +use super::{ + commitment_aux::{CommitmentRootSource, FinalFrontiers, PeerSource}, + ZebraDb, +}; + +/// Embedded verified final note-commitment frontiers for Mainnet. +const MAINNET_FINAL_FRONTIERS: &[u8] = include_bytes!("vct/mainnet-frontier.bin"); + +/// Errors validating serialized VCT final-frontier bytes. +#[derive(Clone, Debug, Eq, Error, PartialEq)] +pub enum FinalFrontiersValidationError { + /// The bytes could not be parsed as [`FinalFrontiers`]. + #[error("invalid VCT final frontier bytes: {error}")] + InvalidBytes { + /// The parser error message. + error: String, + }, + + /// The serialized frontier height does not match the expected checkpoint handoff height. + #[error("embedded VCT final frontier height must match the network's max checkpoint height")] + HeightMismatch { + /// Height encoded in the serialized frontier. + actual: block::Height, + /// Expected checkpoint handoff height. + expected: block::Height, + }, +} + +/// POC state for the verified-commitment-trees experiment +/// (`docs/design/verified-commitment-trees.md`). Shared across +/// [`super::FinalizedState`] clones via `Arc` so the counters are shared. +/// +/// A checkpoint-trusting sync (`checkpoint_sync = true`) uses the peer `tree_aux` source by +/// default on networks with embedded final frontiers; `checkpoint_sync = false` or +/// `vct_fast_sync = false` opts out to the legacy per-block recompute (no VCT state). +#[derive(Debug)] +pub(crate) struct VctState { + /// Fast mode: skip the per-block frontier recompute and fold the source's roots + /// into the anchor set + history tree. + fast: bool, + /// Where the verified per-block roots and handoff frontiers come from. The + /// committer reads roots/handoff/frontiers through this seam only. + source: Box, + /// Whether roots from this VCT state must be confirmed against a buffered successor + /// before they are committed. + requires_verified_successor: bool, + /// Count of blocks that took the fast (skip-recompute) path, for the run summary. + fast_count: AtomicU64, + /// Count of fast blocks whose own commitment check was skipped because the + /// previous block's look-ahead already validated it (the dedup). Lets tests + /// assert the dedup actually engages, so it can't be silently regressed. + prevalidated_count: AtomicU64, +} + +/// Which commitment-root source the committer uses, resolved from the (already read) +/// configuration signals. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +enum SourceMode { + /// Legacy recompute committer (no VCT state). + Legacy, + /// Fetch per-block roots from peers — the default where embedded frontiers exist. + Peer, +} + +/// Resolve the source mode as a pure function, so the peer-source default is +/// unit-testable without touching embedded-frontier files. The fast verified path +/// (peer source) is the default whenever the node syncs under checkpoint trust and +/// the network has an embedded handoff frontier. `checkpoint_sync = false` or +/// `vct_fast_sync = false` selects the legacy recompute; a network with no embedded +/// frontier also falls back to legacy. Storage mode (Archive vs. Pruned) is orthogonal and not +/// an input here. +fn select_source_mode( + checkpoint_sync: bool, + vct_fast_sync: bool, + has_embedded_frontiers: bool, +) -> SourceMode { + if !checkpoint_sync || !vct_fast_sync || !has_embedded_frontiers { + SourceMode::Legacy + } else { + SourceMode::Peer + } +} + +impl VctState { + /// Build the committer state from `checkpoint_sync` (the mirror of + /// `consensus.checkpoint_sync`) and the `vct_fast_sync` knob. + /// On networks with an embedded handoff frontier (Mainnet) a checkpoint-trusting sync + /// defaults to the peer (`tree_aux`) fast source; disabling checkpoint sync, setting + /// `vct_fast_sync = false`, or using a network without an embedded frontier returns `None` for + /// a zero-overhead legacy committer that recomputes the trees per block. + pub(super) fn from_config( + checkpoint_sync: bool, + vct_fast_sync: bool, + network: &Network, + db: ZebraDb, + ) -> Option> { + // Parse the embedded handoff frontier once (None on networks without one, e.g. + // Testnet). The decision below only needs its presence; the peer arm reuses the + // parsed value. + let embedded = embedded_final_frontiers(network); + + match select_source_mode(checkpoint_sync, vct_fast_sync, embedded.is_some()) { + // Default: the peer (`tree_aux`) source on any network with embedded final + // frontiers (Mainnet). Per-block roots arrive from peers into a shared cache + // filled by the driver; the committer reads them per height and folds them in, + // skipping the recompute. A height the peer cannot supply — or any node with no + // serving peers — stays in legacy mode, bit-identical to a legacy committer by + // construction. + SourceMode::Peer => { + let parsed = embedded?; + tracing::info!( + handoff_height = parsed.height.0, + "VCT: peer (tree_aux) source enabled by default — roots fetched from peers" + ); + let source = PeerSource::new_with_db(db, Some(parsed)); + Some(Arc::new(VctState { + fast: true, + source: Box::new(source), + requires_verified_successor: true, + fast_count: AtomicU64::new(0), + prevalidated_count: AtomicU64::new(0), + })) + } + + // Legacy committer: full per-block recompute when checkpoint sync is disabled, the + // force-disable knob is set, or the network has no embedded frontiers. No VCT state, + // zero overhead. + SourceMode::Legacy => None, + } + } + + /// `true` when the fast (skip-recompute) path is active. + pub(super) fn is_fast(&self) -> bool { + self.fast + } + + /// The supplied roots for `height`, when vct mode has a source entry for it + /// (the signal that this block takes the fast path). + pub(super) fn vct_roots_at_height( + &self, + height: block::Height, + ) -> Option<(sapling::tree::Root, orchard::tree::Root)> { + if !self.fast { + return None; + } + + if self + .source + .vct_last_checkpoint_height() + .is_some_and(|handoff| height > handoff) + { + return None; + } + + self.source.vct_root(height) + } + + /// `true` when committing `height` on the vct path needs a buffered successor before + /// it can safely persist this block's supplied roots. + /// + /// Only untrusted peer-supplied roots at or above Heartwood require this. The + /// checkpoint handoff is exempt because its embedded final frontiers are verified + /// against this block's roots before the real tip treestate is written; trusted + /// local fixtures can commit their tip root on the in-arrears check. + pub(super) fn vct_root_needs_successor( + &self, + height: block::Height, + network: &Network, + ) -> bool { + self.fast + && self.vct_roots_at_height(height).is_some() + && self.requires_verified_successor + && self + .source + .final_frontiers() + .is_none_or(|frontiers| frontiers.height != height) + && Some(height) >= NetworkUpgrade::Heartwood.activation_height(network) + } + + /// Discard the supplied root for `height` after it failed verification, so a re-fetch + /// can replace it. See + /// [`CommitmentRootSource::invalidate`](super::commitment_aux::CommitmentRootSource::invalidate). + pub(super) fn invalidate_fast_root(&self, height: block::Height) { + self.source.invalidate(height); + } + + /// Discard peer-supplied roots that are no longer needed after `height` has committed. + pub(super) fn evict_committed_roots_through(&self, height: block::Height) { + self.source.evict_committed_through(height); + } + + /// The checkpoint handoff height: the boundary below which the fast path skips + /// per-height note-commitment trees. `None` unless final frontiers are loaded. + pub(super) fn vct_sync_last_checkpoint_height(&self) -> Option { + self.source.vct_last_checkpoint_height() + } + + /// The verified `(sapling, orchard, sprout)` frontiers to write as the tip + /// treestate, when `height` is the checkpoint handoff height. + #[allow(clippy::type_complexity)] + pub(super) fn final_frontiers_for_last_checkpoint( + &self, + height: block::Height, + ) -> Option<( + Arc, + Arc, + Arc, + )> { + self.source + .final_frontiers() + .filter(|f| f.height == height) + .map(|f| (f.sapling.clone(), f.orchard.clone(), f.sprout.clone())) + } + + /// Record that a block took the fast (skip-recompute) path. + pub(super) fn record_fast_block(&self) { + self.fast_count.fetch_add(1, Ordering::Relaxed); + } + + /// Record a fast block whose own commitment check was skipped by the dedup. + pub(super) fn record_prevalidated(&self) { + self.prevalidated_count.fetch_add(1, Ordering::Relaxed); + } + + /// Number of blocks that took the fast path so far. + pub(super) fn fast_count(&self) -> u64 { + self.fast_count.load(Ordering::Relaxed) + } + + /// Number of fast blocks whose own commitment check the dedup skipped. + #[cfg(test)] + pub(super) fn prevalidated_count(&self) -> u64 { + self.prevalidated_count.load(Ordering::Relaxed) + } + + /// Test-only: build fast-mode state from an arbitrary commitment-root source + /// (e.g. a payload produced from a database), so the producer→consumer round-trip + /// can be exercised without networking. + #[cfg(test)] + pub(super) fn test_with_source( + source: Box, + requires_verified_successor: bool, + ) -> Arc { + Arc::new(VctState { + fast: true, + source, + requires_verified_successor, + fast_count: AtomicU64::new(0), + prevalidated_count: AtomicU64::new(0), + }) + } +} + +/// The verified final frontiers embedded for `network`, if supported. +/// +/// Mainnet uses the constant embedded in the binary. Regtest has no fixed checkpoint — +/// its checkpoint list is derived at runtime from the mined chain — so there is no +/// committed frontier to embed; for deterministic e2e/integration testing of the fast +/// path on Regtest, the frontier is instead loaded from the file named by the +/// `VCT_REGTEST_FRONTIER` env var. This is scoped to **Regtest only** and validated +/// against the configured Regtest checkpoint height, so Mainnet always uses the +/// embedded constant and never reads the env. Other testnets have no frontier. +fn embedded_final_frontiers(network: &Network) -> Option { + match network { + Network::Mainnet => Some(parse_embedded_final_frontiers( + MAINNET_FINAL_FRONTIERS, + network.checkpoint_list().max_height(), + )), + Network::Testnet(params) if params.is_regtest() => { + let path = std::env::var_os("VCT_REGTEST_FRONTIER")?; + Some(load_frontier_file( + path.as_ref(), + network.checkpoint_list().max_height(), + )) + } + Network::Testnet(_) => None, + } +} + +/// Load and validate a final-frontier fixture file (the Regtest path; see +/// [`embedded_final_frontiers`]). Separated from the env read so it is unit-testable +/// without mutating process environment variables. +fn load_frontier_file(path: &std::ffi::OsStr, expected_height: block::Height) -> FinalFrontiers { + let bytes = + std::fs::read(path).expect("VCT_REGTEST_FRONTIER must name a readable final-frontier file"); + parse_embedded_final_frontiers(&bytes, expected_height) +} + +/// Parse embedded final frontiers and verify they match the checkpoint list. +fn parse_embedded_final_frontiers(bytes: &[u8], expected_height: block::Height) -> FinalFrontiers { + parse_final_frontiers_bytes(bytes, expected_height).unwrap_or_else(|error| panic!("{error}")) +} + +fn parse_final_frontiers_bytes( + bytes: &[u8], + expected_height: block::Height, +) -> Result { + let parsed = FinalFrontiers::from_bytes(bytes).map_err(|error| { + FinalFrontiersValidationError::InvalidBytes { + error: error.to_string(), + } + })?; + + if parsed.height != expected_height { + return Err(FinalFrontiersValidationError::HeightMismatch { + actual: parsed.height, + expected: expected_height, + }); + } + + Ok(parsed) +} + +/// Validate serialized VCT final-frontier bytes against an expected checkpoint handoff height. +pub fn validate_final_frontiers_bytes( + bytes: &[u8], + expected_height: block::Height, +) -> Result<(), FinalFrontiersValidationError> { + parse_final_frontiers_bytes(bytes, expected_height).map(|_| ()) +} + +/// Test/developer helper for producing embedded final-frontier bytes from a +/// legacy-computed tip treestate. +#[cfg(test)] +fn final_frontiers_bytes(height: block::Height, trees: &NoteCommitmentTrees) -> Vec { + FinalFrontiers { + height, + sapling: trees.sapling.clone(), + orchard: trees.orchard.clone(), + sprout: trees.sprout.clone(), + } + .to_bytes() +} + +#[cfg(test)] +mod tests { + use super::*; + + const EXPECTED_MAINNET_FINAL_SAPLING_ROOT: [u8; 32] = [ + 5, 88, 219, 64, 134, 21, 57, 124, 234, 59, 83, 8, 7, 143, 19, 29, 247, 58, 105, 80, 119, + 139, 242, 243, 206, 137, 211, 94, 151, 126, 154, 13, + ]; + const EXPECTED_MAINNET_FINAL_ORCHARD_ROOT: [u8; 32] = [ + 177, 173, 139, 203, 63, 186, 47, 172, 148, 107, 150, 204, 211, 212, 33, 155, 172, 108, 132, + 148, 70, 210, 120, 97, 219, 160, 58, 242, 198, 124, 44, 3, + ]; + const EXPECTED_MAINNET_FINAL_SPROUT_ROOT: [u8; 32] = [ + 77, 239, 224, 205, 90, 67, 51, 216, 15, 139, 120, 78, 55, 17, 177, 22, 246, 34, 206, 184, + 49, 7, 97, 172, 28, 178, 69, 208, 13, 101, 55, 169, + ]; + + #[test] + fn source_mode_precedence() { + use SourceMode::*; + // Args are (checkpoint_sync, vct_fast_sync, has_embedded_frontiers). + + // The default: a checkpoint-trusting sync with VCT fast sync on uses the peer source + // wherever embedded frontiers exist (Mainnet). Storage mode (Archive/Pruned) is not an + // input, so this covers both Archive and Pruned. + assert_eq!(select_source_mode(true, true, true), Peer); + // `vct_fast_sync = false` keeps checkpoint sync on but forces the legacy recompute, + // regardless of embedded frontiers. + assert_eq!(select_source_mode(true, false, true), Legacy); + assert_eq!(select_source_mode(true, false, false), Legacy); + // `checkpoint_sync = false` also fully recomputes the trees: legacy, never peer, + // regardless of the fast-sync knob or embedded frontiers. + assert_eq!(select_source_mode(false, true, true), Legacy); + assert_eq!(select_source_mode(false, true, false), Legacy); + assert_eq!(select_source_mode(false, false, true), Legacy); + assert_eq!(select_source_mode(false, false, false), Legacy); + // No embedded frontiers (e.g. Testnet): legacy, never peer, even under checkpoint sync. + assert_eq!(select_source_mode(true, true, false), Legacy); + } + + #[test] + fn successor_policy_is_vct_state_data() { + let network = Network::Mainnet; + let height = NetworkUpgrade::Heartwood + .activation_height(&network) + .expect("mainnet has a Heartwood activation height"); + let root_map = + || std::iter::once((height.0, (Default::default(), Default::default()))).collect(); + + let trusted = VctState::test_with_source( + Box::new(super::super::commitment_aux::FixtureSource::new( + root_map(), + None, + )), + false, + ); + assert!( + !trusted.vct_root_needs_successor(height, &network), + "trusted fixture roots can commit without a buffered successor" + ); + + let untrusted = VctState::test_with_source( + Box::new(super::super::commitment_aux::FixtureSource::new( + root_map(), + None, + )), + true, + ); + assert!( + untrusted.vct_root_needs_successor(height, &network), + "untrusted roots defer until a buffered successor verifies them" + ); + } + + #[test] + fn vct_root_is_bounded_by_handoff_height() { + let handoff = block::Height(10); + let after_handoff = (handoff + 1).expect("test height is valid"); + let roots = std::collections::HashMap::from([ + (handoff.0, (Default::default(), Default::default())), + (after_handoff.0, (Default::default(), Default::default())), + ]); + let frontiers = FinalFrontiers { + height: handoff, + sapling: Arc::new(sapling::tree::NoteCommitmentTree::default()), + orchard: Arc::new(orchard::tree::NoteCommitmentTree::default()), + sprout: Arc::new(sprout::tree::NoteCommitmentTree::default()), + }; + + let bounded = VctState::test_with_source( + Box::new(super::super::commitment_aux::FixtureSource::new( + roots.clone(), + Some(frontiers), + )), + false, + ); + assert!( + bounded.vct_roots_at_height(handoff).is_some(), + "the handoff root remains fast-path eligible" + ); + assert!( + bounded.vct_roots_at_height(after_handoff).is_none(), + "roots above the handoff are ignored" + ); + + let unbounded = VctState::test_with_source( + Box::new(super::super::commitment_aux::FixtureSource::new( + roots, None, + )), + false, + ); + assert!( + unbounded.vct_roots_at_height(after_handoff).is_some(), + "sources without a handoff keep the existing fixture behavior" + ); + } + + #[test] + fn embedded_mainnet_final_frontiers_parse() { + let frontiers = embedded_final_frontiers(&Network::Mainnet) + .expect("mainnet has embedded final frontiers"); + + assert_eq!( + frontiers.height, + Network::Mainnet.checkpoint_list().max_height(), + "embedded frontier is tied to the last mainnet checkpoint" + ); + assert_eq!( + <[u8; 32]>::from(frontiers.sapling.root()), + EXPECTED_MAINNET_FINAL_SAPLING_ROOT, + "embedded mainnet final Sapling frontier root is pinned" + ); + assert_eq!( + <[u8; 32]>::from(frontiers.orchard.root()), + EXPECTED_MAINNET_FINAL_ORCHARD_ROOT, + "embedded mainnet final Orchard frontier root is pinned" + ); + assert_eq!( + <[u8; 32]>::from(frontiers.sprout.root()), + EXPECTED_MAINNET_FINAL_SPROUT_ROOT, + "embedded mainnet final Sprout frontier root is pinned" + ); + } + + #[test] + fn final_frontiers_capture_helper_serializes_tip_trees() { + let height = block::Height(3_358_006); + let trees = NoteCommitmentTrees::default(); + + let parsed = FinalFrontiers::from_bytes(&final_frontiers_bytes(height, &trees)) + .expect("captured final frontiers should parse"); + + assert_eq!(parsed.height, height, "captured height round-trips"); + assert_eq!( + parsed.sapling.root(), + trees.sapling.root(), + "captured sapling frontier round-trips" + ); + assert_eq!( + parsed.orchard.root(), + trees.orchard.root(), + "captured orchard frontier round-trips" + ); + assert_eq!( + parsed.sprout.root(), + trees.sprout.root(), + "captured sprout frontier round-trips" + ); + } + + #[test] + #[should_panic(expected = "embedded VCT final frontier height must match")] + fn embedded_final_frontiers_reject_checkpoint_height_mismatch() { + let frontiers = FinalFrontiers { + height: block::Height(1), + sapling: Arc::new(Default::default()), + orchard: Arc::new(Default::default()), + sprout: Arc::new(Default::default()), + }; + + let _ = parse_embedded_final_frontiers(&frontiers.to_bytes(), block::Height(2)); + } + + #[test] + fn final_frontiers_parser_rejects_short_height() { + let error = + FinalFrontiers::from_bytes(&[0, 1, 2]).expect_err("short height should be rejected"); + + assert_eq!( + error.to_string(), + "missing final frontier height: expected 4 bytes, got 3" + ); + } + + #[test] + fn final_frontiers_parser_rejects_missing_tree_length() { + let bytes = block::Height(1).0.to_le_bytes(); + + let error = + FinalFrontiers::from_bytes(&bytes).expect_err("missing length should be rejected"); + + assert_eq!( + error.to_string(), + "missing sapling frontier length prefix at byte 4: expected 4 bytes, got 0" + ); + } + + #[test] + fn final_frontiers_parser_rejects_truncated_tree_blob() { + let mut bytes = block::Height(1).0.to_le_bytes().to_vec(); + bytes.extend_from_slice(&3u32.to_le_bytes()); + bytes.extend_from_slice(&[0, 1]); + + let error = + FinalFrontiers::from_bytes(&bytes).expect_err("truncated blob should be rejected"); + + assert_eq!( + error.to_string(), + "truncated sapling frontier blob at byte 8: length prefix says 3 bytes, but only 2 remain" + ); + } + + #[test] + fn final_frontiers_parser_rejects_trailing_bytes() { + let bytes = FinalFrontiers { + height: block::Height(1), + sapling: Arc::new(Default::default()), + orchard: Arc::new(Default::default()), + sprout: Arc::new(Default::default()), + } + .to_bytes() + .into_iter() + .chain([0]) + .collect::>(); + + let error = + FinalFrontiers::from_bytes(&bytes).expect_err("trailing bytes should be rejected"); + + assert_eq!( + error.to_string(), + format!( + "unexpected trailing final frontier bytes at byte {}: 1 bytes", + bytes.len() - 1 + ) + ); + } + + #[test] + #[should_panic(expected = "invalid VCT final frontier bytes: truncated sapling frontier blob")] + fn embedded_final_frontiers_reject_malformed_bytes_with_context() { + let mut bytes = block::Height(1).0.to_le_bytes().to_vec(); + bytes.extend_from_slice(&3u32.to_le_bytes()); + bytes.extend_from_slice(&[0, 1]); + + let _ = parse_embedded_final_frontiers(&bytes, block::Height(1)); + } + + #[test] + fn embedded_final_frontiers_are_network_specific() { + assert!( + embedded_final_frontiers(&Network::new_default_testnet()).is_none(), + "testnet has no embedded final frontier until VCT fast sync supports it" + ); + } + + /// The Regtest frontier-file loader (the `VCT_REGTEST_FRONTIER` path) round-trips a + /// captured frontier and ties it to the expected checkpoint height — exercising the + /// producer (`to_bytes`) → loader (`load_frontier_file`) seam without env vars. + #[test] + fn load_frontier_file_round_trips_a_captured_frontier() { + let height = block::Height(123); + let bytes = FinalFrontiers { + height, + sapling: Arc::new(Default::default()), + orchard: Arc::new(Default::default()), + sprout: Arc::new(Default::default()), + } + .to_bytes(); + + let path = + std::env::temp_dir().join(format!("vct-frontier-load-test-{}.bin", std::process::id())); + std::fs::write(&path, &bytes).expect("write temp frontier file"); + + let loaded = load_frontier_file(path.as_os_str(), height); + assert_eq!(loaded.height, height, "loaded frontier height matches"); + assert_eq!( + loaded.sapling.root(), + sapling::tree::NoteCommitmentTree::default().root(), + "loaded sapling frontier round-trips" + ); + + let _ = std::fs::remove_file(&path); + } + + /// A frontier whose height does not match the checkpoint height is rejected, so a + /// stale/wrong Regtest fixture cannot silently mis-seed the handoff. + #[test] + #[should_panic(expected = "embedded VCT final frontier height must match")] + fn load_frontier_file_rejects_height_mismatch() { + let bytes = FinalFrontiers { + height: block::Height(5), + sapling: Arc::new(Default::default()), + orchard: Arc::new(Default::default()), + sprout: Arc::new(Default::default()), + } + .to_bytes(); + let path = std::env::temp_dir().join(format!( + "vct-frontier-mismatch-test-{}.bin", + std::process::id() + )); + std::fs::write(&path, &bytes).expect("write temp frontier file"); + + let _ = load_frontier_file(path.as_os_str(), block::Height(6)); + } +} diff --git a/zebra-state/src/service/finalized_state/vct/mainnet-frontier.bin b/zebra-state/src/service/finalized_state/vct/mainnet-frontier.bin new file mode 100644 index 00000000000..eec96324fe9 Binary files /dev/null and b/zebra-state/src/service/finalized_state/vct/mainnet-frontier.bin differ diff --git a/zebra-state/src/service/finalized_state/zebra_db.rs b/zebra-state/src/service/finalized_state/zebra_db.rs index e7ba5325b2f..be72207dfca 100644 --- a/zebra-state/src/service/finalized_state/zebra_db.rs +++ b/zebra-state/src/service/finalized_state/zebra_db.rs @@ -175,11 +175,25 @@ impl ZebraDb { ) } + db.run_blocking_format_repairs(network); db.spawn_format_change(format_change); db } + /// Run synchronous compatibility repairs before background format checks can read the DB. + pub fn run_blocking_format_repairs(&self, network: &Network) { + if self.debug_skip_format_upgrades { + return; + } + + // Repair incompatible stored history-tree bytes before the background + // format-validity check can read and panic on them. Healthy databases are + // a no-op, and read-only/offline-tool opens keep their existing + // skip-upgrade behavior. + rollback::repair_tip_history_tree_if_incompatible(self, network); + } + /// Launch any required format changes or format checks, and store their thread handle. pub fn spawn_format_change(&mut self, format_change: DbFormatChange) { if self.debug_skip_format_upgrades { diff --git a/zebra-state/src/service/finalized_state/zebra_db/block.rs b/zebra-state/src/service/finalized_state/zebra_db/block.rs index 3177bee75ae..dc0cfc8bb37 100644 --- a/zebra-state/src/service/finalized_state/zebra_db/block.rs +++ b/zebra-state/src/service/finalized_state/zebra_db/block.rs @@ -75,11 +75,29 @@ struct AdvertisedBodySize(u32); /// full per-height trees. #[derive(Clone, Copy, Debug, Eq, PartialEq)] pub struct VctData { - /// Roots to insert into the anchor set instead of writing per-height trees. - pub(in super::super) anchor_roots: (sapling::tree::Root, orchard::tree::Root), + /// When `Some`, skip per-height tree writes and fold these roots into the + /// anchor set instead. `None` on the checkpoint-handoff block, which writes + /// the real final frontier through the legacy tree path. + pub(in super::super) anchor_roots: Option<(sapling::tree::Root, orchard::tree::Root)>, + + /// When `Some`, mark the database as vct-synced: per-height trees are + /// absent below this height. `None` for a non-persistent fast sync (no + /// embedded final frontiers loaded). + pub(in super::super) sync_below: Option, +} - /// Height below which per-height trees are absent in a VCT-synced database. - pub(in super::super) sync_below: Height, +impl VctData { + /// Bundles the per-commit VCT data, or returns `None` for a pure legacy + /// commit with no VCT work. + pub(in super::super) fn new( + anchor_roots: Option<(sapling::tree::Root, orchard::tree::Root)>, + sync_below: Option, + ) -> Option { + (anchor_roots.is_some() || sync_below.is_some()).then_some(Self { + anchor_roots, + sync_below, + }) + } } impl AdvertisedBodySize { @@ -1033,6 +1051,7 @@ impl ZebraDb { /// - Propagates any errors from computing the block's chain value balance change or /// from applying the change to the chain value balance #[allow(clippy::unwrap_in_result)] + #[allow(clippy::too_many_arguments)] pub(in super::super) fn write_block( &mut self, finalized: FinalizedBlock, diff --git a/zebra-state/src/service/finalized_state/zebra_db/block/tests/prune.rs b/zebra-state/src/service/finalized_state/zebra_db/block/tests/prune.rs index 6c24d19b301..536b8dd52eb 100644 --- a/zebra-state/src/service/finalized_state/zebra_db/block/tests/prune.rs +++ b/zebra-state/src/service/finalized_state/zebra_db/block/tests/prune.rs @@ -49,7 +49,7 @@ fn new_state_with_blocks(config: &Config, network: &Network) -> FinalizedState { .expect("test data deserializes"); state - .commit_finalized_direct(block.into(), None, "prune tests") + .commit_finalized_direct(block.into(), None, None, "prune tests") .expect("test block is valid"); } @@ -80,7 +80,7 @@ fn new_state_with_checkpoint_retention( .expect("test data deserializes"); state - .commit_finalized_direct(block.into(), None, "checkpoint retention tests") + .commit_finalized_direct(block.into(), None, None, "checkpoint retention tests") .expect("test block is valid"); } @@ -351,7 +351,7 @@ fn checkpoint_retention_hands_off_to_online_pruning_at_start() { .expect("test data deserializes"); state - .commit_finalized_direct(block.into(), None, "checkpoint handoff tests") + .commit_finalized_direct(block.into(), None, None, "checkpoint handoff tests") .expect("test block is valid"); } @@ -386,7 +386,7 @@ fn checkpoint_retention_hands_off_to_online_pruning_at_start() { .expect("test data deserializes"); state - .commit_finalized_direct(block.into(), None, "checkpoint handoff tests") + .commit_finalized_direct(block.into(), None, None, "checkpoint handoff tests") .expect("handoff block is valid"); let online_prune_until = @@ -630,7 +630,7 @@ fn archive_to_pruned_checkpoint_sync_drains_archive_raw_transactions_before_skip .expect("test data deserializes"); archive_state - .commit_finalized_direct(block.into(), None, "archive phase") + .commit_finalized_direct(block.into(), None, None, "archive phase") .expect("archive block is valid"); } @@ -670,7 +670,7 @@ fn archive_to_pruned_checkpoint_sync_drains_archive_raw_transactions_before_skip .expect("test data deserializes"); pruned_state - .commit_finalized_direct(block.into(), None, "archive to pruned checkpoint") + .commit_finalized_direct(block.into(), None, None, "archive to pruned checkpoint") .expect("checkpoint block is valid"); assert_eq!( @@ -728,7 +728,7 @@ fn archive_backlog_flag_is_recomputed_when_reopening_a_pruned_database() { .expect("test data deserializes"); archive_state - .commit_finalized_direct(block.into(), None, "archive phase") + .commit_finalized_direct(block.into(), None, None, "archive phase") .expect("archive block is valid"); } std::mem::drop(archive_state); @@ -761,7 +761,7 @@ fn archive_backlog_flag_is_recomputed_when_reopening_a_pruned_database() { .zcash_deserialize_into() .expect("test data deserializes"); pruned_state - .commit_finalized_direct(block.into(), None, "archive to pruned checkpoint") + .commit_finalized_direct(block.into(), None, None, "archive to pruned checkpoint") .expect("checkpoint block is valid"); assert_eq!( pruned_state.db.lowest_retained_height(), @@ -842,7 +842,7 @@ fn contextual_commits_keep_raw_transactions_before_checkpoint_retention_start() .zcash_deserialize_into() .expect("genesis test data deserializes"); state - .commit_finalized_direct(genesis.into(), None, "contextual retention tests") + .commit_finalized_direct(genesis.into(), None, None, "contextual retention tests") .expect("genesis block is valid"); let block: Arc = blocks @@ -858,7 +858,7 @@ fn contextual_commits_keep_raw_transactions_before_checkpoint_retention_start() let finalizable = FinalizableBlock::new(contextually_verified, Treestate::default()); state - .commit_finalized_direct(finalizable, None, "contextual retention tests") + .commit_finalized_direct(finalizable, None, None, "contextual retention tests") .expect("contextual block is valid"); assert!( @@ -1197,7 +1197,8 @@ fn reopening_fast_synced_database_in_pruned_mode_with_vct_disabled_succeeds() { } #[test] -fn reopening_interrupted_fast_sync_without_a_root_source_succeeds() { +#[should_panic(expected = "interrupted below the checkpoint handoff")] +fn reopening_interrupted_fast_sync_without_a_root_source_panics() { let _init_guard = zebra_test::init(); let network = Mainnet; @@ -1220,23 +1221,19 @@ fn reopening_interrupted_fast_sync_without_a_root_source_succeeds() { state.db.write_batch(batch).expect("marker batch writes"); } - // This storage-only PR does not enable the fast path or its resume policy. Reopening preserves - // the marker so later fast-sync wiring can decide how to resume or repair. - let reopened = FinalizedState::new( + // Reopening with the fast path disabled must refuse: the on-disk frontier is stale and no + // root source exists, so the committer would otherwise stall on every below-handoff block. + let _state = FinalizedState::new( &config, &network, #[cfg(feature = "elasticsearch")] false, ); - assert_eq!( - reopened.db.vct_synced_below(), - Some(Height(100)), - "the interrupted fast-sync marker is preserved across reopen" - ); } #[test] -fn reopening_interrupted_fast_sync_with_vct_disabled_succeeds() { +#[should_panic(expected = "interrupted below the checkpoint handoff")] +fn reopening_interrupted_fast_sync_with_vct_disabled_panics() { let _init_guard = zebra_test::init(); let network = Mainnet; @@ -1259,19 +1256,14 @@ fn reopening_interrupted_fast_sync_with_vct_disabled_succeeds() { state.db.write_batch(batch).expect("marker batch writes"); } - // The force-disable knob only affects the future fast-sync source selection. The database still - // opens and keeps the marker available for follow-up resume or repair logic. - let reopened = FinalizedState::new( + // Reopening with the VCT force-disable knob must refuse: the on-disk frontier is stale and + // no root source exists, so the committer would otherwise stall on every below-handoff block. + let _state = FinalizedState::new( &config, &network, #[cfg(feature = "elasticsearch")] false, ); - assert_eq!( - reopened.db.vct_synced_below(), - Some(Height(100)), - "the interrupted fast-sync marker is preserved across reopen" - ); } #[test] diff --git a/zebra-state/src/service/finalized_state/zebra_db/block/tests/snapshot.rs b/zebra-state/src/service/finalized_state/zebra_db/block/tests/snapshot.rs index 82a214099c6..b875ed9903d 100644 --- a/zebra-state/src/service/finalized_state/zebra_db/block/tests/snapshot.rs +++ b/zebra-state/src/service/finalized_state/zebra_db/block/tests/snapshot.rs @@ -195,7 +195,7 @@ fn test_block_and_transaction_data_with_network(network: Network) { .expect("test data deserializes"); state - .commit_finalized_direct(block.into(), None, "snapshot tests") + .commit_finalized_direct(block.into(), None, None, "snapshot tests") .expect("test block is valid"); let mut settings = insta::Settings::clone_current(); diff --git a/zebra-state/src/service/finalized_state/zebra_db/chain.rs b/zebra-state/src/service/finalized_state/zebra_db/chain.rs index e7cadb8bc04..ff225f04257 100644 --- a/zebra-state/src/service/finalized_state/zebra_db/chain.rs +++ b/zebra-state/src/service/finalized_state/zebra_db/chain.rs @@ -29,7 +29,7 @@ use zebra_chain::{ use crate::{ request::FinalizedBlock, service::finalized_state::{ - disk_db::DiskWriteBatch, + disk_db::{DiskWriteBatch, ReadDisk}, disk_format::{chain::HistoryTreeParts, RawBytes}, zebra_db::{metrics::value_pool_metrics, ZebraDb}, TypedColumnFamily, @@ -157,6 +157,40 @@ impl ZebraDb { Arc::new(HistoryTree::from(history_tree)) } + /// Returns `Ok(())` if the stored tip history tree decodes with the current + /// `HistoryTreeParts` format. + /// + /// This is a non-panicking compatibility probe used during DB open before + /// background format checks can call [`Self::history_tree`]. It reads raw + /// bytes and performs the same current-key then legacy-key fallback as + /// [`Self::history_tree`]. + pub(crate) fn check_tip_history_tree_decodes(&self) -> Result<(), String> { + let history_tree_cf = self + .db + .cf_handle(HISTORY_TREE) + .expect("column family was created when database was created"); + + let raw_parts: Option = self.db.zs_get(&history_tree_cf, &()); + let raw_parts = raw_parts.or_else(|| { + self.db + .zs_last_key_value::<_, RawBytes, RawBytes>(&history_tree_cf) + .map(|(_height_key, tree_value)| tree_value) + }); + + let Some(raw_parts) = raw_parts else { + return Ok(()); + }; + + let parts = HistoryTreeParts::from_bytes_result(raw_parts.raw_bytes()) + .map_err(|error| format!("stored history tree does not deserialize: {error}"))?; + + parts + .with_network(&self.db.network()) + .map_err(|error| format!("stored history tree is invalid for this network: {error}"))?; + + Ok(()) + } + /// Returns all the history tip trees. /// We only store the history tree for the tip, so this method is only used in tests and /// upgrades. diff --git a/zebra-state/src/service/finalized_state/zebra_db/prune.rs b/zebra-state/src/service/finalized_state/zebra_db/prune.rs index 42813e35378..e32c54f7988 100644 --- a/zebra-state/src/service/finalized_state/zebra_db/prune.rs +++ b/zebra-state/src/service/finalized_state/zebra_db/prune.rs @@ -355,7 +355,7 @@ mod tests { .expect("test data deserializes"); state - .commit_finalized_direct(block.into(), None, "offline prune tests") + .commit_finalized_direct(block.into(), None, None, "offline prune tests") .expect("test block is valid"); } diff --git a/zebra-state/src/service/finalized_state/zebra_db/rollback.rs b/zebra-state/src/service/finalized_state/zebra_db/rollback.rs index e37bee5bf55..d357a614d70 100644 --- a/zebra-state/src/service/finalized_state/zebra_db/rollback.rs +++ b/zebra-state/src/service/finalized_state/zebra_db/rollback.rs @@ -553,6 +553,61 @@ fn block_has_sprout_commitments(block: &Block) -> bool { block.sprout_note_commitments().next().is_some() } +/// Blocking DB-open repair for an incompatible stored `history_tree`. +/// +/// If the stored tip history tree cannot be decoded by the current code, rebuild it from the +/// finalized blocks plus Sapling/Orchard roots using the same algorithm rollback uses, then write +/// it back before the background format-validity check can read the old value. +/// +/// The roots come from the compact per-height root index when present, so post-index VCT +/// fast-synced databases can be repaired without historical tree rows. Pre-index archive +/// databases fall back to deriving roots from per-height trees. Databases missing both sources +/// fail closed with remediation instead of attempting a partial rebuild. +pub(crate) fn repair_tip_history_tree_if_incompatible(db: &ZebraDb, network: &Network) { + let Some(tip_height) = db.finalized_tip_height() else { + return; + }; + // Pre-Heartwood (no history tree) needs no repair. + if NetworkUpgrade::current(network, tip_height) < NetworkUpgrade::Heartwood { + return; + } + // Healthy DBs (the common case) decode fine: no-op, no rebuild. + if let Err(error) = db.check_tip_history_tree_decodes() { + tracing::warn!( + ?tip_height, + ?error, + "stored history tree is incompatible with this binary; rebuilding it from finalized \ + blocks and commitment roots before startup" + ); + } else { + return; + } + + match rebuild_history_tree_from_upgrade_activation(db, network, tip_height) { + Ok(rebuilt) => { + let mut batch = DiskWriteBatch::new(); + batch.update_history_tree(db, &rebuilt); + db.write_batch(batch) + .expect("history-tree repair batch write should succeed"); + tracing::info!( + ?tip_height, + history_root = ?rebuilt.hash(), + "history-tree repair complete; rebuilt tip tree written in the current format" + ); + } + Err(error) => { + panic!( + "cannot repair the incompatible history tree at tip {tip_height:?}: {error}. \ + The repair requires finalized block bodies plus Sapling/Orchard roots from the \ + current network-upgrade activation height through the tip. Roots can come from \ + `commitment_roots_by_height` or from per-height trees. If this database predates \ + the root index and is VCT fast-synced or pruned, re-sync from genesis or repair \ + from an archive-capable database." + ); + } + } +} + fn rebuild_history_tree_from_upgrade_activation( db: &ZebraDb, network: &Network, @@ -1394,7 +1449,7 @@ mod tests { db.write_batch(batch) .expect("seeding the serving index succeeds"); - let served = db.commitment_roots_by_height_range(Height(4)..=Height(6)); + let served = crate::service::finalized_state::serve_block_roots(&db, Height(4)..=Height(6)); assert_eq!( served .into_iter() diff --git a/zebra-state/src/service/finalized_state/zebra_db/shielded.rs b/zebra-state/src/service/finalized_state/zebra_db/shielded.rs index 8422c7ac935..65836f8c1dd 100644 --- a/zebra-state/src/service/finalized_state/zebra_db/shielded.rs +++ b/zebra-state/src/service/finalized_state/zebra_db/shielded.rs @@ -818,8 +818,8 @@ impl DiskWriteBatch { // below the checkpoint handoff height). Written in the same atomic batch as // every vct commit, so a vct-synced database always carries the marker and // the read/validity guards never see absent trees without it. - if let Some(VctData { sync_below, .. }) = vct_data { - self.update_vct_sync_marker(zebra_db, sync_below); + if let Some(handoff) = vct_data.and_then(|vct| vct.sync_below) { + self.update_vct_sync_marker(zebra_db, handoff); } // POC (verified-commitment-trees) vct path: the committer skipped the @@ -829,15 +829,7 @@ impl DiskWriteBatch { // tree CFs and subtrees entirely. The Sprout tree is unchanged below any // modern checkpoint, so it is correctly left untouched here. // See docs/design/verified-commitment-trees.md. - if let Some(VctData { - anchor_roots: (sapling_root, orchard_root), - sync_below, - }) = vct_data - { - // Mark the database as vct-synced in the same atomic batch as every - // fast commit, so the read/validity guards never see absent trees - // without the handoff marker. - self.update_vct_sync_marker(zebra_db, sync_below); + if let Some((sapling_root, orchard_root)) = vct_data.and_then(|vct| vct.anchor_roots) { self.insert_sapling_anchor(zebra_db, &sapling_root); self.insert_orchard_anchor(zebra_db, &orchard_root); // Persist the per-height roots into the serving index even though no per-height diff --git a/zebra-state/src/service/non_finalized_state.rs b/zebra-state/src/service/non_finalized_state.rs index 337cf01ff1c..1207d3e40d1 100644 --- a/zebra-state/src/service/non_finalized_state.rs +++ b/zebra-state/src/service/non_finalized_state.rs @@ -637,6 +637,7 @@ impl NonFinalizedState { block, &network, &history_tree, + // The non-finalized path doesn't precompute the auth data root. None, )); }); diff --git a/zebra-state/src/service/tests.rs b/zebra-state/src/service/tests.rs index 39f75203c36..f598174bbaf 100644 --- a/zebra-state/src/service/tests.rs +++ b/zebra-state/src/service/tests.rs @@ -28,8 +28,9 @@ use crate::{ arbitrary::Prepare, init_test, service::{ - arbitrary::populated_state, chain_tip::TipAction, headers_by_height_range, - non_finalized_state::Chain, StateService, + arbitrary::populated_state, block_roots_by_height_range, chain_tip::TipAction, + headers_by_height_range, non_finalized_state::Chain, root_covered_best_header_tip, + StateService, }, tests::{ setup::{partial_nu5_chain_strategy, transaction_v4_from_coinbase}, @@ -41,17 +42,24 @@ use crate::{ const LAST_BLOCK_HEIGHT: u32 = 10; -fn roots_from_height(start: Height, count: u32) -> Vec { +fn root_at(height: Height) -> BlockCommitmentRoots { + BlockCommitmentRoots { + height, + sapling_root: sapling::tree::NoteCommitmentTree::default().root(), + orchard_root: orchard::tree::NoteCommitmentTree::default().root(), + ironwood_root: zebra_chain::ironwood::tree::NoteCommitmentTree::default().root(), + sapling_tx: 0, + orchard_tx: 0, + ironwood_tx: 0, + auth_data_root: zebra_chain::block::merkle::AuthDataRoot::from([0u8; 32]), + } +} + +fn roots_from_height(start_height: Height, count: usize) -> Vec { (0..count) - .map(|offset| BlockCommitmentRoots { - height: Height(start.0 + offset), - sapling_root: sapling::tree::NoteCommitmentTree::default().root(), - orchard_root: orchard::tree::NoteCommitmentTree::default().root(), - ironwood_root: zebra_chain::ironwood::tree::NoteCommitmentTree::default().root(), - sapling_tx: 0, - orchard_tx: 0, - ironwood_tx: 0, - auth_data_root: zebra_chain::block::merkle::AuthDataRoot::from([0u8; 32]), + .map(|offset| { + let offset = u32::try_from(offset).expect("test root count fits in u32"); + root_at(Height(start_height.0 + offset)) }) .collect() } @@ -663,6 +671,49 @@ async fn header_only_service_requests_preserve_body_boundary() -> std::result::R Ok(()) } +#[tokio::test(flavor = "multi_thread")] +async fn commit_header_range_rejects_missing_tree_aux_roots() -> std::result::Result<(), BoxError> { + let _init_guard = zebra_test::init(); + let network = Network::Mainnet; + let (state_service, _read_state, _, _) = + StateService::new(Config::ephemeral(), &network, Height::MAX, 0).await; + let genesis = + zebra_test::vectors::BLOCK_MAINNET_GENESIS_BYTES.zcash_deserialize_into::>()?; + let block1 = + zebra_test::vectors::BLOCK_MAINNET_1_BYTES.zcash_deserialize_into::>()?; + + let state = Buffer::new(BoxService::new(state_service), 1); + assert_eq!( + state + .clone() + .oneshot(Request::CommitCheckpointVerifiedBlock( + CheckpointVerifiedBlock::from(genesis.clone()), + )) + .await?, + Response::Committed(genesis.hash()), + ); + + let error = state + .oneshot(Request::CommitHeaderRange { + anchor: genesis.hash(), + headers: vec![block1.header.clone()], + body_sizes: vec![0], + tree_aux_roots: Vec::new(), + }) + .await + .expect_err("missing roots must reject a non-empty header range"); + + assert!(matches!( + error.downcast_ref::(), + Some(crate::CommitHeaderRangeError::TreeAuxRootCountMismatch { + headers: 1, + roots: 0, + }) + )); + + Ok(()) +} + /// A node still in the finalized (checkpoint) write phase must be able to commit /// a Zakura header range. /// @@ -776,9 +827,11 @@ async fn header_range_reads_include_non_finalized_best_chain_blocks() -> Result< chain = chain.push(block1.clone().prepare().test_with_zero_spent_utxos())?; chain = chain.push(block2.clone().prepare().test_with_zero_spent_utxos())?; + let chain = Arc::new(chain); + assert_eq!( headers_by_height_range( - Some(Arc::new(chain)), + Some(chain.clone()), &state_service.read_service.db, start, 2, @@ -788,6 +841,45 @@ async fn header_range_reads_include_non_finalized_best_chain_blocks() -> Result< (start.next().unwrap(), block2_hash, block2.header.clone()), ], ); + let roots = block_roots_by_height_range(Some(chain), &state_service.read_service.db, start, 2); + assert_eq!(roots.len(), 2); + assert_eq!(roots[0].height, start); + assert_eq!(roots[1].height, start.next().unwrap()); + let verified_tip = ((start - 1).unwrap(), block::Hash([0; 32])); + let best_header_tip = (start.next().unwrap(), block2_hash); + assert_eq!( + root_covered_best_header_tip( + None::>, + &state_service.read_service.db, + Some(best_header_tip), + Some(verified_tip), + ), + Some(verified_tip), + "rootless durable header tips are capped to the verified block tip" + ); + assert_eq!( + root_covered_best_header_tip( + Some(Arc::new( + Chain::new( + &network, + (start - 1).unwrap(), + Default::default(), + Default::default(), + Default::default(), + Default::default(), + Default::default(), + ValueBalance::fake_populated_pool(), + ) + .push(block1.prepare().test_with_zero_spent_utxos())? + .push(block2.prepare().test_with_zero_spent_utxos())?, + )), + &state_service.read_service.db, + Some(best_header_tip), + Some(verified_tip), + ), + Some(best_header_tip), + "verified non-finalized roots allow the header tip to stay ahead" + ); assert_eq!( headers_by_height_range(None::>, &state_service.read_service.db, start, 2), Vec::new(), diff --git a/zebra-state/src/service/write.rs b/zebra-state/src/service/write.rs index fceafbf42d7..11dfd40b3ee 100644 --- a/zebra-state/src/service/write.rs +++ b/zebra-state/src/service/write.rs @@ -4,7 +4,7 @@ use std::{ collections::VecDeque, path::{Path, PathBuf}, sync::Arc, - time::Duration, + time::{Duration, Instant}, }; use indexmap::IndexMap; @@ -16,7 +16,7 @@ use tokio::sync::{ use tracing::Span; use zebra_chain::{ block::{self, Height}, - parallel::commitment_aux::BlockCommitmentRoots, + parallel::{commitment_aux::BlockCommitmentRoots, tree::NoteCommitmentTrees}, }; use crate::{ @@ -39,6 +39,22 @@ use crate::service::{ non_finalized_state::Chain, }; +/// Delay between retryable VCT root-miss commit attempts while the peer cache refills. +const VCT_ROOT_RETRY_WAIT: Duration = Duration::from_millis(500); + +/// Delay between retryable VCT await-successor commit attempts. Shorter than +/// [`VCT_ROOT_RETRY_WAIT`]: the root is already cached and only the next block needs to be +/// downloaded into the look-ahead, so a tighter poll keeps the one-block commit lag small. +const VCT_AWAIT_SUCCESSOR_WAIT: Duration = Duration::from_millis(20); + +/// How long a single checkpoint height may stay stuck on a retryable VCT root stall before +/// the committer escalates to an error-level log and a `state.vct.root.stalled.height` gauge. +/// Transient waits (a successor still downloading, a root still in flight) clear well within +/// this; staying stuck past it means no peer can serve a root the frozen frontier requires, +/// and — by design — the committer will not recompute against the stale frontier, so the node +/// cannot advance until a peer supplies it. Surfacing that loudly is the operator's only signal. +const VCT_ROOT_STALL_WARN_AFTER: Duration = Duration::from_secs(30); + /// The maximum size of the parent error map. /// /// We allow enough space for multiple concurrent chain forks with errors. @@ -220,7 +236,7 @@ impl From for NonFinalizedWriteMessage { /// `finalized_state` or `non_finalized_state` and channels for sending /// it blocks. #[derive(Clone, Debug)] -pub(super) struct BlockWriteSender { +pub struct BlockWriteSender { /// A channel to send blocks to the `block_write_task`, /// so they can be written to the [`NonFinalizedState`]. pub non_finalized: Option>, @@ -328,9 +344,20 @@ impl WriteBlockWorkerTask { backup_dir_path, } = &mut self; - let mut prev_finalized_note_commitment_trees = None; + let mut prev_finalized_note_commitment_trees: Option = None; let mut deferred_non_finalized_messages = VecDeque::new(); + // One-block look-ahead buffers the next checkpoint block so VCT fast-path + // roots can be authenticated by the successor header before being trusted. + let mut finalized_lookahead: VecDeque = VecDeque::new(); + let mut retry_finalized_block: Option = None; + + // Tracks how long the committer has been stuck retrying a single VCT root stall, so a + // genuine stall (no peer can serve a frozen-frontier height) escalates to a loud, + // observable signal while a transient wait stays quiet. `(height, first-seen)`. + let mut vct_root_stall: Option<(Height, Instant)> = None; + let mut vct_root_stall_logged = false; + // Write all the finalized blocks sent by the state, // until the state closes the finalized block channel's sender. loop { @@ -357,13 +384,19 @@ impl WriteBlockWorkerTask { Err(TryRecvError::Disconnected) => {} } - let ordered_block = match finalized_block_write_receiver.try_recv() { - Ok(block) => block, - Err(TryRecvError::Empty) => { - std::thread::park_timeout(Duration::from_millis(10)); - continue; - } - Err(TryRecvError::Disconnected) => break, + let ordered_block = match retry_finalized_block + .take() + .or_else(|| finalized_lookahead.pop_front()) + { + Some(block) => block, + None => match finalized_block_write_receiver.try_recv() { + Ok(block) => block, + Err(TryRecvError::Empty) => { + std::thread::park_timeout(Duration::from_millis(10)); + continue; + } + Err(TryRecvError::Disconnected) => break, + }, }; // TODO: split these checks into separate functions @@ -394,22 +427,152 @@ impl WriteBlockWorkerTask { Assuming a parent block failed, and dropping this block", ); + // The pipeline is broken; drop any buffered look-ahead so the next + // commit re-seeds from the real tip. + finalized_lookahead.clear(); + finalized_state.clear_vct_prevalidated_next(); + // We don't want to send a reset here, because it could overwrite a valid sent hash std::mem::drop(ordered_block); continue; } - // Try committing the block - match finalized_state - .commit_finalized(ordered_block, prev_finalized_note_commitment_trees.take()) + // Peek the next block so VCT can authenticate this block's supplied + // roots against its successor before committing them. + if finalized_lookahead.is_empty() { + if let Ok(next) = finalized_block_write_receiver.try_recv() { + finalized_lookahead.push_back(next); + } + } + + // A non-handoff VCT fast block's supplied roots are authenticated by + // its successor's header. If the successor is not buffered yet, keep + // this block local and wait instead of surfacing a checkpoint commit + // error through the invalid-block reset path. + if finalized_lookahead.is_empty() + && finalized_state.vct_fast_needs_successor(ordered_block.0.height) { + tracing::trace!( + height = ?ordered_block.0.height, + hash = ?ordered_block.0.hash, + "VCT: deferring fast checkpoint commit until successor is buffered" + ); + retry_finalized_block = Some(ordered_block); + std::thread::park_timeout(Duration::from_millis(10)); + continue; + } + + // The buffered successor (if any) lets the committer verify this block's + // verified-commitment-trees fixture roots before trusting them: a block's + // roots are only committed by the next block's header. Its auth data root + // is already precomputed by the checkpoint verifier. + let next_checkpoint = finalized_lookahead + .front() + .map(|next| (next.0.block.clone(), next.0.auth_data_root)); + let prev_note_commitment_trees = prev_finalized_note_commitment_trees.take(); + let prev_note_commitment_trees_for_retry = prev_note_commitment_trees.clone(); + + let next_block_took_vct_path = + finalized_state.vct_fast_will_apply(ordered_block.0.height); + + // Try committing the block + match finalized_state.commit_finalized( + ordered_block, + prev_note_commitment_trees, + next_checkpoint, + ) { Ok((finalized, note_commitment_trees)) => { + // Whether this successful commit consumed header-carried + // tree-aux roots to skip the note-commitment frontier rebuild. + if next_block_took_vct_path { + metrics::counter!("state.vct.fast_path.hit").increment(1); + } else { + metrics::counter!("state.vct.fast_path.miss").increment(1); + } + + // A successful commit clears any VCT root stall: log recovery and reset + // the stalled-height gauge if it had been raised. + if vct_root_stall.is_some() { + if vct_root_stall_logged { + info!( + stalled_height = ?vct_root_stall.map(|(h, _)| h), + "VCT: checkpoint commit recovered; the stalled height now has a verifiable supplied root" + ); + metrics::gauge!("state.vct.root.stalled.height").set(0.0); + } + vct_root_stall = None; + vct_root_stall_logged = false; + } + let tip_block = ChainTipBlock::from(finalized); prev_finalized_note_commitment_trees = Some(note_commitment_trees); chain_tip_sender.set_finalized_tip(tip_block); } - Err(error) => { + Err((ordered_block, error)) => { + // Retryable VCT root stalls (an absent/evicted root, or one not yet + // verifiable for lack of a buffered successor) park-and-retry the same + // block in place rather than resetting the queue. An absent root waits + // for header sync to deliver it; an await-successor stall just waits for + // the next block to be downloaded into the look-ahead, so it polls faster. + if let Some(height) = error.vct_retryable_height() { + metrics::counter!("state.vct.root.retry.count").increment(1); + let needs_refetch = error.vct_supplied_root_unavailable_height(); + + // Escalate a stall that persists on the same height past the warn + // threshold: a transient wait resolves in a few polls and stays + // quiet, but a height stuck longer means no peer can serve a root the + // frozen frontier requires — the node will not advance (it will not, + // by design, recompute against the stale frontier). Surface it loudly. + match vct_root_stall { + Some((stuck, _)) if stuck == height => {} + _ => { + vct_root_stall = Some((height, Instant::now())); + vct_root_stall_logged = false; + } + } + if !vct_root_stall_logged + && vct_root_stall.is_some_and(|(_, since)| { + since.elapsed() >= VCT_ROOT_STALL_WARN_AFTER + }) + { + tracing::error!( + ?height, + awaiting_refetch = needs_refetch.is_some(), + stalled_for = ?VCT_ROOT_STALL_WARN_AFTER, + "VCT: checkpoint commit stalled with no verifiable supplied root; \ + the node cannot advance until a peer serves this height (it will \ + not recompute against the frozen frontier)" + ); + metrics::gauge!("state.vct.root.stalled.height") + .set(f64::from(height.0)); + vct_root_stall_logged = true; + } else { + tracing::warn!( + ?height, + block_height = ?ordered_block.0.height, + block_hash = ?ordered_block.0.hash, + awaiting_refetch = needs_refetch.is_some(), + "VCT: supplied root not yet verifiable; retrying checkpoint commit in place" + ); + } + + prev_finalized_note_commitment_trees = prev_note_commitment_trees_for_retry; + retry_finalized_block = Some(ordered_block); + std::thread::park_timeout(if needs_refetch.is_some() { + VCT_ROOT_RETRY_WAIT + } else { + VCT_AWAIT_SUCCESSOR_WAIT + }); + continue; + } + let finalized_tip = finalized_state.db.tip(); + let _ = ordered_block.1.send(Err(error.clone())); + + // The commit failed and the queue is being reset, so clear + // the buffered successor. + finalized_lookahead.clear(); + finalized_state.clear_vct_prevalidated_next(); // The last block in the queue failed, so we can't commit the next block. // Instead, we need to reset the state queue, @@ -581,7 +744,7 @@ impl WriteBlockWorkerTask { tracing::trace!("finalizing block past the reorg limit"); let contextually_verified_with_trees = non_finalized_state.finalize(); prev_finalized_note_commitment_trees = finalized_state - .commit_finalized_direct(contextually_verified_with_trees, prev_finalized_note_commitment_trees.take(), "commit contextually-verified request") + .commit_finalized_direct(contextually_verified_with_trees, prev_finalized_note_commitment_trees.take(), None, "commit contextually-verified request") .expect( "unexpected finalized block commit error: note commitment and history trees were already checked by the non-finalized state", ).1.into(); diff --git a/zebra-state/src/tests/setup.rs b/zebra-state/src/tests/setup.rs index 7c4c4a0bd6c..34b1785a84d 100644 --- a/zebra-state/src/tests/setup.rs +++ b/zebra-state/src/tests/setup.rs @@ -113,7 +113,7 @@ pub(crate) fn new_state_with_mainnet_genesis( let genesis = CheckpointVerifiedBlock::from(genesis); finalized_state - .commit_finalized_direct(genesis.clone().into(), None, "test") + .commit_finalized_direct(genesis.clone().into(), None, None, "test") .expect("unexpected invalid genesis block test vector"); assert_eq!(