feat(state): verified commitment trees fast checkpoint sync (state layer)

CHANGELOG.md

@@ -151,6 +151,24 @@ and this project adheres to [Semantic Versioning](https://semver.org).
   duplicate-peer handling scaffolding.
 - Added bounded Zakura header-sync stream-5 wire messages, stateless header
   validation, and the default `network.zakura.header_sync` config surface.
+- Verified-commitment-trees fast checkpoint sync. Below the last checkpoint Zebra
+  now fetches per-block Sapling/Orchard commitment roots from peers over a new
+  header-sync-aligned `tree_aux` stream, verifies each root against the node's own
+  checkpoint-committed block headers (the ZIP-221 ChainHistory MMR plus direct
+  below-Heartwood/below-NU5 checks), and folds the verified roots into the anchor
+  set and history tree — skipping the per-block note-commitment frontier recompute
+  that dominates checkpoint-sync CPU cost. At the checkpoint handoff an embedded
+  final frontier, verified against that block's proven root, is written as the tip
+  treestate and normal per-block recompute resumes. The resulting consensus state
+  is byte-identical to the legacy recompute; a root that cannot be obtained or
+  verified is rejected rather than recomputed against the stale frozen frontier, so
+  no untrusted data can influence consensus state. This is the default whenever
+  `consensus.checkpoint_sync = true` on a network with an embedded handoff frontier
+  (Mainnet), for both Archive and Pruned storage modes. The new
+  `consensus.vct_fast_sync` flag (default `true`) selects this fast path; set it to
+  `false` to keep checkpoint sync enabled while forcing the legacy per-block
+  recompute. Bumps the state database
+  format to 27.3.0 (new column families only; no data migration).
 - Include the `zebra-rollback-state` and `zebra-prune-state` utilities alongside
   `zebrad` in release Docker images and Docker CI builds.
 - Use the `5.0.0-rc.3` release identity for this fork's v5 rollback build.
@@ -211,6 +229,14 @@ and this project adheres to [Semantic Versioning](https://semver.org).
 
 - Use network protocol version 170160 as the NU6.3 minimum on Mainnet, Testnet,
   and Regtest, matching Zebra's advertised current protocol version.
+- Stop the database format-validity check from panicking with "just checked for
+  genesis block" while a verified-commitment-trees fast sync is in progress. The
+  check runs on a background thread, concurrently with block commits, and could
+  read its `is_vct_synced()` guard as `false` and then read an absent genesis
+  note-commitment tree once a concurrent fast-sync commit set the marker in
+  between. It now treats an absent genesis tree as a (mid-flight) fast-synced
+  database — where the genesis-root-caching invariant does not apply — instead of
+  panicking.
 - Avoid panics in the block write task when RPC users invalidate a non-finalized
   root block or reconsider the same invalidated block twice.
 - Compare RPC authentication cookies in constant time after checking their

zebra-chain/src/block/commitment.rs

@@ -396,6 +396,16 @@ pub enum CommitmentError {
         actual: [u8; 32],
     },
 
+    #[error(
+        "invalid pre-NU5 orchard root: expected the empty-tree root {:?}, actual: {:?}",
+        hex::encode(expected),
+        hex::encode(actual)
+    )]
+    InvalidPreNu5OrchardRoot {
+        expected: [u8; 32],
+        actual: [u8; 32],
+    },
+
     #[error("missing required block height: block commitments can't be parsed without a block height, block hash: {block_hash:?}")]
     MissingBlockHeight { block_hash: block::Hash },
 

zebra-state/CHANGELOG.md

@@ -12,8 +12,31 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 - Bumped the state database format to 28 during upgrade, backfilling empty
   Ironwood tree, value pool, and index data, then rebuilding stored history tree
   entries so they use the Ironwood-capable entry size.
-- Extended value-pool disk serialization with an Ironwood slot after the
-  deferred pool, and bumped the state database format version to `27.3.0`.
+- Extended value-pool disk serialization with an Ironwood slot after the deferred pool, with the
+  verified-commitment-trees state database format changes preserved as the intermediate
+  no-migration version `27.3.0`.
+- Extended the `commitment_roots_by_height` serving index (and the `tree_aux` header-sync
+  payload it is served from) with each block's per-height ZIP-244 `auth_data_root`, so a node
+  can serve the co-input a peer needs to authenticate a block's note-commitment roots against
+  its successor's NU5+ header commitment. Consolidated under database format version `27.3.0`
+  (no version bump); serving-index rows from an earlier pre-release build (64 bytes, no
+  auth-data root) remain readable. Carrying the auth-data root bumps the Zakura header-sync
+  stream format to version 5, a breaking wire change (all peers must run the new format).
+- Extended the same serving index and `tree_aux` payload with each block's Ironwood
+  note-commitment root and its three per-pool shielded transaction counts
+  (Sapling/Orchard/Ironwood). Together with the roots and auth-data root, these are the complete
+  set of ZIP-221 history-leaf inputs a recipient needs to rebuild each leaf and verify the
+  supplied roots against its own header commitments during header sync, without downloading the
+  block body. Rows grow to 152 bytes with a backward-compatible `FromDisk`; still consolidated
+  under `27.3.0`. Carried in the (already breaking, still version 5) Zakura header-sync stream
+  format, with `auth_data_root` serialized last. This is the data-carrying half only — nothing
+  consumes the new fields yet.
+- Added the `vct_upgrade_metadata` column family, recording the upgrade height `U` (the lowest
+  height this binary committed). `tree_aux` root serving now stitches the per-height trees below
+  `U` with the serving index at and above `U`, so a node that upgraded mid-chain serves a range
+  crossing `U` as one gap-free batch instead of a short prefix that stalled the fetch client.
+  Historical note-commitment tree RPCs are unavailable only within the band `[U, H)` (where `H`
+  is the checkpoint handoff), and available below `U` and at or above `H`.
 
 ## [8.0.0] - 2026-06-02
 

zebra-state/src/error.rs

@@ -86,6 +86,26 @@ impl CommitBlockError {
         }
     }
 
+    /// Returns the missing VCT supplied-root height for retryable root-fetch stalls.
+    pub fn vct_supplied_root_unavailable_height(&self) -> Option<block::Height> {
+        match self {
+            CommitBlockError::ValidateContextError(error) => {
+                error.vct_supplied_root_unavailable_height()
+            }
+            _ => None,
+        }
+    }
+
+    /// Returns the height for any retryable VCT root stall (absent/evicted root, or one
+    /// not yet verifiable for lack of a buffered successor). See
+    /// [`ValidateContextError::vct_retryable_height`].
+    pub fn vct_retryable_height(&self) -> Option<block::Height> {
+        match self {
+            CommitBlockError::ValidateContextError(error) => error.vct_retryable_height(),
+            _ => None,
+        }
+    }
+
     /// Returns a suggested misbehaviour score increment for a certain error.
     pub fn misbehavior_score(&self) -> u32 {
         0
@@ -149,6 +169,18 @@ impl CommitCheckpointVerifiedError {
     pub fn duplicate_location(&self) -> Option<&KnownBlock> {
         self.0.duplicate_location()
     }
+
+    /// Returns the missing VCT supplied-root height for retryable root-fetch stalls.
+    pub fn vct_supplied_root_unavailable_height(&self) -> Option<block::Height> {
+        self.0.vct_supplied_root_unavailable_height()
+    }
+
+    /// Returns the height for any retryable VCT root stall (absent/evicted root, or one
+    /// not yet verifiable for lack of a buffered successor). See
+    /// [`ValidateContextError::vct_retryable_height`].
+    pub fn vct_retryable_height(&self) -> Option<block::Height> {
+        self.0.vct_retryable_height()
+    }
 }
 
 impl From<ValidateContextError> for CommitCheckpointVerifiedError {
@@ -367,6 +399,23 @@ pub enum ValidateContextError {
     #[non_exhaustive]
     NotReadyToBeCommitted,
 
+    #[error(
+        "verified-commitment-trees fast path has no valid supplied root for height \
+         {height:?}: the note-commitment frontier is frozen, so this block cannot be \
+         committed until a verifiable root is fetched from a peer (retryable)"
+    )]
+    #[non_exhaustive]
+    VctSuppliedRootUnavailable { height: block::Height },
+
+    #[error(
+        "verified-commitment-trees fast path cannot yet verify the supplied root for height \
+         {height:?}: no successor block is buffered to confirm it against the header chain, and \
+         committing it unverified would persist a root that is only checked one block later \
+         (irreversibly, once on disk). Commit is deferred until the successor arrives (retryable)"
+    )]
+    #[non_exhaustive]
+    VctSuppliedRootAwaitingSuccessor { height: block::Height },
+
     #[error("block height {candidate_height:?} is lower than the current finalized height {finalized_tip_height:?}")]
     #[non_exhaustive]
     OrphanedBlock {
@@ -597,6 +646,33 @@ pub enum ValidateContextError {
     },
 }
 
+impl ValidateContextError {
+    /// Returns the missing VCT supplied-root height for retryable root-fetch stalls.
+    ///
+    /// This is the subset of [`Self::vct_retryable_height`] that warrants a peer *refetch*:
+    /// the supplied root is absent or was evicted after failing verification, so a different
+    /// peer must supply a replacement. An await-successor stall ([`Self::vct_retryable_height`]
+    /// but not this) already has its root and only waits for the next block to be downloaded.
+    pub fn vct_supplied_root_unavailable_height(&self) -> Option<block::Height> {
+        match self {
+            ValidateContextError::VctSuppliedRootUnavailable { height } => Some(*height),
+            _ => None,
+        }
+    }
+
+    /// Returns the height for any retryable VCT root stall: either an absent/evicted supplied
+    /// root ([`Self::VctSuppliedRootUnavailable`]) or one not yet verifiable because no successor
+    /// is buffered to confirm it ([`Self::VctSuppliedRootAwaitingSuccessor`]). The write loop
+    /// parks and retries the same block for both; only the former additionally requests a refetch.
+    pub fn vct_retryable_height(&self) -> Option<block::Height> {
+        match self {
+            ValidateContextError::VctSuppliedRootUnavailable { height }
+            | ValidateContextError::VctSuppliedRootAwaitingSuccessor { height } => Some(*height),
+            _ => None,
+        }
+    }
+}
+
 impl From<sprout::tree::NoteCommitmentTreeError> for ValidateContextError {
     fn from(value: sprout::tree::NoteCommitmentTreeError) -> Self {
         ValidateContextError::NoteCommitmentTreeError(value.into())
@@ -657,4 +733,65 @@ mod tests {
         };
         assert_eq!(dup_err.misbehavior_score(), 0);
     }
+
+    #[test]
+    fn checkpoint_error_exposes_retryable_vct_root_height() {
+        let height = Height(42);
+        let retryable: CommitCheckpointVerifiedError =
+            ValidateContextError::VctSuppliedRootUnavailable { height }.into();
+        assert_eq!(
+            retryable.vct_supplied_root_unavailable_height(),
+            Some(height),
+            "checkpoint commit errors expose retryable VCT root misses"
+        );
+
+        let non_retryable: CommitCheckpointVerifiedError =
+            ValidateContextError::NonSequentialBlock {
+                candidate_height: Height(5),
+                parent_height: Height(3),
+            }
+            .into();
+        assert_eq!(
+            non_retryable.vct_supplied_root_unavailable_height(),
+            None,
+            "unrelated validation errors are not treated as VCT root misses"
+        );
+        assert_eq!(
+            non_retryable.vct_retryable_height(),
+            None,
+            "unrelated validation errors are not retryable VCT stalls"
+        );
+    }
+
+    /// An await-successor stall is retryable (the write loop parks and re-commits) but is
+    /// *not* a refetch case: the root is present, only its successor is missing. So it must
+    /// surface through `vct_retryable_height` while `vct_supplied_root_unavailable_height`
+    /// (which gates the peer refetch) stays `None` — otherwise the committer would spam
+    /// pointless refetches for a root it already holds.
+    #[test]
+    fn await_successor_is_retryable_but_not_a_refetch() {
+        let height = Height(7);
+        let awaiting: CommitCheckpointVerifiedError =
+            ValidateContextError::VctSuppliedRootAwaitingSuccessor { height }.into();
+
+        assert_eq!(
+            awaiting.vct_retryable_height(),
+            Some(height),
+            "an await-successor stall is retryable",
+        );
+        assert_eq!(
+            awaiting.vct_supplied_root_unavailable_height(),
+            None,
+            "an await-successor stall must not trigger a peer refetch (the root is present)",
+        );
+
+        // The unavailable case is both retryable and a refetch trigger.
+        let unavailable: CommitCheckpointVerifiedError =
+            ValidateContextError::VctSuppliedRootUnavailable { height }.into();
+        assert_eq!(unavailable.vct_retryable_height(), Some(height));
+        assert_eq!(
+            unavailable.vct_supplied_root_unavailable_height(),
+            Some(height)
+        );
+    }
 }

zebra-state/src/lib.rs

@@ -63,8 +63,10 @@ pub use service::{
     finalized_state::FinalizedState,
     init, init_read_only,
     non_finalized_state::NonFinalizedState,
+    queued_blocks::QueuedCheckpointVerified,
     spawn_init_read_only,
     watch_receiver::WatchReceiver,
+    write::BlockWriteSender,
     OutputLocation, ReadState, State, TransactionIndex, TransactionLocation,
 };
 
@@ -80,6 +82,10 @@ pub use service::finalized_state::{
     preview_rollback_finalized_state, rollback_finalized_state, RollbackBackupSummary,
     RollbackFinalizedStateError, RollbackFinalizedStateOptions, RollbackFinalizedStateSummary,
 };
+pub use service::finalized_state::{
+    produce_final_frontiers_bytes, validate_final_frontiers_bytes, FinalFrontiersGenerationError,
+    FinalFrontiersValidationError,
+};
 pub use service::{
     finalized_state::{DiskWriteBatch, FromDisk, IntoDisk, WriteDisk, ZebraDb},
     ReadStateService,