feat(state): skip the transparent address index in pruned checkpoint-sync mode#357
Draft
p0mvn wants to merge 25 commits into
Draft
feat(state): skip the transparent address index in pruned checkpoint-sync mode#357p0mvn wants to merge 25 commits into
p0mvn wants to merge 25 commits into
Conversation
|
Note Complete: Audit complete. No review-worthy issues remain after automatic triage. One finding was auto-invalidated. Open the full results here. Analyzed two files, diff |
Author
|
@v12-auditor review |
Analyzed eight files, diff |
p0mvn
changed the title
p0mvn
changed the title
Author
|
@cursor review |
p0mvn
marked this pull request as draft
July 1, 2026 06:25
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
There was a problem hiding this comment.
✅ Bugbot reviewed your changes and found no new issues!
Comment @cursor review or bugbot run to trigger another review on this PR
Reviewed by Cursor Bugbot for commit 0b39740. Configure here.
Author
|
@v12-audit review |
Author
I expect to have been addressed v12 runs locally. Will review updates in the morning. If any, I think these should be non-blocking for approval. Will address |
p0mvn
force-pushed
the
state-checkpoint-sync-config
branch
from
July 1, 2026 07:41
c3bdb63 to
824e597
Compare
…oint verifier (#124) * perf(consensus): precompute auth data root concurrently in the checkpoint verifier The ZIP-244 authorizing-data commitment (Block::auth_data_root, a per-transaction auth digest) is one of the two dominant serial costs of the finalized committer on heavy shielded blocks. Unlike the note-commitment tree update it depends only on the block's own transactions, not chain state, so it can be computed ahead of the committer. Computing it inline in `check_block` does NOT help: the checkpoint verifier is wrapped in a tower `Buffer` (single worker), and `check_block` runs on that serialized path, so the work just moves to another single-threaded stage. Instead, compute it in the per-block task the verifier already `tokio::spawn`s to commit each verified block. That task runs off the buffer worker, one per block, so many blocks' auth digests are computed concurrently (via `spawn_blocking`), overlapping with and ahead of the single-threaded committer. The committer uses the precomputed value (carried on `SemanticallyVerifiedBlock::auth_data_root`), falling back to computing it when absent. Only Nu5-onward blocks bind the auth data in their block commitment. Consensus-neutral: the value is byte-identical to recomputing it at commit time; an end-to-end differential mainnet sync is the proof, since a wrong auth data root fails the commitment check and rejects the block. * spawn auth data root pre-compute after verifying checkpoint * simplify comment
Computing a v5+ transaction's txid (`Transaction::hash`) and its ZIP-244 authorizing-data digest (`auth_digest`) each independently convert the whole transaction to its librustzcash representation (re-serialize + re-parse), which dominates the per-transaction cost on heavy shielded blocks. The checkpoint commit path paid this twice: once building the transaction hashes in `CheckpointVerifiedBlock::new`, and again computing the auth data root. Add `Transaction::txid_and_auth_digest`, which performs one conversion and returns both. `SemanticallyVerifiedBlock::with_hash` now computes the transaction hashes and the auth data root together from that single shared conversion (the auth digest is nearly free once the txid is computed), so the auth data root is carried on the block and the separate per-block conversion in the checkpoint verifier's commit task is removed. Byte-identical to the separate computations (differential proptest `txid_and_auth_digest_matches_separate`); an end-to-end mainnet sync is the consensus proof.
Computing a v5+ transaction's txid (`Transaction::hash`) and its ZIP-244 authorizing-data digest (`auth_digest`) each independently convert the whole transaction to its librustzcash representation (re-serialize + re-parse), which dominates the per-transaction cost on heavy shielded blocks. The checkpoint commit path paid this twice: once building the transaction hashes in `CheckpointVerifiedBlock::new`, and again computing the auth data root. Add `Transaction::txid_and_auth_digest`, which performs one conversion and returns both. `SemanticallyVerifiedBlock::with_hash` now computes the transaction hashes and the auth data root together from that single shared conversion (the auth digest is nearly free once the txid is computed), so the auth data root is carried on the block and the separate per-block conversion in the checkpoint verifier's commit task is removed. Byte-identical to the separate computations (differential proptest `txid_and_auth_digest_matches_separate`); an end-to-end mainnet sync is the consensus proof.
…ck writer (#128) * perf(state): serialize raw transactions in parallel when writing blocks * perf(state): compute block size in parallel + run block-write batch prep in dedicated pool * comment
…reshold (#138) The checkpoint committer serializes each block's raw transactions (block.rs) and sums the per-transaction sizes (chain.rs) on the rayon pool. That fan-out is a clear win for the large blocks in the heavy shielded region, but for the small blocks of the early chain the rayon fork-join cost (waking workers, distributing the items, joining) outweighs the work itself. Gate both parallel paths on PARALLEL_BLOCK_TX_THRESHOLD (16 transactions): blocks at or above it keep the parallel path, smaller blocks run sequentially. The output is byte-identical either way, so this is purely a scheduling change. Measured with two fresh-from-genesis mainnet syncs of the same binary, gate toggled, over a matched height window (per-block, committer-thread metrics that are independent of peer/download luck): batch_prep 1.45ms -> 1.31ms (-10%) write_block_total 6.38ms -> 6.08ms ( -5%) Stable across sub-windows (batch_prep -8% to -13%). The heavy shielded region is unaffected: those blocks have >= 16 transactions and keep the parallel path.
In the checkpoint range, per-transaction CPU is dominated by computing the v5 txid and ZIP-244 authorizing-data digest. Both went through `Transaction::to_librustzcash`, which serializes the whole transaction and reparses it — decompressing every Jubjub/Pallas curve point — purely so librustzcash can re-serialize those same bytes into the BLAKE2b digest tree. A `perf` flamegraph of the heavy shielded region (mainnet 1.72M–1.73M) attributes ~44% of all CPU to these reparses (leaves are `bls12_381::Scalar::square` / `sqrt_tonelli_shanks` from point decompression); the BLAKE2b hashing itself is <1%. The decompressed points are never needed in the checkpoint range (no proof/signature verification). Compute the txid and auth digest directly from Zebra's already-parsed `Transaction` fields, feeding their canonical bytes straight into the same BLAKE2b tree (`transaction::zip244`). This removes the reparse entirely for the digest path. v6 transactions (unstable `tx_v6`) still use librustzcash. This is consensus-critical and byte-identical to librustzcash: proven by a differential property test (`native_zip244_matches_librustzcash`) over thousands of random v5 transactions, the existing ZIP-244 known-answer vectors, and a clean differential mainnet checkpoint sync.
#133) V5 transaction deserialization re-ran the full Transaction::to_librustzcash conversion and discarded the result, purely to reject transactions that Zebra can parse but librustzcash cannot. That conversion decompresses every Jubjub and Pallas curve point. A flamegraph of the heavy shielded region attributes about 25 to 30 percent of checkpoint-sync CPU to this single discarded reparse, and after the native ZIP-244 digest change it is the largest remaining cost. The check is redundant for rejecting untrusted transactions. Every transaction from a peer, the mempool, or sendrawtransaction is converted via CachedFfiTransaction::new before the semantic verifier accepts it, so a non-convertible v5 transaction is still rejected there with a clean error, including fully shielded transactions whose bundles are derived from that same conversion. Blocks below the checkpoints are trusted by their hash and validated against the header merkle root built from the native transaction IDs, and the checkpoint commit path no longer calls to_librustzcash for v5. Zebra's own deserializer still rejects the non-canonical encodings it validates (for example an identity-point Orchard rk), so only the librustzcash-specific re-validation moves from parse time to verification time. The pre-NU5 consensus branch id rejection added by the same upstream change is kept, since it is independent and cheap.
…tic path (#136) * proto(chain): defer Sapling value-commitment point decompression PROTOTYPE for benchmarking lever #1. After the native-digest and dropped-reparse changes, a flamegraph of the checkpoint heavy region attributes about 60% of CPU to Sapling Jubjub point decompression (a field square root in jubjub::AffinePoint::from_bytes), almost entirely the value commitment cv on every spend and output. Checkpoint sync never uses cv as a point: it verifies no signatures or proofs, and the note-commitment tree uses cm_u, not cv. Store cv as its canonical 32-byte encoding and decompress lazily, only when a consumer needs the point. Deserialization just copies the bytes, serialization and the txid digest use them directly, and the binding-signature verification in the semantic verifier decompresses on demand via ValueCommitment::commitment. This mirrors what Orchard already does for rk, which is why Orchard decompression is negligible in the profile. Prototype caveat: ValueCommitment::commitment panics on a non-canonical encoding rather than returning an error, and the not-small-order check now happens at the point of use instead of at parse. Correct for checkpoint sync (block hashes are trusted) and exercised by the unit tests, but the production version must make the accessor fallible so the semantic and mempool paths reject a malformed point cleanly instead of panicking. * proto(chain): also defer Sapling ephemeral_key point decompression Extends the lazy value-commitment prototype. With cv deferred, the profile showed the remaining ~50% of heavy-region CPU is the other per-output Jubjub point, the ephemeral_key, decompressed at parse. The validator only needs its bytes (txid digest and serialization); the point is needed only for wallet trial-decryption. Store ephemeral_key as its canonical 32-byte encoding and skip decompression at deserialization, like cv. Same prototype caveat: the not-small-order consensus check is deferred and must be re-added on the semantic and mempool paths in a production version. * proto(chain): validate lazy Sapling cv/epk consensus safety The deferred not-small-order checks for cv and ephemeral_key are not actually missing on the consensus path: librustzcash enforces them for every untrusted transaction, which all go through to_librustzcash (CachedFfiTransaction::new) on the semantic and mempool paths. cv is rejected at read (zcash_primitives read_value_commitment uses from_bytes_not_small_order); epk is rejected at verify (sapling-crypto verifier check_output uses epk.is_small_order). The checkpoint verifier trusts block hashes and does not need them. Add a regression test that constructs a v5 transaction with a small-order cv and epk and asserts both the deferral (Zebra now deserializes it) and the safety net (to_librustzcash rejects it), plus that the exact library detection functions flag the point. Correct the type docs accordingly. * perf(chain): enforce deferred Sapling cv/epk check on the semantic path Hardens the lazy Sapling cv/ephemeral_key prototype into a safer design. The lazy types keep point decompression off the checkpoint-sync hot path (the measured ~2.5x win), but the not-small-order consensus check is now re-enforced explicitly by Zebra on the untrusted boundary instead of relying solely on librustzcash. Add `Transaction::sapling_point_encodings_are_valid` (and the underlying `ShieldedData::point_encodings_are_valid`, `ValueCommitment::is_valid_not_small_order`, `EphemeralPublicKey::is_valid_not_small_order`), and call it from `verify_v4_transaction` / `verify_v5_transaction`, returning `TransactionError::SmallOrder` for a small-order or off-curve cv or epk. This runs on the semantic verification path and the mempool, which process untrusted transactions; the checkpoint verifier never calls it (it trusts block hashes), so the checkpoint throughput is unchanged. This restores a Zebra-side, auditable enforcement of the rule and makes the epk check isolatedly testable (it runs independently of proof verification). Spend rk is still validated at deserialization. Validated by `sapling_point_encodings_check_rejects_bad_points` and the existing lazy-cv/epk tests. * fix(consensus): run the deferred Sapling cv/epk check before to_librustzcash Adversarial review of the lazy Sapling change found one non-consensus issue: a small-order or off-curve cv failed inside CachedFfiTransaction::new (mapped to UnsupportedByNetworkUpgrade, mempool misbehavior score 0) before the explicit SmallOrder check ran, so a peer spamming bad-cv transactions received a lighter penalty than before the change (when it was a deserialization error). Move the sapling_point_encodings_are_valid check into the verifier's early quick checks, before the state lookups and the librustzcash conversion. Now a bad cv or epk fails fast with TransactionError::SmallOrder (score 100), restoring the peer penalty and making the check the primary, version-agnostic enforcer for v4, v5, and v6. Remove the now-redundant per-version copies. No consensus behavior change: the same transactions are accepted and rejected. The review confirmed no path commits or relays a transaction with a bad point without this check or checkpoint hash-trust, the commitment() panic is not reachable in release (no non-test caller), and there is no DoS amplification. * refactor(chain): make ValueCommitment::commitment fallible Removes the latent panic in `ValueCommitment::commitment`, which is the only caller-facing point that could decompress a deferred (unvalidated) value commitment. It now returns `Option`, so a future caller must handle an invalid encoding instead of getting a hidden panic, eliminating a possible DoS if the helper were ever moved onto a production path. `ShieldedData::binding_verification_key` (its only caller, used in tests) now propagates the `Option`. No production code calls either; the consensus encoding check happens on the semantic path via `sapling_point_encodings_are_valid`. * test(consensus): end-to-end reject of a Sapling output with an invalid epk Adds the missing end-to-end test for the deferred Sapling cv/epk check: it takes a real Sapling-output transaction, corrupts the first output's ephemeral key to an off-curve point, and runs it through the full transaction Verifier, asserting TransactionError::SmallOrder. The state service is unreachable!, proving the check fires in the early quick checks before any state lookup, and that the rejection is the explicit SmallOrder error rather than a later proof failure. This closes the last gap from the security review: the epk rejection is now confirmed by execution through the live verifier, not only by the isolated check and the librustzcash backstop. * consensus equivalence tests
…s reads (#140) * Update zebra-state/src/request.rs Co-authored-by: Dev Ojha <ValarDragon@users.noreply.github.com> * Update zebra-state/src/request.rs Co-authored-by: Dev Ojha <ValarDragon@users.noreply.github.com> * perf(state): parallelize and de-duplicate the committer's UTXO/address reads Before building the write batch, the checkpoint committer reads every transparent input's UTXO and every changed address's balance from RocksDB, one `zs_get` at a time on the writer thread. In the transparent-heavy ranges (~100-330K) these cache-served but serial point lookups dominate the per-block write time while the other cores sit idle (CPU ~2/8). The spent-UTXO path also re-derives each input's transaction location twice: once directly and once inside `utxo()`. Two changes in `write_block`: - Read the output location once and reuse it via `utxo_by_location` instead of letting `utxo()` look it up again (3 reads/input -> 2). - Fan the spent-UTXO and address-balance reads across the rayon pool (the writer already runs inside COMMIT_COMPUTE_POOL) once a block has enough inputs/addresses to amortize the fork-join cost, gated by PARALLEL_BLOCK_READ_THRESHOLD (16). The reads are read-only and land in order-independent maps, so the committed batch is byte-identical to the sequential path. Measured over a full mainnet genesis sync, comparing the same binary with and without this change, per-100K committer-thread metrics (peer-independent): range prep_reads write_block_total 100k 7.57 -> 2.64 ms 15.71 -> 10.38 ms 200k 8.94 -> 3.75 ms 19.01 -> 14.30 ms 300k 10.89 -> 3.52 ms 20.32 -> 13.07 ms 400k 2.33 -> 1.05 ms 4.84 -> 3.05 ms prep_reads drops 55-68% and write_block_total 25-37% across the transparent band, moving the bottleneck there onto rocksdb commit. No effect on low-input blocks (gated to sequential) or the heavy shielded region (few transparent inputs). * clean up and tests * comment * clean up comment * fix(state): remove duplicate finalized block import --------- Co-authored-by: Dev Ojha <ValarDragon@users.noreply.github.com>
#144) * perf(state): precompute note-commitment tree hashing off the committer [prototype] Move the dominant per-block committer cost — the Sapling/Orchard note-commitment tree update — off the single serial committer thread. `parallel_append` is split into `precompute_subtree_roots` (the per-leaf Merkle hashing, position-independent: it needs only the starting note count, not the frontier's hashes) and `graft` (the cheap O(log N) merge on the committer). The finalized write loop runs a 1-block look-ahead: before committing block N it spawns block N+1's hashing on the commit-compute pool, so the hashing overlaps N's commit on otherwise-idle cores; the committer then only grafts. A size-match guard makes a stale precompute fall back to inline hashing, so this can only affect speed, never correctness. Byte-identical to the inline append (differential proptests over the split and the tracked-subtree boundary). `NOTE_PRECOMPUTE_DISABLE` env var forces the inline path for single-binary A/B benchmarking. A/B over the sandblast region (1.71M-1.735M): committer update_trees -54% (12.5 -> 5.7 ms/block). Throughput was flat there because that window is feed- bound (downloads buffered, CPU ~3.5/8), not committer-bound; the win applies where the committer is the gate. Prototype pending the feed instrumentation. * tests, clean up, changelog * fix(chain): use checked arithmetic for precompute capacity check The precompute batch path takes a caller-supplied start_size, so start_size + nodes.len() could wrap past the MAX_LEAVES capacity check (and panic on overflow in debug builds) for values near u64::MAX, building an inconsistent precompute that could later panic in graft. Use checked_add and reject over-capacity sizes with a clean MaxDepthExceeded error. Adds a regression test. * fix(chain): return recoverable errors from precompute helpers instead of panicking The precompute and graft helpers enforced caller-controlled preconditions with panicking assertions: precompute_subtree_roots / precompute_append_ batch_with_subtree on an empty batch, and graft on a frontier size that did not match the precompute's start position. Reachable via the public BlockNotePrecompute path, these turned invalid input into a process panic. Replace the assertions with recoverable BatchFrontierError variants (EmptyBatch, PrecomputeStartMismatch) and map them through a new NoteCommitmentTreeError::InvalidPrecompute in the Sapling/Orchard wrappers. The in-node path is unaffected (it guards empty note sets and size-matches before applying). Adds tests. * perf(chain): run Sapling and Orchard precompute concurrently BlockNotePrecompute::compute hashed the two pools sequentially. Although each pool's append is internally parallel, the two no longer overlapped the way update_trees_parallel's per-pool spawn_fifo tasks did. Restore the cross-pool overlap with rayon::join. * perf(chain): gate note-commitment precompute parallelism on batch size Below PARALLEL_HASH_THRESHOLD (16) note commitments, the per-leaf Merkle hashing now runs entirely serially: benchmarks show that for small batches the rayon join/par_iter overhead matches or exceeds the hashing it parallelizes (crossover ~16 for both Sapling Pedersen and Orchard Sinsemilla), and most blocks outside the sandblast region are small. The gate is on the whole-batch decision only; above the threshold each chunk still splits down to the leaves, so medium batches keep their internal parallelism. BlockNotePrecompute::compute likewise only spawns the cross-pool rayon::join when a pool is large enough to repay it. Adds the precompute_threshold benchmark (and bench-only precompute_then_ graft_root shims) used to find the crossover. Correctness is unchanged and covered by the existing differential proptests. * fix(state): make the look-ahead note precompute cancellable The finalized write loop starts the next block's note-commitment precompute before the current block has committed, to overlap the hashing with the commit. A current block that fails to commit (e.g. a checkpoint-range block whose authorizing-data commitment is only rejected at finalized-state commit) leaves that speculative work unwanted, and the spawned task previously had no cancellation path: it hashed the discarded child in full before noticing the receiver was dropped. Thread an Arc<AtomicBool> cancellation flag through spawn_note_precompute into BlockNotePrecompute::compute. The two pools are now hashed sequentially (each still internally parallel) so the flag is checked between them; the writer trips it whenever it drops a pending precompute (commit failure, parent-failure skip, height mismatch, or hash mismatch), bounding the wasted work for a discarded child to at most one pool. Correctness is unaffected (the committer still size-checks before applying). Adds a cancellation test. Also normalizes the prior 'graft' terminology to 'apply_precompute'. * perf(chain): keep the cross-pool join in the cancellable precompute Restore the rayon::join (and small-block sequential gating) for the two pools in BlockNotePrecompute::compute, which was dropped when compute was made cancellable. Cancellation is now done by checking the flag up front and at the start of each pool's hashing rather than strictly between the pools, so the cross-pool overlap is preserved while a cancel that lands before a pool starts still skips its work. * fix(chain): bind note precompute to its block, not just the tree size A BlockNotePrecompute was selected solely by start_size == tree.count(), and in that branch the block's own note-commitment arguments were ignored in favor of the precompute's leaves. A precompute accidentally paired with a different block of the same starting tree size would therefore be grafted, silently producing a wrong note-commitment root. The node avoided this by pairing each precompute with the exact block hash in the write loop, but that invariant lived outside zebra-chain's API. Record the block hash in BlockNotePrecompute::compute and have update_trees_parallel_with apply the precompute only when its block_hash matches the block being committed; a mismatch falls back to inline hashing (correct, just slower). Adds a test that a precompute for a different block at the same starting size is rejected.
…type] (#151) When a required (head-of-line) block registry-misses, re-dispatch its backoff retry as a fan-out to several random ready peers, ignoring inventory markers, and take the first peer that delivers it. This bypasses stale 'missing' markers (pool.route_inv.notfound.all_missing), the measured cause of ordered-commit stalls, where ready peers actually have the block. - zebra-network: new Request::HedgedBlocksByHash routing directive + route_hedge (reuses select_random_ready_peers; rewrites to per-peer BlocksByHash); falls back to the same NotFoundRegistry as route_inv so sync retry/backoff is unchanged. - zebrad sync: download_and_verify_hedged + hol_hedge_fanout, env-gated via SYNC_HOL_HEDGE_FANOUT (default 0 = off). Only the registry-miss retry hedges; the 2s backoff and #105 gating are unchanged. - Test: peer_set_route_hedge_bypasses_missing_markers. Also fixes a pre-existing compile break in sync/tests/vectors.rs (Downloads::new missing Network arg) so the test target builds; the 5 stale Commit-vs- CommitCheckpointPrecomputed failures there are pre-existing precompute drift, unrelated to this change.
…'s UTXO reads (#158) * perf(state): parallelize per-block serialization in the finalized block writer (#128) * perf(state): serialize raw transactions in parallel when writing blocks * perf(state): compute block size in parallel + run block-write batch prep in dedicated pool * comment * perf(state): gate parallel block batch-prep on a transaction-count threshold (#138) The checkpoint committer serializes each block's raw transactions (block.rs) and sums the per-transaction sizes (chain.rs) on the rayon pool. That fan-out is a clear win for the large blocks in the heavy shielded region, but for the small blocks of the early chain the rayon fork-join cost (waking workers, distributing the items, joining) outweighs the work itself. Gate both parallel paths on PARALLEL_BLOCK_TX_THRESHOLD (16 transactions): blocks at or above it keep the parallel path, smaller blocks run sequentially. The output is byte-identical either way, so this is purely a scheduling change. Measured with two fresh-from-genesis mainnet syncs of the same binary, gate toggled, over a matched height window (per-block, committer-thread metrics that are independent of peer/download luck): batch_prep 1.45ms -> 1.31ms (-10%) write_block_total 6.38ms -> 6.08ms ( -5%) Stable across sub-windows (batch_prep -8% to -13%). The heavy shielded region is unaffected: those blocks have >= 16 transactions and keep the parallel path. * perf(state): overlap raw-transaction serialization with the committer's UTXO reads In checkpoint sync through the shielded sandblast region the finalized committer is the serial bottleneck. The `tx_by_loc` raw-transaction serialization (re-serializing each transaction to bytes) runs sequentially after the spent-UTXO reads on the committer's critical path. Run it concurrently with those reads via `rayon::join`: serialization is CPU-bound while the reads wait on disk, so they overlap. The bytes are threaded as `precomputed_raw_txs` into `prepare_block_batch`, which uses them directly; the semantic path passes `None` and serializes inline as before. Output is byte-identical and there is no on-disk-format change. Matched A/B on mainnet 1.81-1.9M (archive mode): ~0.8-1.2 ms less total committer time per block (peer-independent) and ~+5-6% throughput.
…ommit-compute pool (#247) The committer is not a member of COMMIT_COMPUTE_POOL, so install() is a synchronous cross-thread handoff that parks the committer until a pool worker runs the job. The look-ahead note-commitment precompute keeps those workers busy, so this second per-block handoff waits on a contended pool and that wait dominates the isolation it was meant to provide. Run write_block directly on the committer thread; its internal rayon uses the global pool. Measured net win (committer -12%, +5% throughput) on the sandblast region.
The syncer now sends Request::CommitCheckpointPrecomputed for checkpoint-height blocks; the mock expectations still required Request::Commit. Match by block hash, as the multi-block cases in the same file already do. Surfaced once the build break was fixed.
* fix(state): tolerate absent genesis trees in format check during VCT fast sync
`cache_genesis_roots::quick_check` runs on the background format-validity
thread (`DbFormatChange::spawn_format_change`) concurrently with block
commits. It reads its `is_vct_synced()` guard once, then reads the genesis
note-commitment trees via readers (`sapling_tree_by_height` etc.) that return
`None` for any height in a fast-synced database's `[U, H)` absent band by
re-reading the same `vct_synced_below` marker. A fast-sync commit can set that
marker in the window between the two reads, so the function passes the
`is_vct_synced()` guard (marker unset) and then a genesis read returns `None`
(marker now set) -- and the old `.expect("just checked for genesis block")`
panicked. Under the test suite's parallelism this surfaced as a flaky
`vct_mode_switches_continue_from_safe_boundaries` failure (reproduced 2/10).
Treat an absent genesis tree as a (mid-flight) fast-synced database, where the
genesis-root-caching invariant does not apply, and return `Ok` instead of
panicking. A `None` here can only mean the absent-band guard fired; a genuinely
missing genesis tree on a non-fast database makes the readers panic with their
own "must exist" message rather than return `None`, so no real corruption is
masked.
* test(network): bound and harden flaky Zakura networking tests
The Zakura networking tests (`zakura::legacy_gossip::tests::*`,
`zakura::testkit::cluster::tests::*`, `peer::handshake::tests::mutual_p2p_*`,
the testkit gossip mesh) dial real iroh/QUIC endpoints and register peers
within a bounded deadline. Run at the suite's `num-cpus` parallelism they
oversubscribe the CPU and starve those handshakes, so the hardcoded 5s connect
and `await_until` convergence deadlines are exceeded and the upgrade handshake
takes a different path. In the PR lane (`--profile all-tests`, `fail-fast` and
no retries) a single such timeout sinks the whole run. Reproduced at 2x thread
oversubscription: 10/10 iterations had >= 1 failure.
Two complementary changes:
- Introduce a single generous `TEST_NET_TIMEOUT = 30s` constant in the
zebra-network testkit and replace the scattered 5s/15s `connect_native`,
`connect_full_mesh`, `await_all_connected`, `await_until`, and convergence
`timeout` deadlines with it. A slow connect under load is not a correctness
signal, and these helpers return as soon as the awaited condition holds, so a
generous deadline never masks a genuine hang.
- Add a nextest `zakura-network` test group (`max-threads = 4`) and a
default-profile override binding the real-connect modules to it, so they do
not starve each other. The override only assigns a group; it excludes no
tests, so coverage is unchanged, and living on the `default` profile it
applies to both `all-tests` and `full-tests`.
With both changes the same 2x-oversubscription reproduction is 0/5, and a full
`zebra-network`/`zebra-state` run under `--profile all-tests` passes (1007/1007).
…y-bench) (#305) * feat(perf): add offline commit-pipeline replay benchmark (zebra-replay-bench) Adds a standalone benchmark that replays real mainnet blocks through the finalized-state committer (commit_finalized_direct) with no networking, to measure the write-assembler + disk-writer in isolation from header/body sync and head-of-line stalls. Two phases share a flat block cache: `index` reads a height window from a snapshot DB into the cache; `apply` replays it onto a writable fork whose tip is the window start-1 and verifies the final tip hash. Both legacy (full per-block note-tree recompute) and VCT fast-path modes are supported; the VCT path indexes per-height anchor roots itself (`index-roots`) and injects them into the base fork's header-roots column family so the committer folds them and skips the recompute. Adds zebra_state::FinalizedState::new_read_only, the make targets perf-build-replay-bench / perf-replay-index / perf-replay, and the deploy/runner/replay_run.sh harness. * possible CI fixes
…replay-bench (#307) Extends the offline commit-pipeline bench one abstraction level up: the new `apply-worker` subcommand drives blocks through the real zebra-state write worker (BlockWriteSender::spawn / WriteBlockWorkerTask::run) instead of calling commit_finalized_direct directly. Legacy and VCT modes are both supported; the VCT path feeds one trailing sidecar successor so the last window block commits, then verifies via a cloned DB handle. Isolate the commit measurement on both sides with a shared bounded prefetch (prefetch.rs): a producer thread reads, deserializes, and builds each CheckpointVerifiedBlock (the verifier-side prepare_block_data) ahead of the committer into a bounded channel, so block read + parse + prep stay off the timed thread and memory stays flat regardless of window size. apply consumes it directly; apply-worker feeds the worker with a bounded in-flight window, which also avoids a RocksDB write-stall from a large backlog. Depth is tunable via ZRB_PREFETCH_CAP (default 64). zebra-state: export BlockWriteSender and QueuedCheckpointVerified (write and queued_blocks modules made pub(crate)) so an external bench crate can drive the worker. Harness: replay_run.sh run-worker + make perf-replay-worker. RESULTS.md records the commit-only VCT A/B (direct and worker converge at ~124-128 blk/s once prep is pipelined off both) and the methodology.
…ungs to zebra-replay-bench (#308) * feat(perf): add checkpoint-verifier replay rung to zebra-replay-bench Adds `apply-verifier`, a third altitude above the committer and write worker: it drives cached blocks through the real zebra-consensus CheckpointVerifier, which commits to a real zebra-state StateService (-> write worker -> committer) on a multi-thread tokio runtime. This adds the per-block work the lower rungs skip: proof-of-work (difficulty + equihash) and Merkle-root validity, plus checkpoint-range batching. VCT successor boundary: the verifier only releases a block once its checkpoint range is contiguous, and the worker's VCT fast path can't commit a block until its successor is buffered. The final checkpoint's successor is in the dropped tail, so the bench feeds up to the last checkpoint <= end (to deliver the successors the worker needs) but counts/gates to the second-to-last checkpoint, checking that block's committed hash against the embedded checkpoint hash. A periodic progress log (fed/done/front-height) makes any stall observable. zebra-consensus: export CheckpointVerifier unconditionally (was gated to test/proptest-impl) so the external bench crate can drive it. Harness: replay_run.sh run-verifier + make perf-replay-verifier. RESULTS.md records the 30K numbers: legacy 49.9 blk/s (equihash overlaps the legacy recompute for free) and VCT 133.0 blk/s (slightly above the worker/direct VCT rates -- the concurrent verify->commit pipeline hides verification behind the commit). * feat(perf): add block-sync Sequencer replay rung to zebra-replay-bench Adds `apply-sequencer`, a fourth altitude above the committer/worker/verifier: it drives cached mainnet bodies through the real Zakura block-sync Sequencer (zebra-network) -- the body reorder + ordered-submit pipeline -- which submits to the same real CheckpointVerifier -> StateService as apply-verifier. zebra-network: a feature-gated (`internal-bench`) helper `zakura::spawn_bench_sequencer` constructs and spawns the real SequencerTask with no peers/reactor and returns a minimal split handle (feeder / submissions / committer). It lives inside the block_sync module so it reaches the pub(super) internals; no internal types are exposed and the default API is unchanged. The bench feeds bodies into the reorder queue in height order (random/out-of-order multi-peer arrival is a future knob), drives the ordered SubmitBlocks through the verifier with bounded concurrency, and reports each commit back so the sequencer frontier advances. VCT-only; same checkpoint-batching boundary as apply-verifier (feed to the last checkpoint, commit/gate to the second-to-last). Harness: replay_run.sh run-sequencer + make perf-replay-sequencer. RESULTS.md records the 30K VCT number (123.5 blk/s -- ~on the worker, ~7% under the verifier; the in-order feed leaves the reorder buffer idle). Stacked on the checkpoint-verifier branch (depends on its CheckpointVerifier export). * feat(perf): default replay-sequencer to pruned + Zakura JSONL traces Extend the apply-sequencer replay rung: - Pruned storage mode is now the default (matching the production mainnet config); --archive opts back into Archive. Harness: REPLAY_ARCHIVE=1. - --trace-dir writes the real Zakura JSONL trace tables (block_sync.jsonl) via the production ZakuraTrace/JsonlTracer path, including periodic block_sync_state snapshots, so the .cursor/skills/zakura-trace-plots script plots them directly. Flushed at end-of-run; no-op when unset. Harness: REPLAY_TRACE_DIR=<dir>. - Finite 4 GiB in-flight byte budget bounds the applying buffer for large windows (peak RSS 3.8 GiB at 100K, vs unbounded before). Makefile help now lists the worker/verifier/sequencer rungs and the new REPLAY_ARCHIVE / REPLAY_TRACE_DIR knobs. RESULTS documents the 100K pruned run (103 blk/s), the per-phase commit decomposition (VCT eliminates update_trees; committer is RocksDB-write bound), and the throughput-vs- block-size-over-height finding. The zebra-network bench helper stays fully behind the internal-bench feature; the default build is unchanged.
* chore(zakura): add default bootstrap peers * test(zebrad): store bootstrap peer config snapshot
* fix(network): close peers after repeated receive timeouts * fix(network): tolerate more receive timeouts
…sync mode In pruned + checkpoint-sync mode (the minimal fast-validator configuration), the transparent address index — balances, address→utxo, address→tx — is RPC-only state that consensus never reads. Skip building it, the way pruned mode already drops raw-transaction storage. - `Config::skip_archive_indexes()` = `Pruned && checkpoint_sync` gates the write skip; the commit path elides the per-block address-balance reads and the three address index CF writes. The UTXO set, tx-location index, nullifiers (including Ironwood), note commitment trees, and value pool are unchanged. - Rollback and the block-info/address-received format check tolerate the absent index. The finalized-spender migration still records shielded (incl. Ironwood) nullifier tx locations while skipping the transparent spender index. - Address-lookup RPCs (getaddressbalance/utxos/txids) and finalized transparent spender lookups return an explicit "unavailable in pruned storage" error rather than wrong (empty) results. Durability: a pruned database is marked from its first commit under a distinct key in the pruning-metadata column family, independent of the raw-transaction pruning cursor. `ZebraDb::address_index_unavailable` and the archive-reopen guard consult it, so address RPCs stay disabled and the database cannot be reopened as an archive even during the initial-sync window before the pruning cursor is written. Bumps the state database format minor version (added key). Reapplied onto feat/pre-release-main.
p0mvn
force-pushed
the
state-checkpoint-sync-config
branch
from
July 2, 2026 22:06
824e597 to
397910e
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
On a pruned, checkpoint-syncing node (the minimal fast-validator configuration),
the transparent address index — balances, address→utxo, address→tx — is RPC-only
state that consensus never reads. Building it costs per-block address-balance
reads and index writes on the commit path. Pruned mode already drops
raw-transaction storage for the same reason; this extends that to the address
index.
Solution
Config::skip_archive_index()=Pruned && checkpoint_syncgates the behavior.index CF writes. The UTXO set (
utxo_by_out_loc),tx_loc_by_hash, nullifiers,note commitment trees, and value pool are unchanged.
index (they only touched it to maintain the address CFs).
getaddressbalance/getaddressutxos/getaddresstxids)return an explicit "index disabled in pruned mode" error rather than wrong
(empty) results.
Archive nodes, and pruned nodes with checkpoint sync disabled (full semantic
verification), keep the index unchanged.
Test evidence
skip_address_index_only_when_pruned_and_checkpoint_syncing).populated in archive, with
utxo_by_out_locstill written when skipping.zebra-statelib suite: 205 passed, 0 failed. Build + clippy + fmt clean.AI disclosure
Implemented with Claude Code (the skip, RPC guards, rollback/format-check
adaptations, and tests).