perf: block execution pipeline#309
Draft
p0mvn wants to merge 18 commits into
Draft
Conversation
And one more auto-invalidated finding. Analyzed 13 files, diff |
p0mvn
force-pushed
the
perf/block-exec-opt
branch
from
June 30, 2026 22:34
923e17b to
3314965
Compare
* perf(sync): header-authenticated per-block checkpoint release Step 1 of decoupling checkpoint verification from block bodies. Instead of streaming checkpoint-class bodies into the legacy CheckpointVerifier (which re-accumulates ~400 blocks and releases a whole range via the backward walk, causing the burst sawtooth and range-coupled head-of-line stall), commit each checkpoint-class body individually using a state-minted provenance token. - zebra-state: AuthenticatedCheckpointHash provenance newtype (private ctor) + ReadRequest::AuthenticatedCheckpointHash. The token is minted strictly from the Zakura checkpoint-anchored header frontier (zakura_header_hash for both the height and the next-checkpoint anchor C), never from committed body state, so it cannot authenticate a height from finalized bodies. missing_block_bodies clamps checkpoint-class heights by authenticated_header_tip for airtight ordering. - zebra-consensus: CommitCheckpointAuthenticated request routed on height <= max_checkpoint_height to CheckpointVerifier::call_authenticated, a narrow committer that runs check_block, binds block.hash == token, and releases straight to CommitCheckpointVerifiedBlock with no queue, no range walk, and no verifier_progress mutation. No fallback, no mixed mode. - zebrad: Zakura block-sync driver reads the token at apply time and sends the authenticated request; a missing token is a hard invariant violation, not a fallback. Tagged checkpoint_commit_path = "zakura_authenticated". Also includes commit-pipeline/sync instrumentation: floor-gap and commit-frontier diagnostics in the block-sync reactor, timed_commit_phase on the history-tree push, and a commit-metrics-gated DiskWriteBatch::size_in_bytes for batch-size sampling. * build
Overlap the next finalized block's batch assembly with the current block's disk write, raising checkpoint-sync commit throughput from `assemble + flush` per block toward `max(assemble, flush)`. - Split the committer into assemble (`assemble_finalized_direct` / `assemble_block_batch`, no writes) and flush (`flush_finalized_direct` / `flush_block_batch`) halves. - Add `FinalizedPipeline`: in-memory tip state (history tree, note trees, value pool, vct_upgrade marker, tip cursor) + a read-through overlay for not-yet- flushed spent UTXOs and address balances, so a block assembled ahead of the durable write reads its parent's effects from memory. - Wire a dedicated disk-writer thread fed by a bounded channel (depth = backpressure); the finalized tip, commit response, and download budget all trail the flush (ack-after-flush), so crash recovery is unchanged. - Gate behind `Config::finalized_block_pipeline_depth` (default 0 = synchronous, byte-identical to the previous committer). Only the checkpoint (reorg-free) region uses the pipeline.
…heck The run-ahead finalized committer dropped every block after the first when `finalized_block_pipeline_depth > 0`. The write worker's "next valid height" guard (which drops blocks whose parent has not committed) read the *durable* disk tip via `finalized_state.db.finalized_tip_height()`. But in the pipeline the assembler runs ahead of the disk writer (ack-after-flush), so when block N+1 arrives, block N is assembled but not yet flushed — the durable tip still shows N-1, the guard computes a stale `next_valid_height`, and N+1 is dropped as "wrong height". Its dropped response surfaces as `WriteTaskExited`, which cascades the whole checkpoint range and tears the StateService down. Fix: when the pipeline is active, derive `next_valid_height` from the in-memory assembled tip (`pipeline.tip()`), falling back to the durable tip (so the first block, before the pipeline is seeded, is unchanged). The synchronous path is unaffected (no pipeline -> durable tip, as before). This only reproduces through the full StateService write worker, which the existing `FinalizedState`-only equivalence test bypasses. Adds a regression test (`pipelined_state_service_commits_chain`) that commits a chain through the real StateService with the pipeline enabled.
…seeding Add per-phase commit timing for offline profiling: VCT commitment-root verify (fold), assemble-self, queue-wait and batch-commit on the commit-pressure trace, plus disk-writer service/recv-wait and run-ahead assembler park histograms. Add per-block checkpoint verify timing (pow/precompute/merkle) in zebra-consensus. Add ZebraDb::seed_zakura_headers_from_blocks, which seeds the Zakura header store from cached blocks so the header-authenticated checkpoint fast path can run against an offline snapshot with no live header sync. Ungate DiskWriteBatch::size_in_bytes: it is read by the commit-pressure trace independently of the commit-metrics feature, so the workspace did not build without that feature.
…k-sync driver Replace the hand-rolled verify/commit loop in apply-sequencer with the real Zakura block-sync apply driver (drive_block_sync_actions), exposed from zebrad behind a new internal-bench feature via zebrad::bench_api. The bench feeds cached bodies into the real sequencer; the production driver applies them through the consensus router into state using the header-authenticated checkpoint fast path, and the bench gates on the sequencer's verified tip. - zebra-network: BenchSequencerHandle::into_driver_parts + BenchDriverParts (inert BlockSyncHandle + BlockApplyFinished->ApplyFinished shim) and BenchCommitter::wait_for_verified_tip, behind internal-bench. - zebrad: internal-bench feature; bench_api re-exports the (unmodified) driver and BlocksyncThroughputProbe; mod zakura widened to pub(crate). - zebra-replay-bench: rewire apply-sequencer; add a seed-headers subcommand (one-time Zakura header-store seeding of the base); depend on zebrad. - Harness/docs: point at the cleanly-pruned 1.85M base + 1849958..1899957 window; add the commit/verify-timing plot script.
Fix a memory blow-up in the offline apply-sequencer bench and add the instrumentation used to localize it. All instrumentation is env-gated and inert in production. Prefetch OOM: - The parallel deserialize path used an unbounded worker->reorder channel, so when the consumer backpressured (the read-bound sandblast region ~1.86M) the reader raced ahead and deserialized a large swath of the window into RAM -- each block's Orchard Halo2 proof is large -- exhausting memory. Bound read-ahead with a permit channel: at most `capacity` blocks may be read but not yet emitted, while the worker->reorder channel stays unbounded so the in-order reorder thread can never deadlock waiting for the next block. - apply-sequencer: feed backpressure on the committed tip (ZRB_MAX_FEED_AHEAD) bounds the sequencer applying set; bench.rs exposes committed_tip. Profiling instrumentation: - stage_timing: add enabled() so callers can skip trace-only aggregates. - commit_pressure: record process RSS per traced row (separates RocksDB-internal growth from total process growth). - block.rs: per-category commit-batch key attribution (shielded/transparent/trees/...) and spent-UTXO create->spend distance buckets (cache-locality measurement). - disk_db: env-tunable RocksDB knobs (background jobs, subcompactions, memtable budget) for compaction experiments; default behavior unchanged. Tooling: - by-height latency/throughput/keys-by-category plot script for a trace dir.
The transparent address index (balances, address->utxo, address->tx) is RPC-only state, not consensus. A node that is both pruned and checkpoint-syncing now omits it -- just as pruned mode already drops raw-transaction storage -- removing the per-block address-balance reads and the address-index writes. In high address-churn regions (e.g. the 2022 sandblast consolidation) the index was ~74% of the transparent write volume, so dropping it is a large commit speedup; the UTXO set, value pool, and nullifiers are byte-identical. - Gate: `Config::skip_address_index()` = `Pruned && checkpoint_sync`. An archive node keeps the index; so does any node with checkpoint sync disabled (full semantic verification), even when pruned. Unit test covers all three branches. - Commit path: `block.rs`/`transparent.rs` elide the address-balance reads and the three address index column families when skipping; the UTXO-set writes/deletes still run. - Query path: the `getaddressbalance`/`getaddressutxos`/`getaddresstxids` read handlers return a clear "address index disabled in pruned storage mode" error rather than wrong (empty) results. Also refines the offline spend-distance instrumentation buckets and adds an A/B throughput/keys-by-height plot script.
Flip the offline replay bench from Archive to StorageMode::Pruned with checkpoint sync, the real fast-sync configuration. This exercises the pruned-only paths in the bench: raw-transaction pruning and the transparent address-index skip (now gated on pruned + checkpoint_sync, with the env override removed). The per-run fork is a throwaway copy, so online pruning never touches the base snapshot. Verified: the snapshot opens in pruned mode without tripping the one-way pruned reopen guard, the run commits through the gate (hash matches), and the address index is skipped at runtime (sandblast keys_transparent drops to the UTXO-set floor) without any env override.
p0mvn
force-pushed
the
perf/block-exec-opt
branch
from
June 30, 2026 22:46
3314965 to
0bf6fee
Compare
Defer the per-block spent-UTXO resolution (utxo_by_out_loc deletes + the transparent value-pool debit) off the finalized commit critical path and reconcile it in a batched pass on a dedicated worker thread, in the checkpoint-trusted fast-sync range. In the sandblast spam range the per-block spent-UTXO reads (~439 cold random reads/block) are the dominant commit cost; a ceiling probe showed removing them is +209% sandblast throughput. The reads can't be cached (97% of spends are >4096 blocks old) but can be deferred and batched: - per block: record spent outpoints into an in-memory window, write UTXO creates + headers + nullifiers + the VCT root fold inline, skip the spent-side value pool; - every `defer_reconcile_interval` blocks: a reconcile resolves the window's spends from disk (sorted/deduped), recomputes the value pool, and writes the deletes + value pool (chain_value_pools tip + per-height BlockInfo) atomically; - v2 runs the reconcile on a worker thread (capacity-1 handoff) so it overlaps assembly: the worker owns the value pool, processes windows FIFO, and writes keys disjoint from the assembler's creates (no shared-state races). Result (offline replay bench, mainnet sandblast range): the deferred run is byte-identical to the non-deferred run (value pool, ~20M-entry UTXO set, and per-height BlockInfo digests all match), and v2 lifts sandblast throughput +73% (overall +43%). All env/config-gated and default-off; no production path is enabled. This is a measurement PROTOTYPE: see docs/design/deferred-transparent-reconcile.md for the correctness invariants, results, and the production-readiness gaps (handoff drain barrier, crash recovery, RPC guards) that must land before it can be turned on. Adds a cf-dump bench subcommand + column-family verification digests for the byte-match.
`SequencerTask::publish_view` ran on every body and control event and each time recomputed the reserved-byte total via `WorkQueue::reserved_bytes()`, an O(pending + in_flight) scan of the whole work queue, purely to feed the `sync.block.budget.audit_drift` consistency metric. As the apply/commit backlog grew to thousands of blocks during checkpoint sync, that per-event scan became quadratic and saturated the single sequencer task, starving the commit-compute threads and freezing body commits for tens of seconds (observed ~18-30s stalls around the peak-backlog heights via mid-stall thread dumps). The published `SequencerView` already uses the budget's O(1) running counter, so the full scan is only a drift check. Sample it at most once per `BUDGET_AUDIT_INTERVAL` (1s) instead of on every event, keeping the audit while removing it from the hot path.
Make the deferred transparent reconcile prototype safe to benchmark on a real, disposable below-checkpoint node. The reconcile logic is byte-match-verified; these are lifecycle guards around it, all no-ops when the feature is off. - Reachability: expose defer_transparent_reconcile + defer_reconcile_interval as [state] config fields (default off), so a node can opt in from zebrad.toml. - Height-bound gate: defers_transparent_spends_at requires height <= max_checkpoint_height, so every above-checkpoint block commits inline (non-deferred). Unit-tested. - Handoff drain barrier (consensus-critical): before the first above-checkpoint block, flush in-flight blocks, drain the reconcile window (flush_and_join the worker), and reseed the pipeline value pool from disk, so the semantic verifier never reads a stale/superset UTXO set or a lagging value pool. The common handoff (checkpoint-verified channel close) drains the same way. - Format check + clean shutdown: check_new_blocks skips while deferral is configured and the tip is in the deferred range (the BlockInfo/value-pool lag is expected there); the reconcile worker is flush_and_join'd at the handoff and via a Drop safety net. Crash recovery + RPC guards remain out of scope (disposable-node model, documented in docs/design/deferred-transparent-reconcile.md). Byte-match preserved: value pool + UTXO set + per-height BlockInfo identical to a non-deferred run.
The v2 reconcile worker was single-threaded while the committer's other cores sat idle, so it capped the deferral win at +73% sandblast. Parallelize its two heavy per-interval passes on the rayon pool: - spent-UTXO resolution (output_location + utxo_by_location per outpoint) is CPU-bound page-cache reads, embarrassingly parallel over the window's deduped outpoints; - the per-block value-pool change (chain_value_pool_change re-folds every tx's value balance) is independent per block, so compute the (delta, block_size) per block in parallel, then apply the deltas in a sequential running-sum for the per-height BlockInfo + tip pool (chaining stays ordered, so the arithmetic is byte-identical). Uses the global rayon pool, not the committer's treestate pool, so it doesn't steal from the committer's critical path (which is light in the deferred range). Byte-match preserved (value pool + UTXO set + per-height BlockInfo identical to a non-deferred run). Offline replay bench, mainnet sandblast range: sandblast throughput +73% -> +114% vs baseline (210 -> 284 blk/s), ~60% of the ceiling headroom. Remaining serial cost is the per-interval delete-batch build + write (a follow-up).
…r incrementally Replaces the earlier throttle band-aid with the real fix. When headers race far ahead of the body tip, the sequencer's WorkQueue holds the whole lag (100k+ pending heights) in mutex-guarded BTreeMaps, and two O(n) operations ran under that lock on the hot path: - `reserved_bytes()` re-summed reserved request bytes across pending + in_flight on every `publish_view` (i.e. every body/control event); and - `advance_floor()` ran a full-map `retain` to drop committed heights on every floor advance. As the backlog grew both went quadratic, saturating the single sequencer task and serializing the work-queue lock so commit and download stalled together (mid-stall dumps showed the sequencer in `retain`/`reserved_bytes` with peers blocked on the lock). Fixes: - `WorkQueueInner` maintains a `reserved_bytes` running counter, updated at every ledger transition; `reserved_bytes()` returns it in O(1). The per-event budget audit cross-checks it against the independently-maintained ByteBudget, and a new unit test asserts the counter never drifts from a full scan across every transition. - `advance_floor`/`reset_above` pop only the committed prefix/suffix (O(removed · log n)) instead of a full-map scan. Reverts the `BUDGET_AUDIT_INTERVAL` throttle from the previous commit: with `reserved_bytes` O(1), the audit runs every event again with no work-queue scan.
`publish_view` builds the `SequencerView` on every body/control event, and four of its fields each scanned the whole `applying` map (the apply backlog, thousands of entries under stall): `applying_buffered_bytes`, `submitted_applying_bytes`, `submitted_applying_count`, and `unsubmitted_applying_count`. Same O(n)-per-event anti-pattern as the work-queue scans just fixed. Maintain `applying_buffered_bytes`, `submitted_applying_count`, and `submitted_applying_bytes` incrementally on the Sequencer at every applying insert/remove and submit-flag flip (`bytes` is immutable after insert); derive `unsubmitted_applying_count` as `applying.len() - submitted_applying_count`. All four accessors are now O(1). A new unit test asserts the counters never drift from a full scan across insert/submit/unsubmit/remove/commit-release/reset.
Per-phase timing (env-gated ZRB_RECONCILE_DEBUG) showed the reconcile worker is no longer the bottleneck after A1 (assembler backpressure = 0, worker 32-56% idle), but its resolution par_iter (~900k outpoints) shared the global rayon pool with the assembler's raw-transaction serialization par_iters, so the worker's large task starved the assembler's on the shared queue. Run the worker's per-interval reconcile on a dedicated RECONCILE_POOL so it no longer contends with the committer's global-pool work. Byte-match preserved (a scheduling-only change: value pool + UTXO set + per-height BlockInfo identical to a non-deferred run). Offline replay bench, mainnet sandblast range: sandblast throughput +114% -> +143% (284 -> 330 blk/s, ~86% of the +209% ceiling); overall +68%. The remaining sandblast gap (330 vs the ~470 light-region rate) is now assembler-side (the per-block commit of large blocks: creates, tx serialization, nullifiers, flush), not reconcile-side.
…o perf/block-exec-opt
Add panels to the checkpoint-sync dashboard/recorder to distinguish peer/download problems from commit-side stalls: - Download: blocks received / s (rate of `sync.block.body.received`), and the floor-gap available / servable peer gauges (`sync.block.floor_gap.*`) — the head-of-line-on-the-floor signature (available drops to 0 while servable > 0). - Peers: connections accepted / s and closed / s (`zakura.p2p.conn.accepted`, `.conn.closed.neutral`) to surface churn, plus dial failures / s (`zakura.p2p.discovery.dial.failed`) for discovery/connectivity trouble. Recorded into per-run samples.jsonl (via PANEL_KEYS) so past runs can be compared.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.