From 1258d8c6e58ecdcd46cb3d2608f30d792fb6d227 Mon Sep 17 00:00:00 2001 From: Blake Emerson Date: Sat, 20 Jun 2026 23:55:29 -0500 Subject: [PATCH] fix(finalised-state): guard against concurrent background backfills in sync_to_height The ChainIndex sync loop calls FinalisedState::sync_to_height on every iteration. With LONG_RUNNING_SYNC_THRESHOLD = 10, any gap larger than ten blocks takes the background path, which spawns a detached task and returns immediately. begin_background_op() only bumps a counter for wait_until_synced and does not gate re-entry, so during a from-genesis / far-behind catch-up (where the gap stays large for the whole sync) every loop iteration spawns ANOTHER backfill task. The concurrent backfills race the single LMDB writer and the ephemeral-reference swap, producing out-of-order "cannot write block at height X; current tip is Y" (Y > X) errors, exhausting the retry budget into CriticalError, and ultimately a SIGSEGV inside LMDB (the environment is opened without WRITE_MAP and with MDB_NOTLS, so concurrent writers corrupt its dirty-page bookkeeping). Return early when a background op is already in flight so only one backfill runs at a time; the sync loop re-checks on its next iteration and the existing task keeps making progress. Near-tip syncs (gap <= threshold) use the inline path and are unaffected. Validated on Zcash mainnet from genesis: before, the node crashlooped (6 restarts, stuck at h~233k with no forward progress); after, 0 restarts, a single backfill, CPU ~30-40 cores -> <1, memory ~15 GiB -> ~1.3 GiB. Co-Authored-By: Claude Opus 4.8 (1M context) --- .../src/chain_index/finalised_state.rs | 21 +++++++++++++++++++ 1 file changed, 21 insertions(+) diff --git a/packages/zaino-state/src/chain_index/finalised_state.rs b/packages/zaino-state/src/chain_index/finalised_state.rs index 579c9c75b..bca68f08f 100644 --- a/packages/zaino-state/src/chain_index/finalised_state.rs +++ b/packages/zaino-state/src/chain_index/finalised_state.rs @@ -708,6 +708,27 @@ impl FinalisedState { let source = source.clone(); if sync_is_long_running { + // Re-entry guard: only one background backfill may run at a time. + // + // The ChainIndex sync loop calls `sync_to_height` on every iteration, and with + // `LONG_RUNNING_SYNC_THRESHOLD == 10` any gap larger than ten blocks takes this + // background path, which spawns a detached task and returns immediately. During a + // from-genesis (or far-behind) catch-up the gap stays large for the whole sync, so + // without this check every loop iteration spawns *another* backfill task. Multiple + // concurrent backfills then race the single LMDB writer and the ephemeral-reference + // swap below, producing out-of-order writes ("cannot write block at height X; current + // tip is Y" with Y > X), exhausting the retry budget into `CriticalError`, and + // ultimately a SIGSEGV inside LMDB (the environment is opened without `WRITE_MAP` and + // with `MDB_NOTLS`, so concurrent writers corrupt its dirty-page bookkeeping). + // + // `begin_background_op` only increments a counter for `wait_until_synced`; it does not + // gate re-entry. Bail out early when a background op is already in flight — the sync + // loop simply re-checks on its next iteration and the existing task keeps making + // progress. Near-tip syncs (gap <= threshold) take the inline path and are unaffected. + if router.has_background_ops() { + return Ok(()); + } + // Register the background sync in the foreground, before spawning, so `wait_until_synced` // cannot observe a "no work in progress" state between this method returning and the // spawned task starting. The guard is moved into the task and drops when it completes.