fix(finalised-state): guard against concurrent background backfills in sync_to_height

[emersonian]

(AI-authored - this allowed Zaino to deploy in my test environment without crashing, in addition to setting the config sync_write_batch_bytes to 128MiB)

Problem

A from-genesis (or far-behind) mainnet sync crashes repeatedly with SIGSEGV and
never cleanly catches up. Logs show, on a loop:

WARN  write_core: cannot write block at height Height(211873); current tip is 214169
ERROR finalised_state: background sync_to_height failed after 5 attempts, giving up
WARN  finalised_source: background task didn't exit in time – aborting
INFO  chain_index: Starting ChainIndex sync loop
INFO  write_core: syncing finalised blocks 224445..=3385157   # a SECOND backfill loop
<exit 139 / SIGSEGV, no Rust panic>

Root cause

ChainIndex::start_sync_loop calls FinalisedState::sync_to_height on every
iteration. With LONG_RUNNING_SYNC_THRESHOLD = 10, any gap larger than ten blocks
takes the background path, which tokio::spawns a detached backfill task and
returns immediately. begin_background_op() only increments a counter for
wait_until_synced — it does not gate re-entry. During a large catch-up the gap
stays huge for the whole sync, so each loop iteration spawns another backfill task.

The concurrent backfills then race the single LMDB writer and the
init_or_take_ephemeral reference swap, producing the out-of-order
"cannot write block at height X; current tip is Y" (Y > X) errors, exhausting the
retry budget into CriticalError, and ultimately a SIGSEGV inside LMDB — the
environment is opened without WRITE_MAP and with MDB_NOTLS, so concurrent
writers corrupt its dirty-page bookkeeping.

Near-tip operation (gap ≤ threshold) takes the inline path and never hits this,
which is why only large catch-ups are affected.

Fix

Bail out of the background path early when a background op is already in flight, so
only one backfill runs at a time. The sync loop re-checks on its next iteration and
the existing task keeps making progress. Inline (near-tip) syncs are unchanged.

if sync_is_long_running {
    if router.has_background_ops() {
        return Ok(());
    }
    let op_guard = router.begin_background_op();
    ...
}

Validation

Zcash mainnet, from genesis, k3s/LMDB:

• Before: crashlooped — 6 restarts, stuck at h≈233k with no forward progress.
• After: 0 restarts, a single backfill task; CPU ~30–40 cores → <1, RSS ~15 GiB → ~1.3 GiB.

Note for reviewers

has_background_ops() is a counter read, so the check-then-begin_background_op
isn't atomic in general. It's safe here because the only production caller is the
single sequential ChainIndex sync loop. If you'd prefer hardening against future
concurrent callers, a dedicated AtomicBool claimed via compare_exchange (or a
tokio::sync::Mutex/TryLock around the spawn) would make it race-proof.

[idky137]

This must be called whether or not the sync is "long running", as there must only ever be a single active db writer. (see 82aa6d0)

[zancas]

I don't see a specific ask from the reviewer.

I am marking this draft. When the author decides they're ready for another review they can toggle back to review-ready.