v0_to_v1_interrupted migration test can hang on wait_until_ready

[zancas]

Symptom

cargo nextest run -p zaino-state chain_index::tests::finalised_state::migrations::v0_to_v1_interrupted was observed to hang indefinitely (no timeout, no panic) on branch accelerate_unit_tests_replace_sleeps. Source: packages/zaino-state/src/chain_index/tests/finalised_state/migrations.rs:84-148.

The test exercises a three-phase flow:

1. Spawn v0 db, full-sync, wait_until_ready, shutdown. (lines 117-128)
2. Spawn a raw DbBackend::spawn_v1, partial-sync 50 blocks, shutdown. (lines 132-137)
3. Re-spawn ZainoDB at v1 to drive the v0→v1 migration over the partial v1 directory left by phase 2, then wait_until_ready. (lines 142-147)

The hang manifests inside one of the wait_until_ready calls (likely the post-migration one at line 143, but unconfirmed).

Notes from initial investigation

Ruled out: NamedAtomicStatus::store broadcast race

I spent some time wondering whether the load-then-send_replace pattern in NamedAtomicStatus::store (packages/zaino-state/src/status.rs:82-93, pre-#1055-review-fix) could explain the hang. It cannot, in this test:

• All three wait_until_ready implementations on this path are poll-based:
  • ZainoDB::wait_until_ready — chain_index/finalised_state.rs:384-393
  • LmdbLifecycle::wait_until_ready — chain_index/finalised_state/db.rs:117-126
  • DbBackend::wait_until_ready — chain_index/finalised_state/db.rs:256-266
• All three poll NamedAtomicStatus::load(), which is *self.inner.borrow() — a live read of the watch's current value, not a changed() subscription.
• The store race can produce at most (a) a cosmetic double-broadcast for one transition, (b) a no-op suppression against a stale snapshot whose decision is still consistent with the value observed at load time. Neither leaves the stored value stuck at a non-Ready state.

So wait_until_ready polling will always observe the latest write. The hang has another cause.

Plausible candidates (speculation; needs reproduction)

1. Migration logic with partial-v1 state. The v1 directory in phase 3 already contains 50 blocks from phase 2's raw DbBackend::spawn_v1 write. If the migration in ZainoDB::spawn reads pre-existing v1 rows and gets confused (e.g., recomputes a validated-tip cursor that disagrees with the on-disk tip), the sync/validate loop may never mark Ready.
2. Sync loop stalling at the partial tip. The v1 sync loop in chain_index/finalised_state/db/v1/write_core.rs transitions Syncing↔Ready/RecoverableError around per-iteration progress. A stuck "no forward progress, no terminal error" branch would loop without ever calling store(Ready).
3. LMDB env release timing. The two 1000ms tokio::time::sleep calls (lines 130, 139) are intended to let LMDB locks settle between shutdown and the next spawn. On slow hardware or under heavy load, this could be too short — though that should produce a spawn error, not a wait_until_ready hang.

Next steps

• Reproduce locally (it was non-deterministic on my run — first-fail observation, no repro count yet) and identify which wait_until_ready hangs (line 125 or 143).
• Add tracing instrumentation at every status.store(...) site in chain_index/finalised_state/db/v1/write_core.rs and chain_index.rs so the actual status sequence under hang conditions is visible.
• Inspect the partial-v1 directory left after phase 2 (e.g., tempdir contents, metadata row, headers table) to confirm what state the migration is reading.
• Cross-check whether v0_to_v1_partial (lines 150-216, which does a full phase-2 sync) reliably passes — if so, the suspect zone is migration handling of a partial v1 specifically.

Context

• Branch: accelerate_unit_tests_replace_sleeps
• Observed during work on PR ● refactor(status): back NamedAtomicStatus with tokio::sync::watch #1055 review feedback (the send_if_modified refactor for NamedAtomicStatus::store).