Flaky: two_concurrent_backups_one_succeeds_one_returns_already_in_progress races under load

[utof]

Symptom

perima-app::backup_concurrent_returns_already_in_progress::two_concurrent_backups_one_succeeds_one_returns_already_in_progress fails intermittently:

thread '...' panicked at crates/app/tests/backup_concurrent_returns_already_in_progress.rs:49:5:
first backup should succeed: Err(BackupFailed { reason: AlreadyInProgress })

Observed blocking a git push pre-push hook (cargo nextest run --workspace --all-targets) on 2026-08-01. Passed on the immediately preceding run of the identical tree, so it is load-dependent, not deterministic.

Root cause

The test relies on a 10 ms sleep to establish ordering between two spawned tasks:

let h1 = tokio::spawn(async move { uc1.execute(...).await });
tokio::time::sleep(Duration::from_millis(10)).await;
let h2 = tokio::spawn(async move { uc2.execute(...).await });

tokio::spawn only schedules a task — it does not run it. The assertion at line 49 requires h1 to have entered execute and taken the AtomicBool guard before h2 is spawned. Under a loaded machine (full workspace nextest runs many test binaries in parallel), h1 may not be polled within the 10 ms window; h2 then wins the guard and h1 is the one that gets AlreadyInProgress.

The docstring calls this a "deterministic 10ms-stagger window". It is not deterministic — it is a timing assumption that inverts under scheduler pressure. Note the SlowAdmin { sleep_ms: 200 } ratio (20:1) does not help: the failure is starvation before h1 is first polled, not the length of the critical section.

Suggested fix

Replace the sleep with an actual synchronisation point, so h2 is spawned only after h1 is provably inside the critical section. E.g. give common::SlowAdmin a tokio::sync::Notify (or a oneshot::Sender) that fires on entry to backup, and await it before spawning h2. The 200 ms sleep can then shrink substantially, making the test both deterministic and faster.

Assertions themselves are correct and worth keeping — the guard behaviour under contention is real coverage. Only the ordering mechanism is unsound.

Scope note

Pre-existing on main (introduced in 5d1a6e6, backup slice-1). Surfaced during the transcription-v1 slice (#184) but unrelated to it. Concrete instance of the class tracked in #123.