fix(tests): deflake flush-observation tests + tolerate concurrent sweep race

[JustMaier]

Problem

A family of #[serial] concurrent_engine tests observe post-flush state after a fixed thread::sleep or after wait_for_flush(alive_count). Both under-synchronize: the flush loop publishes the snapshot and bumps alive_count/publish_count before it does the post-publish disk I/O — there's a deliberate std::thread::yield_now() between the publish and the ops-log append (concurrent_engine.rs:2937). Under scheduler contention (default --test-threads, CI, or a concurrent bulk load) the observed state hasn't landed yet, so the assert races.

The victim rotates by scheduling: CI hit test_flush_thread_appends_ops_to_shard_stores (at two different asserts across runs); locally under load I reproduced test_sync_filter_values_clear_all, test_cursor_persists_via_merge_thread, test_put_bulk_persists_to_docstore, test_put_bulk_loading_then_persist, test_delete_cleans_filter_and_sort_bits, and test_get_documents_preserves_order. All the same root cause.

src/concurrent_engine.rs is byte-identical to main — this flake is not caused by any recent feature diff (it originally surfaced blocking PR #328, whose diff never touches this file).

Fixes

Test synchronization (concurrent_engine.rs):

• Add wait_until(max_ms, cond) — poll a condition instead of sleeping.
• Disk/docstore-observation tests now poll the actual asserted condition (ops_count > 0, filter/sort shard present, cursor persisted, all docs readable) with a generous deadline.
• In-memory-snapshot tests use wait_for_flush_quiet / wait_for_flush(0) instead of a bare sleep or a wait_for_flush(count) that never changes on an upsert.

Production robustness (shard_store.rs) — a real bug found while deflaking:
Opening a ShardStore runs a recursive orphan-.new sweep. When it races a live writer's compaction rename (packed sort shards are written as <field>.new then renamed on every sort flush), the vanished file returns NotFound, which sweep_dir counted as a hard failure → ShardStore::new returned Err → the test's .unwrap() panicked with sweep failed for 1 orphan .new file(s). NotFound means the orphan is already gone (the desired end state), so it's now treated as success in both the promote-rename and delete branches. Idempotent and correct for the crash-recovery startup sweep too.

Verification

Full lib test suite (--skip test_min_tracked_value_after_expansion, known deadlock) run under a concurrent 107M dump (heavy IO/CPU amplifier):

	branch baseline	this PR
full-suite runs failing	3 / 4	0 / 4

The sweep fix is what closes the last gap — after the test-sync fix alone, test_flush_thread still failed 3/4 at the sweep race; with both, 4/4 pass.

🤖 Generated with Claude Code