fix(cache): close cache-maintenance races behind top-of-feed loss

[JustMaier]

Summary

Three fixes from the 2026-07-23 cache-race audit. Prod symptom: transient top-of-feed loss on cached bucket + reactionCount entries that self-heals at bucket_entry_ttl.

1. Stale-min remove race (CONFIRMED bug)

The async cache worker captures min_tracked_value at collect, evaluates removes unlocked, and applies later. A concurrent pagination expand() (4K→64K) lowers the floor and appends members between collect and apply, so apply removed the freshly-added legitimate members.

Fix: removes are tagged by origin.

• Filter-mismatch removes (removes): the slot no longer matches the entry's filter. Produced by evaluate_filter_work and by evaluate_sort_work's qualifies-but-unmatched branch. Always applied (membership-guarded).
• Sort-bound removes (sort_bound_removes): the slot's value fell below the floor captured at collect. Only these get the stale-min guard.

A sort-bound candidate was, by construction, non-qualifying under the captured floor, so the guard reduces to a single live-bound test: apply the remove only when the value still does not qualify under the current floor (a concurrent expand that lowered the floor to include it ⇒ keep the member). No captured-bound value needs to be carried.

Origin tagging is required (not just cleaner): a filter-mismatch remove can carry a sort value that qualifies under the live floor (e.g. a slot that stopped matching the filter but sorts inside the band expand opened). An origin-blind guard drops it and leaves a no-longer-matching slot in the result = transient false inclusion. (This was the review-caught gap in the first revision, where the guard was applied to all removes via a two-bound captured_min_tracked discriminator.)

Tests:

• test_apply_maintenance_keeps_members_added_by_concurrent_expand — collect→expand→apply; the expand-added member survives, a genuine below-floor drop still applies. Fails under unconditional apply.
• test_apply_maintenance_filter_mismatch_remove_always_applies — a filter-mismatch remove whose value qualifies under the live floor MUST still evict. Verified it fails under the origin-blind guard and passes after.

2. No-op sort-set suppression (flood amplifier)

process_set_op's sort branch and recompute_computed_sorts_for_slot's computed-target branch full-overwrote every bit layer unconditionally. The steady-state re-emitter's same-value sets thus became real sort mutations → bucket_changed_slots → maintain_bucket_membership flood.

Fix: both reconstruct the slot's currently-published value (ConcurrentEngine::reconstruct_sort_value, ~num_bits contains) and skip the clear+set — and the mutated-slot registration — when unchanged. None engine (dump/test mode) keeps the unconditional overwrite.

Staleness (documented at the call site): reconstruct reads the published snapshot (base + unmerged diffs). Within-batch re-sets of the same slot+field are collapsed upstream by dedup_ops (LIFO), so intra-batch staleness can't drop a genuine change. The one known residual is cross-batch: a value that moves away and back to its published value across batches has the return set suppressed and keeps its published sort position until the next genuine change — a narrow, self-healing window accepted in exchange for killing the flood.

Tests: test_set_op_suppresses_same_value_sort_write, test_recompute_suppresses_same_value_computed_sort — same-value set/recompute emit zero sort mutations; changed value still writes the full overwrite.

3. Built-in page-1 divergence canary

Config-gated (cache.page1_canary_sample_rate, default 0.0 = off, hot-reloadable via PATCH /indexes/{name}/config). On a sampled first-page bucket-sort cache hit it re-derives the first page from the slow path — bounded to the query limit, no doc fetches — diffs IDs, and on divergence increments bitdex_cache_page1_divergence_total{sort_field} and tracing::warns once per entry per 60s with the UnifiedKey, min_tracked_value, capacity, needs_rebuild, and bucket cutoff. Sampling is restricted to first-page queries (no cursor, no offset) — the top-of-feed case the prod symptom hit and the only offset-free fast/slow comparison. Counter registration and bridge wiring are #[cfg(feature = "server")]; the detection+warn path runs in any build.

Q4 / known-test note

test_min_tracked_value_after_expansion (the known-deadlocked test, skipped in the suite run) collides nominally with this area (expand + min_tracked_value) but exercises only single-threaded expand() bound bookkeeping — not the apply-time race this PR fixes. Fix #1 touches apply_maintenance_results / the evaluators, not expand() or min_tracked_value(), so it does not make that test fixable/relevant; its hang is environmental (orphaned-cargo target lock per the known-hangs reference), not a code deadlock in a path this PR changes. The new test_apply_maintenance_keeps_members_added_by_concurrent_expand covers the actual race directly.

Testing

cargo test --lib --features pg-sync (skipping the known-deadlocked test_min_tracked_value_after_expansion): 1255 passed, 8 ignored. cargo check --features server,pg-sync --lib clean, no new warnings.

🤖 Generated with Claude Code