fix(activation): gate verification on an observed publish (v1.1.49, step 1 of 3)

[JustMaier]

Step 1 of the verifier rework, shipping alone as v1.1.49. It adds the publish gate; deleting the barrier, Inconclusive, and inconclusive_total is step 2 and is deliberately held until the prediction in §4 is read in prod.

Read the evidence before the diff. The diff is small. What justifies it is a design that was built, broken, and rebuilt — and three bugs that lived where two correct things met.

---

1. Mutation G: survived the design we almost shipped, inexpressible in this one

The first implementation kept a publish counter in an atomic beside the ArcSwap, ordered against the store by hand. It was accepted on the explicit premise that the coupling was pinnable by a test. So the test was written, and then the killing mutation was run:

flush_pub_count.fetch_add(1, Ordering::Release);   // MUTATION G — moved ABOVE the store
inner.store(Arc::new(staging.clone()));

test_publish_count_advances_only_after_publish ... ok      # SURVIVED. Zero tests died.

The invariant is not pinnable. The violation window is nanoseconds; any observational test polls straight past it. The premise was false — and we would have shipped a comment-guarded invariant with a decoration test in front of it, which looks more rigorous than the honest version. A test that cannot fail is worse than no test, because it is trusted.

This PR removes the ability to express the bug rather than testing for it. publish_seq lives inside InnerEngine, so the seq and the state it certifies cross the ArcSwap in one atomic store:

staging.publish_seq = flush_cycle;          // stamped after apply, immediately before...
inner.store(Arc::new(staging.clone()));     // ...the store. One act.

There is no separate increment to hoist. The counter cannot run ahead of the snapshot because it is the snapshot. So no Release/Acquire pair to get right, no dedicated accessor, no ordering signpost, nothing to rot — those requirements do not get satisfied, they cease to exist. The decoration test is deleted, with a note in its place recording that it was mutated, survived, and why no replacement is needed.

flush_publish_count reverts to Relaxed and stays reporting-only; its doc now says why nothing may depend on it — it sits beside the snapshot and can drift from what is published, which is the exact defect removed.

2. Three bugs, all at seams. Zero inside the logic.

The counter's own logic was desk-traced four times and was correct every time. Every real bug was at a seam between two individually-correct features:

#	seam	failure	found by
1	v1.1.44's persisted ring x v1.1.49's gate	flush_publish_count is process-local and restarts at 0; a persisted tag names a count the new process never reaches, so the slot waits forever, unverified	asking, not reading — by someone without the source locally
2	"restored slot" x "in-flight batch"	a restored slot has nothing to wait for (already in the boot snapshot); gating it on a future publish strands it on any index quiet enough not to write	the rehydrate test, on its first run
3	field-unload x seq	the two InnerEngine rebuild sites have no seq; publish_seq: 0 compiles, reads fine, and rewinds it — stranding every slot tagged above it	writing the rebuild sites
4	gate operator x fixture values	> -> >= makes the gate a total no-op and 79 tests pass. The tests used u64::MAX and 0 — two values where > and >= agree — and never published == c, the state every live slot occupies	reviewer5, by mutation
5	is_open x its only caller	the gate was pinned as arithmetic; every test built its gate BY HAND, so VerifyGate::Ready at the enqueue site passed 1234 tests. Operator pinned, argument open	reviewer5, by mutation
6	one unload path x the other	my own bug-3 pin was half a pin — it drove save_and_unload only, so mutating the other rebuild site alone SURVIVED. Mutating both at once proves only that AT LEAST ONE is covered, which is not the claim	me, forced by the double-run rule

All of them fail silently: no log line, no counter, no alarm. A slot that quietly stops being watched looks exactly like a slot with nothing wrong. None of it was reachable by reading either side.

Six seams, zero interior bugs. The logic was desk-traced four times and was correct every time — because nothing was there. Rigour inside a component cannot see between components. Findings 4-6 are the sharper version: they are seams in the tests, not the code. The design got adversarial attention; the tests protecting it got none, and were individually correct and collectively blind.

Methodological, and load-bearing for anyone re-running this: this suite has pre-existing flakes (main fails ~1-in-6 full-suite runs in the same module — bare thread::sleep(50ms) where a flush-wait belongs). That makes mutation testing report FALSE KILLS. A single-run "it kills" is unreliable in the direction that comforts you; a SURVIVES cannot be manufactured by a flake, only a KILL can. Every kill here was run twice. Finding 6 only surfaced because that rule forced me to mutate sites separately rather than together.

Bug 2 is why VerifyGate is an enum, not a number:

enum VerifyGate { Ready, AfterPublish(u64) }

Ready is not a sentinel — it is a different kind of thing. No u64 can say "nothing to wait for" when the gate is "count moved past my tag" and the counter starts at 0. Encoding it as a number forces a false choice: strand restored slots, or loosen the comparison for everyone and judge live slots pre-publish (the quieter, worse option).

Mutation H (Ready -> AfterPublish(5_000) on load — "we're losing information, keep the seq") kills test_activation_verify_ring_survives_restart: checked: 0 vs 1. Run and reverted. Both call sites carry a comment naming that seductive change.

The pre-existing v1.1.44 restart test would not have caught bug 1: it enqueued at tag 0, where persisting the tag is harmless. Green because the fixture dodged the seam — the same species as a test that cannot fail. The new fixture uses a realistic high count.

3. Correction: fourteen publish sites, not three

An earlier report of mine said "3 construction sites". That was InnerEngine construction; the publish surface is fourteen inner.store sites (I first said fifteen; four of the grep hits were comments — reviewer5's count is right, and the point is unchanged: 14 ≫ 3). This cuts against my own earlier recommendation: the external-counter design needed one ordering to hold, at fourteen sites, forever, guarded by a comment. It was nearly adopted on a count of three.

The stamp sits at store-time, which makes "did anything publish between the stamp and the apply?" unaskable rather than answered — for the stamp. But the enqueue does rely on a real property, and I originally overstated this as "the design does not rely on that". It relies on thread confinement: the apply, the tag and the store all happen on the single flush thread within one iteration, with no publish between, so the next publish is necessarily the one carrying the batch. Add a store in that span and the gate opens on a correctly-stamped snapshot that lacks the batch — the seq honest, the verdict wrong. That invariant is now stated at the enqueue site rather than credited to the wrong property.

Bonus: the ForcePublish handlers become correct by construction rather than harmless by luck of direction — they publish staging carrying its own seq, so the seq always describes the snapshot it arrived on.

4. The prod prediction — pre-registered, version-gated, and falsifiable

This is why step 1 ships alone. We have deterministic proof the gate works in a test. We do not have proof that our model of prod is right — that pre-publish reads are what produce publish-lag. This release makes that model falsifiable while the barrier is still in place to catch us.

This is not inert. It changes when the verifier reads: later, and only later. That is the safe direction — a slot verified late is fine, a slot verified early is the bug — but it is a behaviour change, and the prediction only exists because it is one.

⚠️ Version gate: this picture is for v1.1.49 only, and it INVERTS v1.1.48's

On v1.1.48, publish_lag_total climbing was CORRECT — #320 suppressing false orphans, explicitly pre-registered as "do not alarm".
On v1.1.49, the same counter climbing is the BAD outcome, because the gate is supposed to stop those reads happening at all.
Same counter, opposite meaning, one release apart. Check the version before reading this counter. Do not carry v1.1.48's note forward.

🔴 CHECK THE CODE ARM BEFORE THE MODEL ARM

A broken gate and a wrong model currently produce the SAME signature. If the gate is bypassed (see Mutations L/M below), the verifier reads pre-publish, the barrier fires, publish_lag climbs and checked_total holds — which is exactly the ❌ model wrong picture. We would go hunting prod for a cause that isn't there while the defect is one line at the enqueue site.

The discriminator, already exported and free: under a WORKING gate, _pending > _ready transiently on every activation, because slots genuinely wait a cycle for their publish.

⇒ _pending > _ready MUST occur during the window. If it never does, the gate never closed once — that is a CODE bug, not a wrong model. Fix the gate before concluding anything about prod.

Note the gauges alone cannot catch this: _pending/_ready detect under-draining, and a bypassed gate over-drains — ring depth ≈ 0, reads perfectly healthy. The relationship between them is the signal, not either level.

Pre-registered before deploy:

The LOUD term (checked first — without it the rest is unfalsifiable):

• 📈 bitdex_activation_verify_checked_total MUST keep climbing, at roughly its current rate.

  This gate is precisely the kind of change that can silently switch the verifier off: if the seq never advances, or the gate never opens, no slots are drained — and then publish_lag, inconclusive and redriven all go to zero, which is indistinguishable from a perfect result. A prediction made only of zeros cannot be falsified. If checked_total flatlines, this release FAILED, however clean everything else looks.

Then the quiet terms:

• ✅ Model confirmed: publish_lag_total rate drops and inconclusive_total -> ~0, while checked_total holds its rate. Slots stop being read pre-publish, so apparent orphans stop being manufactured ⇒ the barrier is provably redundant, and step 2 deletes something we watched become unnecessary.
• ❌ Model wrong — but ONLY after the code arm clears: checked_total holds, _pending > _ready did occur (so the gate is genuinely working), and publish_lag_total still does not drop. Then pre-publish reads were not the cause. We learn it with the barrier still there. Step 2 does not proceed.
• redriven_total should stay 0 throughout. If it moves, that is a real drop signal, independent of this change.

Why this can't wait for step 2: under Mutations L/M today, the still-present barrier catches the bypass and publish_lag climbs — the release fails loudly, which is exactly why step 1 ships alone and why it works. Step 2 deletes the barrier, which is the only thing covering the enqueue site. After step 2, a bypassed gate reads: ratio 1.0 ✅, ring depth 0 ✅, checked holding ✅, lag gone (deleted) ⇒ every surviving term confirms while the gate does nothing. So pinning the enqueue site is a hard prerequisite for step 2, and it lands now, while the barrier is still here to prove it works.

Reviewers: please attack the prediction as well as the code. If the falsification condition is wrong it needs to be wrong now — pre-registering it is what stops us reading whatever happens as success.

---

Diff

• InnerEngine gains publish_seq; stamped at store-time; carried across both rebuild sites.
• Verify ring: VecDeque<u32> -> VecDeque<PendingVerify { slot, gate }>. drain_activation_verify releases only slots whose gate is open, stopping at the first closed one — sound because ordering is structural: a single flush thread assigns strictly increasing seqs, so no config knob can perturb it (unlike a time-based delay).
• Rehydrated slots re-stamp to Ready; the on-disk format stays a plain slot list — no tag to serialize, so no version bump and no migration.
• published_seq() replaces publish_count_acquire().

Tests: 1292 passed, 1 failed (query_stream_full_channel_drops_oldest — always-allowed), 8 ignored. cargo check --features server,pg-sync --lib clean.

bd21987 (the external-counter shape) is kept beneath 255ecf4 unsquashed on purpose: the Mutation G evidence is the argument against the design we almost shipped, and squashing it would leave the conclusion without the reasoning.

🤖 Generated with Claude Code