live verifier judges worse frames than an offline replay of the same run (23 differing stage votes over 19 episodes)

[heyong4725]

The problem

The live realistic verifier and an offline replay of the same recorded frames reach different stage votes. On a 20-episode run they agree on the final verdict 18 of 19 times, but differ on 23 individual stage votes — and the live path is systematically worse:

stage	live	offline replay (same frames)
identity_overhead fail	18 / 19	13 / 20
containment error	18 / 19	15 / 20
upright error	18 / 19	14 / 20

That matters because A7 drives the loop from the live verdict. If the live path judges worse frames than a replay does, the agent optimises against a verifier that is noisier than the one we measure and report.

It also retracts a claim I made on #116: I reported live and replay agreeing "stage-for-stage" — that was 3 episodes, all failures, where agreement is cheap.

Evidence

fid20-I11 — 20 episodes, seeds 10..29, --verifier both with AISLE_FRAME_CAPTURE_PERIOD_S=5, so the same episodes are judged live by the node and offline by tools/judge_recorded_run.py from the recorded frames.

Also visible: the live run produced 19 sidecar records for 20 episodes, so one episode is still being lost despite the bounded pre-teardown wait added in #116.

Likely cause

The node processes queued events behind real time — its own log shows Discarding event for input joint_state due to queue size limit. judge_frames costs 3–5 s per episode, during which frames keep arriving. When the next goal triggers finish(), the frames it has processed can predate the true episode end, so the terminal frame it judges is not the terminal frame that was rendered.

That would explain the direction of the discrepancy exactly: an earlier terminal frame is one where the med is less likely to be in the tray, which fails identity_overhead, which removes the grounding the geometry stages need — matching containment/upright going to error at the same rate.

What would settle it

Compare, per episode, the sim stamps of the judged frames the live node used against the stamps checkpoint_stamps() selects offline. The VER-14 sidecar already records frames[].sim_time_ns per camera, so this is a few lines against data already on disk — no new runs needed. If the live stamps are systematically older, the diagnosis is confirmed.

Options once confirmed

1. Judge from a bounded buffer keyed by stamp rather than "latest processed". The node already retains frames by stamp; the bug would be that finish() fires before the queue drains. Draining pending camera events before judging is the minimal fix.
2. Judge at checkpoints instead of at episode end — the design VER-9 describes. Removes the end-of-episode rush entirely, at ~3–5 s per checkpoint, which the node cannot currently afford at a 5 s cadence.
3. Accept the divergence and drive A7 from the replay path, treating the live node as observability only. Honest, but gives up the live verdict A7 wants.

I lean 1, then re-measure. Option 2 is the principled answer but needs the judging cost down first.

Not blocking today

--verifier both is a sidecar: nothing consumes its verdict, and the VER-6 numbers published in #114/#117 are from the offline path, which is unaffected. This blocks A7, not current measurement.

IDs: VER-5, VER-6, VER-7 (live and replay must agree), VER-9.