feat: verify a probe's answer key before it reaches a developer

[imkp1]

Adds stage 4. Stage 3 writes the question, every option, the key, and the explanation in one call, in sequence, and never re-reads an early option against a later one. Every failure that produces is already forbidden in its prompt, so more instruction is the approach that has been tried. This is the check instead.

verify.py reads the options back cold — no key, no explanation, no seed, no transcript, no repository — and judges each one true or false with a stated reason. The probe survives only if exactly one is true and it is the stored key. It takes a Probe and nothing else on purpose: a judge that has seen the reasoning behind the answer is the model agreeing with itself, and keeping the transcript unavailable at the type level is what stops that eroding.

Backtested before anything was wired in

All 47 stored probes, against the real CLI:

verified	44
discarded	3 (6%)
two true options	0
cost	$0.041/probe vs a $0.226 stage 3 baseline (~+10% per kept session)

One discard is a genuinely false key: a PEP 503 question whose answer claims fault-line normalises onto faultline. It does not — fault-line normalises to fault-line. The real collision is Faultline.

Sensitivity was measured separately, by hand-planting a known-true option into five clean probes: caught 5/5, each at the exact planted index.

Two designed pieces are deliberately absent

• Repointing correct_idx at the option the judge preferred. That judgment landed 3 times in 47 and would have been right once; the other two were local-file questions where the judge could only reason from the stem's premises. A verifier that installs a false key two times in three is the bug it exists to catch. It discards instead.
• The stage 3a/3b split, and check A with its repair loop. Both existed to fix two-true option sets. Stage 3 does not measurably produce them, so the repair path would be code that never runs.

The original gate for this work required probe 48 to fail verification — key "kernel serialises dirent creation" vs distractor "mkdir is a single syscall, two callers can't both succeed", believed to be two true options. It does not fail, batched or in isolation with the key not even visible, and the judge is right: read/write are single syscalls and are not atomic against concurrent callers, so that distractor asserts a false causal chain. Probe 48 is a good probe.

Behaviour

• A call failure keeps the probe; only a judgment discards. A CLI that cannot run has said nothing about the question, and treating silence as rejection would empty the queue every time the model was unreachable.
• New unverified session verdict and a sixth EmptyReason, with a line on both /grask surfaces. A discarded probe is not silent, not error, not ask.
• CAPTURE_STALE_MINUTES 20 → 30: the pipeline's worst case is now 24 minutes of stage timeouts, and a marker that expires early promises a probe that is still coming. That is a ceiling for crash detection, not a runtime — measured end to end over the real store, capture runs a median of 25s and has never exceeded 166s.

The suite was making real calls

Adding stage 4 gave eight existing tests a real claude -p call each, and they kept passing — the only symptom was test_capture.py going from 0.11s to 49s. The cause is structural: a test injects the stages that existed when it was written and inherits the real default for anything added later.

conftest now refuses any subprocess whose argv[0] ends in claude. It raises through pytest.fail rather than AssertionError, because capture_session catches Exception on purpose and would swallow the guard into an error row — the author of stage 5 would see that instead of a message naming what they did.

Reviewing the branch against itself

Five defects, then two more, all found after the stage above was written:

• A discard spoke for sessions it wasn't. unverified outranks caught_up and was scoped only by age, so one discard suppressed "you are caught up" for seven days — including for a developer who had since earned a probe and answered it. It now also requires that nothing has minted a probe since.
• A discarded question read as free. No probes row exists on that path, so stage 3 and stage 4's spend — the expensive half of the session — was recorded nowhere, in the very report used to decide whether stage 4 earns its price. New sessions.discarded_usd, kept apart from cost_usd so SUM(discarded_usd) stays answerable. outcomes had the mirror-image blind spot: it summed seeds through the probes join, losing stage 2 for any session with a seed and no probe. On a two-session fixture the reported total went from $0.46 to the true $0.83.
• Cost and duration covered different stages. verify billed stage 4 into cost_usd but not duration_ms.
• A judgment's indices were trusted one at a time. Every element in range and every element "index": 0 is individually valid and collectively meaningless; read literally it moved the true verdict onto option 0 and discarded a good probe for disagreeing with a key it never disagreed with. Indices are now accepted only as a permutation, or as position throughout.
• A stage that gave up recorded nothing. seed and probe raise LLMError and capture caught neither, so both fell to the catch-all: an error row with no cost and no seed. Probe exhausting its retries is the pipeline's most ordinary failure and it reported three billed calls as $0.00 while discarding a stage 2 result that was never at fault. LLMError now carries its spend, and capture keeps the seed whenever stage 2 produced one.
• Nothing could use a kept seed. add_probe was reachable only from capture, so "the seed is still stored" was a consolation, not a recovery — a control priced against a redemption nobody had built. grask.reprobe re-runs stages 3 and 4 over unprobed seeds. Both stages, not a shortcut past the check; explicit and --go-gated, bounded to PROBE_TTL_DAYS, and skipped once the transcript has rotated.
• doctor called a stale skill healthy. grask install copies SKILL.md into ~/.claude/skills and upgrading the package does not re-copy it. Stage 4 is where that first bites: a skill enumerating five empty_reason codes invites the model to flatten an unfamiliar sixth into one it recognises, and unverified softened into "you're caught up" is exactly what the new reason exists to prevent. Plugin installs are unaffected.

Stale text went with them: SKILL.md listed five empty-queue reasons in both the packaged copy and the plugin mirror, and stage counts and timings were wrong in storage.py, capture.py, llm.py and design.md.

Release

Bumped to 0.1.0-rc6 (pyproject.toml, .claude-plugin/plugin.json, uv.lock). The plugin cache is keyed by version, so shipping this under rc5 would no-op for anyone already on rc5, and PyPI holds 0.1.0rc5 permanently. The CHANGELOG gained the stage 4 entry it never had.

Verification

408 passed, ruff and mypy clean, green on the 3.8 CI floor. The shipped module reproduces the backtest against the real store: probe 42 rejected, 47 and 48 verified.

🤖 Generated with Claude Code