Calibrate token-guess outcomes against no-learning floors

[Al-does]

Why

Reviewing the mess3_token_guess_cycle_1 setup for conceptual problems turned up one that invalidates the headline reading: neither outcome axis starts near zero, and nothing in the study measured where they do start.

• A randomly initialised transformer that never trained reads belief R² = 0.8823 ± 0.0055 at chance accuracy. The reported reward_only (0.8552) and max_entropy (0.8558) arms are below that floor.
• A policy that echoes the last visible token scores 0.6735 against a Bayes ceiling of 0.6893. The entire usable accuracy range is 1.6 percentage points, and the three PPO arms sit inside noise of the echo baseline.

Every arm plotted against the floors it has to clear

What this adds

baselines.py

• expected_accuracy_band scores the agent, the echo baseline, and exact Bayes on the agent's own belief trajectory by expected hit rate rather than realized hits. All three share one realization of the process, which removes the sampling noise that otherwise swamps a range this narrow, and reports the agent as a fraction of the range.
• trivial_feature_r2 fits the study's own rank-2 affine probe to last-token, last-two-token, and own-action one-hots, so the floors are directly comparable to the reported r_squared.
• residual_r2_beyond_recent_tokens wraps the library's existing conditional_residual_r2, which the belief-geometry study already uses but this one did not.

analysis.probe_checkpoint emits all of it, so every future probe is self-calibrating.

untrained_reference/ is a new leaf that probes three freshly initialised transformers with the identical environment, architecture, probe, and rollout seeds. Only the weights are untouched.

NOTES.md gains a reference-band section that supersedes the earlier interpretation.

Verification

uv run pytest -q -m "not slow" → 60 passed, 1 skipped. Four new tests pin the floors: the echo and Bayes policies land at 0.0 and 1.0 of the range, recent-token features score 0.75–0.96, and a probe fed the exact belief clears the within-branch residual test at >0.99.

untrained_reference was run twice at full budget and reproduced identically (0.8877 / 0.8748 / 0.8844).

Not addressed here

Three further setup problems came out of the review and are left for discussion rather than changed unilaterally, since they alter the recipe:

1. NextTokenGuessTask.resolve_action ignores the action, so the process is passive and only the immediate reward term of the GAE sum carries action-dependent signal. At γ=0.99, λ=0.95 the rest inflates advantage variance 7.7×, costing about 8.7× in sample efficiency.
2. For the same reason the max_entropy bonus depends on state alone and cancels out of the advantage. With entropy_coeff=0.0 the arm is a provable no-op, which is why it matched reward_only to four decimals.
3. delay is not a usable difficulty knob: at delay=0 the graded token is already visible and echoing it scores 1.0000. Widening both axes needs a noisier emission channel and a stickier chain instead.

_{To show artifacts inline, enable in settings.}