Review the token-guess studies and scope the cycle-2 re-run

[Al-does]

Review of mess3_token_guess_cycle_1 and mess_3_kelly_cycle_1/2/3 ahead of a re-run, plus the measurement work needed before that re-run can be designed. No training conditions are added.

What the review found

The headline metric is mostly measuring something trivial. An affine probe fitted to the one-hot encoded raw observations, with no network anywhere in the pipeline, scores belief-probe R² = 0.9668. A randomly initialised copy of the study transformer scores 0.8733. The supervised replication reaches 0.9989. So the metric's usable range is 0.032 wide, and reward_only (0.8552) and max_entropy (0.8558) land below the untrained network, while all eight Kelly cycle-2 wager arms land below the raw-observation floor.

Belief-probe R² landscape

Greedy token accuracy is saturated. The trivial repeat-the-previous-token rule scores 0.6732 and an exact Bayesian filter saturates at 0.6883. Every reported arm falls in 0.6733–0.6860, inside a 0.015 range.

None of the published orderings is statistically supported. The Kelly cycles reported mean ± std where the ± was a population standard deviation over three seeds. Recomputed as a t-based interval and paired across shared seeds, 0 of 34 pairwise orderings survive a Holm correction — including the headline PPO-versus-IQN gap, which is +0.095 with a 95% interval of [−0.017, +0.206].

Three-seed intervals against the references

Choosing a better operating point

Cycle 1's parameters are close to the worst choice in the symmetric MESS3 family. A fast-mixing chain lets the belief be approximated by an exponentially weighted average of recent one-hot observations, which is exactly what an affine probe computes.

Metric range across MESS3 parameters

Moving to α=0.70, p=0.995 triples the belief-probe range (floor 0.967 → 0.906) and multiplies the accuracy range by ten (0.015 → 0.145), keeping the task learnable at a Bayes accuracy of 0.68.

Belief-probe R² declines under continued supervised training at a converged cross-entropy (0.9567 → 0.9318 over 6,000 AdamW steps). The decline cycle 1 saw over 20M PPO steps is therefore not caused by reinforcement learning, which makes optimiser and learning rate larger influences on the headline metric than most of the differences between arms.

Two sizing results

Context length is 6x larger than needed. CausalTransformerEncoder bands attention at every layer, so the receptive field is n_layers × context_len — 192 observations at the current settings, verified by perturbing inputs at increasing distance. The belief needs about 32. Dropping to context_len=16 leaves a 50% margin and makes a learner step 3.7x faster, a cached rollout step 3.4x faster, and a whole PPO run 1.6x faster at matched environment steps.

The probe needs to get bigger as the chain gets slower. Running the identical probe 60 times on fresh rollouts, the spread of the answer is 0.0025 at 30,000 test steps on the slower chain against 0.0004 at the cycle-1 parameters — the same size as the smallest seed-to-seed spreads, so probe noise would be indistinguishable from seed variance. It falls to 0.0005 by 500,000 steps, which cost seconds.

Why a slower chain costs probe precision

This also corrects REVIEW.md: the i.i.d. bootstrap there understates the error whenever the chain is slow. A moving-block bootstrap predicted ±0.0047 and replication measured ±0.0047.

What this PR adds

path	purpose
REVIEW.md	the audit, with every number reproducible
PLAN.md	proposed cycle-2 design and the open questions
metric_references.py, references/	task-intrinsic floors and ceilings for both metrics
operating_point.py, task_parameters/	metric range, precision, and context sizing across MESS3 parameters
statistics.py, audit/	seed-level intervals, paired comparison, Holm correction, noncentral-t power planning, and a re-analysis of the committed results

All three leaves are runnable and their outputs are committed:

uv run rl-harness experiments.mess3_token_guess_cycle_2.references.experiment
uv run rl-harness experiments.mess3_token_guess_cycle_2.task_parameters.experiment
uv run rl-harness experiments.mess3_token_guess_cycle_2.audit.experiment

Testing

uv run pytest -q -m "not slow" — 89 passed, 1 skipped. 33 are new, and several encode the findings directly: that the raw-observation floor exceeds three published cycle-1 scores, that only conditional_decoupled_kelly_iqn clears the floor in Kelly cycle 3, that the headline PPO-versus-IQN gap is unresolved at three seeds, that the receptive field is n_layers × context_len, and that replication confirms the slow chain is the noisier probe.

All three leaves were also run at full size and under --smoke.

Note

Running with B2_* configured writes the bucket name into tracked results/. Pass --no-upload-artifacts from an environment where those are secrets.

_{To show artifacts inline, enable in settings.}