Test no-delay policy learning in MESS3 variant 2

[Al-does]

Summary

• diagnose why original variant 2 cannot express the privileged hidden-state policy with delay=1
• add greedy and sampled action metrics conditioned on hidden state, exact-belief argmax, and a one-step exact-belief oracle
• add and run a 1M-step delay=0 intervention on an RTX 4090
• commit the compact run results and findings

Diagnosis

Actions are 0=noop, 1=positive, and 2=negative. The previous policy collapsed to always positive (action 1). Positive beats noop until reward-state belief exceeds 0.7541, but after a delayed observation the current belief is bounded by the largest transition probability, 0.3324. The intended noop region is unreachable, so entropy annealing cannot solve this condition.

Vast result

Seed 42, 1,022,738 environment steps:

• hidden states 0/1: ~95.2% positive, ~4.8% noop, 0% negative
• hidden state 2: 67.8% noop, 32.2% positive, 0% negative
• state-2-dominant beliefs: 76.7% noop
• greedy one-step-oracle agreement: 96.3%
• greedy reward-state occupancy: 25.97%

The conditioned policy is already visible at 527,868 steps, so a 5–10M-step run is unnecessary for this intervention.

Validation

• uv run pytest -q tests/test_mess3_reward_state_action_symmetry_cycle_1.py — 16 passed
• uv run pytest -q -m "not slow" — 82 passed, 1 skipped, 1 deselected
• local end-to-end smoke run completed
• Vast run manifest status: completed