Observed
Per-task max_steps in specs.py (place_cutlery 60, stack 80, ..., sort_cutlery 200) were authored against the abstract mock world, where an episode completes in ~4 steps. On a physical or simulated embodiment the framework truncates the trial at task.max_steps, so at a 15 Hz control rate place_cutlery gets 4 seconds and the longest task 13 seconds — versus the 120 s operator budget the real-rig protocol uses (and the ~28-120 s that real successful MolmoAct2 episodes actually take).
Field evidence (Isaac sim, bimanual YAM, MolmoAct2 via robolab's inspect-robots bridge, 2026-07-17): end-of-episode frames show the gripper arriving at the correct object exactly as the 60-step budget expires; 0/10 across two tasks at spec horizon. The July 2 full-suite 0/50 (robocurve/robolab, kitchenbench_eval_results/molmoact2_yam_sim.md) has the same signature ("reaching toward the correct objects without completing") and consumed the same horizons. Real-rig sessions never hit this because their successes were ad-hoc instruction runs with max_steps=1200 from config, not registered-task runs.
Why it matters
Any VLA evaluated on the registered tasks with a real-time embodiment is scored against a horizon ~10-30x shorter than the methodology intends; resulting P-hats are dominated by truncation, not competence. The mock is unaffected (its episodes finish in a handful of steps), so CI can't catch it.
Proposal
Make the horizon time-based or embodiment-scaled:
- Preferred:
TaskSpec declares max_seconds (e.g. 120, matching the real-rig operator budget); make_task/the framework converts via the embodiment's declared control_hz at pairing time. Mock keeps tiny effective budgets automatically (its control_hz semantics) or gets an explicit floor.
- Cheaper: keep
max_steps but document it as mock-scale and add a scale hook to make_task(horizon_scale=...); adapters pass their own factor (robolab's bridge now carries --horizon-scale 7.5 as a workaround, robolab/inspect_bridge/run.py).
The workaround lives downstream; filing so the benchmark owns the fix.
Observed
Per-task
max_stepsinspecs.py(place_cutlery 60, stack 80, ..., sort_cutlery 200) were authored against the abstract mock world, where an episode completes in ~4 steps. On a physical or simulated embodiment the framework truncates the trial attask.max_steps, so at a 15 Hz control rate place_cutlery gets 4 seconds and the longest task 13 seconds — versus the 120 s operator budget the real-rig protocol uses (and the ~28-120 s that real successful MolmoAct2 episodes actually take).Field evidence (Isaac sim, bimanual YAM, MolmoAct2 via robolab's inspect-robots bridge, 2026-07-17): end-of-episode frames show the gripper arriving at the correct object exactly as the 60-step budget expires; 0/10 across two tasks at spec horizon. The July 2 full-suite 0/50 (robocurve/robolab,
kitchenbench_eval_results/molmoact2_yam_sim.md) has the same signature ("reaching toward the correct objects without completing") and consumed the same horizons. Real-rig sessions never hit this because their successes were ad-hoc instruction runs withmax_steps=1200from config, not registered-task runs.Why it matters
Any VLA evaluated on the registered tasks with a real-time embodiment is scored against a horizon ~10-30x shorter than the methodology intends; resulting P-hats are dominated by truncation, not competence. The mock is unaffected (its episodes finish in a handful of steps), so CI can't catch it.
Proposal
Make the horizon time-based or embodiment-scaled:
TaskSpecdeclaresmax_seconds(e.g. 120, matching the real-rig operator budget);make_task/the framework converts via the embodiment's declaredcontrol_hzat pairing time. Mock keeps tiny effective budgets automatically (its control_hz semantics) or gets an explicit floor.max_stepsbut document it as mock-scale and add a scale hook tomake_task(horizon_scale=...); adapters pass their own factor (robolab's bridge now carries--horizon-scale 7.5as a workaround,robolab/inspect_bridge/run.py).The workaround lives downstream; filing so the benchmark owns the fix.