Problem
Task.max_steps is an integer step budget fixed at task construction time, but a step only has a duration once an embodiment is paired: the same 600-step horizon is 60s on a 10 Hz mock world and 40s on a 15 Hz arm. Benchmarks that specify physical time budgets (kitchenbench's protocol allots per-task seconds; see robocurve/kitchenbench#28 and the discussion on robocurve/kitchenbench#29) currently have no way to express them. Task factories behind the string registry are zero-arg, so a benchmark cannot thread the paired embodiment's rate into make_task itself, and Task is deliberately embodiment-agnostic (R1: the rollout enforces no wall-clock rate; TaskEnvelope carries no control rate on purpose).
Sketch
Let a task declare its horizon in physical time, and let eval() resolve it against the pair:
Task grows an optional max_seconds: float | None (mutually exclusive with max_steps).
eval() resolves max_steps = ceil(max_seconds * embodiment.info.control_hz) after compatibility checking, before the rollout, and stamps both the resolved value and the declared seconds into EvalLog metadata so runs at different rates remain interpretable.
TaskEnvelope.max_steps keeps its current meaning (adapters see the resolved integer), so bind_task consumers are unaffected.
This does not reintroduce pacing: the rollout still never sleeps; control_hz is only used as a conversion factor for the budget, which is exactly the quantity info.control_hz already documents.
Open questions
- What should happen when
info.control_hz is None or zero (event-driven embodiments)? Refuse at compatibility-check time, or require max_steps for such pairs?
- Should scorers see the declared seconds (for time-normalized metrics) or stay steps-only?
- Does
eval-set need per-pair resolved horizons surfaced in the comparison table?
Found while reviewing robocurve/kitchenbench#29, which attempted this benchmark-side and could not reach the paired embodiment's rate through the zero-arg registry path.
Problem
Task.max_stepsis an integer step budget fixed at task construction time, but a step only has a duration once an embodiment is paired: the same 600-step horizon is 60s on a 10 Hz mock world and 40s on a 15 Hz arm. Benchmarks that specify physical time budgets (kitchenbench's protocol allots per-task seconds; see robocurve/kitchenbench#28 and the discussion on robocurve/kitchenbench#29) currently have no way to express them. Task factories behind the string registry are zero-arg, so a benchmark cannot thread the paired embodiment's rate intomake_taskitself, andTaskis deliberately embodiment-agnostic (R1: the rollout enforces no wall-clock rate;TaskEnvelopecarries no control rate on purpose).Sketch
Let a task declare its horizon in physical time, and let
eval()resolve it against the pair:Taskgrows an optionalmax_seconds: float | None(mutually exclusive withmax_steps).eval()resolvesmax_steps = ceil(max_seconds * embodiment.info.control_hz)after compatibility checking, before the rollout, and stamps both the resolved value and the declared seconds intoEvalLogmetadata so runs at different rates remain interpretable.TaskEnvelope.max_stepskeeps its current meaning (adapters see the resolved integer), sobind_taskconsumers are unaffected.This does not reintroduce pacing: the rollout still never sleeps;
control_hzis only used as a conversion factor for the budget, which is exactly the quantityinfo.control_hzalready documents.Open questions
info.control_hzis None or zero (event-driven embodiments)? Refuse at compatibility-check time, or requiremax_stepsfor such pairs?eval-setneed per-pair resolved horizons surfaced in the comparison table?Found while reviewing robocurve/kitchenbench#29, which attempted this benchmark-side and could not reach the paired embodiment's rate through the zero-arg registry path.