Skip to content

[design] Seconds-based task horizons resolved against the embodiment's control rate at eval() time #160

Description

@jeqcho

Problem

Task.max_steps is an integer step budget fixed at task construction time, but a step only has a duration once an embodiment is paired: the same 600-step horizon is 60s on a 10 Hz mock world and 40s on a 15 Hz arm. Benchmarks that specify physical time budgets (kitchenbench's protocol allots per-task seconds; see robocurve/kitchenbench#28 and the discussion on robocurve/kitchenbench#29) currently have no way to express them. Task factories behind the string registry are zero-arg, so a benchmark cannot thread the paired embodiment's rate into make_task itself, and Task is deliberately embodiment-agnostic (R1: the rollout enforces no wall-clock rate; TaskEnvelope carries no control rate on purpose).

Sketch

Let a task declare its horizon in physical time, and let eval() resolve it against the pair:

  • Task grows an optional max_seconds: float | None (mutually exclusive with max_steps).
  • eval() resolves max_steps = ceil(max_seconds * embodiment.info.control_hz) after compatibility checking, before the rollout, and stamps both the resolved value and the declared seconds into EvalLog metadata so runs at different rates remain interpretable.
  • TaskEnvelope.max_steps keeps its current meaning (adapters see the resolved integer), so bind_task consumers are unaffected.

This does not reintroduce pacing: the rollout still never sleeps; control_hz is only used as a conversion factor for the budget, which is exactly the quantity info.control_hz already documents.

Open questions

  • What should happen when info.control_hz is None or zero (event-driven embodiments)? Refuse at compatibility-check time, or require max_steps for such pairs?
  • Should scorers see the declared seconds (for time-normalized metrics) or stay steps-only?
  • Does eval-set need per-pair resolved horizons surfaced in the comparison table?

Found while reviewing robocurve/kitchenbench#29, which attempted this benchmark-side and could not reach the paired embodiment's rate through the zero-arg registry path.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions