Is your feature request related to a problem?
eval_set(retry_attempts=...) and inspect-robots eval-set --retry-attempts accept the value but never act on it: eval_set just loops eval() once per task and never reads it. The docstring says so plainly ("accepted now so callers don't get retrofitted, but is not yet honored").
The cost shows up on exactly the workload eval-set exists for: a long multi-task benchmark on real hardware. If a 10-task run is interrupted at task 7 (Ctrl-C via #118, an embodiment fault, or a fail-on-error halt), re-running the command redoes all 10 from scratch.
Every already-completed scene is discarded, which on real arms is expensive operator time.
@jeqcho floated this as a natural follow-up when merging #68, so I want to confirm the direction before writing any code.
Describe the solution you'd like
A re-invocation of the same eval-set should skip scenes that already finished and run only the rest, bounded by --retry-attempts.
This touches eval(), eval_set(), and logging, so the interface and the open decisions are spelled out below.
What the current machinery already gives us
-
SceneResult.status is recorded per scene ("success", "error", or "cancelled"), and a halted run's log contains exactly the scenes it attempted (the scene loop breaks on halt). So "which scenes finished" is fully reconstructable from a log on disk.
-
Seeding is scene-deterministic (derive_seed(eval_seed, scene.init_seed, epoch)), so a resumed scene reproduces bit-for-bit. Resumption does not compromise reproducibility.
What is missing
-
No stable run identity.
The log filename is {task-slug}_{random-uuid}.json, and EvalSpec carries no run ID, so a re-invocation has no way to locate the prior run's logs to resume from.
This is the load-bearing piece.
Open decisions (would like your read before I implement)
1. Identity mechanism
Auto-derive a deterministic run_id from (task, policy, embodiment, seed, epochs, scene ids), record it in EvalSpec and the filename, and let a re-run of the same command silently resume (mirrors Inspect AI's zero-ceremony resume), with an explicit --fresh / --no-resume override for a clean run.
Alternative: an explicit --run-id the user must pass.
I lean auto-derive.
2. "Done" semantics
A scene that completed but scored a failure (e.g. success_at_end=0) is done and must not re-run.
Only error, cancelled, and never-reached scenes retry.
I want to confirm resumption keys off scene status, never off the score, so a legitimately-failed scene is never silently re-rolled.
3. Granularity
Scene-atomic for v1: a scene is either fully done (all its epochs) or fully redone.
Epoch-level mid-scene resume is deferred.
(The docstring already frames this as "already-finished scenes".)
4. Where the merge lives
eval_set reads prior logs, filters each task's scenes to the unfinished set, runs those, then merges the new results with the retained ones into one fresh, complete log (metrics recomputed over the union).
Alternative: eval() grows a resume_from / skip_scenes parameter.
The first keeps eval() untouched and confines resumption to eval_set; the second is more direct but widens the core API.
I lean toward keeping it in eval_set.
5. retry_attempts counting
Mirror Inspect AI: retry_attempts bounds how many times the set re-attempts tasks that are still incomplete, and overall success is "every task complete."
Confirm this matches your intent rather than a per-task count.
Scope check
Alternatives considered
Task-level skip only
A completed task is skipped, and an incomplete task re-runs wholesale.
Simpler, but it throws away the completed scenes inside the interrupted task, which are the most valuable work to preserve on hardware.
A separate inspect-robots resume <log> command
More explicit, but it adds a second entry point for what is conceptually the same command run again.
Additional context
Follow-up to #68.
Happy to take the whole thing if the direction lands.
I would isolate the identity / log-id change first, since it is the piece everything else depends on.
Is your feature request related to a problem?
eval_set(retry_attempts=...)andinspect-robots eval-set --retry-attemptsaccept the value but never act on it:eval_setjust loopseval()once per task and never reads it. The docstring says so plainly ("accepted now so callers don't get retrofitted, but is not yet honored").The cost shows up on exactly the workload
eval-setexists for: a long multi-task benchmark on real hardware. If a 10-task run is interrupted at task 7 (Ctrl-C via #118, an embodiment fault, or a fail-on-error halt), re-running the command redoes all 10 from scratch.Every already-completed scene is discarded, which on real arms is expensive operator time.
@jeqcho floated this as a natural follow-up when merging #68, so I want to confirm the direction before writing any code.
Describe the solution you'd like
A re-invocation of the same
eval-setshould skip scenes that already finished and run only the rest, bounded by--retry-attempts.This touches
eval(),eval_set(), and logging, so the interface and the open decisions are spelled out below.What the current machinery already gives us
SceneResult.statusis recorded per scene ("success","error", or"cancelled"), and a halted run's log contains exactly the scenes it attempted (the scene loop breaks on halt). So "which scenes finished" is fully reconstructable from a log on disk.Seeding is scene-deterministic (
derive_seed(eval_seed, scene.init_seed, epoch)), so a resumed scene reproduces bit-for-bit. Resumption does not compromise reproducibility.What is missing
No stable run identity.
The log filename is
{task-slug}_{random-uuid}.json, andEvalSpeccarries no run ID, so a re-invocation has no way to locate the prior run's logs to resume from.This is the load-bearing piece.
Open decisions (would like your read before I implement)
1. Identity mechanism
Auto-derive a deterministic
run_idfrom(task, policy, embodiment, seed, epochs, scene ids), record it inEvalSpecand the filename, and let a re-run of the same command silently resume (mirrors Inspect AI's zero-ceremony resume), with an explicit--fresh/--no-resumeoverride for a clean run.Alternative: an explicit
--run-idthe user must pass.I lean auto-derive.
2. "Done" semantics
A scene that completed but scored a failure (e.g.
success_at_end=0) is done and must not re-run.Only
error,cancelled, and never-reached scenes retry.I want to confirm resumption keys off scene status, never off the score, so a legitimately-failed scene is never silently re-rolled.
3. Granularity
Scene-atomic for v1: a scene is either fully done (all its epochs) or fully redone.
Epoch-level mid-scene resume is deferred.
(The docstring already frames this as "already-finished scenes".)
4. Where the merge lives
eval_setreads prior logs, filters each task's scenes to the unfinished set, runs those, then merges the new results with the retained ones into one fresh, complete log (metrics recomputed over the union).Alternative:
eval()grows aresume_from/skip_scenesparameter.The first keeps
eval()untouched and confines resumption toeval_set; the second is more direct but widens the core API.I lean toward keeping it in
eval_set.5.
retry_attemptscountingMirror Inspect AI:
retry_attemptsbounds how many times the set re-attempts tasks that are still incomplete, and overall success is "every task complete."Confirm this matches your intent rather than a per-task count.
Scope check
Alternatives considered
Task-level skip only
A completed task is skipped, and an incomplete task re-runs wholesale.
Simpler, but it throws away the completed scenes inside the interrupted task, which are the most valuable work to preserve on hardware.
A separate
inspect-robots resume <log>commandMore explicit, but it adds a second entry point for what is conceptually the same command run again.
Additional context
Follow-up to #68.
Happy to take the whole thing if the direction lands.
I would isolate the identity / log-id change first, since it is the piece everything else depends on.