[4/6][rollout] feat: harbor execution backend v2

[mathewjhan]

Part of splitting #272 into reviewable pieces (4/6). Builds on #284 (container contract) and #285 (packaging).

What this adds

HarborBackendV2: runs each rollout as a Harbor trial. The agent can be either a user workflow (packaged into a wheel and installed in the container at trial start) or a registered native Harbor agent (terminus-2, mini-swe-agent, oracle) with the rollout endpoint injected into its environment.

How a rollout flows through it:

1. Task selection (tasks.py): template mode uses one task directory for every rollout; dataset mode routes by metadata["harbor_task_id"] to a folder under tasks_dir (path escapes rejected); metadata["harbor_task"] fetches a task from a local path, git checkout, or registry package, with per-ref locks so concurrent rollouts download once.
2. Materialization: the task is copied into a per-rollout directory; the rollout's input file is staged; if the task has no tests/ and a grader exists, a test.sh is generated that installs and runs the grader. The ground-truth label is staged only into tests/, which Harbor uploads at verification time — the agent phase cannot read it.
3. Image preparation: patch_dockerfile_with_sdk appends a block to the task's Dockerfile that installs a static uv binary and creates /opt/osmosis/venv with the bundle's dependencies pre-installed. Per-trial installs then only add the user's own code (--no-deps), which cuts container startup from minutes to seconds. The patch is deterministic, so identical tasks keep identical image content hashes and share builds.
4. Execution (harness_agent.py): the installed agent uploads the wheel, installs it into the venv, backfills an empty prompt from the task's instruction.md, runs the agent script, and returns the result through the trial's agent metadata.
5. Callbacks: the workflow-complete callback fires when verification starts (agent phase over); the grader-complete callback fires at trial end with the reward parsed from Harbor's verifier result. Callback delivery failures are logged and never abort trial archival.
6. Observability and lifecycle: per-phase timings and failure phases in every result (diagnostics.py), native-agent ATIF parsing with secret redaction, artifact relocation, prewarm() / prewarm_lifespan() to build task images before serving traffic, cancel_rollouts(ids | prefix | all), rollout_status() with terminal outcomes retained in a TtlCache ([1/6][rollout] feat: add TtlCache utility #283), and admission control via max_queue_depth.

Example

backend = HarborBackendV2(
    orchestrator=TrialQueue(n_concurrent=100),
    tasks_dir=Path("tasks"),           # 300 task folders
    task_mode="dataset",
    agent=MyWorkflow,                  # or agent="mini-swe-agent"
    workflow_config=my_config,
    environment_config=EnvironmentConfig(type=EnvironmentType.SKYPILOT),
)
app = create_rollout_server(
    backend=backend,
    lifespan=backend.prewarm_lifespan(task_ids=["task-0000"]),
)

A trainer then POSTs rollouts with metadata={"harbor_task_id": "task-0042"}; each one runs in its own sandbox and reports back through the callbacks.

[mintlify]

Docs PR opened: https://github.com/osmosis-ai/docs/pull/68

Documented the new HarborBackendV2 execution backend, covering task modes, native agents, prewarming, cancellation, status, and admission control.