[1/5][rollout] feat: harbor backend v2 - #272
Merged
Merged
Conversation
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
Codecov Report❌ Patch coverage is 📢 Thoughts on this report? Let us know! |
github-actions
Bot
added
enhancement
This was referenced Aug 6, 2026
mathewjhan
added a commit
that referenced
this pull request
Aug 6, 2026
…ntainer runner (#284) Part of splitting #272 into reviewable pieces (2/6). ## What this adds The data types and the in-container code that let a user's agent workflow run inside a task container and report results back to the host. - `ContainerInput` / `ContainerResult` (`container/files.py`): the two JSON files exchanged with the container. The host stages `container_input.json` (rollout id, prompt, metadata, chat-completions URL, API key) before the agent starts; the agent phase writes `container_result.json` (status, error, workflow output) when it ends. `write_reward` writes the reward file at the path Harbor's verifier reads (`/logs/verifier/reward.json`). - `AgentWorkflowOutput` (`types/output.py`): what a workflow's `run()` may return — one or more named message histories plus metrics. Returning nothing is also valid; the runner then collects the conversation recorded at the chat-completions endpoint. - `RolloutStatus` (`types/sample.py`): one status vocabulary used everywhere — `queued`, `running`, `grading`, `success`, `failure`, `cancelled`, `unknown`. `RolloutSample` gains `extra_fields` for structured diagnostics. - The runner (`container/runner.py`): the entrypoints that the generated bundle scripts call inside the container. `agent_main` reads the input file, runs the workflow, writes the result file, and saves the conversation as an ATIF `trajectory.json` next to the agent logs — the same location and format native Harbor agents use, so tools that read trajectories work on both. `grader_main` loads the agent's messages (from the result file, or from a trajectory file when the agent was not SDK code), runs the user's grader, and writes the reward. The grader reads its input from `tests/` first: that copy may carry the ground-truth label, while the agent-phase copy has the label removed so the model can never read the answer. - `messages_from_trajectory` (`container/trajectories.py`): converts a trajectory document (ATIF steps, or a raw messages list from a native harness) into plain chat messages. ## Example A workflow and grader written against these types: ```python class MyWorkflow(AgentWorkflow): async def run(self, ctx: AgentWorkflowContext) -> None: agent = StrandsAgent(model=ctx.config.model, messages=ctx.prompt) await agent.invoke_async() # returning None is fine: the recorded conversation becomes the sample class MyGrader(Grader): async def grade(self, ctx: GraderContext) -> None: answer = ctx.sample.messages[-1]["content"] ctx.set_reward(1.0 if answer.strip() == ctx.label else 0.0) ``` Inside the container, the bundle's console scripts call `agent_main(MyWorkflow, config)` and `grader_main(MyGrader, config)`; everything else in this PR is the plumbing those two calls rely on. Co-authored-by: Cursor <cursoragent@cursor.com>
mathewjhan
added a commit
that referenced
this pull request
Aug 6, 2026
…dles (#285) Part of splitting #272 into reviewable pieces (3/6). ## What this adds `osmosis_ai/packaging.py`: builds a standard Python wheel from a user's rollout project so the project can be installed inside a task container with one `pip install`. - `build_bundle(project_dir, workflow=..., grader=...)` produces a wheel containing the project's package plus a generated `bundle_main.py` shim. The shim imports the user's classes directly (`from my_harness.solver import MyWorkflow`) and exposes two console scripts, `<package>-agent` and `<package>-grade`, which call the runner entrypoints from #284. Nothing is resolved dynamically at runtime; the class binding happens at build time. - Wheels are cached by a content hash of the project files under the user cache directory (`platformdirs`), so rebuilding an unchanged project is free. - `inspect_bundle(wheel)` reads the wheel's metadata with `importlib.metadata` and returns the declared dependencies (keeping environment markers, dropping extras-gated entries) plus the two script names. The Harbor backend (next in the stack) uses this list to pre-install dependencies into the task image. Also includes the `bench_harness` fixture project the packaging tests build against, and the `platformdirs` dependency. ## Example ```python from osmosis_ai.packaging import build_bundle, inspect_bundle wheel = build_bundle( Path("my_rollout_project"), workflow="my_harness.solver:MyWorkflow", grader="my_harness.grade:MyGrader", ) info = inspect_bundle(wheel) info.agent_script # "my-harness-agent" info.requirements # ["strands-agents>=1.0", "httpx>=0.27", ...] ``` Inside a container, `pip install my_harness-0.1.0-py3-none-any.whl` followed by running `my-harness-agent` executes the user's workflow with no other setup. Co-authored-by: Cursor <cursoragent@cursor.com>
mathewjhan
added a commit
that referenced
this pull request
Aug 6, 2026
Part of splitting #272 into reviewable pieces (4/6). Builds on #284 (container contract) and #285 (packaging). ## What this adds `HarborBackendV2`: runs each rollout as a Harbor trial. The agent can be either a user workflow (packaged into a wheel and installed in the container at trial start) or a registered native Harbor agent (`terminus-2`, `mini-swe-agent`, `oracle`) with the rollout endpoint injected into its environment. How a rollout flows through it: 1. **Task selection** (`tasks.py`): template mode uses one task directory for every rollout; dataset mode routes by `metadata["harbor_task_id"]` to a folder under `tasks_dir` (path escapes rejected); `metadata["harbor_task"]` fetches a task from a local path, git checkout, or registry package, with per-ref locks so concurrent rollouts download once. 2. **Materialization**: the task is copied into a per-rollout directory; the rollout's input file is staged; if the task has no `tests/` and a grader exists, a `test.sh` is generated that installs and runs the grader. The ground-truth label is staged only into `tests/`, which Harbor uploads at verification time — the agent phase cannot read it. 3. **Image preparation**: `patch_dockerfile_with_sdk` appends a block to the task's Dockerfile that installs a static `uv` binary and creates `/opt/osmosis/venv` with the bundle's dependencies pre-installed. Per-trial installs then only add the user's own code (`--no-deps`), which cuts container startup from minutes to seconds. The patch is deterministic, so identical tasks keep identical image content hashes and share builds. 4. **Execution** (`harness_agent.py`): the installed agent uploads the wheel, installs it into the venv, backfills an empty prompt from the task's `instruction.md`, runs the agent script, and returns the result through the trial's agent metadata. 5. **Callbacks**: the workflow-complete callback fires when verification starts (agent phase over); the grader-complete callback fires at trial end with the reward parsed from Harbor's verifier result. Callback delivery failures are logged and never abort trial archival. 6. **Observability and lifecycle**: per-phase timings and failure phases in every result (`diagnostics.py`), native-agent ATIF parsing with secret redaction, artifact relocation, `prewarm()` / `prewarm_lifespan()` to build task images before serving traffic, `cancel_rollouts(ids | prefix | all)`, `rollout_status()` with terminal outcomes retained in a `TtlCache` (#283), and admission control via `max_queue_depth`. ## Example ```python backend = HarborBackendV2( orchestrator=TrialQueue(n_concurrent=100), tasks_dir=Path("tasks"), # 300 task folders task_mode="dataset", agent=MyWorkflow, # or agent="mini-swe-agent" workflow_config=my_config, environment_config=EnvironmentConfig(type=EnvironmentType.SKYPILOT), ) app = create_rollout_server( backend=backend, lifespan=backend.prewarm_lifespan(task_ids=["task-0000"]), ) ``` A trainer then POSTs rollouts with `metadata={"harbor_task_id": "task-0042"}`; each one runs in its own sandbox and reports back through the callbacks. Co-authored-by: Cursor <cursoragent@cursor.com>
mathewjhan
added a commit
that referenced
this pull request
Aug 6, 2026
…ncel endpoints (#287) Part of splitting #272 into reviewable pieces (5/6). Builds on #286 (backend capabilities). ## What this adds Three server-level behaviors that let a trainer manage load and track rollouts over plain HTTP: - **Admission control on `POST /rollout`**: when the backend's queue is at `max_queue_depth`, the server answers `429` with a `Retry-After` header instead of accepting work it cannot start. Accepted rollouts return `202`, which states what actually happens: the rollout is queued and runs after the response is sent. - **`GET /rollout/{id}/status`**: reports `queued`, `running`, or `grading` for live rollouts, and `success` / `failure` / `cancelled` (with reward and error message) for recently finished ones, retained for a fixed window. Anything else returns `unknown`. A trainer polls this as a liveness signal instead of trusting a blind timeout. - **`POST /rollout/cancel`**: takes exactly one selector — `{"ids": [...]}`, `{"prefix": "tenant-a::"}`, or `{"all": true}` — and returns a disposition per rollout (`cancelled_queued`, `cancelled_running`, `not_found`). Prefix cancellation is what a controller uses to stop everything belonging to one adapter when it is deregistered. Cancelling an already-finished rollout returns `not_found`, so the call is safe to repeat. ## Example ```bash curl -X POST $SERVER/rollout -d '{"rollout_id": "tenant-a::r1", ...}' # -> 202 (or 429 + Retry-After: 5 when the queue is full) curl $SERVER/rollout/tenant-a::r1/status # -> {"rollout_id": "tenant-a::r1", "status": "running"} curl -X POST $SERVER/rollout/cancel -d '{"prefix": "tenant-a::"}' # -> {"dispositions": {"tenant-a::r1": "cancelled_running"}} ``` Co-authored-by: Cursor <cursoragent@cursor.com>
mathewjhan
added a commit
that referenced
this pull request
Aug 6, 2026
…288) Part of splitting #272 into reviewable pieces (6/6). Exercises everything below it in the stack. ## What this adds A CLI that measures how long the full rollout lifecycle takes on the Harbor backend: submit, container build/start, agent run, verification, teardown. It reports cold and warm timings separately and per-rollout p50/p95/max, which is how regressions in container startup cost were found and fixed during development. It runs against the bundled `bench_harness` project by default (a minimal workflow + grader), or against any Harbor task folder via `--tasks-dir`. Rollout ids carry a per-invocation nonce so concurrent bench runs cannot collide. ## Example ```bash # default bench task, 3 runs of 5 concurrent rollouts uv run benchmarks/container_lifecycle/container_lifecycle_bench.py --runs 3 --concurrency 5 # against a real dataset task, without baking the SDK into the image uv run benchmarks/container_lifecycle/container_lifecycle_bench.py \ --tasks-dir harbor-datasets/datasets/algotune/algotune-aes-gcm-encryption \ --runs 3 --concurrency 5 --no-patch-dockerfile-with-sdk ``` Output shows per-run wall time, success counts, rollouts/sec, and warm-run latency percentiles. Co-authored-by: Cursor <cursoragent@cursor.com>
mathewjhan
marked this pull request as ready for review
August 6, 2026 22:40
mathewjhan
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Why
How to Test
Checklist
[module] type: descriptionformatenhancement,bug,breaking)ruff check .andruff format --check .passpyright osmosis_ai/passespytestpasses (new tests added if applicable)Summary by cubic
Add
HarborBackendV2to run Harbor tasks with pure task images via bundled workflows or native Harbor agents, plus admission control, rollout status polling, cancellation, and artifact handling. Native Harbor tasks use Harbor’s validator when running native agents.New Features
HarborBackendV2: bundled workflows or native agents (terminus-2,mini-swe-agent,oracle); template/dataset modes; per‑phase timing with failure diagnostics; secrets redacted; diagnostics returned viaextra_fieldson results.osmosis_ai.packaging): builds a wheel from apyproject.tomlharness, injects agent/grader console scripts, caches by content hash in the user cache dir viaplatformdirs, and exposesinspect_bundle/project_dir_for.ContainerInput/ContainerResultplus in‑containerrunner.agent_main/runner.grader_main; extracts messages from trajectories, writes reward files for Harbor, and addsAgentWorkflowOutput.HarborTask+TaskModecopy per rollout, writeinstruction.md, patch Dockerfiles to install SDK deps into an isolated venv (/opt/osmosis/venv) bootstrapped withuv(fallback topip), and generatetests/test.shto install a grader when needed.native_agent_kwargsfor per‑rollout agent settings; use Harbor’s native validator; bundled workflows run from the venv.HarborBackend: content‑addressed image tags to reuse builds; new exports inosmosis_ai.rollout.backend.harbor.__init__.has_capacity()(429 +Retry-Afterwhen full),GET /rollout/{id}/statusfor lifecycle and terminal states,POST /rollout/cancel(ids/prefix/all), andPOST /rolloutreturns 202 when accepted and queued.benchmarks/container_lifecyclefor cold/warm throughput.Migration
HarborBackendV2withtasks_dir; choose a native agent by name or provide a workflow harness (code_diror a prebuiltbundle).pyproject.toml; the bundler creates<package>-agent/<package>-gradescripts automatically./opt/osmosis/venv; images only need Python. For dataset runs, settask_mode="dataset"and includeharbor_task_idin request metadata.native_agent_kwargs.POST /rollout, 429 withRetry-After, may callPOST /rollout/cancel, and can pollGET /rollout/{id}/statusfor lifecycle and terminal states.Written for commit 8bf241a. Summary will update on new commits.