Design: model NeMo Gym environments as an RL system — Environment / Agent-Harness / Sandbox contracts

[ffrujeri]

Summary

Converge on system-architecture contracts for NeMo Gym so environments match how RL systems are modeled, and benchmarks and agent servers compose by addition — not via per-pair wrappers like anyswe.

Three roles on three orthogonal axes:

• Environment — MDP authority (spec, state, reward). Usually realized by one resources server (seed_session → SessionDescriptor, verify, optional MCP tools / step, per-task state), but can be formed from several — e.g. a task/verify RS plus a sandbox broker RS, or a facade RS that delegates. Environment is the role; resources server(s) are the implementation.
• Agent Server — hosts an Agent: policy model, tools, and orchestration (planning, value/reward models, nested control, blackbox CLIs, …). Implemented as a responses_api_agents/ server (e.g. claude_code_agent, simple_agent). Independent of which Environment it is wired to in YAML. Distinct server type from Environment and Sandbox — not a resources server.
• Sandbox — substrate and isolation. The nemo_gym/sandbox/ provider seam today; optionally a sandbox resources server (broker) that holds live handles and hands out serializable references.

The motivating mismatch: PR #1738 parks SWE grading and world spec inside the agent server (responses_api_agents/swe_env/, inline verify_task, no resources_servers/swe_bench/). The target restores the Environment as a first-class resources server while keeping blackbox agent servers like claude_code_agent where they belong — connected to the Environment through an interface-agnostic seed_session descriptor, not an anyswe-style wrapper.

Motivation / context

• Parent: Decouple SWE environment infrastructure from agent harnesses #1249 (decouple SWE infra from agent harnesses).
• PR Converge #1677 swe_env decoupling into anyswe + verify on SWE-bench Verified (#1249) #1738 converges anyswe onto SWE grading code but grades inline in the agent server, deletes resources_servers/swe_env/, and keeps that code under responses_api_agents/swe_env/ instead of the Environment RS. That dissolves the Environment server and the verified: marker.
• Complementary (out of scope here): ansubramania's Blackbox agent integration design doc — training/RL-data half (token-id capture, trajectory stitching, capture store).

Key design decisions

• Agent servers stay under responses_api_agents/. Blackbox CLIs (Claude Code, Codex, …) are hosted by agent servers (e.g. claude_code_agent). They connect to an Environment via seed_session / verify, not by being re-hosted inside a generic run-in-box wrapper agent.
• seed_session returns a SessionDescriptor — placement topology, optional SandboxSpec / box reference, optional MCP connection info, optional model egress hints. The agent server binds to the Environment from this descriptor; it does not hardcode docker, OpenSandbox, or SWE-bench wiring.
• Only data crosses HTTP. Sandbox spec and (when brokered) a box reference/token — never a live handle (SandboxHandle.raw).
• Environment = semantics + substrate (when the task has a world). The resources server owns reward and task definition; the sandbox realizes state and transitions (repo/filesystem for SWE). Reward stays in verify; the box carries no reward authority.
• Hermetic grading (the twin). verify() always grades in its own fresh box, independent of where the agent server ran the episode.
• Composition, not a cross-product. A run names agent server × environment × sandbox backend × topology; combinations add. Drop the anyswe wrapper-per-(agent × benchmark) pattern.

Sandboxing topologies

Topology	What's isolated	Typical attachment	Example
A — None	nothing	agent server on host; terminal verify	math, MCQ
B — Env-sandboxed	env world/state	agent server outside; MCP tools into env box	MCP weather, env-owned DB
C — Agent-in-env	world + agent execution, one box	agent server in-box per descriptor	SWE-bench + Claude Code
D — Whole interaction	full episode boundary	outer orchestrator	untrusted composite agent

Reference pair: claude_code_agent + swe_bench — Environment returns topology C + per-instance SandboxSpec; agent server runs Claude Code inside the box, then POSTs to /verify. Same agent server + example_mcp_weather demonstrates topology B (MCP from descriptor).

Tracked work items

• SessionDescriptor contract — extend seed_session response: placement.topology, optional sandbox.{spec, ref}, optional mcp, optional egress. Document agent-server placement bindings (host / MCP / in-box).
• resources_servers/swe_bench — Environment RS: build_spec from task row, seed_session, verify (fresh eval box). Recover verified: marker and /verify endpoint.
• Relocate SWE domain code to Environment RS — move harnesses, parsing, verify_task, etc. out of responses_api_agents/swe_env/ into resources_servers/swe_bench/ (private modules: harnesses/, parsing/, verify_task.py, …). No top-level nemo_gym/swe/ unless a second RS or non-HTTP consumer appears later.
• In-box binding in agent server — topology C in claude_code_agent first: acquire box via nemo_gym/sandbox, apply descriptor egress, exec agent, pass harvest to /verify. Agent-specific (CLI vs loopback HTTP differs per agent). Extract shared helpers into nemo_gym/sandbox/ only if a second agent duplicates the same pattern — no separate nemo_gym/placement/ package for the prototype.
• Sandbox broker RS (optional) — resources_servers/sandbox/ (or equivalent): hold handles, expose acquire/exec/upload/download/release by reference; backends = local docker/apptainer + remote OpenSandbox/cloud. Client-side RemoteSandboxProvider implementing SandboxProvider.
• Configuration & placement — document WHO/WHAT/WHERE: env supplies world spec; agent server declares intrinsic defaults; driver sets deployment policy (backend, topology override, limits). Precedence: run config > agent defaults; world spec = env ⊕ agent default.
• Topology D / nesting — nested-container capability as broker/provider property validated against spec.
• Publish design note in fern/ after team review.

Open questions

• step() vs terminal-only? First-class step() for env-driven benchmarks, or terminal verify + side channels? Blackbox in-box agent servers force terminal grading; native agent servers may use step.
• Box lifecycle owner for topology C with a broker. Environment-owned state-box + agent server drives by reference (RL-canonical) vs agent-owned box (today's Converge #1677 swe_env decoupling into anyswe + verify on SWE-bench Verified (#1249) #1738, no broker). Prefer env-owned when broker exists; agent-owned as no-broker fallback.
• Sandbox broker failure domain — orphan reaping, TTL keyed by rollout id, reconnect if consumer dies mid-rollout.
• Two-box cost — working + eval boxes at scale; pool/image cache on broker.
• One rollout identity — session cookie, MCP token, box ref, env state, reaper TTL as one first-class object.
• anyswe fate — stepping-stone only; dissolve into swe_bench RS + descriptor-driven claude_code_agent (and other agent servers), not the long-term home.

Out of scope

Training / RL-data half: model-call recording under rollout join key, dialect conversion, token-id/logprob capture, capture store, trajectory stitching — see Blackbox agent integration. Plugs in after these boundaries exist.

Related

#1249 (parent), #1738 (convergence PR), #1677 / #1572 (swe_env), #1682 (MCP RS), #1377 (sandbox provider factory), #1707 (provider config decoupling).

[ffrujeri]

This note is exploratory — a target architecture, not what is implemented today. It sets the training/RL-data half aside (token-id capture, trajectory stitching; see Out of scope). The goal is a design whose boundaries match RL problem decomposition, moving from the current codebase (including PR #1738) toward composable agent server × environment × sandbox.

Why this note

NeMo Gym documents four server types: dataset, agent server, resources server, model. That maps cleanly onto RL — except SWE convergence (PR #1738) collapsed the Environment into the agent server: inline grading, deleted resources_servers/swe_env/, SWE domain code under responses_api_agents/swe_env/.

The question is not “should we delete the resources server?” It is:

How do we keep agent servers under responses_api_agents/ (including those that host blackbox CLIs like Claude Code), restore Environments as resources servers (including swe_bench), and connect them without per-pair wrappers — including when the Environment is sandboxed?

RL mapping

RL concept	NeMo Gym piece	Responsibility
Environment (S, A, P, R)	Resources server + optional sandbox world	Spec, state, reward (verify / optional step), tools
Agent	Agent server (responses_api_agents/)	The decision-making system: policy model, tools, and orchestration (planning, memory, multi-step control, best-of-N, critic/value heads, learned reward models, etc.); drives the episode
State / observation	Sandbox world, conversation, RS session state	What the agent perceives
Action	Model output → tool call, shell, submission	What the agent emits
Reward	verify (terminal) or optional step	Environment authority
Rollout worker	Driver + agent server /run	Pairs agent with env for one episode
Sandbox substrate	nemo_gym/sandbox/ (+ optional broker RS)	Isolation and execution host — not reward

Thesis:

Three roles — Environment, Agent Server, Sandbox — are three contracts on three orthogonal axes. Wire them in YAML; swap any axis independently. The agent server does not embed benchmark grading; the Environment does not embed Claude Code.

Terminology: In RL, the Agent is the full decision-making system — policy model, tools, and whatever orchestration implements learning-relevant behavior (planning loops, critics, value functions, reward models, …). In Gym, an agent server is the FastAPI server type that hosts and runs an agent (responses_api_agents/, /run, optional /v1/responses). It is not a resources server and not a sandbox. The model server is often a separate endpoint the agent calls for inference; the agent may also embed or co-locate model logic depending on design. Colloquial “harness” (e.g. “agent harness” in tutorials) refers to orchestration inside an agent server — not a fourth server type.

Role 1 — Agent Server (responses_api_agents/)

An agent server hosts an Agent: it runs the episode loop, calls the policy (typically via a model server), routes tools, and may include rich orchestration — planners, critics, value heads, reward models, blackbox CLI hosts (Claude Code), nested sub-agents, and more. It exposes /run (and often /v1/responses) as a FastAPI server under responses_api_agents/.

Composable agent servers (planner around coder, best-of-N, orchestration around an untrusted inner agent) remain agent servers or compositions of them. Composability is a property of how agent servers are built, not a reason to relocate blackbox CLI hosts out of responses_api_agents/.

Blackbox CLIs live inside agent servers, not Environments

Claude Code is not a Gym server. It is a subprocess spawned by the claude_code_agent agent server. When SWE-bench measures “Claude Code + model on Verified,” the unit under evaluation is the agent (hosted by that agent server, including its model) — implementation stays responses_api_agents/claude_code_agent/.

The same CLI could appear as an Environment tool if another agent server invoked it via MCP. Role follows what is under test, not where the binary runs. For SWE-bench, the agent server hosts Claude Code; the Environment is swe_bench.

What an agent server must not do

• Own benchmark grading (verify_task inline — PR Converge #1677 swe_env decoupling into anyswe + verify on SWE-bench Verified (#1249) #1738 pattern).
• Hardcode SWE image names, docker vs OpenSandbox, or per-benchmark provisioning.
• Force every benchmark through an anyswe-style outer wrapper agent.

What an agent server must do

• Implement the agent loop for its kind.
• On /run: call Environment seed_session, bind to the returned descriptor, execute, call verify.
• Support placement bindings (below) driven by the descriptor — not by benchmark-specific subclasses.

Role 2 — Environment (MDP authority)

The Environment is the benchmark axis — the MDP authority. In Gym it is usually a single resources server (the one wired on the agent server's resources_server ref for seed_session, tools, and verify). That 1:1 mapping covers most benchmarks today (swe_bench, example_mcp_weather, math servers, etc.).

An environment can also be formed from multiple resources servers:

Pattern	Example	Notes
Task RS + substrate RS	swe_bench + sandbox broker	Semantics/reward on task RS; box lifecycle on broker RS. SessionDescriptor carries spec + ref across the seam.
Facade / composite RS	One RS delegates internally	Single agent wiring; composition hidden behind one HTTP surface.
Many environments in one run	Multi-verifier training	Each row routes via agent_ref to a different agent+RS pair — not one environment made of many RS, but many environments in one batch.

Environment names the role (spec, state, reward). Resources server(s) implement it. Do not equate them blindly: sandbox substrate may be a separate RS, a library in-process, or embedded in the task RS depending on deployment.

Required (on the primary task/verify RS, or on a facade that exposes the same contract):

• seed_session → SessionDescriptor (see below).
• verify → reward — always. Terminal grading for blackbox/in-box agents; optional per-step reward via step for env-driven tasks.
• Per-task state (session cookie / MCP token scope).
• Optional MCP tools (topology B) and optional step(action) (env-driven).

For SWE, the Environment is resources_servers/swe_bench/ (name TBD), not code living under responses_api_agents/. It recovers:

• The verified: YAML marker and baselining workflow.
• An independently addressable /verify endpoint.
• Clear reward authority separate from any agent server.

Domain logic today in responses_api_agents/swe_env/ (verify_task, harnesses, parsing) moves into resources_servers/swe_bench/ as Environment-owned modules — not owned by the agent tree, and not a top-level nemo_gym/swe/ package unless a second RS or non-HTTP consumer needs the same graders later.

In-box orchestration (provision, egress, exec, harvest) is agent-server code using nemo_gym/sandbox/ — not SWE domain code, and not a separate gym-wide placement/ package unless multiple agents later share the exact same binding. Patch harvest for SWE is part of the verify contract (agent passes patch on POST /verify), not generic infrastructure.

Environment = semantics + substrate (layering)

When a task has a world (SWE repo in a container, game state, DB):

• Semantics (reward, task definition, what actions mean) → resources server.
• Substrate (where state lives, where commands run) → sandbox.

State S and transitions P live in the box; reward R lives in verify. The sandbox is not the Environment, but it implements the stateful half when a world exists. Stateless tasks (math, MCQ) skip the substrate layer entirely (topology A).

Role 3 — Sandbox = substrate (+ optional broker)

Implemented today as nemo_gym/sandbox/ — an in-process library, not a Gym server.

Piece	Purpose
SandboxSpec	Serializable creation request (image, workdir, env, ttl, resources)
SandboxHandle	Live binding (sandbox_id, provider_name, opaque raw)
SandboxProvider	create / exec / upload / download / status / close
AsyncSandbox	Lifecycle wrapper used by callers
Backends	docker, apptainer (local), opensandbox (HTTP client to remote service)

Problem it solves: one API and config shape for provisioning boxes across benchmarks and deployment targets, without each agent server reimplementing docker/OpenSandbox wiring.

What it does not solve yet: cross-process sharing, central pooling/TTL, or a first-class Gym endpoint. Those motivate an optional sandbox broker resources server.

Broker model

• Provider seam (always): pluggable backends; live handle in the process that called create.
• Optional broker RS: holds handles in its process; clients hold references only.

Only data crosses HTTP: SandboxSpec + sandbox_ref. Never SandboxHandle.raw.

With a broker, the Environment and the agent server can co-reference the same box (topology C, env-owned state, agent drives by ref) — RL-canonical “environment owns state.” Without a broker, the agent server may own the box directly (today's #1738 path) as a degraded but valid mode.

Hermetic twin unchanged: verify acquires its own fresh box (via broker ref or local provider), regardless of topology.

The connection seam: SessionDescriptor

Today, MCP environments return tool connection metadata from seed_session; claude_code_agent merges it into Claude's MCP config. That is already interface-agnostic at the HTTP layer.

SWE-bench needs a richer descriptor because Claude Code does not act through Environment MCP tools during the loop — it edits /testbed directly. The fix is to generalize seed_session, not to merge agent server and Environment.

Conceptual shape

# Returned from Environment POST /seed_session (conceptual)
placement:
  topology: none | env_sandboxed | agent_in_env | whole_interaction

sandbox:                          # omitted for topology A
  spec:                           # WHAT world (SandboxSpec fields)
    image: docker://swebench/...
    workdir: /testbed
    env: { ... }
    ttl_s: 1800
  ref:                            # WHERE / live handle (optional)
    sandbox_id: ...
    provider: sandbox_broker       # null → client acquires locally from run config

mcp:                              # topology B
  server_name: swe_bench
  url_path: /mcp
  transport: http
  headers: { X-NeMo-Gym-Session-Token: ... }

egress:                           # topology C — model reachable from inside box
  anthropic_base_url: ...
  # or resolved model_server endpoint env vars

verifier_metadata: { ... }        # task ground truth for verify

The agent server reads placement.topology and binds accordingly. It never imports swe_env or chooses docker vs OpenSandbox itself.

Agent-server placement bindings

One agent server (claude_code_agent), three bindings:

flowchart TB
    Run[claude_code_agent /run]
    Seed[POST env /seed_session]
    Run --> Seed
    Seed --> D[SessionDescriptor]

    D --> A[topology A: none]
    D --> B[topology B: env_sandboxed]
    D --> C[topology C: agent_in_env]

    A --> Host[subprocess claude on agent host]
    B --> MCP[wire descriptor.mcp → Claude MCP config]
    C --> InBox[acquire sandbox from spec/ref<br/>exec claude inside box<br/>harvest patch / trajectory]

    Host --> Verify[POST env /verify]
    MCP --> Verify
    InBox --> Verify

Loading

Binding	When	Mechanism
Host	topology A	Today's path: subprocess on agent server host
MCP	topology B	Existing _write_rollout_mcp_config pattern
In-box	topology C	nemo_gym/sandbox acquire/exec + agent-specific launch (e.g. claude -p in box); patch on verify request

The in-box binding uses nemo_gym/sandbox/ plus agent-specific launch logic (from self_drive / anyswe where applicable) — not a separate agent server, not SWE grading imports, and not a gym-wide placement/ package until duplication proves it.

Sandboxing topologies

Topology	Isolated	Attachment	SWE + Claude Code
A — None	nothing	agent server on host; message verify	N/A
B — Env-sandboxed	env world	agent server outside; MCP into box	Possible but not fidelity-preserving for full CLI
C — Agent-in-env	world + agent execution	agent server inside env box	Required for native Claude Code on /testbed
D — Whole interaction	full episode	outer boundary	Untrusted / nested agents

Requirements called out in discussion:

• Agent has a sandboxed instance of the environment → B (MCP into env box) or C (agent server inside box).
• Whole iteration isolated → D (nested boxes; provider must advertise nesting).

Configuration & placement

Two logical boxes are different knobs:

• Agent-isolation box — sandbox the agent server for safety (even without a task world).
• Environment-state box — the task world (SWE per-instance image).

Their relationship is the topology (same box → C; separate → B-ish; nested → D).

Three contributors, with precedence:

Contributor	Set at	Answers
Agent server config	startup	Intrinsic defaults (default_spec, default topology)
Environment seed_session	per task	What world (spec, verifier metadata)
Driver ng_collect_rollouts	per run	Where/how (broker URL, backend, topology override, limits, pooling)

Split:

• WHAT (image, files, task) = env.spec ⊕ agent.default_spec (env wins on conflict).
• WHERE/HOW (backend, broker, topology policy, quotas) = driver/run config, overriding agent defaults.

Graceful degradation: no broker → local docker/apptainer provider; no box → topology A bindings inert.

Target architecture

flowchart TB
    Driver[Rollout driver] -->|POST /run| Agent[Agent server<br/>e.g. claude_code_agent]
    Agent -->|seed_session| Env[Environment RS<br/>e.g. swe_bench]
    Agent <-->|model| Model[Model server]
    Agent -->|verify + trajectory| Env
    Env -->|verify: fresh box| Sbx[Sandbox substrate]
    Agent -->|topology C: exec in box| Sbx
    subgraph SandboxLayer[Sandbox — library or broker RS]
      Sbx
      Prov[docker · apptainer · opensandbox · broker]
    end
    Sbx -.-> Prov
    Env -->|reward| Agent
    Agent -->|result| Driver

Loading

Optional broker RS sits in the Sandbox layer; Environment and Agent Server are peers that consume it by reference.

YAML composition (add, don't multiply)

responses_api_agents:
  claude_code_agent:
    resources_server:
      name: swe_bench              # Environment axis — swap benchmark here
    # intrinsic defaults (topology, egress expectations)

resources_servers:
  swe_bench:
    # verify, seed_session, verified: true
    sandbox_broker: sandbox_rs     # optional; null → local docker

sandbox_topology: agent_in_env    # driver may override
sandbox_backend: docker            # or opensandbox / sandbox_rs ref

No anyswe_claude_code wrapper config — same claude_code_agent entrypoint for math (env A), weather (env B), SWE (env C).

Where PR #1738 sits

#1738 usefully extracts SandboxProvider, verify_task, and self-drive provisioning — but stops at the wrong boundary:

Aspect	#1738	Target
Grading	Inline in anyswe_agent	swe_bench /verify
World spec	Agent server builds SweTask	Environment seed_session
SWE domain code	responses_api_agents/swe_env/	resources_servers/swe_bench/ modules
Claude Code	Dropped in box by anyswe	claude_code_agent + descriptor topology C
verified: marker	Lost	On Environment RS

anyswe is a stepping-stone, not the destination. Keep its good idea (run an agent in the task box); move spec and grading out of the wrapper agent server.

What exists today (inventory)

Component	Location	Role today
Sandbox library	nemo_gym/sandbox/	Provider seam + AsyncSandbox
SWE domain code	responses_api_agents/swe_env/	Grading, harnesses, parsing — misplaced; belongs under swe_bench RS
In-box binding	swe_env/self_drive, anyswe_agent	→ agent server + nemo_gym/sandbox/ (agent-specific; extract shared bits into sandbox only if duplicated)
SWE convergence agent	responses_api_agents/anyswe_agent/	Wrapper + inline verify — interim
Claude Code agent server	responses_api_agents/claude_code_agent/	Host subprocess + MCP env attach — correct home
OpenSandbox	sandbox/providers/opensandbox/	Remote broker client, not Gym RS
MCP Environment example	resources_servers/example_mcp_weather/	Topology B reference
SWE Environment RS	resources_servers/swe_env/	Deleted in #1738 — to restore as swe_bench

Work plan (implementation-facing)

1. SessionDescriptor — schema + seed_session / consumer in claude_code_agent.
2. swe_bench resources server — thin HTTP surface (app.py) over relocated modules (harnesses/, verify_task.py, …); verified: flag.
3. In-box binding — topology C in claude_code_agent via nemo_gym/sandbox + agent-specific exec/harvest; add peers (hermes, etc.) when needed — reuse anyswe loopback pattern per agent or extract tiny shared helper into nemo_gym/sandbox/ if duplicated.
4. Sandbox broker RS — optional; RemoteSandboxProvider for clients.
5. Driver/run config — topology and backend overrides; one rollout identity object.
6. Deprecate anyswe wrapper once descriptor + swe_bench + in-box binding land.

Open questions

• step() investment — worth first-class env-driven stepping, or terminal verify only for v1?
• Topology C ownership with broker — Environment owns state-box lifecycle vs agent-owned fallback (Converge #1677 swe_env decoupling into anyswe + verify on SWE-bench Verified (#1249) #1738).
• Broker ops — reaping, TTL per rollout, pool warm-up for two-box grading at scale.
• In-box binding API surface — how much of harvest (git diff vs output.jsonl) is agent-server-specific vs descriptor-hint?

Out of scope

Training / RL-data: recording model calls, dialect conversion, token-id/logprob capture at served layer, capture store, trajectory stitching — see Blackbox agent integration. Those boundaries plug in after Environment / Agent Server / Sandbox contracts are stable.

Related docs

• fern/.../claude-code-agent-protocol-stack.mdx — wire formats for claude_code_agent today.
• nemo_gym/sandbox/providers/apptainer/README.md — local provider usage.
• resources_servers/example_mcp_weather/ — MCP Environment (topology B) reference implementation.