Skip to content

feat(sandbox): configurable sandbox resources (sandbox_cpus / sandbox_memory_mb) - #455

Draft
rita-aga wants to merge 17 commits into
mainfrom
claude/sandbox-resources-config
Draft

feat(sandbox): configurable sandbox resources (sandbox_cpus / sandbox_memory_mb)#455
rita-aga wants to merge 17 commits into
mainfrom
claude/sandbox-resources-config

Conversation

@rita-aga

Copy link
Copy Markdown
Collaborator

Sandbox resource requests were hardcoded (cpus: 2, memory_mb: 4096 in wasm-helpers/src/sandbox.rs). When the configured provider caps below that — Tensorlake's free tier allows 1 vCPU / 1 GB — every sandbox provisioning call fails and the session silently degrades to running blind (no rendering, no screenshots), which produced structurally broken curation artifacts (ARN-271, contrib-r11 shakeout).

This makes the request configurable with the same precedence pattern as the rest of the sandbox config: per-session entity fields (sandbox_cpus / sandbox_memory_mb) → tenant config (vault secrets of the same names, delivered via the session trigger configs) → compiled defaults.

  • Applied in both provisioning paths: sandbox_provisioner (Resume flow) and monty_repl (lazy provisioning)
  • session.ioa.toml: the two secret keys added to all three sandbox-capable trigger configs
  • Unit test covers the precedence chain, unresolved-{secret:...} templates, and garbage values

Deployed via Genesis as temperpaw/paw-agent@63acb8a6 (blob rebuilt from the deployed tree so it carries exactly this change; prod install pending live verification).

🤖 Generated with Claude Code

https://claude.ai/code/session_01H3fCuggTfCRum7qAWq8337

…x_memory_mb override chain

Per-session entity fields win, then tenant config (vault secrets delivered
via the session trigger configs), then the compiled defaults (2 vCPU /
4096 MB). Providers cap per-sandbox resources differently — Tensorlake
plans limit vCPUs and RAM — so the defaults must be overridable without a
rebuild (ARN-271: every sandbox call failed 'requested 2, max 1' and the
session ran blind).

Applied in both provisioning paths (sandbox_provisioner Resume flow +
monty_repl lazy provisioning) with unit coverage for the precedence chain
and unresolved-secret guards. Published to Genesis as
temperpaw/paw-agent@63acb8a6 (built from the deployed tree so the blob
carries only this change).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H3fCuggTfCRum7qAWq8337
@rita-aga

Copy link
Copy Markdown
Collaborator Author

Status: source patch is correct and unit-tested; the wasm blob build is blocked on build provenance — do NOT install until resolved.

Install attempt of @63acb8a6 failed at module instantiation: unknown import: __wbindgen_placeholder__::__wbindgen_describe. Root cause: a fresh cargo resolve of monty_repl (standalone crate, no committed lockfile) now pulls wasm-bindgen via monty → chrono 0.4.4x (wasmbind default feature on wasm32-unknown-unknown). The deployed blob has zero wbindgen imports — it was built in an environment whose dependency resolution predates that wiring, and pinning chrono back to 0.4.41 does not reproduce it (2,378 wbindgen strings persist). The committed blob's provenance does not match a fresh resolve of the committed sources.

Prod was rolled back to temperpaw/paw-agent@1d831f0e within minutes and session health verified; Genesis LatestVersionHash also reverted to 1d831f0e.

To land this: recover the canonical build recipe/lockfile for the deployed blobs (or vendor a lockfile going forward), rebuild, and gate any install on strings monty_repl.wasm | grep -c wbindgen == 0. The vault overrides (sandbox_cpus=4, sandbox_memory_mb=16384) are already set and become active the moment a good build installs.

@rita-aga

Copy link
Copy Markdown
Collaborator Author

RESOLVED and live-verified. The provenance mystery was the build target: deployed blobs are wasm32-wasip1 (17 WASI imports, zero wbindgen — chrono uses the WASI clock there), not wasm32-unknown-unknown. Rebuilt for wasip1 → byte-profile matches the original (6.7M/17/0).

Shipped as temperpaw/paw-agent@b2515a84 (Genesis published + prod installed + health-checked). Live proof: with vault secrets sandbox_cpus=4 / sandbox_memory_mb=16384, a production curation session provisioned a Tensorlake sandbox that Tensorlake's API reports as {"cpus": 4.0, "memory_mb": 16384} — the override chain works end-to-end, runtime-configurable with zero deploys.

Follow-ups now on the branch record: commit the wasip1 target requirement into the build docs, and vendor lockfiles for the os-app wasm crates so resolves are reproducible.

rita-aga and others added 5 commits July 20, 2026 18:47
Prebaked provider images (e.g. Tensorlake 'katagami-render': chromium +
playwright + pillow) let curation sandboxes boot render-ready instead of
bootstrapping a browser per run. Same precedence as sandbox_cpus/memory:
session field -> vault secret -> provider default. Sent as top-level
"image" in the Tensorlake create body when set.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H3fCuggTfCRum7qAWq8337
…ion sessions (ARN-269)

A tool-less provider turn silently completes a session; for curation jobs
that orphans the job mid-pipeline (observed as a per-run coin flip across
models — 4 consecutive losses on 2026-07-20). Sessions may set the field
tool_choice='required' (spec state + Configure param); provider_caller
forwards it on both the OpenRouter and OpenAI/Codex paths; default 'auto'
preserves chat semantics. Paired with katagami-curation@014f930f which sets
it for all curation spawns. Deployed as paw-agent@a8eb71e5, health-verified
(session field present, provider accepted, completion via typed action).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H3fCuggTfCRum7qAWq8337
…le resolution (ARN-275 ARN-276)

- tensorlake_exec: replace the unbounded 600-iteration fast poll (~6s wall
  clock) with a 240s wall-clock budget paced by an in-sandbox blocking
  waiter; redirect stdin from /dev/null so heredocs cannot hang.
- find_pawfs_file: keyed [WorkspaceId, Path] lookup first; walk fallback is
  error-tolerant (its Name+DirectoryId filter is unkeyed and 413s on large
  tenants).
- find_pawfs_directory: unkeyed Path fallback degrades to not-found.
- verify_file_ready: 10s re-check window before declaring a failed write.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DKJ5oR9ELxyVmzjioKrXHJ
…ing watchdog)

The 300s Executing state_timeout is reset by ProgressMade, but the
per-batch dispatch in monty_repl is gated behind
tool_progress_dispatch_enabled, which nothing sets — so any tool batch
containing a command longer than 300s killed the whole session. Signal
ProgressMade from the tensorlake exec waiter loop (~every 20s while the
command runs), mirroring provider_caller's unguarded stream dispatch.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DKJ5oR9ELxyVmzjioKrXHJ
…nt-side no-tool-call nudge; delete dead call_openrouter (ARN-269)

The live OpenRouter path (call_openai_compatible_chat →
build_chat_completion_body) hardcoded tool_choice='auto'; the earlier
wiring landed in call_openrouter, dead code with zero call sites.
build_chat_completion_body gains a tool_choice param; a tool-less turn
under tool_choice=required is re-called with a bounded correction nudge
(some upstreams ignore the flag); dead function + converters deleted.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DKJ5oR9ELxyVmzjioKrXHJ
@rita-aga

Copy link
Copy Markdown
Collaborator Author

Overnight additions (2026-07-21), all live in production via Genesis + bootstrap pin (paw-agent@3f69bbc):

4bbab6fe — exec budget + keyed file resolution (ARN-275/276)

  • tensorlake_exec: the poll loop ran 600 iterations with no delay (~6s wall clock) then reported "process timed out" while the command was still running — every render died. Now: 240s wall-clock budget paced by an in-sandbox blocking waiter (~10 HTTP calls per 4-min command), stdin redirected from /dev/null (kills the heredoc-hang class).
  • find_pawfs_file: keyed [WorkspaceId, Path] lookup first (O(1) via the ARN-68 oracle); the directory walk's unkeyed Name+DirectoryId filter 413'd on large tenants and killed writes before the keyed fallback was ever reached. Walk failures now degrade to not-found.
  • verify_file_ready: 10s re-check window (StreamUpdated can lag the $value PUT under load).

67288b18 — Executing watchdog fix

  • The 300s Executing state_timeout resets on ProgressMade, but the per-batch dispatch is gated behind tool_progress_dispatch_enabled, which nothing sets — so any tool batch longer than 300s killed the session. The exec waiter now dispatches ProgressMade (~20s cadence) while a command genuinely runs.

86cf436e — tool_choice actually reaches OpenRouter (ARN-269)

  • The live path (call_openai_compatible_chatbuild_chat_completion_body) hardcoded tool_choice:"auto"; the earlier wiring had landed in call_openrouter — dead code, zero call sites. build_chat_completion_body gains a tool_choice param; a tool-less turn under required gets a bounded client-side nudge (2 rounds); the dead function and its converters are deleted (~450 lines).

Production proof: gpt-5.5 produced a finalizer-verified full-artifact language ("Minute of First Light", incl. a real generated hero PNG); qwen 3.6 — which died at turn 1 before the fix — ran 140+ consecutive tool-call turns after it.

🤖 Generated with Claude Code

https://claude.ai/code/session_01DKJ5oR9ELxyVmzjioKrXHJ

rita-aga and others added 11 commits July 21, 2026 20:06
…f-healing action errors

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DKJ5oR9ELxyVmzjioKrXHJ
…on; done-path + readiness-orphan release

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DKJ5oR9ELxyVmzjioKrXHJ
… at spawn on the data plane

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DKJ5oR9ELxyVmzjioKrXHJ
…s (ARN-280)

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DKJ5oR9ELxyVmzjioKrXHJ
…list/action results

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DKJ5oR9ELxyVmzjioKrXHJ
…olution diagnostics

Typed sessions (tool_choice=required) no longer hang their job and leak their
sandbox on a tool-less end_turn: bounded protocol nudge (3), then Fail —
terminal, so the sandbox releases and the job's auto-repair path runs. Every
provision logs which source supplied the sandbox image (entity field vs
trigger config, empty vs unresolved template) so resolution failures are loud.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DKJ5oR9ELxyVmzjioKrXHJ
…acquire 1.99s -> 0.51s median

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DKJ5oR9ELxyVmzjioKrXHJ
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DKJ5oR9ELxyVmzjioKrXHJ
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant