feat: keep a cluster agent's managed memory on its sandbox volume

[zfy0701]

Summary

A pool agent's managed memory (memory/, memory/.history, channels/<key>/…, memory-dreams/<id>/…) was written under the member's ephemeral state root, so a duty move or a rollout lost it (#1078). This PR moves the whole managed tree of a cluster agent onto its sandbox volume — one root beside the checkout, <workspace mount>/.agentconnect/memory, same layout beneath — read and written through the shim's file surface, and makes every managed-memory writer and reader a directory abstraction over one file-system port so a later home is a port swap. Local single-daemon behaviour is unchanged.

Closes #1078. Completes the Memory half of #1070 (Files half: #1077).

Decision (from the issue)

• Option A — files on the agent's sandbox volume; option B (a data-plane table) stays available as another MemoryFs implementation plus a one-directory-per-agent copy, and the rationale for A over B (and over the control-plane database) is recorded in the issue.
• Reads and writes while the sandbox is not running answer sandbox-unavailable — one resolution, no fallback to the member's disk (mirrors createWorkspaceReader.filesOf). The console wakes the sandbox first, exactly like Files (feat: wake a cluster agent's sandbox from the console instead of explaining #1077).
• No migration: existing pool agents' member-local memory is already lost with each rollout; local agents keep their files where they are.

Storage and the file-system port

• agents/memory-fs.ts — the ONE port MemoryFs (readFile / writeFile / readdir / mkdir / rename / rm / utimes, root-relative paths, an identity key for the in-process locks and write ledger, subdir for a channel root), exactly two implementations — LocalMemoryFs(rootDir), which keeps the hardened path walk (component-wise realpath, symlink refusal, O_NOFOLLOW, random exclusive temp + rename, mtime precondition) that memory.ts used to inline, and ShimMemoryFs(channel, rootPath) — and one factory, resolveMemoryFs(agent, sandboxPlane), behind Daemon.memoryFsFor(agentId): the only place that decides (local port without a plane; the bound sandbox's port under --k8s; MemorySandboxUnavailableError when the sandbox is not bound). No string | MemoryFs unions; no memory consumer receives a directory path.
• agents/memory.ts, the managed provider, the distiller, agents/dream-runner.ts and cp/memory-reader.ts no longer touch node:fs; each takes a MemoryFs (the dispatching provider takes a MemoryProviderDeps object with memoryFsFor; the local agentDirByAgent survives only for native memory and the accepted-skills root, which are not managed memory). History append is now canonicalize-then-rewrite (one write instead of compact/append/compact). Dream staging, the adoption swap (backup, replacement, rename, unchanged-file mtimes), the distill rebase and staged reads all run over the port; a dream's extraction cwd is its input dir in the coordinates of the filesystem that holds it (the pod's, for a cluster agent). A staged skill candidate is materialised locally only for the snapshot walker on accept/review.
• Shim — the memory tree rides the read capability beside the workspace ops: shim/memory-fs-channel.ts (payloads shaped for the 256 KiB frame: budgeted slice reads at an offset; writes staged as appended chunks into a sibling temp and committed by one rename carrying the mtime precondition; text as budget-fitted utf8, bytes as base64) and shim/fd-memory-fs.ts (the pod side, descriptor-bound through withDescent like the workspace executor). The daemon side, ShimMemoryFs, is a pass-through that reassembles slices and chunks; the two typed refusals cross the channel as themselves. The memory logic (locks, history, retention, write ledger, dream fence) stays in the daemon process — only primitives cross the wire.
• Plane — K8sRuntimePlane.memoryFsFor(agentId) on the same condition as workspaceFilesFor; sandboxMemoryRoot(mount); onSandboxBound hook. daemon.ts resolves the managed root once per call (managedMemoryRoot): local dir, or the bound sandbox port, else MemorySandboxUnavailableError.
• Session start — memory.ensure is now async and runs after the host is up (a cluster agent's tree is reachable only once the pod is bound); injection was already there.
• Post-turn capture after suspend — managed distillation that finds the tree unreachable is enqueued in the existing memory_capture_outbox (shared-store-safe) under a synthetic per-agent connection (agents/managed-distill-outbox.ts, withManagedDistill); the pump defers the row without spending attempts while the sandbox is down and drains it when the sandbox binds again on the holder (onSandboxBound wakes the pump). The outbox contract was narrowed to what the pump needs (MemoryCapturePumpRegistry / MemoryCaptureClient); plugin connections are untouched. Ordinary turns still distill inline as before.
• Wire — the daemon reports the refusal as BAD_PAYLOAD + reason: sandbox-unavailable on the memory and dream frames; the control plane maps it to 503 with code: WORKSPACE_SANDBOX_UNAVAILABLE on the memory file, channels, history, write and dream routes (never a 400).

Layering

MemoryFs is the file-shaped port for one thing: an agent's managed memory tree (memory/, .history, channels/, dream staging and backups). It answers "where do these files live" — this disk or the agent's sandbox volume — and nothing else; the memory logic (locks, history, retention, write ledger, dream fence) sits above it in the daemon process, and the memory provider dispatch (DispatchingMemoryProvider: managed / native / none / external plugins, per agent) sits above that, deciding which backend serves an agent's recall, capture, tools and console surface.

Org knowledge does not go through either. On the daemon it is a separate path: the findKnowledge / listKnowledge / orgSkills MCP tools call the control plane directly over the knowledge/search / knowledge/list / org-skills frames (daemon.ts → CpClient.knowledgeSearch…), dream proposals reach it through knowledge/suggestion/*, and the console reads it from the CP's own REST surface — search and revision semantics over CP-owned rows, not a file tree, so MemoryFs is the wrong layer for it. If a unification is wanted, it belongs one level up: an org-knowledge contribution to recallForTurn behind the provider dispatch (a follow-up, not in this PR).

Console

• MemoryPanel takes sandboxed (pool placement, like WorkspaceFiles) and wires useSandboxWake on the root read (the file list): the wake is pressed once when the read refuses with the asleep code (or on open for a pool agent), the calm "starting" line replaces the error, the read is polled, and the terminal copy (MEMORY_SANDBOX_ASLEEP_NOTICE) shows with a Start button; unsupported keeps the copy without one; a plain 503 presses nothing; non-managed providers never press.

Docs

• docs/designs/memory-evolution.md §3.2.1 — where the managed tree lives per placement and how the port and shim channel work.
• docs/designs/k8s-daemon-pool.md §11 — one paragraph on managed memory on the sandbox volume.

Test plan

• packages/daemon — memory-fs.test.ts (local port semantics and hardening; resolveMemoryFs: local agent → LocalMemoryFs, bound sandbox → ShimMemoryFs, unbound → MemorySandboxUnavailableError; the shim port over a fake requester: the managed tree runs unchanged on a pod root, multi-frame reads and chunked writes, typed refusals, provider + CP reader over the port, sandbox-unavailable when unbound, shape query still answers); the descriptor-bound executor and exec-handler routing on Linux (scripts/linux-vitest.sh, run in a container); dream-runner.test.ts (new: stage + adopt on a sandbox root through the port, cwd in pod coordinates, nothing on the member's disk; all 66 existing cases green over the port); managed-distill-outbox.test.ts (deferred without attempts while down, drained once bound, plugin registry untouched); k8s-runtime-plane (memoryFsFor only with a bound channel, root beside the checkout); cp/client-dispatch (memory refusal carries the reason); memory / channel-memory / distiller / provider-dispatch / mcp-ops / session-manager suites green. Full daemon suite: only the pre-existing environment failures.
• packages/control-plane — test:unit; test:int agents.memory.route.test.ts (new: every memory file route answers 503 + WORKSPACE_SANDBOX_UNAVAILABLE for a sleeping sandbox).
• packages/web — MemoryPanel.test.tsx (wake pressed once on the asleep 503, starting line, terminal copy with Start that re-presses, unsupported without Start, plain 503 presses nothing, sandboxed presses on open, native never presses); sandbox-wake / FilesPanel suites; full web suite green.
• pnpm typecheck (daemon, control-plane, web), pnpm lint, pnpm format:check.
• On a cluster: write memory in a turn, suspend the pod, roll the pool; open the Memory tab and watch it wake and show the same files; end a turn, let the pod suspend before distillation runs, and confirm the turn is distilled after the next wake.

Notes for review

• The shim image must ship with a daemon that speaks the new payload family: an older shim answers a memory-fs frame with a parse error, which surfaces as a generic failure until the image rolls (both are built from this package).
• While a deferred managed capture waits for a sleeping sandbox, the pump re-checks it every unavailableRetryMs (5 s) — a couple of cheap store queries; a row that never finds its sandbox expires with the outbox's usual retention.
• Accepted dream skills still publish under the member-local agent dir (unchanged, out of scope); the staged candidate is pulled through the port for the snapshot walker.

[agentconnect-md-test]

Requesting changes for one cluster lifecycle regression. Managed memory correctly moves onto the sandbox volume, but scheduled dreams still call DreamRunner.start() directly after recording the schedule fire. start() now requires an already-bound sandbox before it can snapshot memory, so a normally suspended pool agent loses that scheduled dream instead of waking and running it. The scheduled path needs to bind/wake the sandbox before resolving the memory filesystem (while ordinary console/admin reads can retain the new sandbox-unavailable behavior).

Verification: 261 focused daemon tests and 27 focused web tests passed; daemon, web, and control-plane typechecks passed via their underlying Node entry points; changed files passed ESLint. The focused control-plane integration test could not run because this review environment has no container runtime.

_{sent by review-bot (Codex · gpt-5.6-sol) · open in session}

[agentconnect-md-test]

Wake/bind the sandbox for scheduled dreams before resolving this filesystem. onDreamScheduleFire() records dreamLastRun and then calls start() directly; unlike the Memory console or session-start path, it has no wake step. A pool agent whose pod was suspended during the usual overnight idle window therefore throws MemorySandboxUnavailableError here, logs a failed start, and does not retry until the next schedule occurrence. That makes scheduled dreaming effectively skip normal idle cluster agents. Please run the scheduled start through an ensureChannel/sandbox-hold seam (or inject an equivalent resolver that can wake for this authorized background job) before taking the snapshot.

[zfy0701]

Tightened the surface as reviewed (head fbc4fc7): MemoryFs is the single port every memory consumer takes — memory.ts helpers, ManagedMemoryProvider, the distiller, DreamRunner, the daemon-side memory reader; the string | MemoryFs union is gone and no caller passes a directory path. Exactly two implementations, LocalMemoryFs(rootDir) and ShimMemoryFs(channel, rootPath), and one factory, resolveMemoryFs(agent, sandboxPlane) behind Daemon.memoryFsFor(agentId), is the only place that decides (local without a plane, the bound sandbox's port under --k8s, MemorySandboxUnavailableError when unbound). The dispatching provider now takes a MemoryProviderDeps object; tests construct LocalMemoryFs(dir) explicitly, plus a factory test. Also added a Layering section to the body on where org knowledge sits relative to the provider dispatch (separate CP-frame path; unification, if any, belongs one level up — not changed here).

[agentconnect-md-test]

Requesting changes because the cluster scheduled-dream lifecycle regression remains on this revision. The new MemoryFs-only refactor is coherent, but onDreamScheduleFire() still records the occurrence and calls DreamRunner.start() directly. start() immediately resolves the managed filesystem, and resolveMemoryFs() throws MemorySandboxUnavailableError when the pool agent's sandbox is suspended. Since an idle cluster agent will normally have no running pod at an overnight dream schedule, the authorized scheduled job is logged as failed and that occurrence is lost until the next schedule. Please wake/bind the sandbox (and hold it for the background job) before resolving/snapshotting its memory; ordinary console/admin reads can retain the intended sandbox-unavailable behavior.

Verification on fbc4fc729dff125b2205e26cf8e5cc03e0b6a998: 269 focused daemon tests passed, the daemon typecheck passed, and every changed TypeScript file passed ESLint.

_{sent by review-bot (Codex · gpt-5.6-sol) · open in session}

[agentconnect-md-test]

Thanks—the single-port refactor is clear and is not what blocks approval.

At fbc4fc729, onDreamScheduleFire() still stamps the occurrence and directly calls DreamRunner.start(). start() immediately resolves MemoryFs, with no sandbox wake/bind. A suspended cluster agent therefore throws MemorySandboxUnavailableError, logs the failed start, and loses the stamped occurrence.

Please wake/bind the sandbox before the snapshot and keep it available for the background dream lifecycle. Ordinary console/admin reads can continue returning sandbox-unavailable.

_{sent by review-bot (Codex · gpt-5.6-sol) · open in session}

[zfy0701]

Addressed the review-bot item on scheduled dreams (head 2f2c373): a dream is authorized background work like a turn, so DreamRunner now runs its start through a new withMemoryHome seam — under --k8s the daemon binds it to ensureChannel + withSandbox, i.e. wake, bind, hold — around the memory snapshot and again around the whole run including auto-adoption, so a pool agent whose pod is suspended at its overnight schedule is woken instead of failing the start, and the idle sweep cannot suspend the pod mid-dream (the hold is the same busy count the sweep already respects). An in-flight dream also counts as in-flight work for the shutdown duty release (dutyGroupBusy), and the scheduled path already skips while draining. Console/admin reads keep the sandbox-unavailable refusal. Test added: scheduled dream on a suspended pool agent wakes the sandbox, is held during extraction, adopts on the sandbox root, releases the hold afterwards, and an unbound read still refuses. Rebased on main; dream/memory/k8s/duty suites, typecheck, lint, format:check green.

[agentconnect-md-test]

That lifecycle shape addresses the concern. I can’t verify it yet: GitHub still reports PR head fbc4fc729, and revision 2f2c373 is not currently available through revision-addressed reads. I’ll reevaluate once the push updates the PR.

_{sent by review-bot (Codex · gpt-5.6-sol) · open in session}

[agentconnect-md-test]

The scheduled-dream wake regression is fixed: withMemoryHome binds and holds a suspended cluster sandbox across the snapshot and asynchronous run. Requesting changes for one remaining lifecycle hole in the new background-job bookkeeping.

dutyGroupBusy() delegates to DreamRunner.inFlight(), which only reads active. The run clears active before maybeAutoAdopt(), so shutdown can treat the group as idle, revoke its duty, and release the shim channel while auto-adoption is still reading and swapping the sandbox-backed memory tree. The same block also leaves active and the aborter permanently registered if the second withMemoryHome acquisition/ensureChannel rejects before its callback starts, because cleanup is inside that callback and the outer rejection is swallowed. Please track the whole background-home operation separately from the adoption reservation, expose that whole lifetime to the duty-busy predicate, and perform reservation cleanup from a promise-wide finally.

Verification on 2f2c3739e6a344344fa305507d3b6c3b6144f460: 166 focused dream/K8s/duty/schedule tests passed; daemon typecheck and changed-file ESLint passed.

_{sent by review-bot (Codex · gpt-5.6-sol) · open in session}

[agentconnect-md-test]

This clears the only state inFlight() observes before maybeAutoAdopt() runs. During that adoption window, dutyGroupBusy() can return false and shutdown can call releaseClusterSandbox(), which revokes the shim session and discards the driver's hold while adopt() is still swapping the remote memory tree. There is a second cleanup leak on the same promise: if withMemoryHome rejects in ensureSandbox/ensureChannel before entering this callback, this finally never runs and the outer .catch(() => {}) leaves active/aborters stuck. Please keep a separate background-job marker across acquisition → run → auto-adopt, clear it from a promise-wide finally, and have the duty-busy predicate follow that marker.

[zfy0701]

Addressed the remaining lifecycle hole (head 18dba27): DreamRunner keeps a separate backgroundJobs marker that spans home acquisition, the run and any auto-adoption, cleared in a promise-wide finally outside the withMemoryHome callback; inFlight() (and so dutyGroupBusy) follows that marker, while the adoption reservation (active) stays its own thing and is still released before auto-adoption. A rejected acquisition (a wake that failed before the callback ran) now releases the reservation and aborter, marks the dream failed instead of leaving it pending, and unblocks the next start. Tests: (a) shutdown during auto-adoption still sees the job in flight — the sandbox channel is parked on adoption's first frame and inFlight is true until the swap completes; (b) a second withMemoryHome that rejects before its callback leaves no stuck reservation/aborter — the dream is failed, cancel is a plain state answer, and a new dream starts. Rebased on main; dream/duty/k8s suites, typecheck, lint, format:check green.

[agentconnect-md-test]

Approved. The prior lifecycle findings are resolved on 18dba27739e734ab9218a98266191a42ef9e5a2c: scheduled dreams wake and hold a suspended cluster sandbox, the separate background-job marker remains visible to duty shutdown through auto-adoption, and a failed second home acquisition now marks the dream failed and clears its reservation/job state from a promise-wide finally.

Non-blocking follow-up: the job marker is registered after the initial remote snapshot, so shutdown racing that initial snapshot can still release the sandbox and drop that scheduled attempt. That is a narrow shutdown-only window and does not risk a partial store swap, so I would not hold this startup-stage PR for it.

Verification: 168 focused dream/K8s/duty/schedule tests passed; daemon typecheck and changed-file ESLint passed.

_{sent by review-bot (Codex · gpt-5.6-sol) · open in session}

[agentconnect-md-test]

Confirmed on 18dba277; the blocking lifecycle issues are resolved and the revision is approved.

The only remaining note is the non-blocking shutdown window during the initial snapshot, before backgroundJobs is registered. It does not risk a partial store swap and does not hold this PR.

_{sent by review-bot (Codex · gpt-5.6-sol) · open in session}