feat(daemon): pool orphan reconciler for leaked sandbox objects - #1074
Merged
Conversation
Add a periodic sweep on `--k8s` daemons that lists the sandbox namespace's SandboxClaims and Sandboxes carrying the install's agent label, asks the control plane in one batched `agent/exists` read which of those agents still exist, and collects only what is provably orphaned: a claim whose agent has been gone for at least the grace period on both the member's own sweeps and the object's age, a probe claim past its stamped window, and a claimless Sandbox of a gone agent. Objects of a live agent are never touched, every delete is fenced on the listed UID and resourceVersion, and one member sweeps at a time through a single-holder lease in the shared store (`sweep_leases`, no schema step: a new table lives in the CREATE block). It ships dry-run: `AC_K8S_ORPHAN_DELETE=true` enables deletion, and `AC_K8S_ORPHAN_SWEEP_INTERVAL_MS` / `AC_K8S_ORPHAN_GRACE_MS` tune the jittered cadence and grace. Each sweep logs one summary line. The control plane gains the `agent/exists` → `agent/exists/ok` install-wide frame pair (advertised as `agent-exists-v1`); a daemon that does not see the feature skips the sweep. Probe-claim identity moves to `k8s/probe-claim.ts` so the reconciler and the runtime plane share it without a cycle. Refs #1062
…sumes Remove the dedicated expired-probe-claim sweep from the runtime plane (`reapExpiredProbeClaims` and its five-minute timer): the orphan reconciler collects expired probe claims by the same stamped window, with the same UID/resourceVersion fence. Agent removal's sandbox teardown stays one best-effort delete; its failure log now points at the reconciler instead of asking for a manual claim delete. The design notes follow. Refs #1062
This was referenced Aug 16, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Closes #1062 (Kubernetes side). Adds an orphan reconciler to
--k8sdaemons: a periodic sweep that finds sandbox objects nobody will ever clean up and removes them, instead of every teardown path carrying its own durable obligation to survive a member dying mid-way. Two commits: the reconciler first, then the special-purpose logic it subsumes.Design
--k8sdaemon (packages/daemon/src/k8s/orphan-reconciler.ts), by default every 10 minutes with ±25% jitter, one member at a time: a named single-holder lease in the shared store (LocalStore.acquireSweepLease, new tablesweep_leases) is taken or renewed on each sweep and lasts three intervals. No schema step — a newCREATE TABLE IF NOT EXISTSlives in the CREATE block, soSCHEMA_VERSIONstays 9 and fix(daemon): own the session-metadata outbox per member #1068's v10 is untouched.SandboxClaims andSandboxes in the sandbox namespace carrying the install's agent label (agentconnect.md/agenton the pod metadata). Anything without the label — a warm-pool spare, another tenant's object — is not a candidate.agent/exists→agent/exists/okinstall-wide frame pair per sweep ("which of these agent ids exist"), advertised as theagent-exists-v1server feature; a daemon that does not see the feature (older CP mid-rollout) skips the sweep. Handler:packages/control-plane/src/ws/handlers/agent-exists.ts, fenced to the connection's org for an org-scoped daemon.Sandboxof a gone agent under the same grace.candidates,orphaned,deleted,skipped-live,skipped-grace,failed.Safety rules
Sandboxof a live agent included (deleting a claim deletes the workspace volume; a stray of a live agent is reported, not collected).Sandboxlist the Role does not permit (narrows to claims, warned once), and a sweep whose CP read fails all leave everything alone.What was retired (commit 2)
reapExpiredProbeClaimsand its five-minute timer). Probe-claim identity moved tok8s/probe-claim.ts; the reconciler collects expired probe claims by the same stamped window and fence.discardAgentdeletes the claim once and logs a failure that now points at the reconciler rather than asking for a manual claim delete. It already had no retry bookkeeping to remove; the shim generation, drain and duty-ledger paths are untouched.Deployment side
Ships dry-run by default — logs and counters only. After an observation window in which the summary lines show it collecting exactly what an operator would, enable deletion with
AC_K8S_ORPHAN_DELETE=trueon the daemon pool.AC_K8S_ORPHAN_SWEEP_INTERVAL_MSandAC_K8S_ORPHAN_GRACE_MStune the cadence and grace. Listing/deletingsandboxesneeds those verbs on the daemon Role; without them the sweep narrows to claims and says so once.Test plan
packages/daemon/test/k8s-orphan-reconciler.test.tsagainst the fake API server and a fake CP answer: orphaned claim past grace deleted (enabled) / only logged (dry run); a live agent's objects never touched; a just-missing agent skipped inside grace; unreadable age skipped; claimlessSandboxof a gone agent collected, bound and unlabelled ones left; expired probe claims collected by their window without asking the CP; only the lease holder sweeps; CP read failure skips the sweep; ForbiddenSandboxlist narrows to claims; jittered timer arms and stops.local-store.test.ts: two members over one database contend for the sweep lease; the same assertion added to the PostgreSQL pool-store integration test and run against a localpostgres:16(its first test fails onmainfor an unrelated pre-existing reason).agent-exists.test.ts(CP handler): install-wide answer, org fence.typecheck, k8s driver/takeover/runtime-plane/client/cold-start, daemon-k8s-mode, duty suites,test/cp, local-store; CPtypecheck+ ws unit; protocol suite;pnpm lint;pnpm format:check.docs/designs/k8s-daemon-pool.md§4 "Orphan reconciliation" (+ implementation map row),cluster-spawn-and-shim.mdprobe paragraph.