Skip to content

feat(daemon): pool orphan reconciler for leaked sandbox objects - #1074

Merged
zfy0701 merged 2 commits into
mainfrom
feat/orphan-reconciler
Aug 16, 2026
Merged

feat(daemon): pool orphan reconciler for leaked sandbox objects#1074
zfy0701 merged 2 commits into
mainfrom
feat/orphan-reconciler

Conversation

@zfy0701

@zfy0701 zfy0701 commented Aug 16, 2026

Copy link
Copy Markdown
Contributor

Summary

Closes #1062 (Kubernetes side). Adds an orphan reconciler to --k8s daemons: a periodic sweep that finds sandbox objects nobody will ever clean up and removes them, instead of every teardown path carrying its own durable obligation to survive a member dying mid-way. Two commits: the reconciler first, then the special-purpose logic it subsumes.

Design

  • Where it runs. Every --k8s daemon (packages/daemon/src/k8s/orphan-reconciler.ts), by default every 10 minutes with ±25% jitter, one member at a time: a named single-holder lease in the shared store (LocalStore.acquireSweepLease, new table sweep_leases) is taken or renewed on each sweep and lasts three intervals. No schema step — a new CREATE TABLE IF NOT EXISTS lives in the CREATE block, so SCHEMA_VERSION stays 9 and fix(daemon): own the session-metadata outbox per member #1068's v10 is untouched.
  • What it lists. SandboxClaims and Sandboxes in the sandbox namespace carrying the install's agent label (agentconnect.md/agent on the pod metadata). Anything without the label — a warm-pool spare, another tenant's object — is not a candidate.
  • The control-plane read. One batched agent/existsagent/exists/ok install-wide frame pair per sweep ("which of these agent ids exist"), advertised as the agent-exists-v1 server feature; a daemon that does not see the feature (older CP mid-rollout) skips the sweep. Handler: packages/control-plane/src/ws/handlers/agent-exists.ts, fenced to the connection's org for an org-scoped daemon.
  • What it collects. A claim whose agent the CP no longer knows and has not known for at least the grace period (default 10 min) as observed by the sweeping member across its own sweeps, on an object at least that old; a probe claim past the window the probe stamped on it; a claimless Sandbox of a gone agent under the same grace.
  • One summary line per sweep: candidates, orphaned, deleted, skipped-live, skipped-grace, failed.

Safety rules

  • An object of a live agent is never touched — a claimless Sandbox of a live agent included (deleting a claim deletes the workspace volume; a stray of a live agent is reported, not collected).
  • Every delete carries the UID and resourceVersion from the LIST snapshot, so a same-name replacement created after the list is never the object deleted.
  • Skip, never guess: an id the CP cannot be asked about (non-UUID), an object without a readable age, a Sandbox list the Role does not permit (narrows to claims, warned once), and a sweep whose CP read fails all leave everything alone.
  • Idempotent; a lease handover only restarts the missing-since clock (delays, never accelerates, a delete).

What was retired (commit 2)

Deployment side

Ships dry-run by default — logs and counters only. After an observation window in which the summary lines show it collecting exactly what an operator would, enable deletion with AC_K8S_ORPHAN_DELETE=true on the daemon pool. AC_K8S_ORPHAN_SWEEP_INTERVAL_MS and AC_K8S_ORPHAN_GRACE_MS tune the cadence and grace. Listing/deleting sandboxes needs those verbs on the daemon Role; without them the sweep narrows to claims and says so once.

Test plan

  • New packages/daemon/test/k8s-orphan-reconciler.test.ts against the fake API server and a fake CP answer: orphaned claim past grace deleted (enabled) / only logged (dry run); a live agent's objects never touched; a just-missing agent skipped inside grace; unreadable age skipped; claimless Sandbox of a gone agent collected, bound and unlabelled ones left; expired probe claims collected by their window without asking the CP; only the lease holder sweeps; CP read failure skips the sweep; Forbidden Sandbox list narrows to claims; jittered timer arms and stops.
  • local-store.test.ts: two members over one database contend for the sweep lease; the same assertion added to the PostgreSQL pool-store integration test and run against a local postgres:16 (its first test fails on main for an unrelated pre-existing reason).
  • agent-exists.test.ts (CP handler): install-wide answer, org fence.
  • Ran: daemon typecheck, k8s driver/takeover/runtime-plane/client/cold-start, daemon-k8s-mode, duty suites, test/cp, local-store; CP typecheck + ws unit; protocol suite; pnpm lint; pnpm format:check.
  • Design note: docs/designs/k8s-daemon-pool.md §4 "Orphan reconciliation" (+ implementation map row), cluster-spawn-and-shim.md probe paragraph.

Add a periodic sweep on `--k8s` daemons that lists the sandbox namespace's
SandboxClaims and Sandboxes carrying the install's agent label, asks the
control plane in one batched `agent/exists` read which of those agents still
exist, and collects only what is provably orphaned: a claim whose agent has
been gone for at least the grace period on both the member's own sweeps and
the object's age, a probe claim past its stamped window, and a claimless
Sandbox of a gone agent. Objects of a live agent are never touched, every
delete is fenced on the listed UID and resourceVersion, and one member sweeps
at a time through a single-holder lease in the shared store
(`sweep_leases`, no schema step: a new table lives in the CREATE block).

It ships dry-run: `AC_K8S_ORPHAN_DELETE=true` enables deletion, and
`AC_K8S_ORPHAN_SWEEP_INTERVAL_MS` / `AC_K8S_ORPHAN_GRACE_MS` tune the
jittered cadence and grace. Each sweep logs one summary line.

The control plane gains the `agent/exists` → `agent/exists/ok` install-wide
frame pair (advertised as `agent-exists-v1`); a daemon that does not see the
feature skips the sweep. Probe-claim identity moves to `k8s/probe-claim.ts`
so the reconciler and the runtime plane share it without a cycle.

Refs #1062
…sumes

Remove the dedicated expired-probe-claim sweep from the runtime plane
(`reapExpiredProbeClaims` and its five-minute timer): the orphan reconciler
collects expired probe claims by the same stamped window, with the same
UID/resourceVersion fence. Agent removal's sandbox teardown stays one
best-effort delete; its failure log now points at the reconciler instead of
asking for a manual claim delete. The design notes follow.

Refs #1062
@zfy0701
zfy0701 merged commit 2132862 into main Aug 16, 2026
11 checks passed
@zfy0701
zfy0701 deleted the feat/orphan-reconciler branch August 16, 2026 03:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Orphan reconciler for the daemon pool: sweep leaked sandbox objects and store rows instead of per-case obligations

1 participant