feat(daemon): pool orphan reconciler for leaked sandbox objects

docs/designs/cluster-spawn-and-shim.md

@@ -72,10 +72,11 @@ The daemon pool and agent sandboxes are separate namespaces. The runtime plane r
 `AC_K8S_SANDBOX_NAMESPACE` and uses it for every `SandboxClaim` and `Sandbox` request; it never
 defaults to the namespace mounted into the daemon Pod's ServiceAccount.
 
-Runtime probes use a member-hashed claim name plus an expiry annotation. A label-filtered daemon
-sweep deletes expired claims, bounding resources left by a crash or failed teardown without reading
-daemon-local storage or touching ordinary agent claims. UID/resourceVersion delete preconditions
-prevent a stale sweep from deleting a same-name claim recreated by a container restart.
+Runtime probes use a member-hashed claim name plus an expiry annotation. The pool's orphan
+reconciler (k8s-daemon-pool.md §4) collects expired probe claims by that window, bounding resources
+left by a crash or failed teardown without reading daemon-local storage or touching ordinary agent
+claims. UID/resourceVersion delete preconditions prevent a stale sweep from deleting a same-name
+claim recreated by a container restart.
 
 ## 3. Binding: proving which pod accepted the connection
 

docs/designs/k8s-daemon-pool.md

@@ -137,9 +137,8 @@ and Sandboxes. Each member also receives its Pod UID through the Downward API as
 `AC_K8S_MEMBER_ID`; the runtime probe hashes it into
 `agent-ac-runtime-probe-<member-hash>`, so simultaneous member startup never races
 on one probe claim. Probe claims carry a dedicated label and a 15-minute expiry;
-members periodically delete expired claims, so a missed teardown cannot retain a
-Sandbox and volume forever. Each GC delete carries the UID and resourceVersion from
-its LIST snapshot, so a same-name replacement cannot be deleted by a stale sweep.
+the orphan reconciler (§4) collects an expired one, so a missed teardown cannot
+retain a Sandbox and volume forever.
 
 **Org-threading is the end state; instantiation is scaffolding.** The wire
 carries the org, the data plane carries the org, and the process interior
@@ -187,6 +186,49 @@ record at pod-name time — and the holder then dials the shim and binds at its
 term (§7). Cold, warm, and resume paths are already distinguished and metered
 (`LaunchTimer.observedPath`). **No new wake machinery exists here.**
 
+### Orphan reconciliation
+
+Teardown is best-effort and a member can die mid-way — a rollout, an OOM, a
+node loss — leaving a `SandboxClaim`, a `Sandbox`, or a probe claim that no
+process still intends to remove. Rather than one durable obligation per
+failure mode, a single **orphan reconciler**
+(`packages/daemon/src/k8s/orphan-reconciler.ts`) sweeps the sandbox namespace
+on every `--k8s` daemon, by default every 10 minutes with ±25% jitter, and one
+member at a time: a named single-holder lease in the shared store
+(`LocalStore.acquireSweepLease`, table `sweep_leases`) is taken or renewed at
+each sweep and lasts three intervals, so a holder that disappears is replaced
+after at most that.
+
+**What it collects.** It lists the claims and Sandboxes that carry the
+install's agent label (`agentconnect.md/agent` on the pod metadata), asks the
+control plane in **one batched read per sweep** which of those agent ids still
+exist (`agent/exists` → `agent/exists/ok`, install-wide, advertised as the
+`agent-exists-v1` server feature), and deletes only what is provably orphaned:
+
+- a claim whose agent the control plane no longer knows — and has not known for
+  at least the grace period (default 10 minutes) as observed by the sweeping
+  member across its own sweeps, on an object at least that old;
+- a probe claim past the window the probe stamped on it;
+- a Sandbox no claim binds, whose agent the control plane no longer knows,
+  under the same grace.
+
+**Safety rules.** An object of a live agent is never touched, a claimless
+Sandbox included — deleting a claim deletes the workspace volume and is
+irreversible, so a stray of a live agent is reported, not collected. An id the
+control plane cannot be asked about, an object without a readable age, and a
+sweep whose control-plane read fails all skip. Every delete carries the UID and
+resourceVersion from the LIST snapshot, so a same-name replacement created
+after the list is never the object deleted. Each sweep logs one summary line
+(candidates, orphaned, deleted, skipped-live, skipped-grace, failed).
+
+**Dry run by default.** The reconciler ships reporting only; deletion is
+enabled per deployment with `AC_K8S_ORPHAN_DELETE=true` after an observation
+window in which the summary lines show it collecting exactly what an operator
+would (`AC_K8S_ORPHAN_SWEEP_INTERVAL_MS` and `AC_K8S_ORPHAN_GRACE_MS` tune the
+cadence and grace). It replaced the dedicated probe-claim GC, and agent
+removal's sandbox teardown is best-effort because of it: `discardAgent` deletes
+the claim once and logs a failure, and the reconciler collects the leftovers.
+
 ## 5. The duty ledger and lease service (D6, D7)
 
 **The CP is the ledger.** Members claim, renew, and release duties over the
@@ -837,16 +879,17 @@ shrinking every actor's permissions.
 
 ## 17. Implementation map
 
-| Piece                                                                  | Where                                                                                                 |
-| ---------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------- |
-| Frames + member cap                                                    | `packages/protocol/src/frames/duty.ts`, `relay-daemon.ts` (`RD_ACK_NOT_HOLDER`)                       |
-| Schema + repo (CAS claim, renew, release, reconcile, agent-home claim) | `packages/control-plane/prisma/schema.prisma`, `src/persistence/repositories/duty-group.repo.ts`      |
-| Pure group math + reconcile planner                                    | `packages/control-plane/src/orchestrator/dutyGroup.ts`                                                |
-| Lease exchange (digest diff, chunking, lanes, grace)                   | `packages/control-plane/src/orchestrator/dutyLease.ts`                                                |
-| Recompute sweep + mutation kicks + placement fence                     | `packages/control-plane/src/orchestrator/dutyRecompute.ts`                                            |
-| WS handlers                                                            | `packages/control-plane/src/ws/handlers/{heartbeat,duty-release,duty-claim}.ts`                       |
-| Daemon registry + gate + rendezvous claim                              | `packages/daemon/src/cp/duty-registry.ts`, `src/daemon.ts` (`transportAgents`, `claimDutyForTrigger`) |
-| Relay re-route                                                         | `packages/relay/src/relay-ingress-manager.ts` (`sendWithRendezvous`), `relay-browser-connection.ts`   |
-| Shim dial-in                                                           | `packages/daemon/src/shim/{dialer,server}.ts`                                                         |
-| Pod-bound member identity                                              | `packages/control-plane/src/cluster/daemon-identity.ts`                                               |
-| Member readiness (probe sinks)                                         | `packages/daemon/src/readiness.ts`, `src/daemon.ts` (`readinessState`)                                |
+| Piece                                                                  | Where                                                                                                    |
+| ---------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------- |
+| Frames + member cap                                                    | `packages/protocol/src/frames/duty.ts`, `relay-daemon.ts` (`RD_ACK_NOT_HOLDER`)                          |
+| Schema + repo (CAS claim, renew, release, reconcile, agent-home claim) | `packages/control-plane/prisma/schema.prisma`, `src/persistence/repositories/duty-group.repo.ts`         |
+| Pure group math + reconcile planner                                    | `packages/control-plane/src/orchestrator/dutyGroup.ts`                                                   |
+| Lease exchange (digest diff, chunking, lanes, grace)                   | `packages/control-plane/src/orchestrator/dutyLease.ts`                                                   |
+| Recompute sweep + mutation kicks + placement fence                     | `packages/control-plane/src/orchestrator/dutyRecompute.ts`                                               |
+| WS handlers                                                            | `packages/control-plane/src/ws/handlers/{heartbeat,duty-release,duty-claim}.ts`                          |
+| Daemon registry + gate + rendezvous claim                              | `packages/daemon/src/cp/duty-registry.ts`, `src/daemon.ts` (`transportAgents`, `claimDutyForTrigger`)    |
+| Relay re-route                                                         | `packages/relay/src/relay-ingress-manager.ts` (`sendWithRendezvous`), `relay-browser-connection.ts`      |
+| Shim dial-in                                                           | `packages/daemon/src/shim/{dialer,server}.ts`                                                            |
+| Pod-bound member identity                                              | `packages/control-plane/src/cluster/daemon-identity.ts`                                                  |
+| Member readiness (probe sinks)                                         | `packages/daemon/src/readiness.ts`, `src/daemon.ts` (`readinessState`)                                   |
+| Orphan reconciler + existence read                                     | `packages/daemon/src/k8s/orphan-reconciler.ts`, `packages/control-plane/src/ws/handlers/agent-exists.ts` |

packages/control-plane/src/ws/connection.ts

@@ -44,7 +44,10 @@ const INSTALL_WIDE_FRAME_TYPES = new Set([
   'duty/revoke',
   'duty/release',
   'duty/claim',
-  'duty/claim/ok'
+  'duty/claim/ok',
+  // Existence query from the pool's orphan reconciler: the ids it asks about span every org.
+  'agent/exists',
+  'agent/exists/ok'
 ])
 
 export class DaemonConnection implements ConnChannel {

packages/control-plane/src/ws/handlers/agent-exists.test.ts

@@ -0,0 +1,49 @@
+// `agent/exists` — existence only, fenced to the connection's org when it has one.
+import { describe, expect, it, vi } from 'vitest'
+import type { AnyFrame } from '@agentconnect.md/protocol'
+import type { DaemonConnection } from '../connection.js'
+import type { DaemonWsDeps } from '../deps.js'
+import { handleAgentExists } from './agent-exists.js'
+
+const LIVE = 'a0a0a0a0-aaaa-4aaa-8aaa-aaaaaaaaaaaa'
+const OTHER_ORG = 'b0b0b0b0-bbbb-4bbb-8bbb-bbbbbbbbbbbb'
+const GONE = 'c0c0c0c0-cccc-4ccc-8ccc-cccccccccccc'
+
+function existsFrame(agentIds: string[]): AnyFrame {
+  return {
+    v: 1,
+    id: crypto.randomUUID(),
+    ts: '2026-08-14T00:00:00.000Z',
+    type: 'agent/exists',
+    payload: { agentIds }
+  } as AnyFrame
+}
+
+function fakeConn(orgId: string | null) {
+  return { daemonId: 'd', orgId, replyTo: vi.fn(), sendError: vi.fn() } as unknown as DaemonConnection & {
+    replyTo: ReturnType<typeof vi.fn>
+  }
+}
+
+const listByIds = vi.fn(async (ids: readonly string[]) =>
+  [
+    { id: LIVE, orgId: 'org-a' },
+    { id: OTHER_ORG, orgId: 'org-b' }
+  ].filter((agent) => ids.includes(agent.id))
+)
+const deps = { agent: { listByIds } } as unknown as DaemonWsDeps
+
+describe('agent/exists', () => {
+  it('answers an install-wide member with every asked id that exists, deduplicated', async () => {
+    const conn = fakeConn(null)
+    await handleAgentExists(existsFrame([LIVE, LIVE, OTHER_ORG, GONE]), conn, deps)
+    expect(listByIds).toHaveBeenLastCalledWith([LIVE, OTHER_ORG, GONE])
+    expect(conn.replyTo).toHaveBeenCalledWith(expect.anything(), 'agent/exists/ok', { existing: [LIVE, OTHER_ORG] })
+  })
+
+  it('fences an org-scoped connection to its own org', async () => {
+    const conn = fakeConn('org-a')
+    await handleAgentExists(existsFrame([LIVE, OTHER_ORG, GONE]), conn, deps)
+    expect(conn.replyTo).toHaveBeenCalledWith(expect.anything(), 'agent/exists/ok', { existing: [LIVE] })
+  })
+})

packages/control-plane/src/ws/handlers/agent-exists.ts

@@ -0,0 +1,16 @@
+// `agent/exists` handler — the batch existence read behind the pool's orphan reconciler.
+// A member lists sandbox objects in its cluster, reads the agent ids they carry, and asks
+// here in one round trip which of those agents still exist. Existence only: an id absent
+// from the reply is gone and its objects may be collected; a present id is live and its
+// objects are never touched. An org-scoped connection sees only its own org's agents.
+import { isFrame } from '@agentconnect.md/protocol'
+import { AgentId } from '../../domain/ids.js'
+import type { Handler } from './index.js'
+
+export const handleAgentExists: Handler = async (frame, conn, deps) => {
+  if (!isFrame('agent/exists')(frame)) return
+  const asked = [...new Set(frame.payload.agentIds)].map((id) => AgentId(id))
+  const agents = await deps.agent.listByIds(asked)
+  const existing = agents.filter((agent) => conn.orgId === null || agent.orgId === conn.orgId).map((agent) => agent.id)
+  conn.replyTo(frame, 'agent/exists/ok', { existing })
+}

packages/control-plane/src/ws/handlers/index.ts

@@ -30,6 +30,7 @@ import { handleCronReport } from './cron-report.js'
 import { handleDutyRelease } from './duty-release.js'
 import { handleDutyClaim } from './duty-claim.js'
 import { handleDutyFetch } from './duty-fetch.js'
+import { handleAgentExists } from './agent-exists.js'
 import { handleHookReport } from './hook-report.js'
 import { handleChannelAgents } from './channel-agents.js'
 import { handleChildSessionStatus } from './child-session-status.js'
@@ -74,6 +75,7 @@ export class FrameRouter {
       'duty/release': handleDutyRelease,
       'duty/claim': handleDutyClaim,
       'duty/fetch': handleDutyFetch,
+      'agent/exists': handleAgentExists,
       'hook/report': handleHookReport,
       'hook/start': handleHookStart,
       'github/review-authorize': handleGithubReviewAuthorize,

packages/control-plane/src/ws/handlers/register.ts

@@ -12,6 +12,7 @@
  */
 import {
   isFrame,
+  AGENT_EXISTS_FEATURE,
   ORGANIZATION_KNOWLEDGE_FEATURE,
   SESSION_LIVE_TAIL_FEATURE,
   SESSION_METADATA_ACK_FEATURE,
@@ -75,7 +76,8 @@ export const handleRegister: Handler = async (frame, conn, deps) => {
       SESSION_METADATA_ACK_FEATURE,
       SESSION_PURGE_FEATURE,
       SESSION_VISIBILITY_FEATURE,
-      ORGANIZATION_KNOWLEDGE_FEATURE
+      ORGANIZATION_KNOWLEDGE_FEATURE,
+      AGENT_EXISTS_FEATURE
     ]
   })
   deps.connReg.markReady(conn.daemonId, conn)

packages/daemon/src/cp/client.ts

@@ -14,6 +14,7 @@ import type {
   DutyRevoke,
   DutyClaimOk,
   DutyFetchOk,
+  AgentExistsOk,
   FactsRuntimeProfile,
   FactsMcpServer,
   UsageReport,
@@ -156,7 +157,9 @@ const INSTALL_WIDE_FRAME_TYPES = new Set([
   'duty/revoke',
   'duty/release',
   'duty/claim',
-  'duty/claim/ok'
+  'duty/claim/ok',
+  'agent/exists',
+  'agent/exists/ok'
 ])
 
 const ACK_TIMEOUT_MS = 5000
@@ -933,6 +936,23 @@ export class CpClient {
     return rep.payload as DutyFetchOk
   }
 
+  /**
+   * `agent/exists` (D→C REQ → `agent/exists/ok`) — which of these agents the CP still knows.
+   * Asked by the cluster orphan reconciler for the ids it read off sandbox objects, in one
+   * round trip; install-wide, since the objects span every org the member serves. Callers
+   * gate on {@link supportsServerFeature}(`AGENT_EXISTS_FEATURE`) — an older CP rejects it.
+   */
+  async agentsExist(agentIds: string[]): Promise<AgentExistsOk> {
+    if ((this.state !== 'READY' && this.state !== 'DRAINING') || !this.transport) {
+      throw new WireError('INTERNAL', `control plane unreachable (client ${this.state})`, true)
+    }
+    const rep = await this.request('agent/exists', { agentIds })
+    if (rep.type !== 'agent/exists/ok') {
+      throw new WireError('INTERNAL', `expected agent/exists/ok, got ${rep.type}`, false)
+    }
+    return rep.payload as AgentExistsOk
+  }
+
   /** How this connection is tenanted: `connection` = one org (an API-key daemon),
    *  `frame` = install-wide, every frame carries its own org. Duty leases exist
    *  only on the latter. */

packages/daemon/src/daemon.ts

@@ -281,6 +281,7 @@ import {
 import { resolveRuntimeCatalog, type ResolvedRuntimeCatalog } from './runtimes/registry.js'
 import { installedRuntimeCatalog, installedRuntimes, resolveCommandPath } from './runtimes/probe.js'
 import { K8S_ORG_ID_ENV, startK8sRuntimePlane, type K8sRuntimePlane } from './k8s/runtime-plane.js'
+import { ORPHAN_SWEEP_LEASE } from './k8s/orphan-reconciler.js'
 import {
   setSandboxWorkspaceMode,
   setWorkspaceGitRunnerResolver,
@@ -356,6 +357,8 @@ import {
   originKindOf,
   SessionPurgeReason,
   RD_ACK_NOT_HOLDER,
+  AGENT_EXISTS_FEATURE,
+  AGENT_EXISTS_MAX,
   EventSession as EventSessionSchema
 } from '@agentconnect.md/protocol'
 import { isNoResponseBody, isNoResponsePrefix } from './session/no-response.js'
@@ -2976,6 +2979,12 @@ export class Daemon {
           tunnelsFor: (agentId) =>
             this.agents.get(agentId)?.workspace.gitCredential === 'github-app' ? ['gitcred'] : [],
           tunnelSocketPath: (tunnel) => (tunnel === 'gitcred' ? gitcredSocketPath(root) : undefined),
+          // The orphan reconciler's two install-wide seams: the sweep lease lives in the store every
+          // member shares, and existence is the control plane's to answer — one batched read per sweep.
+          orphans: {
+            acquireLease: (ttlMs, now) => this.dataPlane!.store.acquireSweepLease(ORPHAN_SWEEP_LEASE, ttlMs, now),
+            liveAgents: (agentIds) => this.liveAgentsFor(agentIds)
+          },
           log: {
             info: (message) => this.log.info(message),
             warn: (message) => this.log.warn(message),
@@ -20080,6 +20089,22 @@ export class Daemon {
       )
   }
 
+  /** Cluster only: which of these agents the control plane still knows — the orphan reconciler's one read. */
+  // Throws rather than guesses when the CP cannot be asked, so the sweep skips instead of collecting.
+  private async liveAgentsFor(agentIds: string[]): Promise<Set<string>> {
+    const client = this.cpClient
+    if (!client) throw new Error('control plane is not connected')
+    if (!client.supportsServerFeature(AGENT_EXISTS_FEATURE)) {
+      throw new Error('control plane does not answer agent existence queries yet')
+    }
+    const live = new Set<string>()
+    for (let at = 0; at < agentIds.length; at += AGENT_EXISTS_MAX) {
+      const reply = await client.agentsExist(agentIds.slice(at, at + AGENT_EXISTS_MAX))
+      for (const id of reply.existing) live.add(id)
+    }
+    return live
+  }
+
   /** Cluster only: the sandbox half of "no longer served here"; the claim and volume stay. */
   private releaseClusterSandbox(agentId: string): void {
     this.k8sPlane?.releaseAgent(agentId)
@@ -20093,7 +20118,7 @@ export class Daemon {
    *
    * Best effort by construction: the durable local removal has already succeeded, and failing the
    * lifecycle ACK over a leaked claim would leave the CP and this daemon disagreeing about whether
-   * the agent exists. A failure is therefore reported with the command that finishes the job.
+   * the agent exists. One delete, logged on failure; the orphan reconciler collects what is left.
    */
   private async discardClusterSandbox(agentId: string): Promise<void> {
     const plane = this.k8sPlane
@@ -20102,8 +20127,8 @@ export class Daemon {
       await plane.discardAgent(agentId)
     } catch (err) {
       this.log.warn(
-        `cluster: could not delete the sandbox for removed agent "${agentId}" — its pod and workspace volume ` +
-          `are still allocated (${formatErr(err)}); delete sandboxclaim "${plane.driver.claimName(agentId)}" to reclaim them`
+        `cluster: could not delete the sandbox for removed agent "${agentId}" (${formatErr(err)}) — ` +
+          `sandboxclaim "${plane.driver.claimName(agentId)}" is left for the orphan reconciler`
       )
     }
   }