diff --git a/docs/designs/cluster-spawn-and-shim.md b/docs/designs/cluster-spawn-and-shim.md index e47cc156b..473622253 100644 --- a/docs/designs/cluster-spawn-and-shim.md +++ b/docs/designs/cluster-spawn-and-shim.md @@ -72,10 +72,11 @@ The daemon pool and agent sandboxes are separate namespaces. The runtime plane r `AC_K8S_SANDBOX_NAMESPACE` and uses it for every `SandboxClaim` and `Sandbox` request; it never defaults to the namespace mounted into the daemon Pod's ServiceAccount. -Runtime probes use a member-hashed claim name plus an expiry annotation. A label-filtered daemon -sweep deletes expired claims, bounding resources left by a crash or failed teardown without reading -daemon-local storage or touching ordinary agent claims. UID/resourceVersion delete preconditions -prevent a stale sweep from deleting a same-name claim recreated by a container restart. +Runtime probes use a member-hashed claim name plus an expiry annotation. The pool's orphan +reconciler (k8s-daemon-pool.md §4) collects expired probe claims by that window, bounding resources +left by a crash or failed teardown without reading daemon-local storage or touching ordinary agent +claims. UID/resourceVersion delete preconditions prevent a stale sweep from deleting a same-name +claim recreated by a container restart. ## 3. Binding: proving which pod accepted the connection diff --git a/docs/designs/k8s-daemon-pool.md b/docs/designs/k8s-daemon-pool.md index 5f1b48937..cd5c673cd 100644 --- a/docs/designs/k8s-daemon-pool.md +++ b/docs/designs/k8s-daemon-pool.md @@ -137,9 +137,8 @@ and Sandboxes. Each member also receives its Pod UID through the Downward API as `AC_K8S_MEMBER_ID`; the runtime probe hashes it into `agent-ac-runtime-probe-`, so simultaneous member startup never races on one probe claim. Probe claims carry a dedicated label and a 15-minute expiry; -members periodically delete expired claims, so a missed teardown cannot retain a -Sandbox and volume forever. Each GC delete carries the UID and resourceVersion from -its LIST snapshot, so a same-name replacement cannot be deleted by a stale sweep. +the orphan reconciler (§4) collects an expired one, so a missed teardown cannot +retain a Sandbox and volume forever. **Org-threading is the end state; instantiation is scaffolding.** The wire carries the org, the data plane carries the org, and the process interior @@ -187,6 +186,49 @@ record at pod-name time — and the holder then dials the shim and binds at its term (§7). Cold, warm, and resume paths are already distinguished and metered (`LaunchTimer.observedPath`). **No new wake machinery exists here.** +### Orphan reconciliation + +Teardown is best-effort and a member can die mid-way — a rollout, an OOM, a +node loss — leaving a `SandboxClaim`, a `Sandbox`, or a probe claim that no +process still intends to remove. Rather than one durable obligation per +failure mode, a single **orphan reconciler** +(`packages/daemon/src/k8s/orphan-reconciler.ts`) sweeps the sandbox namespace +on every `--k8s` daemon, by default every 10 minutes with ±25% jitter, and one +member at a time: a named single-holder lease in the shared store +(`LocalStore.acquireSweepLease`, table `sweep_leases`) is taken or renewed at +each sweep and lasts three intervals, so a holder that disappears is replaced +after at most that. + +**What it collects.** It lists the claims and Sandboxes that carry the +install's agent label (`agentconnect.md/agent` on the pod metadata), asks the +control plane in **one batched read per sweep** which of those agent ids still +exist (`agent/exists` → `agent/exists/ok`, install-wide, advertised as the +`agent-exists-v1` server feature), and deletes only what is provably orphaned: + +- a claim whose agent the control plane no longer knows — and has not known for + at least the grace period (default 10 minutes) as observed by the sweeping + member across its own sweeps, on an object at least that old; +- a probe claim past the window the probe stamped on it; +- a Sandbox no claim binds, whose agent the control plane no longer knows, + under the same grace. + +**Safety rules.** An object of a live agent is never touched, a claimless +Sandbox included — deleting a claim deletes the workspace volume and is +irreversible, so a stray of a live agent is reported, not collected. An id the +control plane cannot be asked about, an object without a readable age, and a +sweep whose control-plane read fails all skip. Every delete carries the UID and +resourceVersion from the LIST snapshot, so a same-name replacement created +after the list is never the object deleted. Each sweep logs one summary line +(candidates, orphaned, deleted, skipped-live, skipped-grace, failed). + +**Dry run by default.** The reconciler ships reporting only; deletion is +enabled per deployment with `AC_K8S_ORPHAN_DELETE=true` after an observation +window in which the summary lines show it collecting exactly what an operator +would (`AC_K8S_ORPHAN_SWEEP_INTERVAL_MS` and `AC_K8S_ORPHAN_GRACE_MS` tune the +cadence and grace). It replaced the dedicated probe-claim GC, and agent +removal's sandbox teardown is best-effort because of it: `discardAgent` deletes +the claim once and logs a failure, and the reconciler collects the leftovers. + ## 5. The duty ledger and lease service (D6, D7) **The CP is the ledger.** Members claim, renew, and release duties over the @@ -837,16 +879,17 @@ shrinking every actor's permissions. ## 17. Implementation map -| Piece | Where | -| ---------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------- | -| Frames + member cap | `packages/protocol/src/frames/duty.ts`, `relay-daemon.ts` (`RD_ACK_NOT_HOLDER`) | -| Schema + repo (CAS claim, renew, release, reconcile, agent-home claim) | `packages/control-plane/prisma/schema.prisma`, `src/persistence/repositories/duty-group.repo.ts` | -| Pure group math + reconcile planner | `packages/control-plane/src/orchestrator/dutyGroup.ts` | -| Lease exchange (digest diff, chunking, lanes, grace) | `packages/control-plane/src/orchestrator/dutyLease.ts` | -| Recompute sweep + mutation kicks + placement fence | `packages/control-plane/src/orchestrator/dutyRecompute.ts` | -| WS handlers | `packages/control-plane/src/ws/handlers/{heartbeat,duty-release,duty-claim}.ts` | -| Daemon registry + gate + rendezvous claim | `packages/daemon/src/cp/duty-registry.ts`, `src/daemon.ts` (`transportAgents`, `claimDutyForTrigger`) | -| Relay re-route | `packages/relay/src/relay-ingress-manager.ts` (`sendWithRendezvous`), `relay-browser-connection.ts` | -| Shim dial-in | `packages/daemon/src/shim/{dialer,server}.ts` | -| Pod-bound member identity | `packages/control-plane/src/cluster/daemon-identity.ts` | -| Member readiness (probe sinks) | `packages/daemon/src/readiness.ts`, `src/daemon.ts` (`readinessState`) | +| Piece | Where | +| ---------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------- | +| Frames + member cap | `packages/protocol/src/frames/duty.ts`, `relay-daemon.ts` (`RD_ACK_NOT_HOLDER`) | +| Schema + repo (CAS claim, renew, release, reconcile, agent-home claim) | `packages/control-plane/prisma/schema.prisma`, `src/persistence/repositories/duty-group.repo.ts` | +| Pure group math + reconcile planner | `packages/control-plane/src/orchestrator/dutyGroup.ts` | +| Lease exchange (digest diff, chunking, lanes, grace) | `packages/control-plane/src/orchestrator/dutyLease.ts` | +| Recompute sweep + mutation kicks + placement fence | `packages/control-plane/src/orchestrator/dutyRecompute.ts` | +| WS handlers | `packages/control-plane/src/ws/handlers/{heartbeat,duty-release,duty-claim}.ts` | +| Daemon registry + gate + rendezvous claim | `packages/daemon/src/cp/duty-registry.ts`, `src/daemon.ts` (`transportAgents`, `claimDutyForTrigger`) | +| Relay re-route | `packages/relay/src/relay-ingress-manager.ts` (`sendWithRendezvous`), `relay-browser-connection.ts` | +| Shim dial-in | `packages/daemon/src/shim/{dialer,server}.ts` | +| Pod-bound member identity | `packages/control-plane/src/cluster/daemon-identity.ts` | +| Member readiness (probe sinks) | `packages/daemon/src/readiness.ts`, `src/daemon.ts` (`readinessState`) | +| Orphan reconciler + existence read | `packages/daemon/src/k8s/orphan-reconciler.ts`, `packages/control-plane/src/ws/handlers/agent-exists.ts` | diff --git a/packages/control-plane/src/ws/connection.ts b/packages/control-plane/src/ws/connection.ts index 228a8a696..6569e49dc 100644 --- a/packages/control-plane/src/ws/connection.ts +++ b/packages/control-plane/src/ws/connection.ts @@ -44,7 +44,10 @@ const INSTALL_WIDE_FRAME_TYPES = new Set([ 'duty/revoke', 'duty/release', 'duty/claim', - 'duty/claim/ok' + 'duty/claim/ok', + // Existence query from the pool's orphan reconciler: the ids it asks about span every org. + 'agent/exists', + 'agent/exists/ok' ]) export class DaemonConnection implements ConnChannel { diff --git a/packages/control-plane/src/ws/handlers/agent-exists.test.ts b/packages/control-plane/src/ws/handlers/agent-exists.test.ts new file mode 100644 index 000000000..6ab64a2c7 --- /dev/null +++ b/packages/control-plane/src/ws/handlers/agent-exists.test.ts @@ -0,0 +1,49 @@ +// `agent/exists` — existence only, fenced to the connection's org when it has one. +import { describe, expect, it, vi } from 'vitest' +import type { AnyFrame } from '@agentconnect.md/protocol' +import type { DaemonConnection } from '../connection.js' +import type { DaemonWsDeps } from '../deps.js' +import { handleAgentExists } from './agent-exists.js' + +const LIVE = 'a0a0a0a0-aaaa-4aaa-8aaa-aaaaaaaaaaaa' +const OTHER_ORG = 'b0b0b0b0-bbbb-4bbb-8bbb-bbbbbbbbbbbb' +const GONE = 'c0c0c0c0-cccc-4ccc-8ccc-cccccccccccc' + +function existsFrame(agentIds: string[]): AnyFrame { + return { + v: 1, + id: crypto.randomUUID(), + ts: '2026-08-14T00:00:00.000Z', + type: 'agent/exists', + payload: { agentIds } + } as AnyFrame +} + +function fakeConn(orgId: string | null) { + return { daemonId: 'd', orgId, replyTo: vi.fn(), sendError: vi.fn() } as unknown as DaemonConnection & { + replyTo: ReturnType + } +} + +const listByIds = vi.fn(async (ids: readonly string[]) => + [ + { id: LIVE, orgId: 'org-a' }, + { id: OTHER_ORG, orgId: 'org-b' } + ].filter((agent) => ids.includes(agent.id)) +) +const deps = { agent: { listByIds } } as unknown as DaemonWsDeps + +describe('agent/exists', () => { + it('answers an install-wide member with every asked id that exists, deduplicated', async () => { + const conn = fakeConn(null) + await handleAgentExists(existsFrame([LIVE, LIVE, OTHER_ORG, GONE]), conn, deps) + expect(listByIds).toHaveBeenLastCalledWith([LIVE, OTHER_ORG, GONE]) + expect(conn.replyTo).toHaveBeenCalledWith(expect.anything(), 'agent/exists/ok', { existing: [LIVE, OTHER_ORG] }) + }) + + it('fences an org-scoped connection to its own org', async () => { + const conn = fakeConn('org-a') + await handleAgentExists(existsFrame([LIVE, OTHER_ORG, GONE]), conn, deps) + expect(conn.replyTo).toHaveBeenCalledWith(expect.anything(), 'agent/exists/ok', { existing: [LIVE] }) + }) +}) diff --git a/packages/control-plane/src/ws/handlers/agent-exists.ts b/packages/control-plane/src/ws/handlers/agent-exists.ts new file mode 100644 index 000000000..d6fe2ac89 --- /dev/null +++ b/packages/control-plane/src/ws/handlers/agent-exists.ts @@ -0,0 +1,16 @@ +// `agent/exists` handler — the batch existence read behind the pool's orphan reconciler. +// A member lists sandbox objects in its cluster, reads the agent ids they carry, and asks +// here in one round trip which of those agents still exist. Existence only: an id absent +// from the reply is gone and its objects may be collected; a present id is live and its +// objects are never touched. An org-scoped connection sees only its own org's agents. +import { isFrame } from '@agentconnect.md/protocol' +import { AgentId } from '../../domain/ids.js' +import type { Handler } from './index.js' + +export const handleAgentExists: Handler = async (frame, conn, deps) => { + if (!isFrame('agent/exists')(frame)) return + const asked = [...new Set(frame.payload.agentIds)].map((id) => AgentId(id)) + const agents = await deps.agent.listByIds(asked) + const existing = agents.filter((agent) => conn.orgId === null || agent.orgId === conn.orgId).map((agent) => agent.id) + conn.replyTo(frame, 'agent/exists/ok', { existing }) +} diff --git a/packages/control-plane/src/ws/handlers/index.ts b/packages/control-plane/src/ws/handlers/index.ts index 3210c49d8..99b51e280 100644 --- a/packages/control-plane/src/ws/handlers/index.ts +++ b/packages/control-plane/src/ws/handlers/index.ts @@ -30,6 +30,7 @@ import { handleCronReport } from './cron-report.js' import { handleDutyRelease } from './duty-release.js' import { handleDutyClaim } from './duty-claim.js' import { handleDutyFetch } from './duty-fetch.js' +import { handleAgentExists } from './agent-exists.js' import { handleHookReport } from './hook-report.js' import { handleChannelAgents } from './channel-agents.js' import { handleChildSessionStatus } from './child-session-status.js' @@ -74,6 +75,7 @@ export class FrameRouter { 'duty/release': handleDutyRelease, 'duty/claim': handleDutyClaim, 'duty/fetch': handleDutyFetch, + 'agent/exists': handleAgentExists, 'hook/report': handleHookReport, 'hook/start': handleHookStart, 'github/review-authorize': handleGithubReviewAuthorize, diff --git a/packages/control-plane/src/ws/handlers/register.ts b/packages/control-plane/src/ws/handlers/register.ts index fa50c09e2..2917070a8 100644 --- a/packages/control-plane/src/ws/handlers/register.ts +++ b/packages/control-plane/src/ws/handlers/register.ts @@ -12,6 +12,7 @@ */ import { isFrame, + AGENT_EXISTS_FEATURE, ORGANIZATION_KNOWLEDGE_FEATURE, SESSION_LIVE_TAIL_FEATURE, SESSION_METADATA_ACK_FEATURE, @@ -75,7 +76,8 @@ export const handleRegister: Handler = async (frame, conn, deps) => { SESSION_METADATA_ACK_FEATURE, SESSION_PURGE_FEATURE, SESSION_VISIBILITY_FEATURE, - ORGANIZATION_KNOWLEDGE_FEATURE + ORGANIZATION_KNOWLEDGE_FEATURE, + AGENT_EXISTS_FEATURE ] }) deps.connReg.markReady(conn.daemonId, conn) diff --git a/packages/daemon/src/cp/client.ts b/packages/daemon/src/cp/client.ts index fece2ead3..691daf5b0 100644 --- a/packages/daemon/src/cp/client.ts +++ b/packages/daemon/src/cp/client.ts @@ -14,6 +14,7 @@ import type { DutyRevoke, DutyClaimOk, DutyFetchOk, + AgentExistsOk, FactsRuntimeProfile, FactsMcpServer, UsageReport, @@ -156,7 +157,9 @@ const INSTALL_WIDE_FRAME_TYPES = new Set([ 'duty/revoke', 'duty/release', 'duty/claim', - 'duty/claim/ok' + 'duty/claim/ok', + 'agent/exists', + 'agent/exists/ok' ]) const ACK_TIMEOUT_MS = 5000 @@ -933,6 +936,23 @@ export class CpClient { return rep.payload as DutyFetchOk } + /** + * `agent/exists` (D→C REQ → `agent/exists/ok`) — which of these agents the CP still knows. + * Asked by the cluster orphan reconciler for the ids it read off sandbox objects, in one + * round trip; install-wide, since the objects span every org the member serves. Callers + * gate on {@link supportsServerFeature}(`AGENT_EXISTS_FEATURE`) — an older CP rejects it. + */ + async agentsExist(agentIds: string[]): Promise { + if ((this.state !== 'READY' && this.state !== 'DRAINING') || !this.transport) { + throw new WireError('INTERNAL', `control plane unreachable (client ${this.state})`, true) + } + const rep = await this.request('agent/exists', { agentIds }) + if (rep.type !== 'agent/exists/ok') { + throw new WireError('INTERNAL', `expected agent/exists/ok, got ${rep.type}`, false) + } + return rep.payload as AgentExistsOk + } + /** How this connection is tenanted: `connection` = one org (an API-key daemon), * `frame` = install-wide, every frame carries its own org. Duty leases exist * only on the latter. */ diff --git a/packages/daemon/src/daemon.ts b/packages/daemon/src/daemon.ts index 71a3578d4..edc74e3a1 100644 --- a/packages/daemon/src/daemon.ts +++ b/packages/daemon/src/daemon.ts @@ -281,6 +281,7 @@ import { import { resolveRuntimeCatalog, type ResolvedRuntimeCatalog } from './runtimes/registry.js' import { installedRuntimeCatalog, installedRuntimes, resolveCommandPath } from './runtimes/probe.js' import { K8S_ORG_ID_ENV, startK8sRuntimePlane, type K8sRuntimePlane } from './k8s/runtime-plane.js' +import { ORPHAN_SWEEP_LEASE } from './k8s/orphan-reconciler.js' import { setSandboxWorkspaceMode, setWorkspaceGitRunnerResolver, @@ -356,6 +357,8 @@ import { originKindOf, SessionPurgeReason, RD_ACK_NOT_HOLDER, + AGENT_EXISTS_FEATURE, + AGENT_EXISTS_MAX, EventSession as EventSessionSchema } from '@agentconnect.md/protocol' import { isNoResponseBody, isNoResponsePrefix } from './session/no-response.js' @@ -2976,6 +2979,12 @@ export class Daemon { tunnelsFor: (agentId) => this.agents.get(agentId)?.workspace.gitCredential === 'github-app' ? ['gitcred'] : [], tunnelSocketPath: (tunnel) => (tunnel === 'gitcred' ? gitcredSocketPath(root) : undefined), + // The orphan reconciler's two install-wide seams: the sweep lease lives in the store every + // member shares, and existence is the control plane's to answer — one batched read per sweep. + orphans: { + acquireLease: (ttlMs, now) => this.dataPlane!.store.acquireSweepLease(ORPHAN_SWEEP_LEASE, ttlMs, now), + liveAgents: (agentIds) => this.liveAgentsFor(agentIds) + }, log: { info: (message) => this.log.info(message), warn: (message) => this.log.warn(message), @@ -20080,6 +20089,22 @@ export class Daemon { ) } + /** Cluster only: which of these agents the control plane still knows — the orphan reconciler's one read. */ + // Throws rather than guesses when the CP cannot be asked, so the sweep skips instead of collecting. + private async liveAgentsFor(agentIds: string[]): Promise> { + const client = this.cpClient + if (!client) throw new Error('control plane is not connected') + if (!client.supportsServerFeature(AGENT_EXISTS_FEATURE)) { + throw new Error('control plane does not answer agent existence queries yet') + } + const live = new Set() + for (let at = 0; at < agentIds.length; at += AGENT_EXISTS_MAX) { + const reply = await client.agentsExist(agentIds.slice(at, at + AGENT_EXISTS_MAX)) + for (const id of reply.existing) live.add(id) + } + return live + } + /** Cluster only: the sandbox half of "no longer served here"; the claim and volume stay. */ private releaseClusterSandbox(agentId: string): void { this.k8sPlane?.releaseAgent(agentId) @@ -20093,7 +20118,7 @@ export class Daemon { * * Best effort by construction: the durable local removal has already succeeded, and failing the * lifecycle ACK over a leaked claim would leave the CP and this daemon disagreeing about whether - * the agent exists. A failure is therefore reported with the command that finishes the job. + * the agent exists. One delete, logged on failure; the orphan reconciler collects what is left. */ private async discardClusterSandbox(agentId: string): Promise { const plane = this.k8sPlane @@ -20102,8 +20127,8 @@ export class Daemon { await plane.discardAgent(agentId) } catch (err) { this.log.warn( - `cluster: could not delete the sandbox for removed agent "${agentId}" — its pod and workspace volume ` + - `are still allocated (${formatErr(err)}); delete sandboxclaim "${plane.driver.claimName(agentId)}" to reclaim them` + `cluster: could not delete the sandbox for removed agent "${agentId}" (${formatErr(err)}) — ` + + `sandboxclaim "${plane.driver.claimName(agentId)}" is left for the orphan reconciler` ) } } diff --git a/packages/daemon/src/k8s/orphan-reconciler.ts b/packages/daemon/src/k8s/orphan-reconciler.ts new file mode 100644 index 000000000..6a5cc696b --- /dev/null +++ b/packages/daemon/src/k8s/orphan-reconciler.ts @@ -0,0 +1,278 @@ +import { systemClock, type Clock, type TimerHandle } from '@agentconnect.md/connection' +import { K8sApiError } from '@agentconnect.md/k8s-client' +import { AC_LABEL_AGENT } from './driver.js' +import { probeClaimExpiry } from './probe-claim.js' +import type { Sandbox, SandboxApi, SandboxClaim } from './sandbox-api.js' + +/** + * The pool's orphan reconciler: a periodic sweep that finds sandbox objects nobody will ever + * clean up and removes them, instead of every teardown path carrying its own durable obligation + * to survive a member dying mid-way (k8s-daemon-pool.md §4). + * + * Safety over completeness. Deleting a claim deletes the workspace volume, so a candidate is + * collected only when it is PROVABLY orphaned: the control plane no longer knows its agent, and + * has not for at least the grace period as observed by this member across sweeps; a probe claim + * is past its own window; a Sandbox has no claim and no live agent. An object of a live agent + * is never touched — not even a claimless Sandbox — and an unreadable answer skips the sweep. + * Ships dry-run: it logs and counts until the deployment enables deletion. + */ + +/** Deployment-owned settings, env like the rest of the plane's; absent ⇒ the defaults below. */ +export const ORPHAN_SWEEP_INTERVAL_ENV = 'AC_K8S_ORPHAN_SWEEP_INTERVAL_MS' +export const ORPHAN_GRACE_ENV = 'AC_K8S_ORPHAN_GRACE_MS' +/** Deletion is opt-in: `1`/`true` collects, anything else only reports. */ +export const ORPHAN_DELETE_ENV = 'AC_K8S_ORPHAN_DELETE' +export const DEFAULT_ORPHAN_SWEEP_INTERVAL_MS = 10 * 60_000 +export const DEFAULT_ORPHAN_GRACE_MS = 10 * 60_000 +/** Name of the single-holder lease in the shared store; one member sweeps at a time. */ +export const ORPHAN_SWEEP_LEASE = 'k8s-orphan-sweep' +const UUID = /^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$/i + +export interface OrphanReconcilerSettings { + intervalMs: number + graceMs: number + deleteEnabled: boolean +} + +export function resolveOrphanReconcilerSettings(env: NodeJS.ProcessEnv = process.env): OrphanReconcilerSettings { + const positive = (name: string, fallback: number): number => { + const raw = env[name]?.trim() + if (!raw) return fallback + const value = Number(raw) + if (!Number.isInteger(value) || value <= 0) throw new Error(`${name} is not a positive integer: ${raw}`) + return value + } + const flag = env[ORPHAN_DELETE_ENV]?.trim().toLowerCase() + return { + intervalMs: positive(ORPHAN_SWEEP_INTERVAL_ENV, DEFAULT_ORPHAN_SWEEP_INTERVAL_MS), + graceMs: positive(ORPHAN_GRACE_ENV, DEFAULT_ORPHAN_GRACE_MS), + deleteEnabled: flag === '1' || flag === 'true' + } +} + +/** One sweep's counters, also the shape of its summary log line. */ +export interface OrphanSweepSummary { + candidates: number + orphaned: number + deleted: number + skippedLive: number + skippedGrace: number + failed: number +} + +export interface OrphanReconcilerDeps { + api: Pick + /** Take or renew the install-wide sweep lease for `ttlMs`; false ⇒ another member is sweeping. */ + acquireLease: (ttlMs: number, now: number) => boolean + /** Which of these agents the control plane still knows; a throw skips the sweep. */ + liveAgents: (agentIds: string[]) => Promise> + settings: OrphanReconcilerSettings + clock?: Clock + /** Uniform in [0, 1); spreads the members' timers so a rollout does not line them up. */ + jitter?: () => number + log: { info: (m: string) => void; warn: (m: string) => void; debug?: (m: string) => void } +} + +/** A collectable object: what it is, which agent it belongs to, and how to delete exactly it. */ +interface Candidate { + kind: 'claim' | 'probe-claim' | 'sandbox' + name: string + uid: string + resourceVersion?: string + agentId: string + /** Epoch ms, NaN when the object did not say — an age nobody knows never passes the grace. */ + createdAt: number + /** A probe claim's own window, stamped by the probe that made it. */ + probeExpiresAt?: number +} + +export class OrphanReconciler { + private readonly clock: Clock + private readonly jitter: () => number + private timer?: TimerHandle + private stopped = false + private inFlight?: Promise + /** When this member first saw each object's agent missing, by uid; forgotten once seen live. */ + private readonly missingSince = new Map() + private sandboxListDenied = false + + constructor(private readonly deps: OrphanReconcilerDeps) { + this.clock = deps.clock ?? systemClock + this.jitter = deps.jitter ?? Math.random + } + + /** Arm the periodic sweep; the first one is a jittered interval away, like every later one. */ + start(): void { + this.stopped = false + this.arm() + } + + stop(): void { + this.stopped = true + if (this.timer !== undefined) this.clock.clearTimeout(this.timer) + this.timer = undefined + } + + private arm(): void { + if (this.stopped) return + // ±25% around the interval: members drift apart, and the lease holder still renews well + // inside a lease that lasts three intervals. + const delay = Math.round(this.deps.settings.intervalMs * (0.75 + 0.5 * this.jitter())) + this.timer = this.clock.setTimeout(() => { + this.timer = undefined + void this.sweep().finally(() => this.arm()) + }, delay) + // A sweep is housekeeping: it must never be what keeps a stopping process alive. + ;(this.timer as { unref?: () => void }).unref?.() + } + + /** One sweep. Resolves undefined when this member does not hold the lease or the sweep was skipped. */ + sweep(): Promise { + if (this.inFlight) return this.inFlight + this.inFlight = this.runSweep() + .catch((err: unknown) => { + this.deps.log.warn(`k8s orphans: sweep failed — ${(err as Error).message}`) + return undefined + }) + .finally(() => { + this.inFlight = undefined + }) + return this.inFlight + } + + private async runSweep(): Promise { + const { settings, log } = this.deps + const now = this.clock.now() + if (!this.deps.acquireLease(settings.intervalMs * 3, now)) { + log.debug?.('k8s orphans: another member holds the sweep lease') + return undefined + } + const claims = await this.deps.api.listClaims() + const sandboxes = await this.listSandboxes() + const bound = new Set(claims.map((claim) => claim.status?.sandbox?.name).filter((name) => name !== undefined)) + const candidates: Candidate[] = [] + for (const claim of claims) { + const candidate = candidateOf(claim, 'claim') + if (candidate) candidates.push(candidate) + } + for (const sandbox of sandboxes) { + const name = sandbox.metadata?.name + if (!name || bound.has(name)) continue + const candidate = candidateOf(sandbox, 'sandbox') + if (candidate) candidates.push(candidate) + } + const summary: OrphanSweepSummary = { + candidates: candidates.length, + orphaned: 0, + deleted: 0, + skippedLive: 0, + skippedGrace: 0, + failed: 0 + } + // Probe agents are member-local and never known to the control plane, so they are not asked about. + const askable = [ + ...new Set(candidates.filter((c) => c.kind !== 'probe-claim' && UUID.test(c.agentId)).map((c) => c.agentId)) + ] + const live = askable.length > 0 ? await this.deps.liveAgents(askable) : new Set() + const seen = new Set() + const orphans: Candidate[] = [] + for (const candidate of candidates) { + if (candidate.kind === 'probe-claim') { + // Inside its own window the probe may still be running; an unreadable window is never up. + const expiresAt = candidate.probeExpiresAt ?? Number.NaN + if (!Number.isFinite(expiresAt) || expiresAt > now) summary.skippedGrace += 1 + else orphans.push(candidate) + continue + } + // An id the control plane could not even be asked about is treated as live: never guess. + if (!UUID.test(candidate.agentId) || live.has(candidate.agentId)) { + summary.skippedLive += 1 + continue + } + seen.add(candidate.uid) + const firstMissing = this.missingSince.get(candidate.uid) ?? now + this.missingSince.set(candidate.uid, firstMissing) + // Both clocks must agree: missing across this member's sweeps for the grace, AND old enough + // that no in-flight creation could still be racing the control plane's own write. + const aged = Number.isFinite(candidate.createdAt) && now - candidate.createdAt >= settings.graceMs + if (now - firstMissing < settings.graceMs || !aged) { + summary.skippedGrace += 1 + continue + } + orphans.push(candidate) + } + for (const uid of [...this.missingSince.keys()]) if (!seen.has(uid)) this.missingSince.delete(uid) + summary.orphaned = orphans.length + for (const orphan of orphans) { + if (!settings.deleteEnabled) { + log.info(`k8s orphans: would delete ${orphan.kind} ${orphan.name} (agent ${orphan.agentId}) — dry run`) + continue + } + try { + const current = await this.deleteCurrent(orphan) + if (current) { + summary.deleted += 1 + log.info(`k8s orphans: deleted ${orphan.kind} ${orphan.name} (agent ${orphan.agentId})`) + } else { + log.info(`k8s orphans: ${orphan.kind} ${orphan.name} was replaced since it was listed — left alone`) + } + } catch (err) { + summary.failed += 1 + log.warn(`k8s orphans: deleting ${orphan.kind} ${orphan.name} failed — ${(err as Error).message}`) + } + } + log.info( + `k8s orphans: swept ${summary.candidates} candidates — orphaned=${summary.orphaned} deleted=${summary.deleted} ` + + `skipped-live=${summary.skippedLive} skipped-grace=${summary.skippedGrace} failed=${summary.failed}` + + (settings.deleteEnabled ? '' : ' (dry run)') + ) + return summary + } + + // Listing Sandboxes needs a verb the claim path never did; a Role without it just narrows the sweep to claims. + private async listSandboxes(): Promise { + try { + return await this.deps.api.listSandboxes() + } catch (err) { + if (!(err instanceof K8sApiError) || err.status !== 403) throw err + if (!this.sandboxListDenied) + this.deps.log.warn(`k8s orphans: listing sandboxes is not permitted — sweeping claims only`) + this.sandboxListDenied = true + return [] + } + } + + private deleteCurrent(orphan: Candidate): Promise { + const preconditions = { + uid: orphan.uid, + ...(orphan.resourceVersion ? { resourceVersion: orphan.resourceVersion } : {}) + } + return orphan.kind === 'sandbox' + ? this.deps.api.deleteSandboxIfCurrent(orphan.name, preconditions) + : this.deps.api.deleteClaimIfCurrent(orphan.name, preconditions) + } +} + +/** The install's objects carry the agent label on their pod metadata; anything else is not ours. */ +function candidateOf(object: SandboxClaim | Sandbox, kind: 'claim' | 'sandbox'): Candidate | undefined { + const name = object.metadata?.name + const uid = object.metadata?.uid + if (!name || !uid) return undefined + const labels = + kind === 'claim' + ? (object as SandboxClaim).spec?.additionalPodMetadata?.labels + : (object as Sandbox).spec?.podTemplate?.metadata?.labels + const agentId = labels?.[AC_LABEL_AGENT] + if (!agentId) return undefined + const resourceVersion = object.metadata?.resourceVersion + const probeExpiresAt = kind === 'claim' ? probeClaimExpiry(object as SandboxClaim) : undefined + return { + kind: probeExpiresAt === undefined ? kind : 'probe-claim', + name, + uid, + ...(resourceVersion ? { resourceVersion } : {}), + agentId, + createdAt: Date.parse(object.metadata?.creationTimestamp ?? ''), + ...(probeExpiresAt === undefined ? {} : { probeExpiresAt }) + } +} diff --git a/packages/daemon/src/k8s/probe-claim.ts b/packages/daemon/src/k8s/probe-claim.ts new file mode 100644 index 000000000..62aa1aa7e --- /dev/null +++ b/packages/daemon/src/k8s/probe-claim.ts @@ -0,0 +1,23 @@ +import { createHash } from 'node:crypto' + +/** Reserved prefix for member-scoped runtime probes; the Control Plane never assigns it. */ +export const PROBE_AGENT_ID_PREFIX = 'ac-runtime-probe' +export const PROBE_CLAIM_LABEL = 'agentconnect.md/runtime-probe' +export const PROBE_CLAIM_EXPIRES_ANNOTATION = 'agentconnect.md/runtime-probe-expires-at' +/** Bounds an abandoned probe claim while leaving ample room for cold scheduling and the probe. */ +export const PROBE_CLAIM_TTL_MS = 15 * 60_000 + +/** A deterministic, DNS-safe probe identity unique to one daemon member. */ +export function probeAgentId(memberId: string): string { + const memberHash = createHash('sha256').update(memberId).digest('hex').slice(0, 16) + return `${PROBE_AGENT_ID_PREFIX}-${memberHash}` +} + +/** Whether a claim is a runtime probe's, and if so when its window closes (NaN when unreadable). */ +export function probeClaimExpiry(claim: { + metadata?: { labels?: Record; annotations?: Record } +}): number | undefined { + if (claim.metadata?.labels?.[PROBE_CLAIM_LABEL] !== 'true') return undefined + const raw = claim.metadata.annotations?.[PROBE_CLAIM_EXPIRES_ANNOTATION] + return raw ? Date.parse(raw) : Number.NaN +} diff --git a/packages/daemon/src/k8s/runtime-plane.ts b/packages/daemon/src/k8s/runtime-plane.ts index d7b77ba09..74ae311ec 100644 --- a/packages/daemon/src/k8s/runtime-plane.ts +++ b/packages/daemon/src/k8s/runtime-plane.ts @@ -1,7 +1,8 @@ -import { createHash } from 'node:crypto' import { K8sHttp, loadInClusterConfig } from '@agentconnect.md/k8s-client' import { K8sDriver, PROBE_GRANTS, type LaunchGenerations } from './driver.js' import { SandboxApi } from './sandbox-api.js' +import { OrphanReconciler, resolveOrphanReconcilerSettings, type OrphanReconcilerDeps } from './orphan-reconciler.js' +import { PROBE_CLAIM_EXPIRES_ANNOTATION, PROBE_CLAIM_LABEL, PROBE_CLAIM_TTL_MS, probeAgentId } from './probe-claim.js' import { clusterMetrics } from './cluster-metrics.js' import { ShimDialer } from '../shim/dialer.js' import { ShimGitRunner } from '../shim/git-exec.js' @@ -17,49 +18,8 @@ import type { GitRunner } from '../workspace/git-runner.js' const SILENT = { info: () => {}, warn: () => {} } -/** Reserved prefix for member-scoped runtime probes; the Control Plane never assigns it. */ -export const PROBE_AGENT_ID_PREFIX = 'ac-runtime-probe' -export const PROBE_CLAIM_LABEL = 'agentconnect.md/runtime-probe' -export const PROBE_CLAIM_EXPIRES_ANNOTATION = 'agentconnect.md/runtime-probe-expires-at' /** A probe drives every runtime through `initialize` plus a session, on a possibly cold pod. */ const PROBE_TIMEOUT_MS = 180_000 -/** Bounds an abandoned probe claim while leaving ample room for cold scheduling and the probe. */ -export const PROBE_CLAIM_TTL_MS = 15 * 60_000 -const PROBE_CLAIM_GC_INTERVAL_MS = 5 * 60_000 - -/** A deterministic, DNS-safe probe identity unique to one daemon member. */ -export function probeAgentId(memberId: string): string { - const memberHash = createHash('sha256').update(memberId).digest('hex').slice(0, 16) - return `${PROBE_AGENT_ID_PREFIX}-${memberHash}` -} - -/** Delete only expired probe claims; ordinary agent claims never match. */ -export async function reapExpiredProbeClaims( - api: Pick, - now: number, - log: { warn: (message: string) => void } = SILENT -): Promise { - const claims = await api.listClaims(`${PROBE_CLAIM_LABEL}=true`) - await Promise.all( - claims.map(async (claim) => { - const name = claim.metadata?.name - if (!name || claim.metadata?.labels?.[PROBE_CLAIM_LABEL] !== 'true') return - const uid = claim.metadata.uid - if (!uid) return - const rawExpiry = claim.metadata?.annotations?.[PROBE_CLAIM_EXPIRES_ANNOTATION] - const expiresAt = rawExpiry ? Date.parse(rawExpiry) : Number.NaN - if (!Number.isFinite(expiresAt) || expiresAt > now) return - await api - .deleteClaimIfCurrent(name, { - uid, - ...(claim.metadata.resourceVersion ? { resourceVersion: claim.metadata.resourceVersion } : {}) - }) - .catch((err: unknown) => { - log.warn(`k8s: expired probe claim ${name} teardown failed: ${(err as Error).message}`) - }) - }) - ) -} /** How long a pod that is UP may go without a shim channel before the launch counts as lost. */ const DEFAULT_REBIND_GRACE_MS = 20_000 @@ -120,6 +80,8 @@ export interface K8sRuntimePlaneOptions { /** How long a pod that is up may go without a shim channel before the launch counts as lost. * Injected so a test can cross the window in milliseconds rather than waiting out the default. */ rebindGraceMs?: number + /** The orphan reconciler's install-wide seams (sweep lease, control-plane existence read); absent ⇒ no sweep. */ + orphans?: Pick log?: { info: (m: string) => void; warn: (m: string) => void; debug?: (m: string) => void } } @@ -311,20 +273,20 @@ export async function startK8sRuntimePlane(options: K8sRuntimePlaneOptions): Pro // is a wait for an event, not a window being spent, and the pod's arrival restarts the window. const POD_UP_POLL_MS = Math.max(1, Math.min(2_000, Math.floor(REBIND_GRACE_MS / 4))) const lossWatches = new Map() - let stopped = false - let probeGcTimer: NodeJS.Timeout | undefined let probeInFlight: Promise | undefined - async function runProbeClaimGc(): Promise { - await reapExpiredProbeClaims(api, Date.now(), options.log ?? SILENT).catch((err: unknown) => - options.log?.warn(`k8s: probe claim sweep failed: ${(err as Error).message}`) - ) - if (stopped) return - probeGcTimer = setTimeout(() => void runProbeClaimGc(), PROBE_CLAIM_GC_INTERVAL_MS) - probeGcTimer.unref?.() - } - - void runProbeClaimGc() + // Collects what a member that died mid-teardown left behind — an expired probe claim included + // (k8s-daemon-pool.md §4); the seams it needs are the daemon's, so a plane assembled without + // them (a test) simply never sweeps. + const reconciler = options.orphans + ? new OrphanReconciler({ + api, + ...options.orphans, + settings: resolveOrphanReconcilerSettings(options.env ?? process.env), + log: options.log ?? SILENT + }) + : undefined + reconciler?.start() /** * Start the grace window for a channel that dropped, measured from the right event. @@ -420,6 +382,7 @@ export async function startK8sRuntimePlane(options: K8sRuntimePlaneOptions): Pro return K8sRuntimeTableSchema.parse(raw) }) } finally { + // Best-effort: a claim left behind here expires and the orphan reconciler collects it. await driver.removeAgent(runtimeProbeAgentId).catch((err: unknown) => { options.log?.warn(`k8s: probe sandbox teardown failed: ${(err as Error).message}`) }) @@ -477,9 +440,7 @@ export async function startK8sRuntimePlane(options: K8sRuntimePlaneOptions): Pro await driver.removeAgent(agentId) }, stop: async () => { - stopped = true - if (probeGcTimer) clearTimeout(probeGcTimer) - probeGcTimer = undefined + reconciler?.stop() for (const agentId of [...lossWatches.keys()]) cancelLossCheck(agentId) for (const { proxy } of proxies.values()) proxy.stop('daemon is shutting down') proxies.clear() diff --git a/packages/daemon/src/k8s/sandbox-api.ts b/packages/daemon/src/k8s/sandbox-api.ts index 46207cae5..a76be2cee 100644 --- a/packages/daemon/src/k8s/sandbox-api.ts +++ b/packages/daemon/src/k8s/sandbox-api.ts @@ -12,6 +12,7 @@ interface SandboxContainer { } interface SandboxPodTemplate { + metadata?: { labels?: Record } spec?: { containers?: SandboxContainer[] } } @@ -164,10 +165,17 @@ export class SandboxApi { /** Delete only the listed claim incarnation; false means the name now belongs to a replacement. */ async deleteClaimIfCurrent(name: string, preconditions: { uid: string; resourceVersion?: string }): Promise { + return this.deleteIfCurrent(`${this.claims()}/${name}`, preconditions) + } + + private async deleteIfCurrent( + path: string, + preconditions: { uid: string; resourceVersion?: string } + ): Promise { try { await this.http.json({ method: 'DELETE', - path: `${this.claims()}/${name}`, + path, body: { apiVersion: 'v1', kind: 'DeleteOptions', @@ -185,6 +193,23 @@ export class SandboxApi { } } + async listSandboxes(labelSelector?: string): Promise { + const list = await this.http.json>({ + method: 'GET', + path: this.sandboxes(), + query: { ...(labelSelector ? { labelSelector } : {}) } + }) + return list.items ?? [] + } + + /** Delete only the listed Sandbox incarnation; false means the name now belongs to a replacement. */ + async deleteSandboxIfCurrent( + name: string, + preconditions: { uid: string; resourceVersion?: string } + ): Promise { + return this.deleteIfCurrent(`${this.sandboxes()}/${name}`, preconditions) + } + /** `signal` bounds the read: a caller on a deadline must not be pinned by an API server that * accepted the connection and never answered. Same seam a watch aborts through. */ getSandbox(name: string, opts: { signal?: AbortSignal } = {}): Promise { diff --git a/packages/daemon/src/store/local-store.ts b/packages/daemon/src/store/local-store.ts index 3f91e4c42..af22b401d 100644 --- a/packages/daemon/src/store/local-store.ts +++ b/packages/daemon/src/store/local-store.ts @@ -1172,6 +1172,13 @@ export class LocalStore { agentId TEXT PRIMARY KEY, generation INTEGER NOT NULL ); + -- Single-holder leases for install-wide periodic sweeps (the cluster orphan reconciler): the + -- holder renews by re-acquiring, and a lapsed row is any member's to take. + CREATE TABLE IF NOT EXISTS sweep_leases ( + name TEXT PRIMARY KEY, + ownerId TEXT NOT NULL, + expiresAt INTEGER NOT NULL + ); `) // Stamped only once the CREATE block above has actually emitted that schema, so // a store that failed halfway through creation is not left claiming to be current. @@ -4816,6 +4823,20 @@ export class LocalStore { return Number(row.generation) } + /** Take or renew the named single-holder lease; false means another member holds it unexpired. */ + // One atomic upsert, so two members cannot both win; a store nobody shares always holds. + acquireSweepLease(name: string, ttlMs: number, now: number): boolean { + if (!this.shared) return true + const result = this.db + .prepare( + `INSERT INTO sweep_leases (name, ownerId, expiresAt) VALUES (@name, @ownerId, @expiresAt) + ON CONFLICT(name) DO UPDATE SET ownerId = excluded.ownerId, expiresAt = excluded.expiresAt + WHERE sweep_leases.ownerId = excluded.ownerId OR sweep_leases.expiresAt <= @now` + ) + .run({ name, ownerId: this.ownerId ?? '', expiresAt: now + ttlMs, now }) + return Number(result.changes) === 1 + } + close(): void { this.db.close() } diff --git a/packages/daemon/test/k8s-orphan-reconciler.test.ts b/packages/daemon/test/k8s-orphan-reconciler.test.ts new file mode 100644 index 000000000..3c6b6604b --- /dev/null +++ b/packages/daemon/test/k8s-orphan-reconciler.test.ts @@ -0,0 +1,269 @@ +import { afterEach, describe, expect, it } from 'vitest' +import { FakeClock } from '@agentconnect.md/connection' +import { K8sHttp } from '@agentconnect.md/k8s-client' +import { closeFakeApiServers, fakeApiServer } from '@agentconnect.md/k8s-client/testing' +import { + DEFAULT_ORPHAN_GRACE_MS, + DEFAULT_ORPHAN_SWEEP_INTERVAL_MS, + ORPHAN_DELETE_ENV, + ORPHAN_GRACE_ENV, + ORPHAN_SWEEP_INTERVAL_ENV, + OrphanReconciler, + resolveOrphanReconcilerSettings, + type OrphanReconcilerDeps, + type OrphanSweepSummary +} from '../src/k8s/orphan-reconciler.js' +import { AC_LABEL_AGENT, AC_LABEL_ORG } from '../src/k8s/driver.js' +import { PROBE_CLAIM_EXPIRES_ANNOTATION, PROBE_CLAIM_LABEL, probeAgentId } from '../src/k8s/probe-claim.js' +import { SandboxApi, type Sandbox, type SandboxClaim } from '../src/k8s/sandbox-api.js' + +/** + * The orphan reconciler against a fake API server and a fake control-plane answer. The rules + * under test are the safety rules: only a provably orphaned object goes, a live agent's never + * does, grace is measured on this member's own sweeps as well as on the object's age, and the + * default is to report rather than delete. + */ + +afterEach(closeFakeApiServers) + +const LIVE = '11111111-1111-4111-8111-111111111111' +const GONE = '22222222-2222-4222-8222-222222222222' +const JUST_GONE = '33333333-3333-4333-8333-333333333333' +const T0 = Date.parse('2026-08-14T10:00:00.000Z') +const HOUR = 60 * 60_000 +const GRACE = 10 * 60_000 + +function claim( + agentId: string, + opts: { createdAt?: number; sandbox?: string; probeExpiresAt?: number } = {} +): SandboxClaim { + const name = `agent-${agentId}` + return { + metadata: { + name, + uid: `uid-${name}`, + resourceVersion: `rv-${name}`, + creationTimestamp: new Date(opts.createdAt ?? T0 - HOUR).toISOString(), + ...(opts.probeExpiresAt === undefined + ? {} + : { + labels: { [PROBE_CLAIM_LABEL]: 'true' }, + annotations: { [PROBE_CLAIM_EXPIRES_ANNOTATION]: new Date(opts.probeExpiresAt).toISOString() } + }) + }, + spec: { + warmPoolRef: { name: 'pool' }, + additionalPodMetadata: { labels: { [AC_LABEL_ORG]: 'org-1', [AC_LABEL_AGENT]: agentId } } + }, + ...(opts.sandbox ? { status: { sandbox: { name: opts.sandbox } } } : {}) + } +} + +function sandbox(name: string, agentId?: string, createdAt = T0 - HOUR): Sandbox { + return { + metadata: { + name, + uid: `uid-${name}`, + resourceVersion: `rv-${name}`, + creationTimestamp: new Date(createdAt).toISOString() + }, + spec: { + operatingMode: 'Running', + ...(agentId ? { podTemplate: { metadata: { labels: { [AC_LABEL_AGENT]: agentId } } } } : {}) + } + } +} + +/** A cluster holding `claims` and `sandboxes`, recording every delete with its preconditions. */ +async function cluster(claims: SandboxClaim[], sandboxes: Sandbox[], opts: { sandboxList?: number } = {}) { + const deletes: Array<{ path: string; preconditions: unknown }> = [] + const { config } = await fakeApiServer(({ method, url, body }) => { + if (method === 'DELETE') { + deletes.push({ path: url.pathname, preconditions: JSON.parse(body).preconditions }) + return { json: {} } + } + if (url.pathname.endsWith('/sandboxclaims')) return { json: { items: claims } } + if (url.pathname.endsWith('/sandboxes')) { + if (opts.sandboxList) return { status: opts.sandboxList, json: { kind: 'Status', reason: 'Forbidden' } } + return { json: { items: sandboxes } } + } + return { status: 404, json: { kind: 'Status', reason: 'NotFound' } } + }) + return { api: new SandboxApi(new K8sHttp(config), 'agent-sandboxes'), deletes } +} + +function reconciler(over: Partial & { api: OrphanReconcilerDeps['api'] }) { + const clock = new FakeClock(T0) + const asked: string[][] = [] + const infos: string[] = [] + const warns: string[] = [] + const it = new OrphanReconciler({ + acquireLease: () => true, + liveAgents: async (ids) => { + asked.push(ids) + return new Set(ids.filter((id) => id === LIVE)) + }, + settings: { intervalMs: DEFAULT_ORPHAN_SWEEP_INTERVAL_MS, graceMs: GRACE, deleteEnabled: true }, + clock, + jitter: () => 0.5, + log: { info: (m) => infos.push(m), warn: (m) => warns.push(m), debug: () => {} }, + ...over + }) + return { it, clock, asked, infos, warns } +} + +/** Two sweeps a grace apart: the shape every "past grace" case needs. */ +async function twoSweeps(r: ReturnType): Promise { + await r.it.sweep() + r.clock.advance(GRACE) + return r.it.sweep() +} + +describe('orphan reconciler settings', () => { + it('defaults to a ten-minute jittered sweep, a ten-minute grace, and dry run', () => { + expect(resolveOrphanReconcilerSettings({})).toEqual({ + intervalMs: DEFAULT_ORPHAN_SWEEP_INTERVAL_MS, + graceMs: DEFAULT_ORPHAN_GRACE_MS, + deleteEnabled: false + }) + expect( + resolveOrphanReconcilerSettings({ + [ORPHAN_SWEEP_INTERVAL_ENV]: '60000', + [ORPHAN_GRACE_ENV]: '5000', + [ORPHAN_DELETE_ENV]: 'true' + }) + ).toEqual({ intervalMs: 60_000, graceMs: 5_000, deleteEnabled: true }) + expect(() => resolveOrphanReconcilerSettings({ [ORPHAN_GRACE_ENV]: '-1' })).toThrow(ORPHAN_GRACE_ENV) + }) +}) + +describe('orphan reconciler', () => { + it('deletes a claim whose agent the control plane has forgotten, once past grace on both clocks', async () => { + const { api, deletes } = await cluster( + [claim(GONE, { sandbox: 'sb-gone' }), claim(LIVE, { sandbox: 'sb-live' })], + [] + ) + const r = reconciler({ api }) + // First sight of the missing agent starts the grace; nothing goes yet. + expect(await r.it.sweep()).toMatchObject({ candidates: 2, orphaned: 0, skippedLive: 1, skippedGrace: 1 }) + expect(deletes).toEqual([]) + r.clock.advance(GRACE) + expect(await r.it.sweep()).toMatchObject({ candidates: 2, orphaned: 1, deleted: 1, skippedLive: 1, failed: 0 }) + // Exactly the incarnation that was listed, never a same-name replacement. + expect(deletes).toEqual([ + { + path: `/apis/extensions.agents.x-k8s.io/v1beta1/namespaces/agent-sandboxes/sandboxclaims/agent-${GONE}`, + preconditions: { uid: `uid-agent-${GONE}`, resourceVersion: `rv-agent-${GONE}` } + } + ]) + // One existence read per sweep, covering every agent-bearing candidate at once. + expect(r.asked).toEqual([ + [GONE, LIVE], + [GONE, LIVE] + ]) + expect(r.infos.at(-1)).toContain('orphaned=1 deleted=1 skipped-live=1 skipped-grace=0 failed=0') + }) + + it('never touches an object of a live agent, claimless Sandbox included', async () => { + const { api, deletes } = await cluster([claim(LIVE, { sandbox: 'sb-live' })], [sandbox('sb-stray', LIVE)]) + const r = reconciler({ api }) + expect(await twoSweeps(r)).toMatchObject({ candidates: 2, orphaned: 0, deleted: 0, skippedLive: 2 }) + expect(deletes).toEqual([]) + }) + + it('waits out the grace for an agent that only just went missing, even on an old object', async () => { + const { api, deletes } = await cluster([claim(JUST_GONE)], []) + const r = reconciler({ api }) + await r.it.sweep() + r.clock.advance(GRACE - 1) + expect(await r.it.sweep()).toMatchObject({ orphaned: 0, skippedGrace: 1 }) + expect(deletes).toEqual([]) + r.clock.advance(1) + expect(await r.it.sweep()).toMatchObject({ orphaned: 1, deleted: 1 }) + }) + + it('never collects an object whose age it cannot read', async () => { + const undated = claim(GONE) + delete undated.metadata?.creationTimestamp + const { api, deletes } = await cluster([undated], []) + const r = reconciler({ api }) + expect(await twoSweeps(r)).toMatchObject({ orphaned: 0, skippedGrace: 1 }) + expect(deletes).toEqual([]) + }) + + it('only reports in dry run, which is the default', async () => { + const { api, deletes } = await cluster([claim(GONE)], [sandbox('sb-orphan', GONE)]) + const r = reconciler({ api, settings: { intervalMs: 60_000, graceMs: GRACE, deleteEnabled: false } }) + expect(await twoSweeps(r)).toMatchObject({ candidates: 2, orphaned: 2, deleted: 0 }) + expect(deletes).toEqual([]) + expect(r.infos.filter((m) => m.includes('would delete'))).toHaveLength(2) + expect(r.infos.at(-1)).toContain('(dry run)') + }) + + it('collects a claimless Sandbox of a gone agent, and leaves bound and unlabelled ones alone', async () => { + const { api, deletes } = await cluster( + [claim(GONE, { sandbox: 'sb-bound' })], + [sandbox('sb-bound', GONE), sandbox('sb-orphan', GONE), sandbox('warm-spare')] + ) + const r = reconciler({ api }) + // Its claim will go, and the bound Sandbox with it through the claim; only the stray is a Sandbox delete. + expect(await twoSweeps(r)).toMatchObject({ candidates: 2, orphaned: 2, deleted: 2 }) + expect(deletes.map((d) => d.path)).toEqual([ + `/apis/extensions.agents.x-k8s.io/v1beta1/namespaces/agent-sandboxes/sandboxclaims/agent-${GONE}`, + '/apis/agents.x-k8s.io/v1beta1/namespaces/agent-sandboxes/sandboxes/sb-orphan' + ]) + }) + + it('collects expired probe claims by their own window and asks the control plane about none of them', async () => { + const expired = probeAgentId('member-old') + const running = probeAgentId('member-b') + const { api, deletes } = await cluster( + [claim(expired, { probeExpiresAt: T0 - 1 }), claim(running, { probeExpiresAt: T0 + HOUR })], + [] + ) + const r = reconciler({ api }) + expect(await r.it.sweep()).toMatchObject({ candidates: 2, orphaned: 1, deleted: 1, skippedGrace: 1 }) + expect(deletes.map((d) => d.path)).toEqual([ + `/apis/extensions.agents.x-k8s.io/v1beta1/namespaces/agent-sandboxes/sandboxclaims/agent-${expired}` + ]) + expect(r.asked).toEqual([]) + }) + + it('sweeps only while holding the lease, and skips entirely when the control plane cannot answer', async () => { + const { api, deletes } = await cluster([claim(GONE)], []) + const bystander = reconciler({ api, acquireLease: () => false }) + expect(await twoSweeps(bystander)).toBeUndefined() + expect(bystander.asked).toEqual([]) + const unanswered = reconciler({ + api, + liveAgents: async () => { + throw new Error('control plane is not connected') + } + }) + expect(await twoSweeps(unanswered)).toBeUndefined() + expect(unanswered.warns.at(-1)).toContain('sweep failed') + expect(deletes).toEqual([]) + }) + + it('narrows to claims when the Role does not allow listing Sandboxes', async () => { + const { api, deletes } = await cluster([claim(GONE)], [], { sandboxList: 403 }) + const r = reconciler({ api }) + expect(await twoSweeps(r)).toMatchObject({ candidates: 1, orphaned: 1, deleted: 1 }) + expect(deletes).toHaveLength(1) + expect(r.warns.filter((m) => m.includes('not permitted'))).toHaveLength(1) + }) + + it('runs on a jittered interval and stops cleanly', async () => { + const { api } = await cluster([], []) + const r = reconciler({ api }) + r.it.start() + expect(r.clock.pending).toBe(1) + r.clock.advance(DEFAULT_ORPHAN_SWEEP_INTERVAL_MS) + await new Promise((resolve) => setTimeout(resolve, 20)) + expect(r.infos.filter((m) => m.includes('swept 0 candidates'))).toHaveLength(1) + // Re-armed after the sweep, then disarmed by stop. + expect(r.clock.pending).toBe(1) + r.it.stop() + expect(r.clock.pending).toBe(0) + }) +}) diff --git a/packages/daemon/test/k8s-runtime-plane.test.ts b/packages/daemon/test/k8s-runtime-plane.test.ts index 1ced1914f..9de1cbc20 100644 --- a/packages/daemon/test/k8s-runtime-plane.test.ts +++ b/packages/daemon/test/k8s-runtime-plane.test.ts @@ -1,14 +1,7 @@ import { afterEach, describe, expect, it, vi } from 'vitest' import { Backoff, FakeClock } from '@agentconnect.md/connection' -import { - PROBE_CLAIM_EXPIRES_ANNOTATION, - PROBE_CLAIM_LABEL, - k8sPlaneSettings, - probeAgentId, - reapExpiredProbeClaims, - startK8sRuntimePlane, - type K8sRuntimePlane -} from '../src/k8s/runtime-plane.js' +import { k8sPlaneSettings, startK8sRuntimePlane, type K8sRuntimePlane } from '../src/k8s/runtime-plane.js' +import { PROBE_CLAIM_EXPIRES_ANNOTATION, PROBE_CLAIM_LABEL, probeAgentId } from '../src/k8s/probe-claim.js' import { ShimClient, type ShimTransport } from '../src/shim/client.js' import { ShimServer } from '../src/shim/server.js' import { K8sApiError } from '@agentconnect.md/k8s-client' @@ -184,43 +177,6 @@ describe('k8s plane settings', () => { expect(probeAgentId('member-a')).toMatch(/^ac-runtime-probe-[a-f0-9]{16}$/) expect(probeAgentId('member-a')).not.toBe(probeAgentId('member-b')) }) - - it('reaps only expired probe claims from previous members', async () => { - const deleted: string[] = [] - const current = `agent-${probeAgentId('member-a')}` - const expired = `agent-${probeAgentId('member-old')}` - const live = `agent-${probeAgentId('member-b')}` - const ordinary = 'agent-customer' - const expiry = (name: string, at: string, probe = true): SandboxClaim => ({ - metadata: { - name, - uid: `uid-${name}`, - resourceVersion: `rv-${name}`, - labels: probe ? { [PROBE_CLAIM_LABEL]: 'true' } : {}, - annotations: { [PROBE_CLAIM_EXPIRES_ANNOTATION]: at } - } - }) - await reapExpiredProbeClaims( - { - listClaims: async (selector?: string) => { - expect(selector).toBe(`${PROBE_CLAIM_LABEL}=true`) - return [ - expiry(current, '2026-08-14T11:00:00.000Z'), - expiry(expired, '2026-08-14T09:00:00.000Z'), - expiry(live, '2026-08-14T11:00:00.000Z'), - expiry(ordinary, '2026-08-14T09:00:00.000Z', false) - ] - }, - deleteClaimIfCurrent: async (name: string, preconditions) => { - expect(preconditions).toEqual({ uid: `uid-${name}`, resourceVersion: `rv-${name}` }) - deleted.push(name) - return true - } - }, - Date.parse('2026-08-14T10:00:00.000Z') - ) - expect(deleted).toEqual([expired]) - }) }) describe('k8s runtime plane assembly', () => { diff --git a/packages/daemon/test/local-store.test.ts b/packages/daemon/test/local-store.test.ts index 434f72ef8..f3178e94d 100644 --- a/packages/daemon/test/local-store.test.ts +++ b/packages/daemon/test/local-store.test.ts @@ -2108,3 +2108,22 @@ describe('sandbox generations', () => { reopened.close() }) }) + +describe('sweep leases', () => { + it('lets one member hold and renew, hands over only once the lease lapses, and always holds locally', () => { + const [first, second] = sharedMembers('member-1', 'member-2') + expect(first.acquireSweepLease('orphans', 1_000, 10_000)).toBe(true) + expect(second.acquireSweepLease('orphans', 1_000, 10_500)).toBe(false) + // The holder renews past what the contender saw, so the contender keeps losing. + expect(first.acquireSweepLease('orphans', 1_000, 10_900)).toBe(true) + expect(second.acquireSweepLease('orphans', 1_000, 11_500)).toBe(false) + expect(second.acquireSweepLease('orphans', 1_000, 11_900)).toBe(true) + expect(first.acquireSweepLease('orphans', 1_000, 12_000)).toBe(false) + // Leases are named: another sweep is unrelated. + expect(first.acquireSweepLease('other', 1_000, 12_000)).toBe(true) + const local = store() + expect(local.acquireSweepLease('orphans', 1_000, 0)).toBe(true) + expect(local.acquireSweepLease('orphans', 1_000, 0)).toBe(true) + local.close() + }) +}) diff --git a/packages/daemon/test/postgres-pool-store.int.test.ts b/packages/daemon/test/postgres-pool-store.int.test.ts index 5811cbd6a..0ebe7e39c 100644 --- a/packages/daemon/test/postgres-pool-store.int.test.ts +++ b/packages/daemon/test/postgres-pool-store.int.test.ts @@ -286,6 +286,14 @@ describe.skipIf(!databaseUrl)('PostgreSQL pool member store', () => { second.store.gcRuntimeCatalog(1, 150) expect(first.store.getRuntimeCatalogMeta(runtimeId)).toBeUndefined() expect(second.store.getRuntimeCatalogMeta(runtimeId)).toMatchObject({ fingerprint: 'fp-2' }) + + // The single-holder sweep lease decides through the same upsert on PostgreSQL as on SQLite. + const lease = `sweep-${suffix}` + expect(first.store.acquireSweepLease(lease, 1_000, 10_000)).toBe(true) + expect(second.store.acquireSweepLease(lease, 1_000, 10_500)).toBe(false) + expect(first.store.acquireSweepLease(lease, 1_000, 10_900)).toBe(true) + expect(second.store.acquireSweepLease(lease, 1_000, 11_950)).toBe(true) + expect(first.store.acquireSweepLease(lease, 1_000, 12_000)).toBe(false) } finally { second.store.gcRuntimeCatalog(Number.MAX_SAFE_INTEGER, Number.MAX_SAFE_INTEGER) await second.close() diff --git a/packages/k8s-client/src/watch.ts b/packages/k8s-client/src/watch.ts index 33abe4b40..e3ebc75b7 100644 --- a/packages/k8s-client/src/watch.ts +++ b/packages/k8s-client/src/watch.ts @@ -7,7 +7,13 @@ export interface WatchMetrics { } export interface K8sObject { - metadata?: { name?: string; uid?: string; resourceVersion?: string; annotations?: Record } + metadata?: { + name?: string + uid?: string + resourceVersion?: string + creationTimestamp?: string + annotations?: Record + } } export interface K8sList { diff --git a/packages/protocol/src/consts.ts b/packages/protocol/src/consts.ts index 3fdc10ea8..420165ec7 100644 --- a/packages/protocol/src/consts.ts +++ b/packages/protocol/src/consts.ts @@ -245,3 +245,8 @@ export const CP_IDENTITY_TOKEN_PATH = '/var/run/ac-cp-identity/token' export const CP_URL_ENV = 'AC_CP_URL' /** A pool member's rollout generation (its pod-template hash), reported on register (frames/register.ts). */ export const POD_TEMPLATE_HASH_ENV = 'AC_POD_TEMPLATE_HASH' + +/** CP answers the `agent/exists` batch existence query the pool's orphan reconciler + * asks before collecting a leaked sandbox object; a daemon that does not see it + * skips the sweep rather than emit a frame an older CP rejects as `UNKNOWN_FRAME`. */ +export const AGENT_EXISTS_FEATURE = 'agent-exists-v1' diff --git a/packages/protocol/src/frame.ts b/packages/protocol/src/frame.ts index de822a03c..3091bec94 100644 --- a/packages/protocol/src/frame.ts +++ b/packages/protocol/src/frame.ts @@ -10,6 +10,8 @@ import { AgentStop, AgentUpsert, AgentRemove, + AgentExists, + AgentExistsOk, AgentDetach, AgentActivate, AgentActivity, @@ -219,6 +221,8 @@ export const FRAME_SCHEMAS = { 'agent/stop': AgentStop, 'agent/upsert': AgentUpsert, 'agent/remove': AgentRemove, + 'agent/exists': AgentExists, + 'agent/exists/ok': AgentExistsOk, 'agent/detach': AgentDetach, 'agent/activate': AgentActivate, 'agent/activity': AgentActivity, @@ -478,6 +482,8 @@ export const AnyFrame = z.discriminatedUnion('type', [ frame('agent/stop', FRAME_SCHEMAS['agent/stop']), frame('agent/upsert', FRAME_SCHEMAS['agent/upsert']), frame('agent/remove', FRAME_SCHEMAS['agent/remove']), + frame('agent/exists', FRAME_SCHEMAS['agent/exists']), + frame('agent/exists/ok', FRAME_SCHEMAS['agent/exists/ok']), frame('agent/detach', FRAME_SCHEMAS['agent/detach']), frame('agent/activate', FRAME_SCHEMAS['agent/activate']), frame('agent/activity', FRAME_SCHEMAS['agent/activity']), diff --git a/packages/protocol/src/frames/agent.ts b/packages/protocol/src/frames/agent.ts index b7e678670..e831c9686 100644 --- a/packages/protocol/src/frames/agent.ts +++ b/packages/protocol/src/frames/agent.ts @@ -425,6 +425,26 @@ export const AgentRemove = z.object({ }) export type AgentRemove = z.infer +/** Hard cap on one existence query; a sweep with more candidates chunks. */ +export const AGENT_EXISTS_MAX = 1000 + +/** + * D→C REQ (reply: `agent/exists/ok`) — which of these agents the control plane still knows. + * Install-wide: a pool member's orphan reconciler reads agent ids off cluster objects that + * span every org it serves and asks in one round trip. Existence only, no spec: the answer + * decides whether a leaked sandbox object may be collected, nothing more. + */ +export const AgentExists = z.object({ + agentIds: z.array(z.string().uuid()).min(1).max(AGENT_EXISTS_MAX) +}) +export type AgentExists = z.infer + +/** C→D REP to `agent/exists`: the subset of the asked ids that exist. An id absent here is gone. */ +export const AgentExistsOk = z.object({ + existing: z.array(z.string().uuid()).max(AGENT_EXISTS_MAX) +}) +export type AgentExistsOk = z.infer + /** * Safe move lifecycle (C→D REQ → generic `ack`). `agent/detach` fences the * source or stages the target and archives any daemon-local root; diff --git a/packages/protocol/src/index.ts b/packages/protocol/src/index.ts index 0c594ae18..370e1bf95 100644 --- a/packages/protocol/src/index.ts +++ b/packages/protocol/src/index.ts @@ -30,6 +30,7 @@ export { ORGANIZATION_SUGGESTION_REVIEW_FEATURE, RESERVED_RESTART_CODE, K8S_SUPERVISOR, + AGENT_EXISTS_FEATURE, SESSION_LIVE_TAIL_FEATURE, SESSION_METADATA_ACK_FEATURE, SESSION_PURGE_FEATURE,