You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Implementation plan and tracking issue for the k8s daemon pool: install-wide multi-org daemon members, the duty ledger, and the retirement of the per-org execution envelope. The mechanism design is docs/designs/k8s-daemon-pool.md (#954). The plan is restructured around a walking skeleton: the first milestone is an end-to-end running pool, and every later milestone upgrades a running system.
How to read this
Survey provenance. Counts, paths, and line references come from a code survey originally taken at 09d528a53 and re-verified at 51c317a6b; treat them as pointers to re-verify, not as current facts.
The ordering principle is E2E first: get one pool serving real turns as early as possible, then harden and generalize on top of a system that runs. Perfection items — contract suites, protocol replacements, chart test batteries — attach to the milestone that needs them, never earlier. Three properties hold throughout:
Every milestone ships a running system. M0 runs end to end; each later milestone replaces one piece of scaffolding with the target mechanism while the rest keeps serving.
Nothing lands as a mode flag. Capability knobs and injected drivers, never if (isPool).
The single-org daemon keeps working the entire way. Scaffolding choices are chosen so local/self-hosted code paths are untouched.
Effort labels: S ≈ one PR, days. M ≈ a few PRs, 1–2 weeks. L ≈ a workstream, several weeks.
The trust and tenancy seams (landed)
The pool-facing identity work deliberately landed first, as target seams rather than scaffolding:
CP ↔ pool daemon authentication uses an audience-scoped, Pod-bound Kubernetes ServiceAccount token. TokenReview establishes the ServiceAccount subject and Pod UID; each Deployment Pod gets its own org-less daemon row and stable daemonId for that Pod lifetime.
The daemon ↔ CP WebSocket is install-wide. orgId is optional in the common frame envelope, but mandatory on every org-scoped frame over an install-wide connection. There is no org room, org-specific connection, or per-org socket.
Daemon ↔ shim authentication keeps the existing direct Kubernetes-identity mechanism. The CP does not issue a shim signing key, grant, public key, JWKS document, or key set. This settles the earlier open question outright: no CP-signed shim grant exists or is planned. Audience separation prevents a token for one hop from authenticating at another.
Why a skeleton is reachable early
Two decisions unlock it. The shared-table data plane meant multi-org members did not have to wait for Postgres — the plan allowed one SQLite file per org as scaffolding — but #958 made that moot by putting the complete cloud store on Postgres directly. Fleet identity and frame-scoped organization context land at the first application seam: each Pod authenticates with Kubernetes identity, owns one org-less daemon row, runs one CpClient, and carries orgId on org-scoped frames.
Three survey findings shape the order: dial-in is the prerequisite for tier-shared warm pools (AC_SHIM_ENDPOINT is baked into cloned templates and claim env bypasses adoption), so both land together in M3. Duty-gated ingress is nearly free at the connection layer (all platforms already reconcile per bot at runtime; the filter attaches at transportAgents()) — the real work was the third lifecycle state, since "absent from this.agents" previously meant both not mine and deleted, and the deleted path tears down workspaces; that was M2. Fleet identity is smaller than its daemonId reference count suggests — the WS handlers are daemon-fenced, not org-fenced; only 3 handlers derive org from the connection; the per-Pod identity and common orgId envelope make M4 a bounded threading change rather than a scheduler rewrite.
Goal: two orgs' agents serving real turns from one multi-tenant member, in a three-member Deployment, in the two-namespace layout. This is also the canary: everything after it upgrades a system that demonstrably works.
Multi-org boot: one CpClient authenticated by the Pod-bound ServiceAccount token; org-less daemon row; org-scoped frames carry orgId (frame-mode registration)
Member composition: one Pod runs one multi-org Daemon graph, not one graph per organization. Workspace and Git operations route by agent into that agent sandbox volume; workspace module settings and the member root are install-wide. Per-org workspace roots and per-org workspace-service instances are not part of this design.
Static assignment: agent.daemonId remains the placement seam for the skeleton's lifetime (no ledger enforcement yet)
Minimal chart — not a deliverable of this repository. Every application chart lives on the deployment side (this repository ships no chart for the pool), so the item as written was aimed at the wrong place. What it listed landed there instead: the sandbox→member and member-ingress rules; the runtime SandboxTemplate and warm pool rendered by the chart under release-derived names; and, last, the two-namespace layout — a dedicated agents namespace created by the cluster-admin-owned foundation release under the restricted Pod Security level with a default-deny baseline, the members' claim RBAC bound into it, template, warm pool, and shim-ingress policy rendered into it, the template hardened to pass restricted (deferred until after enforcement was verified so the two changes could not be confounded). Verified 2026-08-15: a cold turn minted its sandbox in the agents namespace, on the pinned node group, under restricted, in ~12 s. kubeconform validation of the rendered chart landed there too; a namespace-wide quota was considered and deliberately not added (see the risk register); a rendered-golden test remains optional.
Kubernetes hygiene that cannot wait (shared namespace): removed the orphaned drain/Sandbox watch after its producer retired, made runtime-probe claim names member-unique with bounded UID/resourceVersion-fenced garbage collection, and made the sandbox namespace explicit. fix(daemon): clean up Kubernetes runtime plane hygiene #970S.
Accepted imperfections, named so nobody mistakes them for the design: per-member sandbox templates are a per-member object (bounded by pool size, closed by M3). Fleet identity, frame-scoped orgId, and direct Kubernetes authentication are target seams from the outset, not temporary scaffolding.
Exit: a Telegram message to org A and a webchat turn for org B are served by the same member process; a second member serves a third org; an idle agent suspends and cold-wakes.
M1 — Shared state: the Postgres store — complete
The plan originally called for an async-store refactor (repository extraction → UnitOfWork → a dedicated async freeze forcing ~145 daemon.ts methods async → a separate PG driver). #958 took a different, cheaper route: a synchronous-facade Postgres store over a worker-thread bridge (store/postgres-sync-database.ts + postgres-store-worker.js), so LocalStore's synchronous surface survives unchanged and the async freeze never happens. As landed:
fix(daemon): use Postgres as the cloud store #958 — the complete durable LocalStore surface runs in Postgres for every --k8s daemon (pooled members and single-org envelope daemons alike): sessions/gates/purge outboxes, transcript rows + recipients + tool bodies + attachments + cursors, durable inbox, hook receipts, activation rendezvous, loop guards, memory-capture and remote-MCP outboxes, cron watermarks, orchestration, dreams, channel-introduction state, and the caches
SQLite is forbidden for cloud daemons — never opened, not a write-through cache, not a partial replica; Postgres is the only durable write target and the source of every restart/read-back decision (docs/designs/cloud-data-plane-postgres.md, rewritten in fix(daemon): use Postgres as the cloud store #958)
Shared-Postgres operations are fenced and store recovery is scoped
Staged-move fence and workspace-conversion leftovers from the fix(daemon): keep the agent-removal obligation off the ephemeral state root #962 inventory — re-examined 2026-08-16. The workspace conversion/materialization markers are no longer a fail-open: on the cluster path a missing marker means unproven and the volume is interrogated (origin converged, or re-cloned) before anything runs on it. The staged-move fence is member state by design (a detached source must not serve the agent again); the one residual is a two-hop move (pool → machine → pool) while the same member pod is alive, which left that member fenced and dark-holding the lease until the pod recycles — Pool member keeps a stale staged-move fence across a two-hop move (pool → machine → pool) #1093, closed by fix: clear a pool member's stale staged-move fence on a move back onto the pool #1098: a commit onto a set now releases the stale fence on every eligible member through a token-less agent/activate, and an unstage restores the replica without starting a host unless the member holds the duty at the host-start boundary.
The cross-driver behavior contract is the store suites themselves, which test(daemon): run the store suites on real PostgreSQL #1080 runs against real Postgres in CI. The per-token transcript write posture against networked Postgres has run the test environment's real turns for days without a symptom; no dedicated load test is planned, so this is closed as observed rather than left open indefinitely.
Exit (met, verified across the test-environment rollouts of 2026-08-15/16): a member restart loses no org state; no per-org SQLite files exist to retire.
M2 — Dynamic ownership: duty groups and the rendezvous — complete; enforced, unconditional, verified on a live rollout
feat(duty): install a granted agent the member does not have #972 — install-on-grant: a grant opened the serving gate but installed nothing, so a member that won a duty for an agent it never had could not serve it. duty/fetch pulls the bundle, authorized by holding the duty, and the grant is applied only after the install succeeds
feat(duty): self-fence held duties when the CP link outlives the lease horizon #976 — daemon self-fence (T_fence): per-group deadlines anchored on a CP-issued duty/renewed confirmation (never on a heartbeat merely sent — a half-open socket must not extend the fence), a general withdrawal guard so a late admission cannot resurrect what a fence, revoke or drain took away, and the discovery that duty/revoke never actually closed platform connections because a duty change did not mark them dirty
Enforcement flip — on in one environment, and it earned its keep immediately by surfacing two defects no unit test could see:
fix(duty): derive a duty group for every agent, not only for edge-bearing ones #983 — an agent with no socket-transport integration and no enabled cron appeared in no computed duty component, so the sweep deleted the singleton the rendezvous had just minted for it and revoked it as gone; the next trigger minted it again; each revoke interrupted the in-flight turn. Every webchat-only agent on the pool was unusable. Fixed by deriving a component for every agent (crons subsumed — a cron's agent is an agent), which also made recompute total and turned the rendezvous mint into a fallback. Verified after deploy: the group is stable, term unchanged, zero revokes.
Then a rollout: the holder's lease lapsed, the group became a claimable vacancy — and nothing claimed it. Three live members, zero grants. incumbent only grants to agent.daemonId's daemon, and after a rollout that names a Pod that no longer exists; healing waited on a trigger, and for webchat the trigger could not even be sent (Webchat cannot reach a pool member after a rollout: readiness and delivery are placement-keyed, so the rendezvous never fires #987) because the console judges "offline" from that same dead id.
Placement becomes a target; incumbent is deleted, not widened.feat(duty): make the pool heal itself by making placement a target #991. incumbent was feat(control-plane): duty-group recompute sweep and the incumbent grant policy #942's soak scaffolding, built when install-on-grant did not exist; every reason for it is gone. But it was also the only thing keeping pool members from claiming agents that live on local daemons — after fix(duty): derive a duty group for every agent, not only for edge-bearing ones #983 every such agent has a vacant group — so its removal needs a real notion of which agents belong to the pool: Agent.placementKind ∈ {daemon, pool} (additive; every existing row keeps its meaning) plus one eligibility resolver that every claim path, the fence, delivery, the roster, the peer directory, hooks, relay ingress and webchat readiness read. Eligibility is FORALL over the group's agents (a group mixing a pool agent with a machine-placed one is claimable by nobody). grantPolicy/incumbentOnly are removed rather than given an any value — a one-member policy enum is a lie about there being a choice. Load-bearing tests: the sweep re-grants a lapsed holder's group to a live member with no trigger; a pool member never claims a local-daemon agent's group; a full member does not claim; refused-install rotation is bounded (a refused grant is invisible on the wire and renewHeld renews by holder alone, so without an explicit surrender the group would wedge on the one member that cannot install it — incumbent was masking that too). Also from its review: a holder change must re-converge the compiled hook / HTTP-bot / collab-routing projections, or ingress keeps targeting the dead member; that kick now lives in one place (orchestrator/agentRouting.ts) shared with the placement-move path.
Proactive healing observed on a real rollout, no message sent. A pool agent was granted 9 s after being placed on the pool. The pool was then rolled: the holder drained and stopped beating; its lease stayed valid for the horizon (correct — T_reassign > T_fence means no reassignment inside the window); on lapse the sweep granted a live member, which fetched, installed, opened the gate and confirmed via digest. Old holder silent → new holder serving in ~85 s, versus permanently offline before. The 30 groups belonging to local-daemon agents stayed vacant throughout — no pool member claimed one.
Self-fence reverse acceptance on the live pool: restarted the control plane with three stable members holding a pool agent; every member's sessionEpoch bumped by one, the agent's duty term did not move, its holder did not change, and reconnect completed in well under the fence horizon. A CP restart is not mistaken for a partition.
Rollout drain is real — feat(daemon): drain duty groups on SIGTERM with acknowledged releases and a draining bit #1021 (closes the ~85 s takeover once listed under known issues: stop() never called releaseAllDuties, leases just lapsed). On SIGTERM a member announces draining (the control plane then refuses its claims), lets in-flight turns finish, releases each held group with an acknowledged duty/release only after that group's host teardown and connection convergence are confirmed, leaves what it cannot confirm to lapse, and settles late grants in one batch after the loop. Verified on live rollouts: "released 1 group, 1 acknowledged, 0 left to lapse", duty term moved exactly once. Daemon pool rolling update: surge/drain control so capacity never dips and each agent moves at most once #1016 (closed) has the design; six review rounds converged on a global rule rather than per-case teardown proofs.
Only the newest live generation of a member set may claim vacated groups — feat(control-plane): only the newest live generation of a member set claims vacated duty groups #1022. maxSurge: 100% alone is not a barrier (the controller scales the old ReplicaSet down as soon as one replacement is Available), so the barrier lives in the ledger: members report their pod-template generation, folded into the claim SQL so no registration can interleave; null generations (local daemons) are never held back. Verified: three members reported the same generation, term moved once.
Shim binding generation is durable and install-shared — fix(daemon): allocate shim binding generations from the shared store #1017. It was a per-member in-memory counter, and the sandbox shim refuses any hello below the highest generation it has seen, so a successor restarted at 1 and was refused (4403 stale generation) until the pod recycled. Now an atomic store-backed counter per agent; verified: after a rollout the successor bound a Running sandbox its predecessor had bound at generation 3, at generation 4.
Members publish a real readiness signal — feat(daemon): publish a real pool-member readiness signal #1056 (/readyz + marker file: startup complete + registered + runtime probe done + not draining; false at SIGTERM through the drain); the deployment side probes it and dropped the timed minReadySeconds stand-in.
M3 — Dial-in, Kubernetes identity, tier-shared warm pools — complete, minus two checks deferred with per-tier pools
The dial-in half has landed (shim/dialer.ts dials, shim/server.ts listens in the sandbox, generation-fenced; K8sDriverDeps.awaitChannel kept its signature, so the driver's tests stand). Authentication is asymmetric by design (design §7): the shim presents its audience-restricted projected ServiceAccount token and the dialing daemon TokenReviews it (implemented — the dialer's PodIdentityVerifier, including the presented-identity-vs-dialed-pod check); toward the shim the pre-disclosure boundary is the sandbox-namespace NetworkPolicy (one ingress rule: pool namespace → shim port), a single active connection, pre-auth frame-size caps, term fencing, and audience separation — an earlier revision's CP-signed grant / JWKS distribution was removed outright, so there is deliberately no shim-side TokenReview to build. Status:
Deprecated:The direct Kubernetes-identity handshake in both directions. Legacy shim→daemon compatibility and the bidirectional compatibility suite are out of scope; M3 retains only the daemon→shim Kubernetes-identity handshake.
The NetworkPolicy flip: the single coarse pool→shim ingress rule replaces sandbox→member egress allowances
The warm pool is a release-level chart object shared by every member (the per-member-template coupling retired with the chart consolidation; warmPoolRef adoption is live). The per-tier dimension stays skipped until agent resource tiers exist at all — warm pods' resources are fixed at creation, so tiers force per-tier pools, and none of that machinery is worth building for today's single size.
The Kubernetes-identity handshake suite — test(daemon): cover the dial-in Kubernetes-identity handshake invariants #1096: the §7 invariants are pinned on the daemon→shim path (identity presented or the dial fails, TokenReview refusal, presented-identity-vs-dialed-pod, audience assertion, single active connection with its queued slot, the fence rules, and the size cap in the direction that is safe today), each new case mutation-checked. It also found that one pre-existing test was passing on its own 1 ms dial deadline rather than on the pod check, and produced two findings: the accept-side WebSocket paths register no 'error' listener, so an oversized frame from an unauthenticated peer takes the shim process down (fix in flight; the shim-side half of the size-cap invariant lands with it), and §7's term is in fact enforced as the generation, with supersession at the dialer rather than at the shim — the design/implementation drift was Dial-in handshake: §7's term fencing is implemented as generation fencing, with supersession at the dialer #1097, closed by docs(designs): the dial-in handshake fences on the generation, not the duty term #1110: §7 now states the shipped shape — bind() carries the generation rules, the listener refuses a second dial outright, supersession happens at the dialer — and records the lingering-socket takeover delay as an accepted trade-off, covered by the duty gate and row ownership rather than by the generation.
Two empirical checks, deferred — only relevant if per-tier shared adoption is ever revisited: whether additionalPodMetadata.labels reach an adopted warm pod, and that an adopted pod never returns to the pool (with tier-shared pools that turns from waste into a cross-tenant leak)
Exit (met): the daemon→shim path passes the Kubernetes-identity handshake suite (#1096). (The original "wake adopts from a shared warm pool" criterion is retired as already met in substance: the warm pool is a release-level shared object and adoption is live; per-tier pools wait on resource tiers existing.) Legacy shim→daemon compatibility is deprecated and out of scope.
M4 — Frame-scoped organization hardening — M (needs M0) — complete: lint fence, frame contract, and the reconnect snapshot all landed
The fleet link already exists — every Deployment Pod gets one org-less daemon row resolved from (ServiceAccount subject, Pod UID) and one install-wide WebSocket; a container restart in the same Pod reuses the row, a replacement Pod gets a new daemonId, and Pod IP is metadata, not identity. M4 completes org-threading across the remaining handlers and registries:
A frame-mode connection requires orgId on every org-scoped request, event, control frame, and correlated reply; both peers validate the frame org against the targeted resource — feat(protocol): require and validate the frame organization on every org-scoped frame and reply (#955 M4) #1089. One classification of every frame type now lives in protocol/frame-scope.ts (install-wide: auth/register/bootstrap, heartbeat, capabilities, facts/*, relay roster, collaboration routes, drain/restart/upgrade/config-push, duty/* except duty/fetch, agent/exists; org-scoped: everything else including typed replies) and both peers run the same two checks at their decode edge: an org is required on org-scoped and forbidden on install-wide frames, a correlated reply must carry exactly its request's org (a mismatch fails the pending request with SCOPE_DENIED and applies nothing), and uncorrelated ack/error are dropped instead of answered. Gaps it closed on the way: the two peers kept divergent lists; replies bypassed the gate on both sides; hook/report and hook/start were unfenced; stray errors could ping-pong SCOPE_DENIED; the daemon's refusals were unlogged and cron/* controls unscoped.
Reconnect state as a combined multi-org snapshot or revision-fenced stream on the same member connection — never subscribe(org), an org room, an org-specific socket, or an org-specific data-plane schema. Do not invent the watermark machinery: Agent.configRevision with the daemon-side stale|conflict|idempotent|apply compare and SessionMeta.visibilityRev/visibilityAckedRev + replay-on-register already exist in production. Closed by the survey + proof in test(control-plane): prove reconnect as a combined multi-org snapshot (#955 M4) #1095: the mechanism already shipped as three pieces on one wire — register/ok is the combined install-wide snapshot (union roster pinned ∪ duty-held across every served org, per-entry orgs, revision-stamped specs applied through the daemon compare, ownership-aware drops), the register-time visibility replay converges visibilityRev per org on org-scoped session/visibility/snapshot frames, and the first beat's duty exchange re-issues missing/stale-term grants revision-stamped. test/protocol/multi-org-reconnect.test.ts pins the end-to-end property (two orgs mutated while a member is away converge through one reconnect); the tests flushed out no production gap.
Exit: one WS per Pod member; two orgs' scoped frames share it without cross-tenant reads or writes.
M5 — Per-org envelope retirement — done, and it was a deletion, not a migration
M5 was planned as a migration: a durable-state importer, workspace accounting, a rehearsed rollback, then the operator stops. None of that was built, because there was nothing to migrate. No production or staging envelope ever existed; the only consumers were disposable test organizations. A migration path would have been machinery written for an empty set, so the milestone collapsed to deleting the model outright (2026-08-14).
Cluster: every AgentConnectOrg CR and the CRD, the operator Deployment/RBAC/Lease, the envelope-fence admission policy, every per-org envelope namespace, and the per-org TokenReview bindings deleted from the one environment that had them. The others never ran an envelope.
chore: remove the per-org operator and envelope provisioner #964 — packages/operator, charts/operator (with the CRD), the operator image and its build/release/bake/component-version wiring, the CP's CR-writing half of src/cluster/ (crd, org-api, service, spec, maintenance-loop), the org_cluster_execution repository/routes/DTOs, and the console's cluster-execution card. docs/designs/agentconnect-org-operator.md is now a tombstone.
Deployment side: the operator release became a trimmed cluster-foundation release keeping only the TokenReview RBAC and the vendored agent-sandbox bootstrap — since folded back into the single application chart (one chart, one release; the CI deployer gained a narrow cluster-scoped grant instead of a second release).
Kept, verified by review: k8s-client, cluster/access.ts + cluster/daemon-identity.ts (Pod-bound TokenReview identity — how a pool member authenticates), the --k8s spawn driver, sandbox claims, the duty ledger.
Items the migration plan owned that simply ceased to exist: the durable-state importer, workspace accounting before cutover, the rollback order, the drain-annotation sweep, and named owners for orphaned operator behaviors (suspend quiescing, org offboarding). They are recorded here as not done and not needed rather than dropped silently.
One ordering mistake worth keeping. The cluster objects were deleted before the control plane that writes them, so between the two changes the deployed CP logged a 404 envelope re-apply on every maintenance tick for the four disposable orgs. Noise rather than damage, and it could not be quieted by turning the flag off (see the decoupling item above) — but the consumer should have been retired before the API it consumes.
Daemon groups — designed, first half in flight
The pool is the degenerate case of a member set — a named set of daemons within which an agent's duty may be claimed. Every prerequisite the pool design listed for generalizing it landed with M2, so the "future direction" became a design: docs/designs/daemon-groups.md (#994, six review rounds). One concept, member_set(id, orgId NULL-able, name), where null means cross-org (the pool — one per install, never per org) and an orgId means one organization's set of its own daemons. Tenancy lives in three write-time invariants (which daemons a set may contain; which set an agent may reference), so the read path is a single rule — claimant ∈ agent's set — and the placement resolver ends with fewer branches than it has today. daemon stays its own kind (a pinned machine has no replaceability; it joins a set, it does not become one). Membership changes are two-phase and generation-fenced in the drain/move order — stop and confirm the old authority, then commit — and enrolling a pinned machine is the existing agent move applied N times inside one fence.
PR 1 — fold the pool into member_set — refactor(control-plane): fold the pool into the unified member-set model #1003, merged and verified after deploy: one org-less member_set row, every pool agent rewritten to set + that row, org-less daemons auto-enrolled on auth (FK cascade removes the membership when the reaper retires the row), enum contracted to {daemon, set}, the ledger now reads a claimant's set from its own tables instead of trusting a caller-asserted scope, and every feat(duty): make the pool heal itself by making placement a target #991 test passed unchanged. Original scope note: zero behavior change, every feat(duty): make the pool heal itself by making placement a target #991 test passes unchanged; the pool becomes the org-less row, org-less daemons auto-enroll on auth, placementKind drops pool for set, {kind:'pool'} survives as API sugar at the edge. Coordinated cutover for the migration (single CP replica, pre-release).
Member-replacement audit (2026-08-16) — closed out
After #1017/#1019/#1023/#987 all turned out to be one class — code assuming one daemon owns an agent, sandbox, or session for its lifetime — a systematic audit of per-member state and agent.daemonId reads produced #1025–#1041 (three mechanisms: per-member memory nobody re-derives on a duty move; control-plane authorization/targeting on agent.daemonId instead of the placement resolver; tables that were per-process under SQLite and shared by accident on the pool store). All seventeen are closed: #1042 (orchestration deadlines duty-gated + CAS), #1044 (hook-completion outbox owned per member), #1045 (sandbox launches duty-scoped, re-derived on takeover), #1046 (boot recovery scoped to owned rows), #1047/#1061 (hooks and PR review through the placement resolver, completion accepted from the serving member), #1048 (relay actions through the rendezvous), #1049/#1064 (inbox backlog replayed on every duty gain, kept on handoff), #1053 (missed cron/dream fires compensated on handover, definition-fenced), #1054 (memory-capture gate keyed by agent), #1055 (the agent.daemonId sweep for 409s/dropped reports/visibility replay), #1057 (multi-agent webchat + delegated MCP placement), #1058 (model catalog keyed by member), #1063 + #1075 (the low-severity batch, transcript tables org-fenced), #1065 (session TTL/GC sweeps holder-only, purge receipts leased), #1068 (session-metadata outbox owned per member, parked not failed), #1069 (loop guard: atomic counters, member-scoped trip). Follow-ups it produced: #1050 (done, #1064), #1051 (done, #1061), #1073 (done, #1080 — the store suites now run on real Postgres in CI, which immediately caught NUL-joined activation keys that Postgres rejects), #1078 (done, #1081), #1062 (reconciler, done; the store half — orphan rows whose agent is gone — landed as #1085, which also folded the seven scattered prune/expire routines into one declarative retention rule table).
What moved off the critical path
Stated explicitly so it is not mistaken for omission: the store contract suite gates M1, not M0. Chart golden tests start minimal in M0 and grow with M3. Version-aware placement exists nowhere (there is no placement). The *Unscoped fence, full explicit-org hygiene, and the reconnect-snapshot proof gated M4 and have all landed (#1066, #1089, #1095). The connection-pool org-dimension fix (two orgs pasting the same bot token collide on the pool keys — copy the relay's tenant-fence) is required before untrusted tenants share a member, i.e. M5, not M0. And the canary is not a phase: M0 is the canary.
Remaining decisions and checks
Ownership assignments once blocking M5 (suspend quiescing, org offboarding) are moot — those behaviors were deleted with the operator, not reassigned. Empirical checks blocking M3: adoption-label propagation, adopted-pod-never-returns. Authentication is settled: CP ↔ daemon and daemon ↔ shim use direct Kubernetes identity; pool members are per-Pod org-less records; org context travels on the common frame; no CP-signed shim grant or public-key distribution exists. Everything else previously listed as a gate is settled in the design document (isolation unit, cron authority — holder-fired as an ingress edge, quota deferral, term/sessionEpoch independence, the duty-group ledger shape).
Risk register
Risk
Milestone
Mitigation
Skeleton org state dies with its member (SQLite on member disk)
M0
Closed by #958: cloud daemons have no SQLite; durable state is in shared Postgres
Two orgs sharing a bot token collapse onto one connection
M5 gate
Org dimension on the pool keys; the relay's tenant-fence fix is the template
Duty release runs the removal path and destroys a workspace
M2 ✅
The release lifecycle state landed in #948 with a test that release preserves the workspace
Partitioned ex-holder serves a group a successor has claimed
M2
The daemon self-fence (above) — the open blocker for the enforcement flip
dispatch() admission race / ACP update reordering after the async flip
M1
Moot: #958's sync-over-worker bridge keeps the store surface synchronous, so the async flip never happens
Long-lived async branch vs store churn
M1
Moot for the same reason — no async branch exists
A term derived from sessionEpoch churns every duty on each CP deploy
M2 ✅
Independent fencing domains; covered by tests in #939
Missing org predicate in a repository (row-tenancy's classic leak)
M1
Predicate injected in exactly one layer; contract-suite coverage; RLS as optional enforcement
One member suspends another org's sandboxes
M0
Client-side watch filtering via the claim→Sandbox join
Stranded drain annotation leaves a sandbox permanently un-wakeable
M5
Moot: the envelope model was deleted, never migrated — no drain annotations exist to strand
Pod killed mid-removal strands a deleted agent's sandbox volume
M1
Still open.#962 diagnosed it and proposed an install-wide agent_removal_obligation row unioned with the FS mirrors at boot, but was closed unmerged; its review also found the approach incomplete, since a store-admission failure exits the process after the CP row is already deleted. The branch survives as a starting point
Stale staged-move fence dark-holds an agent after pod replacement
M5 gate
Scope-key decision from the #962 inventory before pool members carry production agents
Shim listener pre-disclosure
M3
Direct Kubernetes identity authenticates the daemon before disclosure; rate/size limits before authentication; projected-token rotation
Data-plane PG availability becomes pool availability
The original k8s-daemon umbrella (D0–D10: remote spawn, shim, workspace-over-shim, sandbox lifecycle, k8s-as-supervisor) — complete, and it assumed one daemon per org, which #964 retired
Three enforcement-path findings the reviewer could not submit before #948 merged: duty changes not converging platform connections, agentsLost ignored on replacement, drain not closing connections
Closed by #976 and #977 — two of the three were rediscovered independently before the cross-check
MCP proxy and external-memory definitions did not follow the duty holder
Closed by #989 — the duty/fetch bundle carries both, the roster scopes them by the same union as agents; and it found that every MCP reader picked the retiring grant during rotation ([0] on an ascending query), now one currentMcpGrant selector plus an issuedAt fence so a bundle can never regress a fresh key
Webchat could not reach a pool member after a rollout
Delivery/readiness by #991; the install-window residual by #1060 (not_ready, pending directory entry, bounded retry). Webchat continuation of a session recorded on a retired member is a content-ownership question handled in #1019
Rename the "cloud daemon" code identifiers to the pool / member-set vocabulary; identity/storage contract names (the pool ServiceAccount, the store schema) deliberately excluded
Closed by #1008 (code) and #1012 (POOL_NAMESPACE, a hard rename — no compatibility alias, per the same reasoning as #971)
Daemon-pool rolling update: surge-then-drain, capacity never dips, each agent moves at most once
Closed: #1021 + #1022 (application), deployment side on maxSurge: 100%, a long grace period, a Helm timeout above the drain budget, AC_POD_TEMPLATE_HASH; two live rollouts verified. Readiness follow-up #1043 closed by #1056
Files half by #1077 (POST /agents/:id/wake, wake when the tab is active); memory half by #1081 — verified on the test environment at rc.150 (tab wakes the sandbox, a console write lands on the agent's volume)
Deployment-side counterparts live on the deployment side: the Cloud Daemon
was missing two environment settings the runtime plane made mandatory, which had
been failing every deploy and pinning that environment several releases back, and
the enforcement switch needed plumbing before it could be set at all.
Implementation plan and tracking issue for the k8s daemon pool: install-wide multi-org daemon members, the duty ledger, and the retirement of the per-org execution envelope. The mechanism design is docs/designs/k8s-daemon-pool.md (#954). The plan is restructured around a walking skeleton: the first milestone is an end-to-end running pool, and every later milestone upgrades a running system.
How to read this
Survey provenance. Counts, paths, and line references come from a code survey originally taken at
09d528a53and re-verified at51c317a6b; treat them as pointers to re-verify, not as current facts.The ordering principle is E2E first: get one pool serving real turns as early as possible, then harden and generalize on top of a system that runs. Perfection items — contract suites, protocol replacements, chart test batteries — attach to the milestone that needs them, never earlier. Three properties hold throughout:
if (isPool).Effort labels: S ≈ one PR, days. M ≈ a few PRs, 1–2 weeks. L ≈ a workstream, several weeks.
The trust and tenancy seams (landed)
The pool-facing identity work deliberately landed first, as target seams rather than scaffolding:
daemonIdfor that Pod lifetime.orgIdis optional in the common frame envelope, but mandatory on every org-scoped frame over an install-wide connection. There is no org room, org-specific connection, or per-org socket.Why a skeleton is reachable early
Two decisions unlock it. The shared-table data plane meant multi-org members did not have to wait for Postgres — the plan allowed one SQLite file per org as scaffolding — but #958 made that moot by putting the complete cloud store on Postgres directly. Fleet identity and frame-scoped organization context land at the first application seam: each Pod authenticates with Kubernetes identity, owns one org-less daemon row, runs one
CpClient, and carriesorgIdon org-scoped frames.Three survey findings shape the order: dial-in is the prerequisite for tier-shared warm pools (
AC_SHIM_ENDPOINTis baked into cloned templates and claimenvbypasses adoption), so both land together in M3. Duty-gated ingress is nearly free at the connection layer (all platforms already reconcile per bot at runtime; the filter attaches attransportAgents()) — the real work was the third lifecycle state, since "absent fromthis.agents" previously meant both not mine and deleted, and the deleted path tears down workspaces; that was M2. Fleet identity is smaller than itsdaemonIdreference count suggests — the WS handlers are daemon-fenced, not org-fenced; only 3 handlers derive org from the connection; the per-Pod identity and commonorgIdenvelope make M4 a bounded threading change rather than a scheduler rewrite.Milestones
graph LR M0[M0 · Walking skeleton<br/>multi-org member, static assignment] --> M1[M1 · Shared state<br/>async store + PG big table] M1 --> M2[M2 · Dynamic ownership<br/>duty groups, rendezvous] M0 --> M3[M3 · Dial-in, Kubernetes identity,<br/>shared warm pools] M0 --> M4[M4 · Frame-scoped org<br/>hardening] M2 --> M5[M5 · Envelope + operator<br/>DELETED, not migrated] M3 --> M5 M4 --> M5M1 and M3 are independent and can run concurrently.
M0 — Walking skeleton — complete, exit verified 2026-08-15
Goal: two orgs' agents serving real turns from one multi-tenant member, in a three-member Deployment, in the two-namespace layout. This is also the canary: everything after it upgrades a system that demonstrably works.
CpClientauthenticated by the Pod-bound ServiceAccount token; org-less daemon row; org-scoped frames carryorgId(frame-mode registration)Member composition: one Pod runs one multi-org
Daemongraph, not one graph per organization. Workspace and Git operations route by agent into that agent sandbox volume; workspace module settings and the member root are install-wide. Per-org workspace roots and per-org workspace-service instances are not part of this design.agent.daemonIdremains the placement seam for the skeleton's lifetime (no ledger enforcement yet)Minimal chart— not a deliverable of this repository. Every application chart lives on the deployment side (this repository ships no chart for the pool), so the item as written was aimed at the wrong place. What it listed landed there instead: the sandbox→member and member-ingress rules; the runtimeSandboxTemplateand warm pool rendered by the chart under release-derived names; and, last, the two-namespace layout — a dedicated agents namespace created by the cluster-admin-owned foundation release under therestrictedPod Security level with a default-deny baseline, the members' claim RBAC bound into it, template, warm pool, and shim-ingress policy rendered into it, the template hardened to passrestricted(deferred until after enforcement was verified so the two changes could not be confounded). Verified 2026-08-15: a cold turn minted its sandbox in the agents namespace, on the pinned node group, underrestricted, in ~12 s.kubeconformvalidation of the rendered chart landed there too; a namespace-wide quota was considered and deliberately not added (see the risk register); a rendered-golden test remains optional.Accepted imperfections, named so nobody mistakes them for the design: per-member sandbox templates are a per-member object (bounded by pool size, closed by M3). Fleet identity, frame-scoped
orgId, and direct Kubernetes authentication are target seams from the outset, not temporary scaffolding.Exit: a Telegram message to org A and a webchat turn for org B are served by the same member process; a second member serves a third org; an idle agent suspends and cold-wakes.
M1 — Shared state: the Postgres store — complete
The plan originally called for an async-store refactor (repository extraction →
UnitOfWork→ a dedicated async freeze forcing ~145daemon.tsmethods async → a separate PG driver). #958 took a different, cheaper route: a synchronous-facade Postgres store over a worker-thread bridge (store/postgres-sync-database.ts+postgres-store-worker.js), soLocalStore's synchronous surface survives unchanged and the async freeze never happens. As landed:LocalStoresurface runs in Postgres for every--k8sdaemon (pooled members and single-org envelope daemons alike): sessions/gates/purge outboxes, transcript rows + recipients + tool bodies + attachments + cursors, durable inbox, hook receipts, activation rendezvous, loop guards, memory-capture and remote-MCP outboxes, cron watermarks, orchestration, dreams, channel-introduction state, and the cachesdocs/designs/cloud-data-plane-postgres.md, rewritten in fix(daemon): use Postgres as the cloud store #958)reconcile --onceCronJob with observer registration) collects sandbox claims/Sandboxes whose agent is gone and probe claims past their window, so a member dying mid-removal no longer needs a durable obligation; the deployment side runs it every 10 min, dry-run until an observation window passes. What still lived on the state root was moved: managed agent memory (Pool agents' managed memory lives on the member's ephemeral state root, so a duty move or rollout loses it #1078 — the one confirmed data-loss path; closed by feat: keep a cluster agent's managed memory on its sandbox volume #1081: memory lives on the agent's sandbox volume at<workspace>/.agentconnect/memorybehind oneMemoryFsport with local and shim implementations, dreams and the console Memory tab wake the sandbox to reach it), the shim generation (fix(daemon): allocate shim binding generations from the shared store #1017), boot recovery scoped to owned rows (fix(daemon): recover only the interrupted work this member owns #1046), per-member caches keyed by owner (fix(daemon): key the runtime model catalog cache by its owning member #1058), outboxes owned per member (fix(daemon): own the hook-completion outbox per member #1044 hooks, fix(daemon): scope the session TTL/GC sweeps to the duty holder and lease purge receipts per member #1065 purge receipts, fix(daemon): own the session-metadata outbox per member #1068 session metadata).agent/activate, and an unstage restores the replica without starting a host unless the member holds the duty at the host-start boundary.Exit (met, verified across the test-environment rollouts of 2026-08-15/16): a member restart loses no org state; no per-org SQLite files exist to retire.
M2 — Dynamic ownership: duty groups and the rendezvous — complete; enforced, unconditional, verified on a live rollout
feat(control-plane): duty_group ledger for k8s daemons #937 — the
duty_groupledger: connected-component claim units, membership projection, the merge rule, CAS claimsfeat(protocol,control-plane): duty lease exchange riding the heartbeat #939 — the lease exchange riding the heartbeat: held digest + headroom,
duty/grant/duty/revoke/duty/release, chunked emission, per-daemon lanes, CP-side recovery gracefeat(control-plane): duty-group recompute sweep and the incumbent grant policy #942 — the recompute sweep, the incumbent-only grant policy, the placement fence
feat(daemon): hold duty leases and serve only the agents they cover #948 — the daemon duty registry:
transportAgents()as the enforcement gate, cron/dream schedule scoping, the release lifecycle state (release never runs the removal path that tears down workspaces), drain-time releasefeat(control-plane): recompute duty groups when their inputs change #949 — mutation-time recompute kicks (integration/cron/placement changes)
feat: the duty activation rendezvous #951 — the
not_holderrendezvous onrd/msg: claim-on-receipt, one-hop relay re-route, honest drop accountingfix(control-plane): scope agent lifecycle frames to the agent org #965 — agent lifecycle frames carry an explicit
orgId, so a move to an install-wide member is not refused withSCOPE_DENIEDbefore the frame leaves the processfeat(duty): install a granted agent the member does not have #972 — install-on-grant: a grant opened the serving gate but installed nothing, so a member that won a duty for an agent it never had could not serve it.
duty/fetchpulls the bundle, authorized by holding the duty, and the grant is applied only after the install succeedsfeat(duty): self-fence held duties when the CP link outlives the lease horizon #976 — daemon self-fence (T_fence): per-group deadlines anchored on a CP-issued
duty/renewedconfirmation (never on a heartbeat merely sent — a half-open socket must not extend the fence), a general withdrawal guard so a late admission cannot resurrect what a fence, revoke or drain took away, and the discovery thatduty/revokenever actually closed platform connections because a duty change did not mark them dirtyfix(duty): apply a refused replacement's removals instead of keeping the old composition #977 — a refused replacement grant shrinks the held entry to the shared members at the old term instead of keeping departed ones serviceable
feat(duty): route agent updates and the reconnect roster to the duty holder #978 — updates and the reconnect roster follow the holder (closes Agent updates and the reconnect roster must follow the duty holder, not the placement #973): one
AgentDeliveryseam owns every agent/integration/cron lifecycle send, the roster becomespinned-to-me ∪ duties I hold, andconfigRevisionreplaces mere presence as the refetch signalfeat(daemon): make duty enforcement reachable from the environment #980 — the flag is reachable from the environment (
AGENTCONNECT_DUTY_ENFORCEMENT); a pool member's config file is regenerated from defaults on every Pod start, so it could not otherwise be set at allEnforcement flip — on in one environment, and it earned its keep immediately by surfacing two defects no unit test could see:
gone; the next trigger minted it again; each revoke interrupted the in-flight turn. Every webchat-only agent on the pool was unusable. Fixed by deriving a component for every agent (crons subsumed — a cron's agent is an agent), which also made recompute total and turned the rendezvous mint into a fallback. Verified after deploy: the group is stable, term unchanged, zero revokes.incumbentonly grants toagent.daemonId's daemon, and after a rollout that names a Pod that no longer exists; healing waited on a trigger, and for webchat the trigger could not even be sent (Webchat cannot reach a pool member after a rollout: readiness and delivery are placement-keyed, so the rendezvous never fires #987) because the console judges "offline" from that same dead id.Placement becomes a target;
incumbentis deleted, not widened. feat(duty): make the pool heal itself by making placement a target #991.incumbentwas feat(control-plane): duty-group recompute sweep and the incumbent grant policy #942's soak scaffolding, built when install-on-grant did not exist; every reason for it is gone. But it was also the only thing keeping pool members from claiming agents that live on local daemons — after fix(duty): derive a duty group for every agent, not only for edge-bearing ones #983 every such agent has a vacant group — so its removal needs a real notion of which agents belong to the pool:Agent.placementKind ∈ {daemon, pool}(additive; every existing row keeps its meaning) plus one eligibility resolver that every claim path, the fence, delivery, the roster, the peer directory, hooks, relay ingress and webchat readiness read. Eligibility is FORALL over the group's agents (a group mixing a pool agent with a machine-placed one is claimable by nobody).grantPolicy/incumbentOnlyare removed rather than given ananyvalue — a one-member policy enum is a lie about there being a choice. Load-bearing tests: the sweep re-grants a lapsed holder's group to a live member with no trigger; a pool member never claims a local-daemon agent's group; a full member does not claim; refused-install rotation is bounded (a refused grant is invisible on the wire andrenewHeldrenews by holder alone, so without an explicit surrender the group would wedge on the one member that cannot install it —incumbentwas masking that too). Also from its review: a holder change must re-converge the compiled hook / HTTP-bot / collab-routing projections, or ingress keeps targeting the dead member; that kick now lives in one place (orchestrator/agentRouting.ts) shared with the placement-move path.Proactive healing observed on a real rollout, no message sent. A pool agent was granted 9 s after being placed on the pool. The pool was then rolled: the holder drained and stopped beating; its lease stayed valid for the horizon (correct —
T_reassign > T_fencemeans no reassignment inside the window); on lapse the sweep granted a live member, which fetched, installed, opened the gate and confirmed via digest. Old holder silent → new holder serving in ~85 s, versus permanently offline before. The 30 groups belonging to local-daemon agents stayed vacant throughout — no pool member claimed one.Follow-ups from feat(duty): make the pool heal itself by making placement a target #991's own review: fix(duty): scope the routing confirmation to the grant it proves #992 scopes the routing-projection confirmation to
(holder, term)so a same-holder re-take, a composition rewrite, or a stale-term digest cannot inherit a confirmation it did not earn; fix(web): let the edit modal place an agent on the pool #993 fixes the edit modal, whose readiness guard was still member-keyed and refused every pool placement from the console.chore(duty): retire the dutyEnforcement soak flag #995 — the
dutyEnforcementflag is gone (closes Retire the dutyEnforcement flag once enforcement has soaked and placement no longer names a member #982). Both retirement conditions held: placement no longer names a member (feat(duty): make the pool heal itself by making placement a target #991), and the only environment with a pool had enforced and healed a real rollout; environments without a pool have only org-scoped daemons, for which the predicate is false regardless. And "off" had become wrong, not inert — a pool agent is placed on no member, so off meant nobody served it.dutyEnforced()is now the tenancy predicate alone: an install-wide member enforces, an org-scoped daemon never does, there is no switch. (The deployment side keeps the old env var hard-coded for rollback compatibility until no flag-aware prerelease is a redeploy candidate.)fix(knowledge): scope the organization-suggestion replay to the org that owns it #996 — the org-suggestion replay from a pool member (closes Pool member: organization suggestion sync fails — handler derives org from an org-less daemon #968): the daemon no longer falls back to an unscoped frame when it cannot name an org, and — the real finding — the CP's sync handler and the review-decision route both authorized on
agent.daemonId, which is null for a pool agent, so a pool member's suggestions were silently dropped and its review decisions could never be applied. Both now go through the placement resolver (placement ∪ live duty holders). Org-knowledge read handlers still fence on agent.daemonId, so a pool member's agent cannot search or read org knowledge #999 records the same blind spot in the four org-knowledge read handlers.Self-fence reverse acceptance on the live pool: restarted the control plane with three stable members holding a pool agent; every member's
sessionEpochbumped by one, the agent's duty term did not move, its holder did not change, and reconnect completed in well under the fence horizon. A CP restart is not mistaken for a partition.Rollout drain is real — feat(daemon): drain duty groups on SIGTERM with acknowledged releases and a draining bit #1021 (closes the ~85 s takeover once listed under known issues:
stop()never calledreleaseAllDuties, leases just lapsed). On SIGTERM a member announcesdraining(the control plane then refuses its claims), lets in-flight turns finish, releases each held group with an acknowledgedduty/releaseonly after that group's host teardown and connection convergence are confirmed, leaves what it cannot confirm to lapse, and settles late grants in one batch after the loop. Verified on live rollouts: "released 1 group, 1 acknowledged, 0 left to lapse", duty term moved exactly once. Daemon pool rolling update: surge/drain control so capacity never dips and each agent moves at most once #1016 (closed) has the design; six review rounds converged on a global rule rather than per-case teardown proofs.Only the newest live generation of a member set may claim vacated groups — feat(control-plane): only the newest live generation of a member set claims vacated duty groups #1022.
maxSurge: 100%alone is not a barrier (the controller scales the old ReplicaSet down as soon as one replacement is Available), so the barrier lives in the ledger: members report their pod-template generation, folded into the claim SQL so no registration can interleave; null generations (local daemons) are never held back. Verified: three members reported the same generation, term moved once.Shim binding generation is durable and install-shared — fix(daemon): allocate shim binding generations from the shared store #1017. It was a per-member in-memory counter, and the sandbox shim refuses any hello below the highest generation it has seen, so a successor restarted at 1 and was refused (
4403 stale generation) until the pod recycled. Now an atomic store-backed counter per agent; verified: after a rollout the successor bound a Running sandbox its predecessor had bound at generation 3, at generation 4.Members publish a real readiness signal — feat(daemon): publish a real pool-member readiness signal #1056 (
/readyz+ marker file: startup complete + registered + runtime probe done + not draining; false at SIGTERM through the drain); the deployment side probes it and dropped the timedminReadySecondsstand-in.Retryable install-window verdict — fix(collab): retry a cross-daemon peer wake through the pool install window #1060 closes Webchat cannot reach a pool member after a rollout: readiness and delivery are placement-keyed, so the rendezvous never fires #987:
rd/agentmsggainsnot_ready, the directory publishes a pending entry at grant time, the source re-sends the same deliveryId with backoff for a bounded window (exactly-once preserved).M3 — Dial-in, Kubernetes identity, tier-shared warm pools — complete, minus two checks deferred with per-tier pools
The dial-in half has landed (
shim/dialer.tsdials,shim/server.tslistens in the sandbox, generation-fenced;K8sDriverDeps.awaitChannelkept its signature, so the driver's tests stand). Authentication is asymmetric by design (design §7): the shim presents its audience-restricted projected ServiceAccount token and the dialing daemon TokenReviews it (implemented — the dialer'sPodIdentityVerifier, including the presented-identity-vs-dialed-pod check); toward the shim the pre-disclosure boundary is the sandbox-namespace NetworkPolicy (one ingress rule: pool namespace → shim port), a single active connection, pre-auth frame-size caps, term fencing, and audience separation — an earlier revision's CP-signed grant / JWKS distribution was removed outright, so there is deliberately no shim-side TokenReview to build. Status:The direct Kubernetes-identity handshake in both directions. Legacy shim→daemon compatibility and the bidirectional compatibility suite are out of scope; M3 retains only the daemon→shim Kubernetes-identity handshake.warmPoolRefadoption is live). The per-tier dimension stays skipped until agent resource tiers exist at all — warm pods' resources are fixed at creation, so tiers force per-tier pools, and none of that machinery is worth building for today's single size.'error'listener, so an oversized frame from an unauthenticated peer takes the shim process down (fix in flight; the shim-side half of the size-cap invariant lands with it), and §7's term is in fact enforced as the generation, with supersession at the dialer rather than at the shim — the design/implementation drift was Dial-in handshake: §7's term fencing is implemented as generation fencing, with supersession at the dialer #1097, closed by docs(designs): the dial-in handshake fences on the generation, not the duty term #1110: §7 now states the shipped shape —bind()carries the generation rules, the listener refuses a second dial outright, supersession happens at the dialer — and records the lingering-socket takeover delay as an accepted trade-off, covered by the duty gate and row ownership rather than by the generation.additionalPodMetadata.labelsreach an adopted warm pod, and that an adopted pod never returns to the pool (with tier-shared pools that turns from waste into a cross-tenant leak)Exit (met): the daemon→shim path passes the Kubernetes-identity handshake suite (#1096). (The original "wake adopts from a shared warm pool" criterion is retired as already met in substance: the warm pool is a release-level shared object and adoption is live; per-tier pools wait on resource tiers existing.) Legacy shim→daemon compatibility is deprecated and out of scope.
M4 — Frame-scoped organization hardening — M (needs M0) — complete: lint fence, frame contract, and the reconnect snapshot all landed
The fleet link already exists — every Deployment Pod gets one org-less daemon row resolved from
(ServiceAccount subject, Pod UID)and one install-wide WebSocket; a container restart in the same Pod reuses the row, a replacement Pod gets a newdaemonId, and Pod IP is metadata, not identity. M4 completes org-threading across the remaining handlers and registries:orgIdon every org-scoped request, event, control frame, and correlated reply; both peers validate the frame org against the targeted resource — feat(protocol): require and validate the frame organization on every org-scoped frame and reply (#955 M4) #1089. One classification of every frame type now lives inprotocol/frame-scope.ts(install-wide: auth/register/bootstrap, heartbeat, capabilities,facts/*, relay roster, collaboration routes, drain/restart/upgrade/config-push,duty/*exceptduty/fetch,agent/exists; org-scoped: everything else including typed replies) and both peers run the same two checks at their decode edge: an org is required on org-scoped and forbidden on install-wide frames, a correlated reply must carry exactly its request's org (a mismatch fails the pending request withSCOPE_DENIEDand applies nothing), and uncorrelatedack/errorare dropped instead of answered. Gaps it closed on the way: the two peers kept divergent lists; replies bypassed the gate on both sides;hook/reportandhook/startwere unfenced; stray errors could ping-pongSCOPE_DENIED; the daemon's refusals were unlogged andcron/*controls unscoped.subscribe(org), an org room, an org-specific socket, or an org-specific data-plane schema. Do not invent the watermark machinery:Agent.configRevisionwith the daemon-sidestale|conflict|idempotent|applycompare andSessionMeta.visibilityRev/visibilityAckedRev+ replay-on-register already exist in production. Closed by the survey + proof in test(control-plane): prove reconnect as a combined multi-org snapshot (#955 M4) #1095: the mechanism already shipped as three pieces on one wire —register/okis the combined install-wide snapshot (union roster pinned ∪ duty-held across every served org, per-entry orgs, revision-stamped specs applied through the daemon compare, ownership-aware drops), the register-time visibility replay convergesvisibilityRevper org on org-scopedsession/visibility/snapshotframes, and the first beat's duty exchange re-issues missing/stale-term grants revision-stamped.test/protocol/multi-org-reconnect.test.tspins the end-to-end property (two orgs mutated while a member is away converge through one reconnect); the tests flushed out no production gap.*Unscopedlint fence tosrc/ws/**; give the 3 org-deriving handlers explicit arguments — chore(control-plane): fence the daemon WS surface on the frame org #1066 (frameOrgId(frame, conn)is the one place the org is resolved; a dozen handlers now fence their reads on it; three self-row reads andduty/claimallow-listed with a justification). One of them showed up as a live symptom (Pool member: organization suggestion sync fails — handler derives org from an org-less daemon #968,knowledge/suggestions/syncon an org-less member) and is closed by fix(knowledge): scope the organization-suggestion replay to the org that owns it #996; the four org-knowledge read handlers in the same file carry the sameagent.daemonIdblind spot and are Org-knowledge read handlers still fence on agent.daemonId, so a pool member's agent cannot search or read org knowledge #999. After feat(duty): make the pool heal itself by making placement a target #991 this class recurs wherever a handler still authorizes onagent.daemonId— the fix is always the placement resolver, never a per-handler special case.Exit: one WS per Pod member; two orgs' scoped frames share it without cross-tenant reads or writes.
M5 — Per-org envelope retirement — done, and it was a deletion, not a migration
M5 was planned as a migration: a durable-state importer, workspace accounting, a rehearsed rollback, then the operator stops. None of that was built, because there was nothing to migrate. No production or staging envelope ever existed; the only consumers were disposable test organizations. A migration path would have been machinery written for an empty set, so the milestone collapsed to deleting the model outright (2026-08-14).
AgentConnectOrgCR and the CRD, the operator Deployment/RBAC/Lease, the envelope-fence admission policy, every per-org envelope namespace, and the per-org TokenReview bindings deleted from the one environment that had them. The others never ran an envelope.packages/operator,charts/operator(with the CRD), the operator image and its build/release/bake/component-version wiring, the CP's CR-writing half ofsrc/cluster/(crd,org-api,service,spec,maintenance-loop), theorg_cluster_executionrepository/routes/DTOs, and the console's cluster-execution card.docs/designs/agentconnect-org-operator.mdis now a tombstone.k8s-client,cluster/access.ts+cluster/daemon-identity.ts(Pod-bound TokenReview identity — how a pool member authenticates), the--k8sspawn driver, sandbox claims, the duty ledger.CLUSTER_EXECUTION_ENABLED— refactor(control-plane): delete CLUSTER_EXECUTION_ENABLED, POOL_NAMESPACE is the switch #1014, gated on M0's exit as planned. chore: remove the per-org operator and envelope provisioner #964 had already removed the envelope keys and the seed-image boot requirement; the flag's last meaning was "this control plane may TokenReview an in-cluster daemon" — how a pool member registers — which the pool namespace key already asserted whenever set. Renaming was tried and rejected (refactor(control-plane): name the cluster switch after what it still does #971: an honest name at the cost of a deprecated alias). The switch is nowDAEMON_POOL_ENABLED=true(refactor(control-plane): replace POOL_NAMESPACE with DAEMON_POOL_ENABLED #1015; the interimPOOL_NAMESPACEfrom refactor(control-plane): rename CLUSTER_CLOUD_DAEMON_NAMESPACE to POOL_NAMESPACE #1012 carried one bit dressed as a namespace and was replaced, no alias): true ⇒ in-cluster client from the pod's ServiceAccount, still fail-loud outside a pod, pool identities accepted from the control plane's own namespace; unset ⇒ API-key daemon auth only, so a self-hosted control plane on Kubernetes with no cluster access is untouched. Deployment side: the transitionalclusterExecution.enabledblock and the deadCLUSTER_*seed envs come out of the chart in the same cut, ordered after the control plane carrying refactor(control-plane): delete CLUSTER_EXECUTION_ENABLED, POOL_NAMESPACE is the switch #1014 is deployed.Items the migration plan owned that simply ceased to exist: the durable-state importer, workspace accounting before cutover, the rollback order, the drain-annotation sweep, and named owners for orphaned operator behaviors (suspend quiescing, org offboarding). They are recorded here as not done and not needed rather than dropped silently.
One ordering mistake worth keeping. The cluster objects were deleted before the control plane that writes them, so between the two changes the deployed CP logged a 404 envelope re-apply on every maintenance tick for the four disposable orgs. Noise rather than damage, and it could not be quieted by turning the flag off (see the decoupling item above) — but the consumer should have been retired before the API it consumes.
Daemon groups — designed, first half in flight
The pool is the degenerate case of a member set — a named set of daemons within which an agent's duty may be claimed. Every prerequisite the pool design listed for generalizing it landed with M2, so the "future direction" became a design: docs/designs/daemon-groups.md (#994, six review rounds). One concept,
member_set(id, orgId NULL-able, name), where null means cross-org (the pool — one per install, never per org) and anorgIdmeans one organization's set of its own daemons. Tenancy lives in three write-time invariants (which daemons a set may contain; which set an agent may reference), so the read path is a single rule — claimant ∈ agent's set — and the placement resolver ends with fewer branches than it has today.daemonstays its own kind (a pinned machine has no replaceability; it joins a set, it does not become one). Membership changes are two-phase and generation-fenced in the drain/move order — stop and confirm the old authority, then commit — and enrolling a pinned machine is the existing agent move applied N times inside one fence.member_set— refactor(control-plane): fold the pool into the unified member-set model #1003, merged and verified after deploy: one org-lessmember_setrow, every pool agent rewritten toset+ that row, org-less daemons auto-enrolled on auth (FK cascade removes the membership when the reaper retires the row), enum contracted to{daemon, set}, the ledger now reads a claimant's set from its own tables instead of trusting a caller-asserted scope, and every feat(duty): make the pool heal itself by making placement a target #991 test passed unchanged. Original scope note: zero behavior change, every feat(duty): make the pool heal itself by making placement a target #991 test passes unchanged; the pool becomes the org-less row, org-less daemons auto-enroll on auth,placementKinddropspoolforset,{kind:'pool'}survives as API sugar at the edge. Coordinated cutover for the migration (single CP replica, pre-release).Member-replacement audit (2026-08-16) — closed out
After #1017/#1019/#1023/#987 all turned out to be one class — code assuming one daemon owns an agent, sandbox, or session for its lifetime — a systematic audit of per-member state and
agent.daemonIdreads produced #1025–#1041 (three mechanisms: per-member memory nobody re-derives on a duty move; control-plane authorization/targeting onagent.daemonIdinstead of the placement resolver; tables that were per-process under SQLite and shared by accident on the pool store). All seventeen are closed: #1042 (orchestration deadlines duty-gated + CAS), #1044 (hook-completion outbox owned per member), #1045 (sandbox launches duty-scoped, re-derived on takeover), #1046 (boot recovery scoped to owned rows), #1047/#1061 (hooks and PR review through the placement resolver, completion accepted from the serving member), #1048 (relay actions through the rendezvous), #1049/#1064 (inbox backlog replayed on every duty gain, kept on handoff), #1053 (missed cron/dream fires compensated on handover, definition-fenced), #1054 (memory-capture gate keyed by agent), #1055 (theagent.daemonIdsweep for 409s/dropped reports/visibility replay), #1057 (multi-agent webchat + delegated MCP placement), #1058 (model catalog keyed by member), #1063 + #1075 (the low-severity batch, transcript tables org-fenced), #1065 (session TTL/GC sweeps holder-only, purge receipts leased), #1068 (session-metadata outbox owned per member, parked not failed), #1069 (loop guard: atomic counters, member-scoped trip). Follow-ups it produced: #1050 (done, #1064), #1051 (done, #1061), #1073 (done, #1080 — the store suites now run on real Postgres in CI, which immediately caught NUL-joined activation keys that Postgres rejects), #1078 (done, #1081), #1062 (reconciler, done; the store half — orphan rows whose agent is gone — landed as #1085, which also folded the seven scattered prune/expire routines into one declarative retention rule table).What moved off the critical path
Stated explicitly so it is not mistaken for omission: the store contract suite gates M1, not M0. Chart golden tests start minimal in M0 and grow with M3. Version-aware placement exists nowhere (there is no placement). The
*Unscopedfence, full explicit-org hygiene, and the reconnect-snapshot proof gated M4 and have all landed (#1066, #1089, #1095). The connection-pool org-dimension fix (two orgs pasting the same bot token collide on the pool keys — copy the relay's tenant-fence) is required before untrusted tenants share a member, i.e. M5, not M0. And the canary is not a phase: M0 is the canary.Remaining decisions and checks
Ownership assignments once blocking M5 (suspend quiescing, org offboarding) are moot — those behaviors were deleted with the operator, not reassigned. Empirical checks blocking M3: adoption-label propagation, adopted-pod-never-returns. Authentication is settled: CP ↔ daemon and daemon ↔ shim use direct Kubernetes identity; pool members are per-Pod org-less records; org context travels on the common frame; no CP-signed shim grant or public-key distribution exists. Everything else previously listed as a gate is settled in the design document (isolation unit, cron authority — holder-fired as an ingress edge, quota deferral, term/
sessionEpochindependence, the duty-group ledger shape).Risk register
Skeleton org state dies with its member (SQLite on member disk)dispatch()admission race / ACP update reordering after the async flipLong-lived async branch vs store churnsessionEpochchurns every duty on each CP deployStranded drain annotation leaves a sandbox permanently un-wakeableagent_removal_obligationrow unioned with the FS mirrors at boot, but was closed unmerged; its review also found the approach incomplete, since a store-admission failure exits the process after the CP row is already deleted. The branch survives as a starting pointIssues spawned from this plan
Work that earned its own issue rather than a checkbox here, each because it is a
distinct mechanism with its own gate:
#804/#815#950agentsLostignored on replacement, drain not closing connections#973#975#979duty/fetchbundle carries both, the roster scopes them by the same union as agents; and it found that every MCP reader picked the retiring grant during rotation ([0]on an ascending query), now onecurrentMcpGrantselector plus anissuedAtfence so a bundle can never regress a fresh key#987not_ready, pending directory entry, bounded retry). Webchat continuation of a session recorded on a retired member is a content-ownership question handled in #1019#982dutyEnforcementflag itself#1001POOL_NAMESPACE, a hard rename — no compatibility alias, per the same reasoning as #971)#999agent.daemonId, so a pool member's agent cannot search or read org knowledge#1010#1007SandboxClaimregardless of itswarmPoolRef, so a renamed warm pool never migrates existing agents#1009k8s-clientwatch unit test#1016maxSurge: 100%, a long grace period, a Helm timeout above the drain budget,AC_POD_TEMPLATE_HASH; two live rollouts verified. Readiness follow-up #1043 closed by #1056#1023event/session-syncoutbox row forever when its organization could not be resolved#1024duty-lease.handlerrenewal test#1025–#1041#1050#1051#1062reconcile --onceCronJob (observer registration, no lease), dry-run by default#1070POST /agents/:id/wake, wake when the tab is active); memory half by #1081 — verified on the test environment at rc.150 (tab wakes the sandbox, a console write lands on the agent's volume)#1073#1078#1085reconcile --oncealso reaps store rows whose agent no longer exists (dry-run behindAC_STORE_ORPHAN_DELETE); the daemon's hourly tick runs the same rules age-only, replacing seven hand-written prune routines (net −93 lines)#1018/#1020Deployment-side counterparts live on the deployment side: the Cloud Daemon
was missing two environment settings the runtime plane made mandatory, which had
been failing every deploy and pinning that environment several releases back, and
the enforcement switch needed plumbing before it could be set at all.
Known issues
Flaky daemon unit test: shim-dial-in 'waits for the replacement connection' times out on slow CI runners #938— flakyshim-dial-inunit test — closed by test: deflake the timing-based daemon and control-plane suites #1090, which also moved the duty-drain/duty-fence/dream-scheduler suites onto injected clocks and replaced the CP route tests' sleep-and-hope with apg_stat_activitywaiterSandboxClaims do not follow a warm-pool rename or a sandbox-namespace move (a claim is not re-resolved after it binds); on the environments that had them the claims were recreated by hand (K8s driver reuses an existing SandboxClaim regardless of its warmPoolRef, so a renamed warm pool never migrates existing agents #1007, closed as one-time). Nothing to build unless a live-migration path is ever wanted.