epic: make AgentOS Fleet operations resilient by construction

[robinbraemer]

Outcome

Make Fleet operations recover automatically or deterministically from fresh PVCs, provider/model bootstrap failures, Pod and native-session restarts, credential expiry, stream failures, capacity/PVC topology constraints, listener loss, partially completed Kubernetes/PostgreSQL lifecycle operations, authorization-plane failures, memory-index degradation, telemetry-pipeline outages, workload-plan failures, and optional ACP/A2A delivery failures.

Keep AgentOS native-tools-first: Kubernetes is runtime truth, PostgreSQL is custody/workflow truth, Herdr and native sessions are execution truth, Mate PVC Markdown is private-memory truth, Git is delivered-code truth, and OpenTelemetry is diagnostic evidence rather than another authority. This epic does not introduce a custom Kubernetes controller.

Incident evidence

• A fresh Pi PVC could start with Unknown provider: unknown because environment wiring did not create a deterministic provider entry.
• A Second Mate could supervise an existing child Pod through exact resourceNames, but Kubernetes RBAC cannot constrain future Pod creation by object name.
• Client-side Secret apply flows could persist credentials in kubectl.kubernetes.io/last-applied-configuration; projected GitHub data was observed with mode 0644; a manually minted token expired before a Scout launched.
• Terminated, overloaded, credential, transport, provider, listener, capacity, and retry failures were collapsed into generic or non-resumable signals.
• Lifecycle progress stored ad hoc could drift after partial Kubernetes or SQL failure.
• Per-Agent Kustomize overlays require many manually patched placeholders without one typed input/digest contract joined to lifecycle recovery.
• The existing private-memory selector adds a model call and can miss topics whose concise index hook is weak; any improvement must preserve per-Mate privacy, forgetting, and Markdown authority.
• Existing AI telemetry is a strong foundation but does not yet correlate access, budgets, MCP, memory, delegation, readiness, protocol delivery, and repair-forward recovery.

Architectural guardrails

• Give every persistent Second Mate a First-Mate-managed namespace containing that Second Mate and its Crewmates.
• Let Second Mate expand, modify, and shrink approved child workloads only inside its namespace; keep Namespace, RBAC, NetworkPolicy, quotas, admission, and ceilings under First Mate/platform control.
• First Mate maintains broad Second-Mate domains using load, dependency, failure, capacity, idle, and delivery signals; domains split, merge, expand, shrink, and retire without becoming silos.
• Treat every Secret in a Second-Mate namespace as reachable by that Second Mate; never place Fleet-root or provider-root credentials there.
• Use explicit workload-identity routes for credentialed providers while preserving ordinary Internet access.
• Separate agentgateway enforcement, AgentOS/OpenFGA decisions, and provider-scoped credential delivery. Do not add one universal token, OpenBao/Vault, TLS interception, or blanket egress denial.
• Keep Pi's native OpenAI compaction lifecycle; evaluate gateway request-local compression only for stateless routes.
• Keep private typed Markdown topics as memory authority. Any lexical/vector index is optional, local to the owning Mate PVC, rebuildable, and eval-gated; no shared vector service or unapproved remote embedding.
• Retain ordinary Kubernetes resources, dedicated Agent ServiceAccounts, retained home PVCs, pod-local Herdr, and native kubectl; a pure Effect Schema workload compiler may emit reviewable Kustomize overlays but never apply them or become a controller.
• Use ACP only as an evaluated coding-harness control adapter and A2A only as an optional PostgreSQL-first live delivery/interoperability edge.
• Preserve one direct parent-child Agent-authored delivery edge; cross-domain work routes through the common ancestor and lateral sibling/Crewmate channels remain denied.
• Keep Kagent as an optional external A2A provider and disposable benchmark; no Kagent controller, CRD, memory, conversation loop, AgentHarness, or Agent Substrate production dependency is implied.
• Extend privacy-safe OTel across Agent, Assignment, memory, access, cost, topology, protocol delivery, and recovery. Never export prompts, memory, tool payloads, credentials, or dynamic IDs as metric labels.
• Fail closed at dispatch or credentialed-provider authorization when identity, capacity, storage, provider configuration, budget, policy, or workload plan is invalid; prefer repair-forward idempotent reconciliation.
• Migrate AgentOS-owned effectful TypeScript incrementally to Effect while leaving pure computation and presentational TSX pure.

Workstreams

Runtime isolation and placement

• feat(runtime): isolate each Second Mate and its Crewmates in a managed namespace #75 — isolate each Second Mate and its Crewmates in a managed namespace
• refactor(runtime): make Agent workloads and Fleet service wiring namespace-neutral #76 — make Agent workloads and Fleet service wiring namespace-neutral
• feat(runtime): enforce safe child-workload and capacity admission per Second Mate #77 — enforce safe child-workload and capacity admission per Second Mate
• epic(runtime): make Agent workload declarations typed, reviewable, and recoverable #121 — make Agent workload declarations typed, reviewable, and recoverable
• [Idea] Harden disposable full-Fleet development environments #24 — harden disposable full-Fleet development environments

Bootstrap, readiness, and recovery

• fix(ai-gateway): make Pi Gateway provider bootstrap deterministic on fresh PVCs #78 — make Pi Gateway provider bootstrap deterministic on fresh PVCs
• feat(runtime): add semantic readiness for providers, sessions, listeners, and ingress #79 — add semantic readiness for providers, sessions, listeners, ingress, and authorization dependencies
• feat(runtime): add a SQL-backed runtime operation journal and repair-forward reconciliation #82 — add a SQL-backed runtime operation journal and repair-forward reconciliation
• feat(supervision): make retry exhaustion and native-session recovery explicit and resumable #83 — make retry exhaustion and native-session recovery explicit and resumable

Identity and credential safety

• fix(auth): remove credential-bearing Secret apply state and harden projections #80 — remove credential-bearing Secret apply state and harden projections
• Add a team GitHub App identity path #26 — add a team GitHub App identity path
• epic(auth): workload-identity API access with reusable Mate profiles #27 — workload-identity API access, provider isolation, budgets, MCP, and reusable profiles

Agent protocols and interoperability

• epic(protocol): add structured ACP harness control and authorized A2A delivery #122 — add structured ACP harness control and authorized PostgreSQL-first A2A delivery

Private memory

• epic(memory): improve private Mate recall with eval-gated hybrid retrieval #108 — improve private Mate recall with eval-gated local hybrid retrieval

Effect migration

• epic(effect): migrate AgentOS TypeScript to Effect #86 — migrate AgentOS TypeScript effectful paths to Effect

Observability and verification

• feat(observability): end-to-end OpenTelemetry for AgentOS Fleets #56 — end-to-end operational OpenTelemetry for AI, access, memory, delegation, protocols, and recovery
• test(resilience): prove workload, identity, session, protocol, and recovery failure conformance #84 — add restart, credential, stream, capacity, PVC, listener, identity, workload-plan, protocol, and telemetry failure conformance

Recommended delivery order

1. Land Secret safety (fix(auth): remove credential-bearing Secret apply state and harden projections #80) and deterministic provider bootstrap (fix(ai-gateway): make Pi Gateway provider bootstrap deterministic on fresh PVCs #78).
2. Establish namespace ownership (feat(runtime): isolate each Second Mate and its Crewmates in a managed namespace #75), namespace-neutral wiring (refactor(runtime): make Agent workloads and Fleet service wiring namespace-neutral #76), and admission/capacity controls (feat(runtime): enforce safe child-workload and capacity admission per Second Mate #77).
3. Add semantic readiness (feat(runtime): add semantic readiness for providers, sessions, listeners, and ingress #79), operation journaling (feat(runtime): add a SQL-backed runtime operation journal and repair-forward reconciliation #82), and explicit resumable recovery (feat(supervision): make retry exhaustion and native-session recovery explicit and resumable #83).
4. Start the Effect foundation (epic(effect): migrate AgentOS TypeScript to Effect #86), then deliver the typed workload contract/compiler and profiles under epic(runtime): make Agent workload declarations typed, reviewable, and recoverable #121 against feat(runtime): add a SQL-backed runtime operation journal and repair-forward reconciliation #82 rather than adding a controller.
5. Run the agentgateway/OpenFGA gate under epic(auth): workload-identity API access with reusable Mate profiles #27, then deliver workload identity, profiles, provider adapters, GitHub, budgets/kill switches, MCP, and AI Gateway integration.
6. Under epic(protocol): add structured ACP harness control and authorized A2A delivery #122, evaluate ACP independently; prove PostgreSQL-first A2A through the selected gateway/authorization path before implementing it; run the Kagent comparison last and keep it non-adopting.
7. Baseline current memory under epic(memory): improve private Mate recall with eval-gated hybrid retrieval #108; land lexical retrieval before optional local vectors, and adopt hybrid recall only if it wins the evaluation.
8. Build feat(observability): end-to-end OpenTelemetry for AgentOS Fleets #56 throughout every workstream rather than adding observability after behavior ships.
9. Use test(resilience): prove workload, identity, session, protocol, and recovery failure conformance #84 and [Idea] Harden disposable full-Fleet development environments #24 as the disposable-Fleet release gate.

Program success criteria

• A Second Mate provisions and supervises Crewmates in its namespace without per-worker First Mate apply, while cross-namespace access remains denied.
• First Mate topology changes are explainable, bounded, journaled, recoverable, and periodically reevaluated to prevent silos.
• Fresh/restored PVCs start with deterministic provider configuration and cannot become Ready with an unknown provider.
• Provider credentials never enter Agent namespaces, responses, logs, traces, journals, request logs, or dynamic configuration; one provider adapter cannot access another provider's credential.
• Credentialed access is authorized from exact workload/Mate/Assignment identity; reusable profiles, budgets, and kill switches remain within Captain ceilings and meet a revocation SLO.
• Native GitHub tools survive installation-token expiry, and native OpenAI compaction remains replay-compatible through the gateway topology.
• Ordinary Internet continues during access-plane outage.
• A typed workload plan deterministically produces reviewable Kustomize overlays, fails before apply on invalid inputs, and joins exact input/render digests to repair-forward recovery without storing YAML in PostgreSQL.
• Persistent and interactive Agents retain their dedicated ServiceAccount, StatefulSet/PVC, pod-local Herdr, native session, and one-writer semantics.
• ACP for Pi/Codex either passes the supervised lifecycle with Herdr/native-session custody intact or is rejected without weakening the native path.
• A2A discovery/delivery is PostgreSQL-first, workload-identity authorized, hierarchy constrained, and recoverable through the existing listener/Herdr wake path during outage.
• Kagent remains optional interoperability/benchmark evidence unless a later approved design proves its runtime and identity model superior.
• Private-memory retrieval improvements beat the current selector on numerical quality/task/resource gates; Markdown remains authoritative and pause/forget invalidate all derivative hits.
• One privacy-safe trace can correlate Mate, Assignment, memory, authorization, gateway/provider, protocol delivery, topology, and recovery outcomes while forbidden content and high-cardinality metric labels are rejected.
• Dispatch validates memory, quota, node affinity, PVC access mode, and placement feasibility before committing work.
• Partial lifecycle operations reconcile without duplicate Agents, Pods, sessions, listeners, credential grants, memory truth, or lost Assignments.
• Retry exhaustion is durable, diagnosable, and explicitly resumable.
• AgentOS-owned effectful TypeScript uses Effect outside approved runtime edges.
• Disposable-Fleet conformance covers restart, credential expiry, identity/revocation, profile reload, budget kill switch, stream interruption, capacity, PVC topology, listener loss, authorization dependency, workload-plan conflicts, ACP/A2A outage, memory index corruption/forgetting, and Collector failure.

Non-goals

• Custom Kubernetes controller/operator, AgentOS CRD, autonomous wrapper runtime, service mesh, transparent universal proxy, or TLS interception.
• Blocking normal Internet, adding OpenBao/Vault, or migrating AgentOS manifests wholesale from Kustomize to Helm.
• Replacing PostgreSQL coordination with A2A task state or using ACP as lateral inter-Agent messaging.
• Kagent controller/CRDs, conversation loop, memory, AgentHarness, or Agent Substrate as a core production dependency.
• Shared cross-Mate vector memory, PostgreSQL memory mirroring, mandatory semantic indexing, or remote embedding of real memory by default.
• Prompt, transcript, memory, tool-payload, or credential capture for observability; agentgateway request logs are not a Fleet authority.
• Production rollout, infinite retries, or a shadow copy of Kubernetes/native-session/workspace state in PostgreSQL.

Design record

docs/superpowers/specs/2026-07-31-resilient-delegation-access-effect-design.md on branch docs/resilience-access-effect-architecture.