AgentOS is an agent-native fleet system for persistent coding agents in Kubernetes. The model remains the decision-maker; repository code implements only deterministic, reviewable mechanics.
AgentOS occupies the layer above the agent harness: a harness such as Pi turns a model into an agent, and AgentOS turns agents into an organization. Its canonical coordination primitive is the Assignment — one outcome bound to one accountable Agent. The remaining durable model exists to keep Assignments truthful.
Each kind of state has one authority:
| Concern | Authority |
|---|---|
| Human planning and provider workflow | Captain-selected tracker or provider |
| Durable fleet data | PostgreSQL |
| Live workload state | Kubernetes |
| Terminal and harness runtime | Herdr inside each runtime pod |
| Agent home and unfinished work | Agent-owned PVC |
| Delivered code | Git and its remote |
| Optional pooled AI credentials and routing state | Fleet AI Gateway PVC and process |
AgentOS does not continuously mirror one authority into another. PostgreSQL stores pod locators, not heartbeats; Kubernetes is inspected when live state matters. Terminal output stays in the runtime unless a deliberate workflow archives a specific artifact.
AgentOS connects these authorities without turning any one of them into a universal ledger. A provider issue, board change or comment records human planning and external workflow state. A responsible Mate reconciles that intent; a PostgreSQL Task may preserve deliberate backlog, while accepted execution begins with its first accountable Assignment. Raw model reasoning, harness transcripts and terminal output remain in their runtime authorities. Delivered changes become durable only in Git and its remote, after which the responsible Agent reconciles the human tracker through its native provider tool.
Stable references join the chain: a Task may carry external tracker links, its Assignment history records ownership and handoff, and its report identifies the delivered result. AgentOS does not continuously mirror issue bodies, transcripts or repositories into PostgreSQL merely to make querying convenient.
PostgreSQL is therefore the enforceable coordination ledger, not a claim that every thought or mutation is a forensic audit event. Versioned SQL defines the specific guarantees AgentOS can promise: authenticated identity, scoped writes, atomic handoff, durable Captain decisions, selected immutable history and transactional wake hints. Core owns guarantees; integrations own workflows and surfaces.
Benchmark evidence is a bounded, immutable derivative of the authorities
above, never another Fleet authority, transcript mirror or analytics table in
the coordination database. A scenario declares which events and stable
references it needs; unavailable telemetry remains unobserved. Measurement
does not mutate the subject, and improvement begins only after the evidence is
frozen. The portable contracts and AgentOS evidence mapping live under
benchmarks/.
First Mates own the fleet. Second Mates supervise delegated domains. Crewmates perform bounded work. The initial supported First- and Second-Mate harness is Pi because AgentOS customizes its skills, extensions, lifecycle and health reporting. Working agents may use Pi, Codex, or another harness verified by the selected AgentOS release.
Each agent has its own identity and persistent home PVC.
Each agent runs in its own runtime pod with a dedicated ServiceAccount, home
PVC, database principal and pod-local Herdr server. A responsible Mate selects
the workload image and creates the reviewed per-agent Kustomize overlay with
native kubectl commands.
First and Second Mates use the same small agentos image and the same shared
StatefulSet base. Their Kustomize overlays explicitly select the role working
directory, environment, lifecycle tasks and credentials. The image contains a
credential-free, shallow Git seed at its exact source revision, Mise, pinned
Bun and the locked AgentOS
tool definitions. First start clones that seed to the home PVC and Pi runs from
the persistent role directory; an init container
installs Node and the remaining startup-critical Fleet tools onto the Mate's persistent home.
Ordinary tool additions remain in that Mise-managed home. Crewmates instead
receive a task-suited image selected by the responsible Mate. The judgment-based
dispatch profile records the concrete harness and may preserve its other
resolved native choices in an opaque settings object. Model, effort, fast
mode, compaction, context limits and image are examples rather than AgentOS
schema fields. An omitted image choice uses the released lightweight default;
a remote image must come from an approved registry and be pinned by digest.
Large language- or Codex-oriented images are therefore explicit task costs,
not the universal fleet runtime.
Read-heavy Scouts may opt into a separate ArtifactFS image and a reviewed
platform-specific FUSE Pod profile for fast lazy access to large or multiple
repositories. ArtifactFS is never installed in the common Mate image, and its
scratch overlay is not a delivery path for ship work.
One Herdr server runs per runtime pod and owns that pod's workspaces, tabs, panes, terminal processes, runtime status and native harness session references. Client detach leaves processes running. After a pod replacement, Herdr restores its layout and asks supported harnesses to resume from their own persistent sessions.
Herdr is not a fleet database.
Its CLI and socket API are used locally for attach, read, send, wait, layout and debugging.
AgentOS uses Herdr's documented CLI for one-shot supervision waits; those
commands already resolve targets and wait through the public socket protocol,
so AgentOS ships no parallel Herdr client. A future long-lived subscriber must
first justify why the CLI wait surface is insufficient, negotiate the installed
protocol/schema, bootstrap from session.snapshot, and take a fresh snapshot
after every reconnect before trusting subsequent events.
Outside the cluster, a human or seed agent resolves the target Pod with native
kubectl using an explicit Kubernetes context, enters that Pod, and invokes
its real Herdr terminal. A running Mate first distinguishes its current
cluster, namespace, Pod, container and identity from the intended target. It
invokes a native command directly when it already occupies that execution
boundary, and uses kubectl with its in-cluster credentials when the target is
another Kubernetes boundary. Neither direct execution nor kubectl exec is a
universal rule.
A First Mate may arrange central fleet workspaces whose panes attach to Herdr sessions in other pods through Kubernetes exec. This is only a user-facing view; the remote pod-local sessions remain authoritative.
The Captain has one regular fleet interface: First Mate.
First Mate does not perform project-specific coding, investigation, planning, bug reproduction or audits itself.
It may inspect projects read-only to understand and route work, and it may mutate reviewed Fleet operational state, but it delegates project work to a charter-matched Second Mate or a bounded Crewmate.
A persistent writable AgentOS development checkout is First Mate's narrow
self-maintenance exception. Its role contract and delegation Skill own the
exact authority and procedure, including the bounded Captain-authorized
capacity-recovery path while direct reports remain preserved and supervised.
The exception restores shared AgentOS capacity through the normal reviewed
path; it transfers no Mate credential and grants no merge authority.
The root-owned /opt/agentos tree is the immutable image Git seed, not the
active checkout. The harness reads AgentOS instructions, Skills and Mise files
from $HOME/projects/agentos; unfinished changes and its worktrees therefore
survive Pod replacement. Reviewed Markdown and Skill updates can be loaded from
that Git checkout at a safe Pi turn boundary, while OS, runtime and Kubernetes
changes still reach a running Mate through a tested immutable image digest.
Outside that bounded recovery, or whenever a report actively needs attention,
First Mate delegates AgentOS source changes too because hands-on work competes
with supervision.
A Second Mate uses the same architecture inside one persistent charter. It delegates project work to its own Crewmates, manages only its direct subtree and returns Captain-relevant outcomes to First Mate. An empty Second-Mate queue is a healthy idle state, not permission to invent work. Second Mates never create further Second Mates.
Every accepted work item has one durable Task and exactly one active accountable Assignment before an asynchronous worker begins. Creating a new accepted outcome or accepting a recorded backlog Task creates that first Assignment atomically through the released PostgreSQL Function; an unassigned Task remains backlog. A Task keeps one stable identity across handoff. Each Assignment stores its complete authoritative brief and final or handoff report; its PVC copy is only the harness view. A handoff ends the prior Assignment and creates the replacement atomically instead of rewriting history or cloning the Task. A ship Crewmate works in an isolated worktree until its changes are durably landed or handed off. A ship is complete only when the selected project workflow has produced its declared durable delivery artifact: a remote-backed workflow normally commits and pushes the task branch and creates or updates its review artifact, while a local-only workflow produces a clean committed branch. Accepted ship authority never includes a default-branch push or merge. A contradictory brief that forbids every selected delivery path is rejected or reclassified before dispatch; an uncommitted worktree is not review-ready. A scout's durable output is its report; its scratch worktree or ArtifactFS mount may then be discarded. No Mate merges without the Captain's explicit approval or a previously recorded standing authorization, and no agent or workspace with active or unlanded work is retired by implication. Loss of one shared model-capacity path is resolved per Assignment and does not create a dependency between otherwise independent accepted outcomes. A worker's approved direct authentication remains a complete recovery path; only work with no viable authorized capacity is blocked.
Task and Assignment state is the primary channel for delegated work. Inbox is
reserved for durable speech acts that state cannot express, and Agent-authored
delivery crosses only one direct parent-child hierarchy edge. A cross-domain
request escalates to the common ancestor, which creates or routes a Task in the
target subtree; sibling Agents never establish a lateral coordination channel.
Persistent First and Second Mates wake through PostgreSQL LISTEN/NOTIFY and
query the durable rows. Crewmates are not required to run AgentOS supervision
or a PostgreSQL listener: for a downward Crewmate delivery, the owning Mate
commits the Inbox row first, then submits one concise Herdr doorbell containing
only its kind, UUID and subject. The full body is never duplicated into the
terminal. A visible supervisor label followed by U+2063 distinguishes this
ephemeral routing hint from likely direct human input, but never authenticates
it or grants authority. Direct terminal delivery to another Mate is an
exceptional recovery path for a broken listener, not ordinary communication.
Direct Captain intervention in any attached terminal remains authoritative and is reconciled into Fleet state.
Each persistent Mate owns private, typed Markdown memory on its PVC. A bounded
index and selected topics load as fallible context; restricted post-turn
extraction and Dream may maintain only that memory. Memory never authorizes an
action, proves current state or replaces shared Fleet coordination. Guidance
for another Mate crosses the direct hierarchy edge as an Inbox proposal and
the recipient decides whether to edit its own memory. Genuine unresolved
Captain choices live in Inbox under stable keys. Investigations attest their
complete choice set, including none, before completion; the exact answer later
releases linked Task dependencies atomically without a separate decisions
service.
That idempotent transaction—record the response, close the speech act and apply
its coupled state effect—is the template for any future Inbox act that changes
durable state.
Each Mate supervises only its direct reports and keeps status changes sparse: decisions, blockers, material phase changes, completion and failure.
While direct reports are active, their Mate keeps the smallest verified set of
situation-appropriate waits: normally one targeted durable Fleet notification
wait for the current Mate plus native Kubernetes, Herdr-status or bounded
terminal waits for concrete live risks. Waits are deduplicated by authority,
target and predicate and re-armed only while their condition remains useful.
If the selected release lacks that wake capability, the Mate reports the unsupported boundary instead of claiming unattended supervision.
Project checkouts and provider repositories are operated through native Git and reviewed provider CLIs. The project's selected delivery workflow owns its validation, task-branch push, review artifact and approval rigor; AgentOS does not add a wrapper CLI or a parallel review gate. Project metadata records that workflow, artifact and merge authority as durable prose rather than an AgentOS mode enum. The Task links the resulting remote review artifact and the Assignment report records branch, commit, URL, validation and delivery state. External webhook payloads remain immutable JSON, coalesce into short bounded bursts and are reconciled pull-first by the responsible Mate. Provider writes remain synchronous and visible to that Agent rather than being hidden behind an outbox worker.
Provider identity stays at the same native boundary. An individual may keep
native gh authentication on the owning Mate's PVC. A team may instead mount a
dedicated GitHub App private key into First Mate only and mint short-lived
installation tokens on demand for native git, gh-axi and gh calls. First
Mate may reduce a child Agent's token to the repositories and permissions needed
by its active Assignment, deliver it through one Agent-specific projected
Kubernetes Secret, and replace that Secret before the provider expiry. The
child reads the projected file for each provider command; it never receives the
App key or a cached long-lived token. A request for broader access travels up
the durable hierarchy and never enlarges itself. The key and token never enter
Agent home, Fleet rows, task briefs or source control; only non-secret scope,
expiry and Secret references may be recorded. Repository permissions are a
Captain-reviewed provider boundary, not Fleet authority; the accepted delivery
workflow still controls which write or merge is allowed.
Credential bytes that AgentOS places in Kubernetes follow one managed-Secret
lifecycle: create only when absent, retry or rotate through a
resourceVersion-guarded replacement that preserves UID, and require explicit
takeover for conflicting ownership, scope, schema, key sets or annotations.
Managed Secrets contain non-secret ownership labels and no annotations;
generated objects stream directly to the API and are never retained. File
projections are read-only 0440 under an explicit workload fsGroup (or an
atomic runtime-owned 0600 file where a group projection is impossible).
Verification observes metadata, key names, projection rollover and provider
health without reading credential values. $agentos-secrets owns the native
operational procedure, including rollback and revocation.
Pi's native package and resource configuration is the extension boundary for a deployed AgentOS Mate. AgentOS adds no second plugin registry. A reviewed Pi package may load additively beside the released AgentOS resources or provide a replacement distribution whose one selected entrypoint assembles released and organization-specific behavior. Pi configuration is the loaded-resource authority.
@akua-dev/agentos is one package with two explicit surfaces. Importing its
root module exposes inert TypeScript registration functions. Selecting the
package through Pi activates its declared extension and Skills. A replacement
may depend on the package API without selecting the released extension.
A complete replacement exposes one exact Pi-discovered AgentOS entrypoint. The entrypoint validates the explicit deployed Agent role, loads exactly one ordinary First- or Second-Mate setup module and fails closed rather than guessing from the working directory or loading both roles. It may reuse only public exports present in its exact AgentOS release; unavailable behavior is not permission to deep-import private paths.
Packages may contribute extensions, instruction resources, Skills, prompts,
Crewmate setups, database migrations and runtime assets. A complete
distribution may also supply First- and Second-Mate role mise.toml files,
images and native Kubernetes overlays. Package installation makes all of these
available but grants no PostgreSQL, Kubernetes, provider or Git authority.
Several independently loaded extensions may call the public API. Registration values are plain structural data; public functions accept Pi explicitly and keep no cross-package singleton state. Before attaching handlers, the selected owner validates versions and package-qualified IDs and rejects collisions across tools, commands, Skills, messages and persisted entries. Ordering and overlapping ownership are explicit rather than resolved by load order.
Pi selection and process bootstrap are separate activation boundaries. Pi
configuration plus safe /reload can select extensions, instructions, Skills
and prompts. It cannot retroactively select the role Mise configuration, image,
ServiceAccount, RBAC, mounts or Kubernetes workload that started its own
process. Those changes use the workload's explicit
AGENTOS_DISTRIBUTION_ROOT, native Mise process lifecycle and native
Kubernetes apply and rollout while preserving and verifying the Mate home PVC.
AGENTOS_AGENT_CWD must resolve to the selected distribution's exact role
directory. Package presence is not proof of either activation boundary.
An extension may trigger a bounded Pi turn at startup or reload and direct the model to one delivered Skill; the model then inspects and reconciles native authorities. For an authorized in-place distribution update, that Skill may guide a native image and Kubernetes rollout and verify the resumed Herdr and Pi session. The extension cannot perform first-boot selection from inside a process that has not started. AgentOS adds no Fleet-state watcher, synthetic Assignment event stream, automatic migration service or package activation record.
PostgreSQL owns accepted Tasks, Assignments, complete briefs, reports and authority. The Agent row selects the harness; the Assignment brief records the accepted outcome, constraints and chosen runtime expectations. Pi and native harness catalogs prove what was loaded. AgentOS does not duplicate those facts in a second setup state.
$agentos-customization owns the conditional inspection, design, staging,
selection, reload, verification and rollback workflow. The public
CUSTOMIZATION.md page explains this product boundary
without duplicating that procedure.
Mise supplies tools to every AgentOS agent, including First Mates, Second Mates and Crewmates.
The release-owned mise.toml and mise.lock define Bun, Node and
the pinned Fleet tools for Pi, Codex, Herdr, Treehouse, Kubernetes, GitHub,
validation, AXI helpers and command-line inspection. The pair resolves exact
Bun revision 1.4.0-canary.1+3979cbe80 through Mise's checksummed HTTP backend. Because
upstream's moving canary release deletes superseded assets, every Bun
platform entry in the reviewed lock uses its upstream checksum and a direct
URL to an immutable AgentOS toolchain prerelease containing the unmodified
archive plus license and source/relink notices. A stable Bun 1.4 release
supersedes this temporary mirror once available.
AgentOS images install that pair as /etc/mise/config.toml and
/etc/mise/mise.lock, and bake its pinned Bun so typed bootstrap programs do
not require a first-start download. The persistent AgentOS Git checkout
provides the current repository and role Mise configuration; agent-owned
additions live separately under ~/.config/mise/conf.d/.
Release-owned Node dependencies are installed in the immutable image at
/opt/agentos/node_modules. packages/agentos/runtime/run-mate.ts passes that
release path through NODE_PATH to Herdr and its Pi child, so Pi extensions
loaded from the persistent checkout can resolve release dependencies without
installing them into the checkout or PVC.
Before starting a Mate, a direct Mise init step installs the remaining small
startup-critical set: Node, kubectl, Herdr and Pi. A second init step uses
Mise to run the typed home-reconciliation program. Both init containers and the
Mate use one image and one PVC; identical image layers are pulled only once per
node. The Crewmate base additionally installs gh, no-mistakes, Codex and
gh-axi for its reviewed pull-request delivery path; the default and exception
for Agent-authored delivery are owned by the agentos-projects Skill.
Other remaining released Fleet tools stay locked and discoverable but are
installed explicitly when the running Mate's task needs them. The Mate image
carries only PostgreSQL's official pinned
postgresql-client-18 package, so agents can invoke psql immediately without
compiling or embedding a PostgreSQL server.
Mise itself remains part of the running First Mate, not merely its bootstrap.
Its shims come first on PATH for interactive and non-interactive processes, so released tools win over unmanaged global installations.
Agents invoke tools by their ordinary names without a mise exec prefix.
Resolution follows the current working directory, including after a Crewmate enters an isolated worktree.
Pinned operating-system transport and database-client utilities in the reviewed
Mate image remain ordinary commands but are not duplicated into the PVC by Mise.
The existing agent-owned $HOME/.local/bin path may expose exact reviewed
integration entrypoints retained on the home PVC. Those entrypoints do not
become part of the immutable AgentOS tool baseline, and their integration must
verify provenance, activation and rollback through its own native boundary.
Mise configuration is deliberately layered:
/etc/mise/config.tomlprovides the lowest-precedence immutable image baseline.- The persistent AgentOS checkout supplies its reviewed root and role configuration;
~/.config/mise/conf.d/may add approved persistent tools for that agent. - A repository or nested-worktree
mise.toml,.tool-versions, or supported idiomatic version file adds tools and overrides conflicting versions for that project.
A repository with no Mise configuration receives the AgentOS baseline unchanged.
A repository-owned configuration remains project authority; AgentOS does not copy its own project file into foreign worktrees.
Untrusted repository configuration is inspected before trust because Mise configuration may carry executable behavior.
Agents do not install parallel global toolchains through ad hoc npm-global, Homebrew, apt or curl | sh paths.
Treehouse owns reusable detached worktrees inside each Crewmate pod. AgentOS acquires a durable UUID-labelled lease and lets the reviewed Treehouse return workflow own later cleanup instead of reimplementing worktree pooling.
The Fleet baseline deliberately excludes tasks-axi because PostgreSQL is the
task authority, excludes tmux because Herdr is the initial runtime, and excludes
Helm and additional harnesses until implemented behavior requires them.
Direct provider authentication inside each Agent's persistent harness is the complete minimal topology and recovery path. For a delegation-ready Fleet, AgentOS recommends the optional Fleet AI Gateway when several approved Agents or trusted harness automations need model capacity and the Captain accepts the additional credential authority and service lifecycle. The gateway avoids repeated worker logins and credential-file copying while keeping provider credentials out of Agent homes. It is capacity infrastructure, not the Fleet coordination kernel: PostgreSQL still owns work and communication, the harness still chooses the requested provider/model, and the provider response still returns synchronously to the calling harness.
AgentOS evaluates capacity postures from the least distributed complete path:
- direct per-Agent OAuth for the minimal topology and recovery;
- the in-cluster Fleet AI Gateway for a shared multi-subscription pool;
- a mixed in-cluster posture where selected Mates remain direct and only approved workers or trusted automation use the Gateway; and
- an external Cloudflare Worker only when the same deliberately external router must also serve non-AgentOS clients.
For an AgentOS-only Fleet, the in-cluster or mixed posture is stronger than the
external Worker posture. Agents can operate the Gateway through native
Kubernetes, the service needs no public Ingress, and the Fleet-local
OpenTelemetry Collector receives the complete privacy-bounded request path.
Cloudflare is a standalone deployment surface for codex-router, not an
upgrade or dependency for AgentOS.
The first implementation is a single-replica Bun service with one retained
ReadWriteOnce PVC. Its mode-0600 OAuth vault owns fresh server-created Codex
refresh chains and authentication eligibility. Bounded quota observations stay
process-local and are refreshed after restart; the canonical codex-router
Bun/SQLite state owns only opaque session assignments, quota or transient
blocks and renewable request reservations.
It stores no prompts, model responses or harness transcripts. An OpenAI API key
may be mounted separately as an explicitly enabled last-resort fallback and is
never copied into the mutable OAuth vault.
The product, package, Skill and executable are named Fleet AI Gateway and
ai-gateway. Agent Pods never authenticate to it with a Fleet-shared secret.
They call the private agentgateway-openai PEP with a kubelet-rotated,
audience-bound ServiceAccount token. HTTP external authorization performs
TokenReview, resolves the canonical Mate or Assignment, asks the OpenFGA-backed
PDP, and returns a closed grant lasting at most 15 seconds. Agentgateway
overwrites every grant header and forwards the request to the AI Gateway; it
mounts no AI Gateway client credential. A separate core-only operator Secret
protects status endpoints and is never accepted on the inference request path.
Every client request is authenticated and authorized before its body reaches
the provider controller. The AI Gateway validates the bounded grant before
selection, removes projected identity and all x-agentos-* headers, selects
and reserves an eligible account, normalizes the OpenAI Responses path, injects
that account's upstream credential and streams the actual response. Existing
sessions remain sticky while eligible. A sent request is never retried silently
on another account; upstream 401, 429, timeout and provider failures remain
visible to Pi or Codex and affect only later selection.
Pi routes its built-in openai-codex provider through Agentgateway with native
models.json base-URL settings. For First and
Second Mate, prepare-home atomically reconciles only an AgentOS-marker-owned
provider override, validates the staged provider and any separately selected
exact model with pinned Pi before commit, preserves auth.json and unrelated
configuration, and persists no workload or provider credential. The released
before_provider_headers behavior rereads the projected token for every
request, replaces Pi's public transport placeholder, and fails closed when the
token cannot be read or validated.
The Gateway client patch never selects a provider/model merely because the
service exists. Codex Crewmates use a marker-owned native model_providers
entry with command-backed authentication; Codex rereads the same projected
token every 60 seconds and on an authentication retry.
Independently, the persistent First- and
Second-Mate Pi projects load one shared AgentOS-owned session-lifecycle
extension that asks OpenAI for a native compaction result and persists the
complete canonical compact output—including its opaque compaction artifact and
provider metadata—in Pi's existing compaction entry alongside Pi's portable
summary. It persists that result only after one completed terminal response
with valid full output and exactly one compaction artifact. On subsequent
requests, the extension replays that saved output only for the exact provider
and model that created it; a newer local-only compaction supersedes an older
result. Provider failure keeps the portable summary, while unverifiable or
incomplete results are never persisted. Setting
AGENTOS_OPENAI_SERVER_COMPACTION_ENABLED=0 restores Pi's native compaction
path.
The extension preserves Pi's built-in openai-codex transport and supports both
the configured direct OpenAI Responses endpoint and the gateway's Codex
Responses endpoint. It uses bounded HTTP/SSE for Codex, bounded JSON for direct
OpenAI Responses, and introduces no WebSocket runtime dependency. The gateway is
body-blind: after authenticating, it strips credential, gateway/session-routing,
Host, Content-Length, and hop-by-hop request headers, then requests upstream
Accept-Encoding: identity to avoid Bun's intermediary body-decoder boundary.
Because Bun may expose a decoded response body with stale upstream metadata, the
gateway also strips response Content-Encoding and Content-Length with the
hop-by-hop response headers. It forwards every remaining header, the request
body and Bun-exposed response body without inspecting or storing either body.
AgentOS ships no provider-transport replacement or harness wrapper for either
path. Enabling the route is an explicit configuration change; it
never selects or changes a provider or model. quota-axi remains an
observation-only tool and has no routing, login or mutation authority.
An approved Responses client that cannot supply command-backed authentication
uses the AgentOS ai-gateway-workload-proxy as a loopback-only sidecar; the
exact Hermes contract and Pod wiring live in
docs/integrations/hermes-ai-gateway.md.
The sidecar rereads projected workload identity for each request, removes the
client's placeholder authorization and all caller-supplied AgentOS authority
headers, optionally injects a validated Assignment ID from its own trusted
configuration, and forwards only the two reviewed Responses paths once to
Agentgateway. It preserves the upstream status, end-to-end headers after
required hop-by-hop and stale Content-Encoding/Content-Length metadata
removal, and stream without following redirects. W3C trace correlation passes
through unchanged. Its
process-only liveness and local projected-token readiness never contact an
upstream. It owns no credential,
selection, retry, fallback, prompt, response, or deployment authority. The
client keeps exact model selection and disables request replay and provider
fallback for this route; its workload owner composes the sidecar and projected
token into that client's reviewed native manifest.
This is not a universal AgentOS proxy. Git, PostgreSQL, Kubernetes, Herdr,
registries and other provider tools continue through their native interfaces.
Adding another mediated protocol requires a separate review of its authority,
credentials and failure semantics; similarity to HTTP traffic is insufficient.
The gateway's account vault uses locked atomic updates and its canonical
codex-router routing state uses transactional SQLite on the retained PVC, so
account login, rotation, reservations and blocks do not require a reverse-proxy
reload. Provider adapters and selection semantics remain reviewed source and
change through the normal image lifecycle; there is no generic dynamic-route
control plane.
The released service, tests and optional Kubernetes topology live together in
services/ai-gateway/. First Mate may operate that topology through its
reviewed Skill and RBAC without owning the component's source directory.
Each default role owns an additive ai-gateway-client.yaml patch in its
Kubernetes patches/ directory. A reviewed per-Agent overlay composes that
patch only for an approved client; it adds the selected-client label, private
Agentgateway Service URL and explicit Gateway provider
mode without choosing a model or thinking level. Composing every approved
client produces the pooled posture; composing selected clients only produces
the mixed posture. A First Mate, Second Mate, or Crewmate returns to direct authentication by
replacing the client patch with its role's ai-gateway-direct-auth.yaml for one
successful rollout. Pi and Codex reconciliation remove only the owned provider
entry and marker while preserving direct authentication and unrelated
retained-home state; only then is the rollback patch removed. Omitting both
modes while a marker remains fails startup rather than retaining an ambiguous
route.
An exceptional AgentOS client may use a standalone Cloudflare Worker only as a separate external credential and routing authority. The Worker never forwards into the in-cluster Gateway, the Gateway never forwards through the Worker, and one OAuth refresh chain is never shared between their state authorities. Cloudflare Workers Observability owns the Worker's native router logs and traces; the AgentOS Collector continues to own the in-cluster client signals. External telemetry delivery or cross-system correlation is an explicit integration with its own plan, privacy and propagation constraints, not an implicit property of using the Worker.
Claude, Gemini, WebSockets, multi-replica authority, public ingress and general-purpose egress proxy behavior remain outside the first contract.
AgentOS telemetry is a diagnostic control plane, never a second workload, coordination, memory, access or inference data plane. The runtime, Pi, Codex, memory, access services, protocol adapters, topology controller, readiness and recovery paths, optional Fleet AI Gateway and Collector pipeline all consume one contract. They export asynchronously to one Fleet-local OpenTelemetry Collector Service. Collector or remote-export failure never blocks a provider request or Fleet control operation and never participates in Agent or Gateway readiness. The Collector remains a separate single-replica StatefulSet with its own retained PVC so credentials, routing state, Agent state and telemetry storage do not share a process or volume.
Contract version 1 owns the only AgentOS-specific telemetry vocabulary across
the runtime, ai, compaction, memory, access, protocol, topology,
readiness, recovery and telemetry_pipeline domains. Instrumentation
imports its executable definitions from
packages/agentos/src/telemetry/contract.ts; components do not invent local
resource attributes, span names, attribute names or metric names. Each
attribute definition declares its owner, source, signals, sensitivity,
cardinality and executable value rule. Each metric declares its owner,
instrument, UCUM unit, exact allowed labels, histogram boundaries and value
semantics. agentos.ai.cost is explicitly modeled catalog cost in {USD}, not
invoice truth.
Additive bounded values may extend version 1 only after the owning domain, privacy, cardinality, Collector and dashboard reviewers approve the same change. Renaming a field, changing its meaning, changing a metric unit or value semantics, or broadening a bounded value into arbitrary input requires a new contract version, dual-read migration where applicable and an explicit dashboard migration. A dashboard or alert may use only published contract fields; it cannot make a local telemetry vocabulary authoritative.
Every resource carries the applicable standard OpenTelemetry identity:
service.name,service.namespace=agentos,service.versionanddeployment.environment.name;k8s.cluster.name,k8s.namespace.name,k8s.workload.name,k8s.pod.nameandk8s.container.name; andagentos.fleet.name,agentos.ai.runtime(piorcodex) andagentos.ai.runtime.version.
Fleet and Kubernetes values are deployment-controlled resource attributes.
Pod UID is useful inside the Collector for association but is not exported as a
metric label. Provider account identity, email, credential identity and raw
provider account IDs have no telemetry representation. A deployment that must
compare routes may assign an opaque slot-[A-Za-z0-9_-]+ route slot; it appears
only on protected spans and correlated logs.
The bounded per-operation vocabulary is:
| Attribute | Values | Signals | Cardinality and sensitivity |
|---|---|---|---|
agentos.ai.runtime |
pi, codex |
all | 2, public operational |
agentos.ai.route |
direct, ai_gateway |
all | 2, public operational |
agentos.ai.provider.family |
openai, other |
all | 2, public operational |
agentos.ai.request.kind |
main, compaction, memory_extract, memory_consolidate, extension |
all | 5, public operational |
agentos.ai.model.family |
gpt-5, gpt-4.1, o-series, other |
all | 4, public operational |
agentos.ai.session.state |
fresh, resumed |
all | 2, operational |
agentos.ai.stream.mode |
streaming, non_streaming |
all | 2, operational |
agentos.ai.status_class |
success, client_error, server_error, cancelled, error |
all | 5, operational |
agentos.ai.error.class |
none, authentication, rate_limit, overload, timeout, abort, transport, protocol, decode, unavailable, unknown |
all | 11, operational |
agentos.ai.stream.outcome |
not_streamed, completed, client_disconnect, aborted, upstream_error |
all | 5, operational |
| operation, attempt, trace, span, session and provider request IDs | opaque, capped at 128 characters | spans and protected correlated logs only | unbounded, restricted |
agentos.ai.route.slot |
opaque slot-*, capped at 32 characters |
spans and protected correlated logs only | deployment-bounded, restricted |
HTTP status code, numeric token counts, chunk count and byte count may be span
attributes. Metrics use only the bounded dimensions in the table; they never
carry a request, trace, span, attempt, operation, thread, session, pod UID,
provider request or route-slot identifier. Instrumentation records no arbitrary
exception message or stack. 401/403, 429, 502/503/529, timeout,
abort, transport, protocol and decode failures map to the bounded error classes
before export.
The canonical span trees are:
Direct or Gateway AI
agentos.ai.operation
└── agentos.ai.provider.attempt {runtime, route, request kind}
└── ai-gateway.request # Gateway route only
├── ai-gateway.authenticate
├── ai-gateway.route.acquire
├── ai-gateway.quota.refresh
├── ai-gateway.upstream
│ └── ai-gateway.stream
└── ai-gateway.route.release
Codex through Gateway
codex conversation/turn span
└── Codex native API attempt
└── ai-gateway.request ...
Native server compaction
agentos.compaction.operation {path=native_server}
└── agentos.ai.operation {request kind=compaction}
└── agentos.ai.provider.attempt
Memory extraction, consolidation or retrieval
agentos.memory.operation {operation, method, outcome}
└── agentos.ai.operation {request kind=memory_extract|memory_consolidate}
└── agentos.ai.provider.attempt
Authorized external access
agentos.access.http | agentos.access.mcp # Mate client boundary
└── agentos.access.agentgateway # workload-identity boundary
├── agentos.access.authorization
├── agentos.access.provider_adapter
│ └── agentos.access.credential.release
├── agentos.protocol.operation # MCP when applicable
└── agentos.access.provider # upstream client boundary
First Mate topology and semantic readiness
agentos.topology.decision {action, reason}
└── agentos.resilience.operation
├── agentos.resilience.topology_decision
├── agentos.resilience.workload_plan
├── agentos.resilience.render
├── agentos.resilience.apply
└── agentos.readiness.check
Repair-forward recovery
agentos.resilience.operation
├── agentos.resilience.reconciliation
├── agentos.resilience.provider|listener|protocol|session
└── agentos.resilience.outcome
A retry or failover creates a new provider-attempt span and unique AgentOS
attempt ID under the same operation trace. Stream chunks are aggregated, not
spanned individually. Direct routes never manufacture a Gateway span.
Compaction and memory work is a child of the active operation when possible and
otherwise uses a safe standalone operation or span link. Access spans record a
bounded decision and protocol outcome but never the credential, request body,
MCP arguments or response. They correlate authorization through an opaque
decision reference and profile version; provider identity and provider
resource identifiers remain forbidden even on protected signals. Provider
access route, adapter, provider and terminal outcome use finite contract enums;
Mate, Assignment, decision and profile identifiers remain span-only and never
become metric labels. A valid W3C traceparent joins the native AgentGateway,
authorizer and provider-adapter spans without granting authority. Topology and
recovery spans correlate proposal, Agent, Assignment, Kubernetes object,
protocol and native-session identifiers only in protected traces and logs;
metrics receive only their explicitly declared bounded projection.
Repair-forward creates new phase evidence under the same resilience operation
instead of rewriting or erasing the failed attempt.
Contract metrics are monotonic counters unless noted:
| Metric | Unit/type | Allowed labels |
|---|---|---|
agentos.ai.operations |
{operation} counter |
runtime, route, request kind, model family, status class, error class |
agentos.ai.provider.attempts |
{attempt} counter |
runtime, route, request kind, compaction path, model family, status class, error class |
agentos.ai.operation.duration |
s histogram |
runtime, route, status class |
agentos.ai.provider.duration |
s histogram |
runtime, route, request kind, compaction path, status class |
agentos.ai.upstream.headers.duration |
s histogram |
route, status class |
agentos.ai.stream.first_byte.duration |
s histogram |
route |
agentos.ai.stream.duration |
s histogram |
route, stream outcome |
agentos.ai.streams.active |
{stream} up/down counter |
route |
agentos.ai.stream.chunks |
{chunk} counter |
route, stream outcome |
agentos.ai.stream.bytes |
By counter |
route, stream outcome |
agentos.ai.streams |
{stream} counter |
route, stream outcome |
agentos.ai.route.acquire.duration |
s histogram |
route, status class |
agentos.ai.route.events |
{event} counter |
route, route operation, status class, error class |
agentos.ai.route.reservations.active |
{reservation} up/down counter |
route |
agentos.ai.quota.observation.age |
s histogram |
route, stale boolean |
agentos.ai.quota.refreshes |
{refresh} counter |
route, quota outcome, error class |
Duration histograms use seconds with boundaries 0.005, 0.01, 0.025,
0.05, 0.1, 0.25, 0.5, 1, 2.5, 5, 10, 30, 60, 120 and
300. Backend-derived dashboards may calculate provider calls per operation
from trace structure; operation IDs are never copied into metric labels.
The remaining metric families are equally contractual:
| Owner | Metric names | Units and purpose |
|---|---|---|
| AI and budget | agentos.ai.tokens, agentos.ai.cost, agentos.ai.budget.events |
{token}, {USD} and {event} counters for normalized usage, modeled catalog cost and bounded budget-state transitions |
| memory | agentos.memory.operations, agentos.memory.operation.duration, agentos.memory.candidates, agentos.memory.attachments, agentos.memory.attached.bytes, agentos.memory.index.age |
operation counters plus s, {candidate}, {attachment}, By and s histograms; query, topic and embedding values are never labels |
| access | agentos.access.decisions, agentos.access.decision.duration, agentos.access.revocation.duration, agentos.access.profile_reload.duration, agentos.access.credential.releases, agentos.access.provider.operations, agentos.access.protocol.operations |
decision/release/provider/operation counters and s histograms scoped only by bounded operation, route, adapter, provider, decision, reason, dependency, credential outcome, terminal outcome and protocol dimensions |
| protocol | agentos.protocol.operations, agentos.protocol.operation.duration, agentos.protocol.fallbacks |
{operation} and {fallback} counters plus an s histogram for bounded HTTP, MCP, ACP and A2A outcomes |
| topology, readiness and recovery | agentos.topology.decisions, agentos.readiness.checks, agentos.resilience.observations, agentos.resilience.operations, agentos.resilience.operation.duration |
{decision}, {check}, {observation} and {operation} counters plus an s histogram; proposal, Agent, Assignment, Pod, PVC, session, protocol and digest identifiers are excluded |
| Collector pipeline | agentos.telemetry.pipeline.events, agentos.telemetry.queue.size, agentos.telemetry.export.duration, agentos.telemetry.dropped.batches |
{event} and {batch} counters, {batch} up/down counter and s histogram for bounded signal, stage, outcome and reason dimensions |
The executable metric registry is the exact name-to-owner, instrument, UCUM unit, label and bucket table. Runtime helpers project attributes through that specific metric definition; passing a generally metric-safe attribute is not sufficient. Unknown metric names receive no AgentOS labels. Count histograms use the published count buckets, byte histograms use the published byte buckets, and all wall-time and age histograms use seconds.
The executable event registry applies the same rule to structured logs and audit projections. It owns each event name, domain, signal, exact attribute set and value semantics. Version 1 includes bounded AI Gateway and compaction failures, memory degradation and forgetting, access decisions and credential release, protocol fallback, topology decisions, readiness transitions, resilience observations and telemetry-export failures. An unknown event receives no AgentOS attributes; an attribute that is safe for some log or audit is still rejected unless that exact event declares it.
Signal ownership is explicit:
- resources describe deployment-controlled service, Fleet, Kubernetes and runtime identity; resource enrichment cannot add content-bearing fields;
- spans describe causal work using only the published span tree and bounded operational outcomes, with protected opaque correlation when needed;
- metrics are aggregate operational evidence and accept only the exact labels declared for that metric;
- logs are registry-defined bounded state transitions or failures correlated by trace and span ID; arbitrary messages, serialized errors and console-shaped payloads are not telemetry contracts; and
- audit records remain durable domain evidence in the authoritative PostgreSQL/repository contract. An OTLP audit projection is diagnostic and privacy-bounded; it never replaces the source record or grants authority.
The privacy boundary is an allowlist at both instrumentation and Collector.
After the defense-in-depth known-content denylist, the Collector keeps only the
contract's resource keys, span keys, metric-label union or event-log keys in
the corresponding OTTL context. Scope and span-event attributes are cleared,
as are metric exemplar attributes. Arbitrary log bodies are cleared because
structured event fields—not message text—are the log contract. A metrics-only
Collector processor additionally deletes every contract field classified as
restricted or unbounded, so a foreign or lagging instrument cannot turn a
protected correlation value into a remote metric label. Instrumentation still
enforces the stricter exact per-metric and per-event projection before export.
Telemetry never includes prompts, system prompts,
transcripts, message content, database query text, request or response bodies,
tool arguments or results, memory queries, snippets, bodies or embeddings,
Assignment briefs, inbox bodies, repository names, authorization values,
cookies, API/OAuth keys, OTLP exporter headers, provider credentials or account
identity, arbitrary upstream error bodies or content-bearing stack traces.
HTTP headers are never captured wholesale. Provider
x-request-id or x-oai-request-id may be copied only into a capped protected
span/log field. A unique AgentOS attempt value may be sent as
x-client-request-id where the provider contract supports it; long-lived
Codex thread identity is kept separate.
public_operational and operational attributes may be exported only on their
declared signals. restricted or unbounded fields are permitted only for the
declared protected correlation signals and are categorically excluded from
metrics. Cardinality is assessed on possible distinct values, not string
length: a capped UUID is still unbounded. Instrumentation tests seed every
forbidden key, metric tests assert exact per-instrument labels, Collector config
tests compare every overlay with the executable signal, event, metric,
forbidden and protected catalogs. The live OTLP outage test proves that known
forbidden fields, unknown safe-looking attributes, arbitrary log bodies and
dynamic Mate identity do not reach the remote sink while bounded route,
runtime and status attributes do.
AgentOS extracts and injects W3C traceparent and tracestate. Baggage is
disabled unless a reviewed key allowlist exists; arbitrary inbound baggage and
internal route metadata are stripped. Malformed or oversized context is
discarded and replaced without failing inference. Propagation and telemetry
recording failures are always fail-open.
Workloads consume the official OpenTelemetry environment surface:
OTEL_SERVICE_NAME, OTEL_RESOURCE_ATTRIBUTES,
OTEL_EXPORTER_OTLP_ENDPOINT and signal-specific endpoint overrides,
OTEL_EXPORTER_OTLP_PROTOCOL, OTEL_EXPORTER_OTLP_COMPRESSION,
OTEL_EXPORTER_OTLP_TIMEOUT, OTEL_EXPORTER_OTLP_HEADERS,
OTEL_TRACES_SAMPLER, OTEL_TRACES_SAMPLER_ARG,
OTEL_PROPAGATORS=tracecontext,baggage and the signal exporter selectors.
OTEL_SDK_DISABLED=true is the explicit disabled mode. Secret exporter headers
are sourced from Secrets and never rendered in ConfigMaps or diagnostics.
Codex does not enable exporters from these variables by itself, so AgentOS
atomically reconciles its native [otel] configuration from the same surface
and forces log_user_prompt=false. The rendered Crewmate gives the complete
surface to both prepare-home and the long-running runtime container; this is
required because preparation writes the native config before Codex starts.
Compatibility is verified against the Fleet-pinned codex-cli 0.144.5 and
the peeled rust-v0.144.5 upstream source commit
87db9bc18ba5bc82c1cb4e4381b44f693ee35623. Codex continues to own its native
spans, metrics, logs and W3C propagation; AgentOS does not fork or wrap its
provider client. The bridge preserves unrelated user configuration and maps
OTEL_SDK_DISABLED=true to none for all three native exporters, including
the otherwise default Statsig metrics path. It always writes the metrics
exporter explicitly, so Codex never falls back to that default. AgentOS leaves
Codex's separate [analytics] product preference untouched: pinned 0.144.5
uses that preference as the gate for every metrics exporter, so an operator's
explicit analytics.enabled=false also deliberately disables native Codex
metrics while logs and traces continue. This coupling is a pinned-upstream
compatibility limitation, not an AgentOS fork point.
Native Codex resources do not honor the complete generic resource string.
The Collector therefore extracts deployment-controlled runtime/version labels,
adds Fleet/cluster/namespace/Pod/container identity, maps the owning Kubernetes
controller to k8s.workload.name, and only then applies privacy and contract
allowlists. Before raw codex.api_request attributes are discarded, the
Collector projects bounded request kind, status class, sanitized error class,
HTTP status and a capped upstream request ID. Duration remains available in
Codex's native request-duration metric. Raw errors, prompts, message bodies,
tool payloads, credentials and exporter headers never cross the sink boundary.
The pinned support matrix and executable evidence are recorded in
docs/conformance/codex-native-otel.md.
Collector persistent sending queues retain accepted batches only until remote delivery or bounded queue eviction. The optional local diagnostic archive is off by default, size/time rotated on the Collector PVC and never a supported query backend. Authoritative audit events follow the retention and deletion policy of their owning PostgreSQL/repository domain; Collector queues, diagnostic rotation and remote-backend expiry cannot delete or replace that evidence. OTLP audit projections are diagnostic copies. The remote OTLP backend owns its own searchable retention and access control. Collector queue exhaustion or storage failure may drop telemetry but cannot consume the AI Gateway volume or change inference health.
Kubernetes liveness means the pod runtime is technically alive. An agent waiting for a human, hitting a model limit, or losing provider access must remain attachable and must not be restarted merely to hide the failure.
Readiness may report a required agent as degraded after supported retries classify a provider, quota or rate-limit failure.
Ordinary blocked status is not enough to fail readiness.
First or Second Mate inspects Kubernetes and Herdr on demand and decides whether to wait, attach, change model or credentials, restart a process, or take another recovery action.
There are no heartbeats, automatic retry loops or database-maintained liveness mirrors. A Mate inspects current state and invokes native recovery commands when its judgment or the Captain requires them.
PostgreSQL is the durable fleet authority for at least:
- agent identities, hierarchy and roles;
- tasks and backlog;
- inbox requests, questions, decisions, replies and read state;
- durable status, handoff and coordination history;
- curated, evidence-backed Fleet learnings;
- pod and PVC locators;
- schema and release metadata required for safe recovery.
Every agent receives a PostgreSQL identity.
Agents use SQL or psql directly; AgentOS does not add a database wrapper service.
First Mate uses the Fleet owner role and is therefore the administrator of the
Fleet database and released AgentOS schema. This does not require PostgreSQL
cluster SUPERUSER, CREATEDB, CREATEROLE or BYPASSRLS privileges.
The migration chain runs as that Fleet-owner login, creates or adopts the root
First-Mate row and binds it to the same session_user; there is no separate
migrator identity or manual First-Mate mapping. The authorization migration
binds each remaining Agent to an existing,
non-privileged PostgreSQL session_user and applies grants plus Row-Level
Security to the Fleet tables. Every active registered Agent receives the same
complete read view with no hidden rows. Writes remain narrower: First Mate can
administer the Fleet, Second Mates their subtrees, and Crewmates themselves;
Agent-authored Inbox delivery requires an authentic sender and a direct
parent-child recipient, then content follows sender, recipient and immutability
rules. Fleet tables
without a reviewed runtime write policy remain mutable only by First Mate as
owner. Released acceptance Functions let Mates atomically create the first
Assignment inside their managed hierarchy; assigned Crewmates update only work
state.
Completed Assignments are immutable. Retiring an Agent is an explicit function
that refuses active Assignments or active child Agents, so handoff is never an
automatic cascade. Recording hierarchy alone is not authorization, and
migrations never create login credentials.
Messages may be edited by their sender until first read and become immutable
afterward; corrections are follow-up messages. The recipient calls
agentos.receive_inbox to atomically load the row and set read_at; First Mate
retains owner-level administrative repair. read_at means the delivery entered
the recipient's model context, while resolved_at means its requested action or
disposition was durably handled. A read but unresolved row therefore remains
recoverable work after a crash.
Transactional Fleet triggers publish small table-and-operation hints on a
deterministic non-secret channel for each responsible persistent Mate. A
current Pi Mate readiness-gates only its targeted native pg-listen wait, then
catches up through the read-only durable bearings
projection before relying on the one-shot listener. Deterministic SQL maps
Inbox recipients, Assignment owners, hierarchy edges and external-event
claims; an unowned or unresolved edge falls back to First Mate
instead of waking every current Mate. The detailed re-arm and catch-up judgment
belongs to $agentos-supervision; the wake contains no Fleet row data and
LISTEN/NOTIFY never starts a pod or replaces Inbox, Task or external-event
truth. The same generic tool may own additional native blocking commands for
selected Kubernetes resources, Herdr state transitions or bounded pane output;
their completion is only a wake to re-query the named authority. Terminal output
is never a generic wake path.
One PostgreSQL database is one Fleet. Core tables therefore carry no fleet_id; a developer who intentionally needs an isolated second Fleet creates another database. Released objects live in the agentos schema. The local schema is an approved First-Mate playground whose objects are never treated as released AgentOS behavior until they return through a reviewed migration.
The initial durable model stays deliberately small:
- each persistent Mate's PVC stores its own private, typed memory; it is fallible context, not PostgreSQL authority;
agentsstores hierarchy, role and runtime locators, but not Kubernetes or Herdr health;projectsstores non-exclusive work scopes without assigning one permanent owner;tasksstores durable backlog and accepted outcomes, dependencies and its small array of external tracker links;task_assignmentsprotects active Agent-to-Task relationships and makes completed assignment history immutable;inboxstores durable delivery to an Agent under the closed speech-act vocabulary defined by released SQL. It is not a raw model or terminal transcript. A request becomes accepted execution only when the Task's first accountable Assignment exists;learningsstores curated, evidence-backed Fleet knowledge;external_eventsstores external deliveries and their reconciliation state.
All core rows have immutable identifiers and created_at values plus a PostgreSQL-maintained updated_at. Status values always carry explanatory status text. Core history is archived or retired rather than hard-deleted, and foreign keys restrict removal while durable relationships remain.
External providers are bidirectional human surfaces, not independent fleet authorities and not one-way projections. A GitHub or Linear comment, description edit, status change, assignment or related action is untrusted human intent until a responsible First or Second Mate reconciles it with Fleet state. Provider authentication proves who performed an action and what that account could do in the provider; it does not by itself grant authority over an AgentOS project scope.
Every accepted delivery is appended to external_events with the complete provider payload in raw jsonb. AgentOS adds only routing and coordination columns such as provider, delivery identifier, event type, actor identifier, coalescing key, batch identifier and reconciliation status. It does not strip or normalize the payload into a lossy internal event format. Agents project only the JSON fields needed by the current SQL query; a GIN index supports containment and JSON-path lookup without loading every payload into model context.
Ingress persists first and never invokes a model directly. Deliveries for the same provider resource share a coalescing key. A short quiet window combines a burst of related actions, while a hard maximum window makes the batch eligible after 30 seconds even if events continue arriving. Exact provider delivery identifiers are idempotent.
Batch coordination also stays in external_events; there is no second reconciliation table. A short SQL transaction claims all currently pending rows for one coalescing key with the authenticated Mate's session_user, expiry and opaque fencing token, then commits immediately. A caller-supplied Agent ID can never impersonate another Mate. No database lock remains open while a model reasons. Only First and Second Mates may own reconciliation; a cheaper delegated Agent may help inspect a claimed set but cannot complete it.
New events for a claimed key remain durable and pending. Before an external effect or final commit, the owner checks for them and absorbs them into the claim, then re-evaluates from the last successfully reconciled state instead of chasing an obsolete batch. If the owner disappears, the claim expires and another eligible Mate can reclaim every unresolved event with a new token. The former owner is fenced: its old token cannot complete anything. This recovery does not depend on the First Mate remaining alive.
The final local mutation is one short PostgreSQL transaction. It updates all coupled Tasks and Inbox rows and completes the claimed external rows together; a stale token or a newly pending event aborts the whole transaction. Model reasoning and provider commands never run while that transaction is open.
Agents invoke provider tools such as gh-axi directly and synchronously instead of writing an AgentOS outbox. They observe the actual exit status and remain responsible for failure and recovery in their persistent harness session. Cross-system atomicity is impossible: after a successful provider command and before the local transaction, a crash can still occur. State-setting provider operations should be idempotent; non-idempotent operations such as comments use a deterministic action identifier when supported. Recovery checks the already persisted webhook payloads first and queries the provider only when local evidence is inconclusive, avoiding routine duplicate API and model work.
The exact tables, indexes, Functions, Triggers, grants, RLS policies, retention
and immutability guarantees are defined only by versioned SQL and SQL tests
after schema review.
The database/ workspace uses Drizzle Kit only to create and
apply journaled custom SQL migrations; it has no Drizzle ORM schema or runtime
database client.
This document does not duplicate them.
PostgreSQL may run at a developer-selected external or managed endpoint or be self-hosted in the selected cluster. AgentOS treats both as complete paths and does not rank one ahead of the other. For self-hosting it uses CloudNativePG rather than maintaining a raw PostgreSQL StatefulSet. A compatible existing controller can be reused; installing or upgrading the cluster-scoped controller requires explicit approval. When a compatible installed controller is older than the newest stable release, First Mate presents reuse and upgrade as choices.
The released database manifest is version-neutral. At installation time First
Mate finds the newest stable official CloudNativePG release compatible with the
selected Kubernetes server and the newest stable PostgreSQL operand it supports,
verifies immutable image identities, and shows the exact selection before asking
for approval. It then renders one PostgreSQL instance with a 20 GiB PVC, data
checksums, the unprivileged agentos application owner and no network-enabled
superuser. This minimal topology is not HA and has no reviewed backup policy yet.
CNPG generates the application Secret; First Mate uses its pgpass entry with
psql and injects its URI only into the Drizzle migration process. Pinned
migration dependencies are installed from bun.lock into a content-addressed
workspace on the agent PVC when first needed, keeping them out of the runtime
image. After CNPG is Ready, one role-owned additive strategic patch wires its
application identity and private CA into the already running First Mate. This
explicit stage-two handoff preserves the selected image and unrelated workload
state, requires approval for the intentional Pod restart, and is not a second
database topology or release manifest. External PostgreSQL receives only an
installation-specific equivalent after its endpoint, TLS and Secret shape are
known. Topology does not change schema or security semantics.
Kubernetes owns pod existence, phase, readiness and failure state. Each workload uses an explicit namespace, ServiceAccount, PVC mapping and immutable AgentOS image reference.
Persistent Mates that supervise child Pods use the kubelet-rotated projected
ServiceAccount identity mounted by their dedicated Pod. First Mate provisions
one labeled domain namespace per persistent Second Mate and binds that Second
Mate to a reviewed namespaced workload Role. The Role permits native creation
and operation of Crewmate StatefulSets, Services and ServiceAccounts plus Pod
inspection, replacement and approved exec, but excludes Namespace, Secret,
RBAC, quota, LimitRange, NetworkPolicy and cluster-scoped mutation. Kubernetes
RBAC cannot constrain create by a future resource name, so namespace
ownership replaces the previous exact-child-name model. A Second Mate may mount
any Secret placed in its namespace through a child workload even without
get secrets; First Mate therefore keeps Fleet-root credentials in agentos
and places only domain-visible credentials in a domain namespace.
The domain Namespace enforces the restricted Pod Security profile, carries a bounded object, CPU, memory and retained-storage ResourceQuota plus per-container and per-claim LimitRange, and applies ingress isolation that admits only same-namespace Pods. One First-Mate/platform-owned cluster admission bundle uses stable CEL ValidatingAdmissionPolicy and namespace-selected bindings to protect every labeled managed domain. A Second Mate may create and update only one-replica retained Crewmate StatefulSets with matching Agent, owner, Task and Assignment labels, a dedicated ServiceAccount with child token automount off, digest-pinned remote images, bounded requests and limits, restricted security and no host access. It cannot mutate its own persistent Mate workload or any of the namespace, RBAC, quota, LimitRange, NetworkPolicy or cluster admission controls. The bundle is installed separately from each namespaced domain render because its policies and bindings are cluster-scoped.
The domain deliberately defines no egress policy: Agents retain ordinary
internet access. Provider-root credentials never enter a domain; an approved
short-lived native or scoped gateway mechanism remains a separately reviewed
capability. Labeled domain clients reach only explicitly
admitting Fleet services through fully qualified names under
agentos.svc.cluster.local. Sibling namespaces have neither Kubernetes RBAC
nor ingress reachability. First Mate is bound into every domain for credential,
policy, inspect, attach, stop and repair operations. All steady-state access
uses projected ServiceAccount identity; AgentOS never mints or persists a
separate supervision bearer token. This is a Kubernetes supervision boundary,
not the broader provider-credential mediation proposed in the workload-identity
exploration, which remains open.
Before a Crewmate apply, the owning Mate uses native Kubernetes reads to build
the versioned Effect Schema capacity snapshot. It covers namespace quota,
current requested CPU and memory, node allocatable resources and scheduling
constraints, StorageClass binding/topology, retained PVC state and bound-PV
node affinity. The pure classifier returns only fits, provably_blocked or
inconclusive, with bounded evidence and reservation=false. Incomplete or
unsupported observations can never become a fit. Portable versus node-local
storage is explicit; an unbound node-local claim is inconclusive, while a
retained one-writer claim already in use or bound to incompatible node affinity
is blocked. A Second Mate may use its namespace-scoped observations; when a
cluster-scoped fact is required, First Mate supplies the read-only observation
instead of broadening sibling access. Server-side dry-run remains the final
admission check, and the scheduler remains authoritative after apply. A race or
later scheduling failure preserves the same Agent, Task, Assignment, desired
workload and retained PVC for re-observation and repair rather than creating a
duplicate Agent.
StatefulSet deletion and replacement retain Agent homes. Domain retirement removes active workloads and authority but leaves the Namespace while retained PVCs exist. Deleting the Namespace is a separate destructive action because it deletes its PVC objects regardless of StatefulSet retention policy.
After explicit RBAC approval, the default dedicated-cluster path grants First Mate cluster-administrator access so it can inspect and recover agents. Shared or sensitive clusters must offer a scoped mode and explain which recovery operations become unavailable. Workers do not inherit First Mate authority automatically.
Agent runtimes remain ordinary versioned Kubernetes resources; AgentOS does not introduce its own CRDs or autonomous operator. Only the optional self-hosted database path uses the external CloudNativePG CRDs and controller. The optional Fleet AI Gateway is an ordinary single-replica StatefulSet and ClusterIP Service; it adds no Kubernetes controller or CRD.
The public README is the human and seed-agent entry point.
A developer gives its short prompt to an existing coding agent without cloning AgentOS first.
That local agent loads agentos-bootstrap, inspects read-only, explains viable paths and asks before credentials, login, cost, cluster creation, RBAC or installation.
Bootstrap has two handoff stages:
- Establish the smallest persistent Kubernetes runtime: Herdr, Pi First Mate, agent PVC, immutable AgentOS revision, attach path and working model authentication.
- Hand control to that First Mate, which selects or provisions PostgreSQL with approval, applies the versioned database assets and verifies the complete fleet identity.
Stage one may come online without PostgreSQL so the temporary seed needs fewer tools and the developer can meet First Mate sooner. That is an incomplete bootstrap boundary, not an alternate AgentOS operating mode. First Mate may inspect, explain and finish provisioning, but it must not accept durable work, create Assignments or delegate a Crewmate until stage two and the released database security checks pass.
The initial model path uses Pi with the developer's Codex subscription through
provider openai-codex. Existing Pi settings on the persistent home remain
authoritative; AgentOS does not seed a release-wide model or thinking level.
Login happens inside the persistent Pi runtime, not by copying a local token directory.
Exact package versions and authentication commands belong to release assets and the auth skill.
Source-provider authentication is a separate choice. Personal native gh
login remains complete; teams may select the GitHub App path defined by the
auth Skill, with its private key mounted only into First Mate and short-lived
installation tokens consumed directly by native provider tools.
Direct First-Mate authentication remains the fastest initial handoff and a
verified recovery path. After Fleet identity exists, First Mate presents two
worker-capacity postures: the recommended Fleet AI Gateway for a delegation-ready
Fleet, or direct authentication owned separately by every worker harness. The
Captain's selection is fallible guidance in the owning Mate's private
context; exact approval and coupled state changes remain durable Inbox
decisions. Installing the gateway, creating its operator Secret, starting
provider login and enabling a workload's access profile remain explicit or
standing authorizations; recommendation never makes them implicit. Bootstrap
may finish in minimal single-Mate mode
without the gateway, but AgentOS does not call the Fleet delegation-ready
until one approved worker or trusted harness automation has completed a
harmless real model request through the selected capacity path.
For a stable install, the seed resolves the latest published GitHub release,
verifies that release is immutable, and applies only its fixed-name assets
under that versioned release URL. An explicitly chosen preview instead uses an
exact Git commit, a locally rendered manifest and an immutable OCI digest; it
never installs from a branch URL or mutable tag.
External database dependencies are discovered and verified by First Mate when the developer selects self-hosting.
The default manifest grants First Mate administration within agentos;
the separately named dedicated-cluster manifest adds cluster-administrator
access only after explicit approval.
An existing Kubernetes cluster is fully supported. Akua Zero-to-Cluster is an optional path selected by the developer, never an implicit dependency or contact.
Repository-development and operational resources have separate discovery boundaries:
.agents/skills/contains repository development, evaluation and improvement workflows applicable from AgentOS source checkouts;packages/agentos/skills/is the default distribution's Pi-declared shared operational Skill root;packages/agentos/resources/roles/firstmate/skills/contains the default First-Mate-only workflows; the selected entrypoint adds them only to First Mate; and- another distribution declares its own Skill roots through its ordinary Pi package manifest.
The public README points at BOOTSTRAP.md, which forwards to the canonical
First-Mate bootstrap Skill in the default distribution. Bootstrap explicitly
loads other Skills when it reaches their boundary; normal Mate sessions can
load the same declared Skills independently.
Persistent First and Second Mate run from the selected distribution's exact
role directory in the retained Git checkout. Pi starts with
--no-context-files, so repository, ancestor and role-local AGENTS.md or
CLAUDE.md files do not become operational identity by accidental discovery.
The distribution's single entrypoint validates AGENTOS_AGENT_ROLE, injects
exactly one role's versioned instructions.md, and exposes only that role's
resources. The role's native .pi/settings.json selects the distribution
package through Pi's package system.
Repository AGENTS.md files remain contributor boundaries. A Mate changing
AgentOS source loads $agentos-development, which explicitly reads the
applicable contributor instructions before mutation. Operational identity and
repository-development instructions therefore remain independently
replaceable and testable.
Crewmates differ: their harness working directory is the isolated project
workspace, normally a Treehouse worktree and optionally a reviewed ArtifactFS
mount for an eligible Scout. The owning Mate renders the selected
distribution's resources/crewmates/default/BRIEF.md; the project's own
AGENTS.md then supplies codebase instructions without becoming the Fleet role
contract.
The repository root is both the public AgentOS product surface and one Bun workspace. The tree reflects ownership, not deployment order:
.
├── README.md product story and onboarding entrypoint
├── VISION.md direction, priorities and non-goals
├── ARCHITECTURE.md system boundaries and repository map
├── BOOTSTRAP.md stable pointer to the bootstrap Skill
├── CUSTOMIZATION.md public AgentOS customization orientation
├── CONTRIBUTING.md contributor setup and verification
├── AGENTS.md identity-neutral repository rules
├── .agents/skills/
│ ├── agentos-development/ workflow for changing AgentOS itself
│ ├── effect-ts/ repository Effect conventions and references
│ ├── agentos-evaluation/ benchmark execution and evidence
│ └── agentos-improvement-review/ reviewed learning from frozen evidence
├── benchmarks/
│ ├── SPEC.md portable metrics and reporting rules
│ ├── schemas/ scenario and evidence JSON contracts
│ ├── scenarios/ versioned portable evaluations
│ └── profiles/agentos/ AgentOS authority and evidence mapping
├── packages/
│ ├── AGENTS.md importable-package boundary
│ └── agentos/ public API and Pi-native distribution
│ ├── extensions/agentos.ts sole Pi-discovered AgentOS entrypoint
│ ├── extensions/agentos-observability.ts
│ │ explicit diagnostic-only entrypoint
│ ├── src/ inert API, registrations and role setups
│ ├── runtime/ lifecycle, K8s, image seed and tests
│ ├── skills/ shared operational Skills
│ └── resources/
│ ├── roles/ role identity, Skills, Mise and K8s
│ └── crewmates/default/ brief, image and workload assets
├── clis/
│ ├── AGENTS.md admission boundary for shipped commands
│ ├── github-app-token/ short-lived installation-token minting
│ └── pg-listen/ one-notification PostgreSQL primitive
├── database/
│ ├── AGENTS.md SQL-first schema-development boundary
│ ├── kubernetes/cloudnative-pg/ optional self-hosted PostgreSQL topology
│ ├── migrations/ released schema and authorization truth
│ ├── runtime/ deterministic migration preparation
│ └── tests/ behavioral PGlite contract tests
├── services/
│ ├── AGENTS.md admission boundary for optional services
│ ├── ai-gateway/ service, tests and optional K8s topology
│ └── otel-collector/ Collector topology, overlays and tests
├── release/kubernetes/ reviewed manifest release assembly
├── tooling/effect-migration/ Effect inventory, policy, examples and CI gate
├── docs/ supporting assets and stable pointers
├── website/apps/docs/ public landing, Docs and Learn application
├── Dockerfile common AgentOS image build
├── mise.toml / mise.lock reviewed Fleet tool baseline
├── package.json / bun.lock workspace and dependency lock
└── THIRD_PARTY_* redistributed-license obligations
- Put repository contributor boundaries in the nearest
AGENTS.md. Persistent operational identity belongs in the selected distribution's explicit injected role instructions, never a contributor file discovered by accident. - Put conditional operational judgment in one Pi-declared Skill root owned by the narrowest distribution scope shared by every intended role.
- Put a small executable in
clis/<name>/only when a reviewed native tool lacks that primitive. A CLI must not hide capable tools, Agent policy or shadow state. - Put the inert public Pi API, released entrypoint and operational resources in
packages/agentos/. Keep activation explicit through Pi's package settings. - Put a reviewed optional long-running network capability in
services/<name>/only when native tools cannot safely provide its cross-Pod state and request lifecycle. Keep it authenticated, independently testable and outside Fleet coordination authority. - Put retained-home, Pod-security and role-neutral Herdr mechanics shared by
persistent Agents in
packages/agentos/runtime/kubernetes/base/. Add the persistent Pi Mate lifecycle inpackages/agentos/runtime/kubernetes/mate/; put role identity, credentials, RBAC, harness choice and role-specific probes in the selected distribution's role resource directory. - Put released database objects, authorization and transactional coordination
in SQL migrations under
database/. - A deployable component owns its implementation, behavior tests and
Kubernetes shape. Skills and RBAC define who may operate it. Role-specific
workload manifests remain under the owning distribution role; release
assembly belongs under
release/. - Put project orientation in
README.md, direction and product bets inVISION.md, architectural decisions here and contributor procedure inCONTRIBUTING.md. Link across those owners instead of copying a workflow. - Keep generated release artifacts and local review state out of source ownership. Their generator, immutable release or ignored workspace remains authoritative.
The default distribution's executable First/Second-Mate lifecycle, image-seed
builder, observable tests and shared native runtime assets live together under
packages/agentos/runtime/; this is not an agent role, external CLI or generic
importable runtime package. Its kubernetes/base/ contains only semantics
common to persistent Agents, while kubernetes/mate/ adds Pi and Mate health
behavior. Each role owns its Kubernetes workload patch and surrounding
ServiceAccount, Service, identity, credentials, harness choice and authority
under its distribution role.
Stateless workers do not inherit the retained-home base. Optional component
topology stays with that component even when First Mate is its normal operator.
There is no speculative CLI or placeholder application. A real missing native
primitive may enter clis/<name>/ through the admission boundary in
clis/AGENTS.md; reusable imported code belongs in packages/<name>/. The
workspace keeps one bun.lock.
README.mdcontains product orientation and the copyable onboarding prompt.VISION.mdcontains project direction, current priorities, product principles and explicit non-goals.ARCHITECTURE.mdcontains canonical system boundaries, architectural decisions and repository placement rules.- Root
AGENTS.mdowns identity-neutral repository boundaries and instruction-placement rules. CONTRIBUTING.mdcontains repository setup, development conventions and disposable-cluster smoke testing.BOOTSTRAP.mdpoints to the canonical First-Mate bootstrap skill without duplicating its procedure.CUSTOMIZATION.mdexplains trusted AgentOS customization and points to its canonical operational Skill without duplicating the procedure.docs/architecture.mdis a compatibility pointer to this document, not a second architecture source..agents/skills/agentos-development/contains the repository-development workflow shared by contributors and every Agent role working on AgentOS..agents/skills/effect-ts/contains the repository's Effect conventions and references; Effect code must follow its pinned guidance and source checkout..agents/skills/agentos-evaluation/owns benchmark execution and sanitized evidence collection without changing the measured subject..agents/skills/agentos-improvement-review/owns causal review and the smallest reviewed change after evidence is frozen.benchmarks/SPEC.mdowns portable evaluation semantics, metrics, gates and reporting; its schemas, scenarios and AgentOS profile own their corresponding machine-readable or product-specific contracts.packages/agentos/src/owns the inert public API and released Pi behavior; importing it registers nothing and touches no external authority.packages/agentos/extensions/agentos.tsis the default distribution's sole Pi-discovered AgentOS entrypoint.packages/agentos/skills/contains operational workflows shared by First and Second Mate without exposing them to contributor or runtime-development sessions.packages/agentos/skills/agentos-memory/owns private, fallible per-Mate memory behavior, including recall, maintenance, session privacy and routed proposals; it never replaces Fleet authority.packages/agentos/skills/agentos-customization/owns trusted package and complete-distribution inspection, design, staging, native selection, reload or workload rollout, verification and rollback for additive and replacement AgentOS customizations.packages/agentos/skills/agentos-upgrade/owns exact stable one-Mate release updates, run-once released-database sequencing and First-Mate-only frozen-roster Fleet order; its conditional references keep database, atomic declarative/live member and Fleet mechanics separate.packages/agentos/resources/roles/contains the default persistent role instruction surfaces, native Pi selection, role-scoped Skills, Mise and Kubernetes resources.packages/agentos/resources/crewmates/default/BRIEF.mdis the canonical default bounded-worker contract rendered into each Assignment brief.packages/agentos/resources/crewmates/default/images/owns optional task-specific worker images; it never expands the common Mate image or grants runtime permissions by implication.clis/AGENTS.mdadmits only narrow executable primitives and rejects wrappers, policy and shadow state.clis/<name>/owns each admitted command's implementation, package dependencies and behavior tests.services/AGENTS.mdadmits only explicitly reviewed optional network processes and rejects native-tool wrappers, prompt queues and shadow Fleet state.services/ai-gateway/owns the optional authenticated Fleet AI request data plane, its private credential vault, canonical routing database, behavior tests and Kubernetes topology; it does not own harness choice, PostgreSQL state or general Fleet traffic.database/AGENTS.mdgoverns SQL-first schema development without selecting an Agent role.database/kubernetes/cloudnative-pg/is authoritative only for the optional self-hosted CloudNativePG topology; it does not own SQL schema, controller installation or third-party version selection.packages/agentos/runtime/AGENTS.mdgoverns shared container lifecycle mechanics without selecting an Agent role.packages/agentos/runtime/kubernetes/base/owns only the retained-home StatefulSet mechanics shared by persistent Agents.packages/agentos/runtime/kubernetes/mate/owns the Pi andmate:*lifecycle shared by First and Second Mate.packages/agentos/resources/roles/*/kubernetes/andpackages/agentos/resources/crewmates/default/kubernetes/are authoritative for the default distribution's role-owned Kubernetes patches and surrounding resources. A role-owned client patch wires that workload to a component; it does not move the component's topology into the role subtree.services/ai-gateway/kubernetes/is authoritative for the optional single-replica gateway topology; its Secret values and local overlays are not repository state.services/otel-collector/is authoritative for the optional Fleet-local Collector topology, persistent buffering and explicit export overlays.services/ai-gateway/package.jsonpins the publicakua-dev/codex-routerroot Git package by full commit SHA. Its/core,/codex, and/bunentry points own shared routing policy and runtime mechanics; shared behavior changes upstream before the pin is updated.release/kubernetes/is authoritative for human-readable First-Mate and database manifest rendering; stable generated assets belong to immutable GitHub releases, while previews remain exact-commit builds.packages/agentos/runtime/owns the default distribution's executable First/Second-Mate lifecycle, shared persistent-Agent Kubernetes mechanics, image-seed creation and observable runtime tests; it does not select an Agent role.database/migrations/and its Drizzle migration journal are authoritative for database semantics, security and applied order;database/drizzle.tooling.tsis deliberately empty and non-authoritative.- Release assets pin exact versions, digests and checksums.
THIRD_PARTY_NOTICES.mdandTHIRD_PARTY_SOURCES.mdare authoritative for redistributed third-party licensing and source offers.
When documentation and executable assets disagree, stop and reconcile them; do not invent missing production mechanics from prose.
AgentOS remains MIT licensed. Herdr remains a separate AGPL-3.0-or-later or commercially licensed program. The open-source distribution path uses an unmodified, pinned Herdr executable through documented CLI and socket interfaces and ships the corresponding license, notice and source offer. Patching, linking, embedding Herdr source, or otherwise tightening that boundary requires a fresh license review before publication.
AgentOS does not introduce autonomous schedulers, heartbeat infrastructure, AgentOS-specific Kubernetes CRDs or operators, a PostgreSQL wrapper API, prompt queues, universal traffic proxies, transcript-capturing AI gateways, task-specific PVCs or mandatory semantic indexing.