Make Fleet AI Gateway safe for Hermes clients

ARCHITECTURE.md

@@ -474,6 +474,25 @@ path. Enabling the route is an explicit configuration change; it
 never selects or changes a provider or model. `quota-axi` remains an
 observation-only tool and has no routing, login or mutation authority.
 
+An approved Responses client that cannot supply command-backed authentication
+uses the AgentOS `ai-gateway-workload-proxy` as a loopback-only sidecar; the
+exact Hermes contract and Pod wiring live in
+[`docs/integrations/hermes-ai-gateway.md`](./docs/integrations/hermes-ai-gateway.md).
+The sidecar rereads projected workload identity for each request, removes the
+client's placeholder authorization and all caller-supplied AgentOS authority
+headers, optionally injects a validated Assignment ID from its own trusted
+configuration, and forwards only the two reviewed Responses paths once to
+Agentgateway. It preserves the upstream status, end-to-end headers after
+required hop-by-hop and stale `Content-Encoding`/`Content-Length` metadata
+removal, and stream without following redirects. W3C trace correlation passes
+through unchanged. Its
+process-only liveness and local projected-token readiness never contact an
+upstream. It owns no credential,
+selection, retry, fallback, prompt, response, or deployment authority. The
+client keeps exact model selection and disables request replay and provider
+fallback for this route; its workload owner composes the sidecar and projected
+token into that client's reviewed native manifest.
+
 This is not a universal AgentOS proxy. Git, PostgreSQL, Kubernetes, Herdr,
 registries and other provider tools continue through their native interfaces.
 Adding another mediated protocol requires a separate review of its authority,

Dockerfile

@@ -225,6 +225,7 @@ RUN chmod 0644 \
     /opt/agentos/services/a2a/src/main.ts \
     /opt/agentos/services/agentgateway/src/readiness-main.ts \
     /opt/agentos/services/ai-gateway/src/main.ts \
+    /opt/agentos/services/ai-gateway/src/workload-client-proxy-main.ts \
     /opt/agentos/services/egress-authz/src/main.ts \
     /opt/agentos/services/github-broker/src/main.ts \
     /opt/agentos/services/openfga/src/bootstrap.ts \
@@ -247,6 +248,9 @@ RUN chmod 0644 \
   && ln -s \
     /opt/agentos/services/ai-gateway/src/main.ts \
     /usr/local/bin/ai-gateway \
+  && ln -s \
+    /opt/agentos/services/ai-gateway/src/workload-client-proxy-main.ts \
+    /usr/local/bin/ai-gateway-workload-proxy \
   && ln -s \
     /opt/agentos/services/egress-authz/src/main.ts \
     /usr/local/bin/agentos-egress-authz \

docs/integrations/hermes-ai-gateway.config.yaml

@@ -0,0 +1,20 @@
+# Hermes Agent 0.20.0 (v2026.8.3) deterministic integration fixture.
+model:
+  provider: custom:agentos-gateway
+  default: gpt-exact-fixture
+
+providers:
+  agentos-gateway:
+    name: AgentOS Gateway
+    api: http://127.0.0.1:8790/v1
+    api_key: agentos-workload-identity-placeholder
+    transport: codex_responses
+    discover_models: false
+    models:
+      - gpt-exact-fixture
+
+agent:
+  api_max_retries: 0
+
+fallback_providers: []
+fallback_model: ""

docs/integrations/hermes-ai-gateway.md

@@ -0,0 +1,193 @@
+# Hermes Agent through the Fleet AI Gateway
+
+This contract covers Hermes Agent `0.20.0` from release tag `v2026.8.3` and
+other approved clients that speak the OpenAI Responses API but cannot reread a
+Kubernetes projected token for every request. Direct per-agent OAuth remains
+the recovery path.
+
+> Deployment gate: this Hermes revision is not eligible for this route. Its
+> `codex_responses` runtime hardcodes one retry around `responses.create`, so a
+> transport failure after partial output can replay the request;
+> `HERMES_STREAM_RETRIES=0` does not change that behavior. Do not deploy the
+> fixture or Pod wiring below until a pinned Hermes revision with a verified
+> zero-retry Responses path is available.
+
+## Client contract
+
+The client keeps authority over the exact model and reasoning configuration. It
+sends only `POST /v1/responses` or `POST /v1/responses/compact` to a loopback
+`ai-gateway-workload-proxy` sidecar. The sidecar:
+
+- accepts no provider credential and listens on loopback only;
+- rereads the kubelet-rotated `agentos-egress-authz` ServiceAccount token for
+  every request;
+- removes caller-supplied authorization, assignment, decision, grant, and every
+  other `x-agentos-*` header, then supplies only projected identity and an
+  optional validated assignment ID from sidecar-owned configuration to
+  `agentgateway-openai`; W3C `traceparent` and `tracestate` correlation remain
+  unchanged;
+- forwards the request once, without inspecting or storing its body; and
+- returns the actual upstream status, end-to-end headers after required
+  hop-by-hop and stale `Content-Encoding`/`Content-Length` metadata removal,
+  and stream without retrying, following redirects, or selecting another model
+  or account. A transport failure returns the adapter's stable `502`; it never
+  becomes apparent success.
+
+Agentgateway and `agentos-egress-authz`, not the sidecar, authenticate the Pod
+and authorize its registered Agent or Assignment. Do not expose the loopback
+listener as a Service. Do not copy an AI Gateway client Secret, OAuth file, or
+operator token into the client namespace.
+
+`GET /livez` is process-only. `GET /readyz` rereads the projected token and
+returns ready only when it is present, at most 16 KiB, strict UTF-8, already
+trimmed, and JWT-like. Neither endpoint contacts Agentgateway or a provider.
+
+## Hermes 0.20.0 configuration
+
+Hermes 0.20 supports named providers with `transport: codex_responses`, but a
+configured API key is a process-lifetime value. Point the named provider at the
+loopback adapter and use a non-secret placeholder; never put the projected token
+in `config.yaml` or `.env`.
+
+```yaml
+model:
+  provider: custom:agentos-gateway
+  default: <exact-approved-model>
+
+providers:
+  agentos-gateway:
+    name: AgentOS Gateway
+    api: http://127.0.0.1:8790/v1
+    api_key: agentos-workload-identity-placeholder
+    transport: codex_responses
+    discover_models: false
+    models:
+      - <exact-approved-model>
+
+agent:
+  api_max_retries: 0
+
+fallback_providers: []
+fallback_model: ""
+```
+
+These fields are load-bearing for Hermes 0.20's named-provider resolver. The
+input must select `provider: custom:agentos-gateway`; the matching provider entry's
+`transport: codex_responses` retains the Responses transport, and its `api`
+points that transport at the loopback `/v1` base. Hermes normalizes this named
+input at runtime to the named custom provider with `api_mode: codex_responses`
+and `base_url: http://127.0.0.1:8790/v1`. Putting bare `provider: custom`
+directly in the input falls back to Chat Completions. The checked fixture
+[`hermes-ai-gateway.config.yaml`](./hermes-ai-gateway.config.yaml) is exercised
+against the named input contract and the sidecar allowlist in the Gateway test
+suite.
+
+`api_max_retries: 0` and the empty fallback configuration are part of the
+contract: after Hermes sends a request, it must surface the real `401`, `403`,
+`429`, timeout, or provider failure rather than replaying the turn through a
+route that may acquire another account. Auxiliary requests are out of scope for
+this blocked Hermes revision. Do not enable them or configure `provider: auto`,
+`provider: custom`, or a fallback chain until Hermes provides a verified native
+Responses path with no automatic retry, redirect, or fallback.
+
+## Pod wiring
+
+The owning Hermes deployment supplies this wiring. AgentOS owns the adapter
+binary and the governed backend topology, but does not own or apply a foreign
+Hermes StatefulSet. Add the selected-client label, projected identity volume,
+and loopback sidecar to the reviewed workload manifest:
+
+The current Agentgateway NetworkPolicy admits client Pods from the `agentos`
+namespace, or from an already approved namespace labeled
+`agentos.akua.dev/managed-by: agentos-firstmate`, only when the Pod has an
+`agentos.akua.dev/agent` label. Deploy Hermes in one of those approved
+namespaces and preserve that reachability label. These labels only permit
+network reachability; the projected ServiceAccount token and Agentgateway
+authorizer provide authentication and authorization.
+
+```yaml
+spec:
+  template:
+    metadata:
+      labels:
+        agentos.akua.dev/agent: "hermes"
+        agentos.akua.dev/agentgateway-client: "true"
+    spec:
+      automountServiceAccountToken: false
+      containers:
+        - name: hermes
+        - name: ai-gateway-workload-proxy
+          image: <same-reviewed-agentos-image-digest-as-the-gateway-release>
+          command: ["ai-gateway-workload-proxy"]
+          env:
+            - name: AI_GATEWAY_URL
+              value: http://agentgateway-openai.agentos.svc.cluster.local:8788
+            - name: AI_GATEWAY_IDLE_TIMEOUT_SECONDS
+              value: "255"
+            - name: AI_GATEWAY_GRACEFUL_SHUTDOWN_MILLIS
+              value: "20000"
+            - name: AGENTOS_EGRESS_TOKEN_FILE
+              value: /var/run/secrets/agentos-egress/token
+            # Optional: set only from the workload owner's trusted Assignment.
+            - name: AGENTOS_ASSIGNMENT_ID
+              value: <exact-assignment-uuid>
+          ports:
+            - name: workload-proxy
+              containerPort: 8790
+          livenessProbe:
+            exec:
+              command:
+                - /usr/bin/curl
+                - --fail
+                - --silent
+                - --show-error
+                - http://127.0.0.1:8790/livez
+          readinessProbe:
+            exec:
+              command:
+                - /usr/bin/curl
+                - --fail
+                - --silent
+                - --show-error
+                - http://127.0.0.1:8790/readyz
+          securityContext:
+            allowPrivilegeEscalation: false
+            capabilities:
+              drop: ["ALL"]
+            readOnlyRootFilesystem: true
+            runAsNonRoot: true
+          volumeMounts:
+            - name: agentos-egress-identity
+              mountPath: /var/run/secrets/agentos-egress
+              readOnly: true
+      volumes:
+        - name: agentos-egress-identity
+          projected:
+            defaultMode: 0440
+            sources:
+              - serviceAccountToken:
+                  audience: agentos-egress-authz
+                  expirationSeconds: 600
+                  path: token
+```
+
+Use the workload's existing explicit `fsGroup` so the sidecar can read the
+`0440` projection. Only the sidecar mounts that identity volume; Hermes cannot
+read it. Keep the workload's dedicated ServiceAccount, register that identity
+and its approved access profile through the normal AgentOS access-plane
+procedure, and preserve the existing NetworkPolicy. The label grants only network
+reachability to Agentgateway; it is not authorization.
+
+The proxy's idle timeout defaults to the shown maximum of 255 seconds and its
+graceful-shutdown timeout defaults to 20,000 milliseconds. The idle timeout is
+an inactivity limit, not a total response limit; retain the 255-second value
+when the approved provider stream can have gaps that long. The proxy uses
+manual redirect handling, so a `3xx` response is returned to Hermes rather than
+followed.
+
+## Operator workflow
+
+Use `$agentos-ai-gateway` for the approval, migration, rollout, verification,
+rollback, and retirement workflow, including migration from the live v0.1.24
+shared-token topology. This page owns only the Hermes client contract and Pod
+wiring.

docs/security/workload-identity.md

@@ -11,7 +11,7 @@ AgentOS workloads receive a second, explicit projected ServiceAccount token for
 - requested lifetime: 600 seconds; and
 - volume and mount: `agentos-egress-identity`, read-only with mode `0440`.
 
-This is not the workload's Kubernetes API token. Persistent Second Mates retain their separately managed Kubernetes supervision identity. Crewmates keep `automountServiceAccountToken: false` and receive only the explicit audience-scoped projection. Only the long-running Agent container mounts this token; tool-install and home-preparation init containers do not, and domain admission rejects any Crewmate init-container mount. The kubelet rotates projected tokens, so clients must read the token file for each request rather than retaining its contents for the process lifetime.
+This is not the workload's Kubernetes API token. Persistent Second Mates retain their separately managed Kubernetes supervision identity. Crewmates keep `automountServiceAccountToken: false` and receive only the explicit audience-scoped projection. Only the long-running provider client or a reviewed provider sidecar/adapter mounts this token; tool-install and home-preparation init containers do not, and domain admission rejects any Crewmate init-container mount. The kubelet rotates projected tokens, so the component that presents the identity must read the token file for each request rather than retaining its contents for the process lifetime.
 
 The authorization service built by #90 supplies the raw token only to `KubernetesTokenReviewer`. It requests exactly the dedicated audience and accepts a result only when `authenticated` is true and the same audience appears in `status.audiences`. Kubernetes documents that a TokenReview client setting audiences must check the returned intersection; accepting an empty result would fall back to API-server audience semantics and is forbidden here. See the Kubernetes [TokenReview API](https://kubernetes.io/docs/reference/kubernetes-api/definitions/token-review-v1-authentication/).