From 5eec7dc07ff4b0308ee8e24d5a39b7829eaad2b6 Mon Sep 17 00:00:00 2001 From: "akua-agentos[bot]" <4359249+akua-agentos[bot]@users.noreply.github.com> Date: Mon, 10 Aug 2026 12:36:32 +0000 Subject: [PATCH 01/28] feat(ai-gateway): add Hermes workload proxy --- ARCHITECTURE.md | 14 + Dockerfile | 4 + .../hermes-ai-gateway.config.yaml | 20 ++ docs/integrations/hermes-ai-gateway.md | 170 ++++++++++++ package.json | 2 +- services/ai-gateway/README.md | 10 + services/ai-gateway/package.json | 3 +- .../src/workload-client-proxy-main.ts | 114 ++++++++ .../ai-gateway/src/workload-client-proxy.ts | 215 +++++++++++++++ .../workload-client-proxy.effect.test.ts | 244 ++++++++++++++++++ tooling/effect-migration/exceptions.json | 12 + tooling/effect-migration/inventory.json | 2 +- 12 files changed, 807 insertions(+), 3 deletions(-) create mode 100644 docs/integrations/hermes-ai-gateway.config.yaml create mode 100644 docs/integrations/hermes-ai-gateway.md create mode 100644 services/ai-gateway/src/workload-client-proxy-main.ts create mode 100644 services/ai-gateway/src/workload-client-proxy.ts create mode 100644 services/ai-gateway/tests/workload-client-proxy.effect.test.ts diff --git a/ARCHITECTURE.md b/ARCHITECTURE.md index 8a13e914..c1833e32 100644 --- a/ARCHITECTURE.md +++ b/ARCHITECTURE.md @@ -474,6 +474,20 @@ path. Enabling the route is an explicit configuration change; it never selects or changes a provider or model. `quota-axi` remains an observation-only tool and has no routing, login or mutation authority. +An approved Responses client that cannot supply command-backed authentication +uses the AgentOS `ai-gateway-workload-proxy` as a loopback-only sidecar. The +sidecar rereads projected workload identity for each request, removes the +client's placeholder authorization and all caller-supplied AgentOS authority +headers, optionally injects a validated Assignment ID from its own trusted +configuration, and forwards only the two reviewed Responses paths once to +Agentgateway. W3C trace correlation passes through unchanged. Its process-only +liveness and local projected-token readiness never contact an upstream. It +owns no credential, selection, retry, fallback, prompt, response, or deployment +authority. The +client keeps exact model selection and disables request replay and provider +fallback for this route; its workload owner composes the sidecar and projected +token into that client's reviewed native manifest. + This is not a universal AgentOS proxy. Git, PostgreSQL, Kubernetes, Herdr, registries and other provider tools continue through their native interfaces. Adding another mediated protocol requires a separate review of its authority, diff --git a/Dockerfile b/Dockerfile index e26941bb..5cf716d4 100644 --- a/Dockerfile +++ b/Dockerfile @@ -225,6 +225,7 @@ RUN chmod 0644 \ /opt/agentos/services/a2a/src/main.ts \ /opt/agentos/services/agentgateway/src/readiness-main.ts \ /opt/agentos/services/ai-gateway/src/main.ts \ + /opt/agentos/services/ai-gateway/src/workload-client-proxy-main.ts \ /opt/agentos/services/egress-authz/src/main.ts \ /opt/agentos/services/github-broker/src/main.ts \ /opt/agentos/services/openfga/src/bootstrap.ts \ @@ -247,6 +248,9 @@ RUN chmod 0644 \ && ln -s \ /opt/agentos/services/ai-gateway/src/main.ts \ /usr/local/bin/ai-gateway \ + && ln -s \ + /opt/agentos/services/ai-gateway/src/workload-client-proxy-main.ts \ + /usr/local/bin/ai-gateway-workload-proxy \ && ln -s \ /opt/agentos/services/egress-authz/src/main.ts \ /usr/local/bin/agentos-egress-authz \ diff --git a/docs/integrations/hermes-ai-gateway.config.yaml b/docs/integrations/hermes-ai-gateway.config.yaml new file mode 100644 index 00000000..99dd714f --- /dev/null +++ b/docs/integrations/hermes-ai-gateway.config.yaml @@ -0,0 +1,20 @@ +# Hermes Agent 0.20.0 (v2026.8.3) deterministic integration fixture. +model: + provider: agentos-gateway + default: gpt-exact-fixture + +providers: + agentos-gateway: + name: AgentOS Gateway + api: http://127.0.0.1:8790/v1 + api_key: agentos-workload-identity-placeholder + transport: codex_responses + discover_models: false + models: + - gpt-exact-fixture + +agent: + api_max_retries: 0 + +fallback_providers: [] +fallback_model: "" diff --git a/docs/integrations/hermes-ai-gateway.md b/docs/integrations/hermes-ai-gateway.md new file mode 100644 index 00000000..38f6849c --- /dev/null +++ b/docs/integrations/hermes-ai-gateway.md @@ -0,0 +1,170 @@ +# Hermes Agent through the Fleet AI Gateway + +This contract covers Hermes Agent `0.20.0` from release tag `v2026.8.3` and +other approved clients that speak the OpenAI Responses API but cannot reread a +Kubernetes projected token for every request. Direct per-agent OAuth remains +the recovery path. + +## Client contract + +The client keeps authority over the exact model and reasoning configuration. It +sends only `POST /v1/responses` or `POST /v1/responses/compact` to a loopback +`ai-gateway-workload-proxy` sidecar. The sidecar: + +- accepts no provider credential and listens on loopback only; +- rereads the kubelet-rotated `agentos-egress-authz` ServiceAccount token for + every request; +- removes caller-supplied authorization, assignment, decision, grant, and every + other `x-agentos-*` header, then supplies only projected identity and an + optional validated assignment ID from sidecar-owned configuration to + `agentgateway-openai`; W3C `traceparent` and `tracestate` correlation remain + unchanged; +- forwards the request once, without inspecting or storing its body; and +- returns the actual upstream status, headers, and stream without retrying or + selecting another model or account. + +Agentgateway and `agentos-egress-authz`, not the sidecar, authenticate the Pod +and authorize its registered Agent or Assignment. Do not expose the loopback +listener as a Service. Do not copy an AI Gateway client Secret, OAuth file, or +operator token into the client namespace. + +`GET /livez` is process-only. `GET /readyz` rereads the projected token and +returns ready only when it is present, at most 16 KiB, strict UTF-8, already +trimmed, and JWT-like. Neither endpoint contacts Agentgateway or a provider. + +## Hermes 0.20.0 configuration + +Hermes 0.20 supports named providers with `transport: codex_responses`, but a +configured API key is a process-lifetime value. Point the named provider at the +loopback adapter and use a non-secret placeholder; never put the projected token +in `config.yaml` or `.env`. + +```yaml +model: + provider: agentos-gateway + default: + +providers: + agentos-gateway: + name: AgentOS Gateway + api: http://127.0.0.1:8790/v1 + api_key: agentos-workload-identity-placeholder + transport: codex_responses + discover_models: false + models: + - + +agent: + api_max_retries: 0 + +fallback_providers: [] +fallback_model: "" +``` + +These fields are load-bearing for Hermes 0.20's named-provider resolver. The +input must select `provider: agentos-gateway`; the matching provider entry's +`transport: codex_responses` retains the Responses transport, and its `api` +points that transport at the loopback `/v1` base. Hermes normalizes this named +input at runtime to `provider: custom`, `api_mode: codex_responses`, and +`base_url: http://127.0.0.1:8790/v1`. Putting `provider: custom` directly in the +input falls back to Chat Completions. The checked fixture +[`hermes-ai-gateway.config.yaml`](./hermes-ai-gateway.config.yaml) is exercised +against the named input contract and the sidecar allowlist in the Gateway test +suite. + +`api_max_retries: 0` and the empty fallback configuration are part of the +contract: after Hermes sends a request, it must surface the real `401`, `403`, +`429`, timeout, or provider failure rather than replaying the turn through a +route that may acquire another account. Keep auxiliary model slots direct or +configure each approved slot explicitly through the same no-retry contract; +`auto` inherits the main route. + +## Pod wiring + +The owning Hermes deployment supplies this wiring. AgentOS owns the adapter +binary and the governed backend topology, but does not own or apply a foreign +Hermes StatefulSet. Add the selected-client label, projected identity volume, +and loopback sidecar to the reviewed workload manifest: + +```yaml +spec: + template: + metadata: + labels: + agentos.akua.dev/agentgateway-client: "true" + spec: + automountServiceAccountToken: false + containers: + - name: hermes + - name: ai-gateway-workload-proxy + image: + command: ["ai-gateway-workload-proxy"] + env: + - name: AI_GATEWAY_URL + value: http://agentgateway-openai.agentos.svc.cluster.local:8788 + - name: AGENTOS_EGRESS_TOKEN_FILE + value: /var/run/secrets/agentos-egress/token + # Optional: set only from the workload owner's trusted Assignment. + - name: AGENTOS_ASSIGNMENT_ID + value: + ports: + - name: workload-proxy + containerPort: 8790 + readinessProbe: + httpGet: + path: /readyz + port: workload-proxy + securityContext: + allowPrivilegeEscalation: false + capabilities: + drop: ["ALL"] + readOnlyRootFilesystem: true + runAsNonRoot: true + volumeMounts: + - name: agentos-egress-identity + mountPath: /var/run/secrets/agentos-egress + readOnly: true + volumes: + - name: agentos-egress-identity + projected: + defaultMode: 0440 + sources: + - serviceAccountToken: + audience: agentos-egress-authz + expirationSeconds: 600 + path: token +``` + +Use the workload's existing explicit `fsGroup` so the sidecar can read the +`0440` projection. Only the sidecar mounts that identity volume; Hermes cannot +read it. Keep the workload's dedicated ServiceAccount, register that identity +and its approved access profile through the normal AgentOS access-plane +procedure, and preserve the existing NetworkPolicy. The label grants only network +reachability to Agentgateway; it is not authorization. + +## Migration from the live v0.1.24 topology + +The observed pre-migration topology uses one `ai-gateway` replica with a shared +`AI_GATEWAY_TOKEN`, while Hermes uses direct in-Pod provider authentication and +has no projected egress identity. Do not mutate it in place. + +1. Keep Hermes on verified direct OAuth while deploying the reviewed + Agentgateway plus `agentos-egress-authz` topology and registering the Hermes + workload identity/access profile. +2. Build and publish the reviewed AgentOS revision, then update the Hermes + manifest with the exact immutable adapter image digest and projected token + wiring. Do not copy the legacy shared token or any `ai-gateway-client` Secret. +3. Render and review the StatefulSet and NetworkPolicies. Confirm the proxy is + loopback-only, Agentgateway is the only AI Gateway ingress, and ordinary + Internet egress is unchanged. +4. After explicit rollout approval, start a new Hermes session with the exact + selected model and authorize one short fixed no-tool response. Verify the + effective provider/model and failure fidelity without reading prompts, + responses, tokens, account IDs, or vault state. +5. Keep direct OAuth intact until that evidence is accepted. Roll back by + restoring the prior Hermes config/workload revision; do not delete provider + auth or retained homes as part of route rollback. +6. Retire the legacy shared-token request path only after every selected client + has either passed the workload-identity route or returned to verified direct + authentication. Secret removal, deployment, restart, and provider login are + separate human approval gates. diff --git a/package.json b/package.json index 0a25c22b..308d0651 100644 --- a/package.json +++ b/package.json @@ -19,7 +19,7 @@ "resilience:hard-gate": "bun packages/agentos/runtime/resilience-hard-gate-main.ts", "effect:tooling:test": "vitest run clis/pg-listen/tests/pg-listen.effect.test.ts release/kubernetes/tests/release.effect.test.ts", "effect:agentgateway:test": "vitest run services/agentgateway/tests/kubernetes.effect.test.ts services/agentgateway/tests/readiness.effect.test.ts", - "effect:test": "vitest run --no-file-parallelism mise.effect.test.ts tooling/effect-migration clis/github-app-token/tests/github-app-token.effect.test.ts packages/agentos/tests/*.effect.test.ts services/openfga/tests/*.effect.test.ts benchmarks/tests/contracts.effect.test.ts benchmarks/tests/pi-session-adapter.effect.test.ts benchmarks/tests/runner.effect.test.ts database/kubernetes/cloudnative-pg/tests/manifest.effect.test.ts database/tests/*.effect.test.ts packages/agentos/resources/crewmates/default/kubernetes/tests/manifest.effect.test.ts packages/agentos/resources/roles/firstmate/kubernetes/tests/manifests.effect.test.ts packages/agentos/resources/roles/secondmate/kubernetes/tests/*.effect.test.ts packages/agentos/runtime/kubernetes/tests/*.effect.test.ts packages/agentos/runtime/tests/kubernetes.effect.test.ts services/otel-collector/kubernetes/tests/*.effect.test.ts services/otel-collector/tests/*.effect.test.ts packages/agentos/src/shared/tests packages/agentos/src/access/tests packages/agentos/src/harness-control/tests packages/agentos/src/protocol/tests packages/agentos/src/resilience/tests packages/agentos/src/topology/tests packages/agentos/src/telemetry/tests packages/agentos/src/openai-server-compaction/tests/extension.effect.test.ts packages/agentos/src/openai-server-compaction/tests/messages.effect.test.ts packages/agentos/src/openai-server-compaction/tests/remote-runtime.effect.test.ts packages/agentos/src/openai-server-compaction/tests/schemas.effect.test.ts packages/agentos/src/openai-server-compaction/tests/session.effect.test.ts packages/agentos/src/mate-memory/tests/extension.effect.test.ts packages/agentos/src/mate-memory/tests/maintenance.effect.test.ts packages/agentos/src/mate-memory/tests/model.effect.test.ts packages/agentos/src/workloads/tests/compiler.effect.test.ts packages/agentos/src/workloads/tests/compiler-kustomize.effect.test.ts packages/agentos/src/workloads/tests/profiles.effect.test.ts packages/agentos/src/workloads/tests/recovery-conformance.effect.test.ts packages/agentos/src/workloads/tests/disposable-recovery.effect.test.ts packages/agentos/runtime/tests/crewmate-readiness.effect.test.ts packages/agentos/runtime/tests/health.effect.test.ts packages/agentos/runtime/tests/readiness.effect.test.ts packages/agentos/runtime/tests/github-provider.effect.test.ts packages/agentos/runtime/tests/github-workload-auth.effect.test.ts packages/agentos/runtime/tests/memory-activity.effect.test.ts packages/agentos/runtime/tests/memory-store.effect.test.ts packages/agentos/runtime/tests/production-dependencies.effect.test.ts services/a2a/tests services/agentgateway/tests/contract.effect.test.ts services/ai-gateway/tests/accounts.effect.test.ts services/ai-gateway/tests/attribution.effect.test.ts services/ai-gateway/tests/cli.effect.test.ts services/ai-gateway/tests/codex-oauth.effect.test.ts services/ai-gateway/tests/config.effect.test.ts services/ai-gateway/tests/forward.effect.test.ts services/ai-gateway/tests/gateway-service.effect.test.ts services/ai-gateway/tests/kubernetes.effect.test.ts services/ai-gateway/tests/main.effect.test.ts services/ai-gateway/tests/managed-account-live.effect.test.ts services/ai-gateway/tests/observability.effect.test.ts services/ai-gateway/tests/otlp.effect.test.ts services/ai-gateway/tests/selection.effect.test.ts services/ai-gateway/tests/telemetry.effect.test.ts services/ai-gateway/tests/telemetry-overhead.effect.test.ts services/ai-gateway/tests/provider-http.effect.test.ts services/ai-gateway/tests/quota.effect.test.ts services/ai-gateway/tests/response-usage.effect.test.ts services/ai-gateway/tests/state.effect.test.ts services/ai-gateway/tests/storage.effect.test.ts services/egress-authz/tests services/github-broker/tests", + "effect:test": "vitest run --no-file-parallelism mise.effect.test.ts tooling/effect-migration clis/github-app-token/tests/github-app-token.effect.test.ts packages/agentos/tests/*.effect.test.ts services/openfga/tests/*.effect.test.ts benchmarks/tests/contracts.effect.test.ts benchmarks/tests/pi-session-adapter.effect.test.ts benchmarks/tests/runner.effect.test.ts database/kubernetes/cloudnative-pg/tests/manifest.effect.test.ts database/tests/*.effect.test.ts packages/agentos/resources/crewmates/default/kubernetes/tests/manifest.effect.test.ts packages/agentos/resources/roles/firstmate/kubernetes/tests/manifests.effect.test.ts packages/agentos/resources/roles/secondmate/kubernetes/tests/*.effect.test.ts packages/agentos/runtime/kubernetes/tests/*.effect.test.ts packages/agentos/runtime/tests/kubernetes.effect.test.ts services/otel-collector/kubernetes/tests/*.effect.test.ts services/otel-collector/tests/*.effect.test.ts packages/agentos/src/shared/tests packages/agentos/src/access/tests packages/agentos/src/harness-control/tests packages/agentos/src/protocol/tests packages/agentos/src/resilience/tests packages/agentos/src/topology/tests packages/agentos/src/telemetry/tests packages/agentos/src/openai-server-compaction/tests/extension.effect.test.ts packages/agentos/src/openai-server-compaction/tests/messages.effect.test.ts packages/agentos/src/openai-server-compaction/tests/remote-runtime.effect.test.ts packages/agentos/src/openai-server-compaction/tests/schemas.effect.test.ts packages/agentos/src/openai-server-compaction/tests/session.effect.test.ts packages/agentos/src/mate-memory/tests/extension.effect.test.ts packages/agentos/src/mate-memory/tests/maintenance.effect.test.ts packages/agentos/src/mate-memory/tests/model.effect.test.ts packages/agentos/src/workloads/tests/compiler.effect.test.ts packages/agentos/src/workloads/tests/compiler-kustomize.effect.test.ts packages/agentos/src/workloads/tests/profiles.effect.test.ts packages/agentos/src/workloads/tests/recovery-conformance.effect.test.ts packages/agentos/src/workloads/tests/disposable-recovery.effect.test.ts packages/agentos/runtime/tests/crewmate-readiness.effect.test.ts packages/agentos/runtime/tests/health.effect.test.ts packages/agentos/runtime/tests/readiness.effect.test.ts packages/agentos/runtime/tests/github-provider.effect.test.ts packages/agentos/runtime/tests/github-workload-auth.effect.test.ts packages/agentos/runtime/tests/memory-activity.effect.test.ts packages/agentos/runtime/tests/memory-store.effect.test.ts packages/agentos/runtime/tests/production-dependencies.effect.test.ts services/a2a/tests services/agentgateway/tests/contract.effect.test.ts services/ai-gateway/tests/accounts.effect.test.ts services/ai-gateway/tests/attribution.effect.test.ts services/ai-gateway/tests/cli.effect.test.ts services/ai-gateway/tests/codex-oauth.effect.test.ts services/ai-gateway/tests/config.effect.test.ts services/ai-gateway/tests/forward.effect.test.ts services/ai-gateway/tests/gateway-service.effect.test.ts services/ai-gateway/tests/kubernetes.effect.test.ts services/ai-gateway/tests/main.effect.test.ts services/ai-gateway/tests/managed-account-live.effect.test.ts services/ai-gateway/tests/observability.effect.test.ts services/ai-gateway/tests/otlp.effect.test.ts services/ai-gateway/tests/selection.effect.test.ts services/ai-gateway/tests/telemetry.effect.test.ts services/ai-gateway/tests/telemetry-overhead.effect.test.ts services/ai-gateway/tests/provider-http.effect.test.ts services/ai-gateway/tests/workload-client-proxy.effect.test.ts services/ai-gateway/tests/quota.effect.test.ts services/ai-gateway/tests/response-usage.effect.test.ts services/ai-gateway/tests/state.effect.test.ts services/ai-gateway/tests/storage.effect.test.ts services/egress-authz/tests services/github-broker/tests", "effect:runtime:test": "vitest run --no-file-parallelism packages/agentos/src/background-tasks/tests packages/agentos/src/coordination-readiness/tests packages/agentos/src/supervision-guard/tests/extension.effect.test.ts", "effect:bun:test": "bun ./node_modules/vitest/vitest.mjs run --no-file-parallelism tooling/testing/tests/bun-http.effect.test.ts packages/agentos/runtime/tests/capacity-preflight.effect.test.ts packages/agentos/runtime/tests/codex-otel.effect.test.ts packages/agentos/runtime/tests/codex-native-otel.effect.test.ts packages/agentos/runtime/tests/codex-provider.effect.test.ts packages/agentos/runtime/tests/codex-token.effect.test.ts packages/agentos/runtime/tests/distribution-migration.effect.test.ts packages/agentos/runtime/tests/github-native-clients.effect.test.ts packages/agentos/runtime/tests/image-seed.effect.test.ts packages/agentos/runtime/tests/pi-observability-control.effect.test.ts packages/agentos/runtime/tests/pi-provider.effect.test.ts packages/agentos/runtime/tests/prepare-home.effect.test.ts packages/agentos/runtime/tests/readiness-state.effect.test.ts packages/agentos/runtime/tests/runtime.effect.test.ts services/agentgateway/tests/a2a-v1.effect.test.ts services/agentgateway/tests/conformance.effect.test.ts services/ai-gateway/tests/router-package.effect.test.ts services/ai-gateway/tests/routing-state.effect.test.ts && bun run effect:runtime:test", "check": "bun run effect:check && bun run effect:tooling:test && bun run test && bun run typecheck && bun run site:lint && bun run site:typecheck", diff --git a/services/ai-gateway/README.md b/services/ai-gateway/README.md index 965a83ce..2347633f 100644 --- a/services/ai-gateway/README.md +++ b/services/ai-gateway/README.md @@ -59,6 +59,16 @@ composes only the approved clients. Repeated Gateway-owned device logins create the multi-subscription pool; `$agentos-ai-gateway` owns the exact lifecycle and native Pi/Codex configuration. +Responses clients that cannot reread a projected Kubernetes token on every +request use the loopback-only `ai-gateway-workload-proxy` sidecar. It replaces +the client's non-secret transport placeholder with fresh workload identity, +forwards only `/v1/responses` and `/v1/responses/compact` once, and preserves +the actual upstream response. Its `/livez` is process-only and `/readyz` +validates projected identity locally without upstream traffic. The pinned +Hermes 0.20 contract, Pod wiring, and migration from the legacy shared-token +topology are documented in +[`docs/integrations/hermes-ai-gateway.md`](../../docs/integrations/hermes-ai-gateway.md). + `GET /healthz` proves only that the process is serving. `GET /readyz` requires both usable pooled/provider fallback capacity and the current `agentos-provider-budget-settlement` Pod token to pass the authorizer's diff --git a/services/ai-gateway/package.json b/services/ai-gateway/package.json index 513a7a1b..92000fd2 100644 --- a/services/ai-gateway/package.json +++ b/services/ai-gateway/package.json @@ -5,7 +5,8 @@ "private": true, "type": "module", "bin": { - "ai-gateway": "./src/main.ts" + "ai-gateway": "./src/main.ts", + "ai-gateway-workload-proxy": "./src/workload-client-proxy-main.ts" }, "scripts": { "test": "bun ../../node_modules/vitest/vitest.mjs run tests", diff --git a/services/ai-gateway/src/workload-client-proxy-main.ts b/services/ai-gateway/src/workload-client-proxy-main.ts new file mode 100644 index 00000000..b79184b2 --- /dev/null +++ b/services/ai-gateway/src/workload-client-proxy-main.ts @@ -0,0 +1,114 @@ +#!/usr/bin/env bun + +import * as BunFileSystem from "@effect/platform-bun/BunFileSystem"; +import * as BunHttpClient from "@effect/platform-bun/BunHttpClient"; +import * as BunHttpServer from "@effect/platform-bun/BunHttpServer"; +import * as BunRuntime from "@effect/platform-bun/BunRuntime"; +import { + ConfigProvider, + Console, + Effect, + Layer, + Stream, +} from "effect"; +import { + HttpRouter, + HttpServerRequest, + HttpServerResponse, +} from "effect/unstable/http"; + +import { + AIProviderHttp, + AIProviderHttpLive, + type AIProviderResponse, +} from "./provider-http.ts"; +import { + loadWorkloadClientProxyConfig, + makeWorkloadClientProxyHandler, + workloadClientProxyReadinessResponse, + workloadClientProxyErrorResponse, + WorkloadClientProxyError, +} from "./workload-client-proxy.ts"; + +function responseFromUpstream( + upstream: AIProviderResponse, +) { + return Effect.gen(function*() { + const headers = new Headers(upstream.headers); + headers.delete("content-encoding"); + headers.delete("content-length"); + if (upstream.body === null) { + return new Response(null, { status: upstream.status, headers }); + } + const body = yield* Stream.toReadableStreamEffect(upstream.body); + return new Response(body, { status: upstream.status, headers }); + }); +} + +const startup = Effect.gen(function*() { + const config = yield* loadWorkloadClientProxyConfig(); + const http = yield* AIProviderHttp; + const handler = yield* makeWorkloadClientProxyHandler({ + upstreamBaseUrl: config.upstreamBaseUrl, + tokenPath: config.tokenPath, + assignmentId: config.assignmentId, + forward: (request) => + http.execute(request).pipe( + Effect.mapError(() => + WorkloadClientProxyError.make({ code: "upstream_unavailable" }) + ), + Effect.flatMap(responseFromUpstream), + ), + }); + const routes = Layer.effectDiscard(Effect.gen(function*() { + const router = yield* HttpRouter.HttpRouter; + yield* router.add("GET", "/livez", () => + Effect.succeed(HttpServerResponse.fromWeb(Response.json({ status: "alive" })))); + yield* router.add("GET", "/readyz", () => + workloadClientProxyReadinessResponse(config.tokenPath).pipe( + Effect.map(HttpServerResponse.fromWeb), + )); + yield* router.add("*", "/*", (request) => + HttpServerRequest.toWeb(request).pipe( + Effect.flatMap(handler), + Effect.catchTag("WorkloadClientProxyError", (error) => + Effect.succeed(workloadClientProxyErrorResponse(error))), + Effect.catch(() => + Effect.succeed(Response.json({ error: "proxy_unavailable" }, { + status: 503, + })) + ), + Effect.map(HttpServerResponse.fromWeb), + )); + })); + const server = HttpRouter.serve(routes, { disableListenLog: true }).pipe( + Layer.provide(BunHttpServer.layer({ + hostname: config.hostname, + port: config.port, + gracefulShutdownTimeout: config.gracefulShutdownMillis, + })), + ); + yield* Console.log(JSON.stringify({ + event: "agentos.ai_gateway_workload_proxy.listening", + hostname: config.hostname, + port: config.port, + })); + return yield* Layer.launch(server); +}); + +if (import.meta.main) { + const platform = Layer.mergeAll( + BunFileSystem.layer, + BunHttpClient.layer, + ConfigProvider.layer(ConfigProvider.fromEnv()), + AIProviderHttpLive.pipe(Layer.provide(BunHttpClient.layer)), + ); + BunRuntime.runMain(startup.pipe( + Effect.tapError(() => + Console.error(JSON.stringify({ + event: "agentos.ai_gateway_workload_proxy.failed", + })) + ), + Effect.provide(platform), + ), { disableErrorReporting: true }); +} diff --git a/services/ai-gateway/src/workload-client-proxy.ts b/services/ai-gateway/src/workload-client-proxy.ts new file mode 100644 index 00000000..7874a2d8 --- /dev/null +++ b/services/ai-gateway/src/workload-client-proxy.ts @@ -0,0 +1,215 @@ +import { Config, Effect, FileSystem, Schema } from "effect"; + +const maximumTokenBytes = 16 * 1024; +const jwtLike = /^[A-Za-z0-9_-]+\.[A-Za-z0-9_-]+\.[A-Za-z0-9_-]+$/; +const allowedPaths = new Set(["/v1/responses", "/v1/responses/compact"]); +const assignmentIdPattern = + /^[0-9a-f]{8}-[0-9a-f]{4}-4[0-9a-f]{3}-[89ab][0-9a-f]{3}-[0-9a-f]{12}$/i; +export const workloadClientProxyHostname = "127.0.0.1"; +const removedHeaders = new Set([ + "authorization", + "connection", + "content-length", + "host", + "keep-alive", + "proxy-authenticate", + "proxy-authorization", + "te", + "trailer", + "transfer-encoding", + "upgrade", +]); + +export class WorkloadClientProxyError extends Schema.TaggedErrorClass()( + "WorkloadClientProxyError", + { + code: Schema.Literals([ + "invalid_request", + "token_unavailable", + "upstream_unavailable", + ]), + }, +) {} + +export interface WorkloadClientProxyOptions { + readonly upstreamBaseUrl: URL; + readonly tokenPath: string; + readonly assignmentId?: string; + readonly forward: ( + request: Request, + ) => Effect.Effect; +} + +const Port = Schema.Number.pipe( + Schema.check(Schema.isInt(), Schema.isGreaterThan(0)), + Schema.check(Schema.isLessThanOrEqualTo(65_535)), +); +const Configuration = Schema.Struct({ + port: Port, + tokenPath: Schema.String.pipe(Schema.check(Schema.isMinLength(1))), + upstreamBaseUrl: Schema.URL, + assignmentId: Schema.Union([ + Schema.Literal(""), + Schema.String.pipe(Schema.check(Schema.isPattern(assignmentIdPattern))), + ]), + gracefulShutdownMillis: Schema.Number.pipe( + Schema.check(Schema.isInt(), Schema.isGreaterThan(0)), + ), +}); + +export class WorkloadClientProxyConfigurationError extends Schema.TaggedErrorClass()( + "WorkloadClientProxyConfigurationError", + { code: Schema.Literal("invalid_configuration") }, +) {} + +function configurationError() { + return WorkloadClientProxyConfigurationError.make({ + code: "invalid_configuration", + }); +} + +export const loadWorkloadClientProxyConfig = Effect.fn( + "agentos.aiGateway.workloadClient.loadConfig", +)(function*() { + const raw = yield* Config.all({ + port: Config.int("AI_GATEWAY_WORKLOAD_PROXY_PORT").pipe( + Config.withDefault(8_790), + ), + tokenPath: Config.string("AGENTOS_EGRESS_TOKEN_FILE").pipe( + Config.withDefault("/var/run/secrets/agentos-egress/token"), + ), + upstreamBaseUrl: Config.url("AI_GATEWAY_URL"), + assignmentId: Config.string("AGENTOS_ASSIGNMENT_ID").pipe( + Config.withDefault(""), + ), + gracefulShutdownMillis: Config.int( + "AI_GATEWAY_WORKLOAD_PROXY_GRACEFUL_SHUTDOWN_MILLIS", + ).pipe(Config.withDefault(10_000)), + }).pipe(Effect.mapError(configurationError)); + const config = yield* Schema.decodeUnknownEffect(Configuration)(raw).pipe( + Effect.mapError(configurationError), + ); + if ( + config.upstreamBaseUrl.protocol !== "http:" && + config.upstreamBaseUrl.protocol !== "https:" + ) { + return yield* configurationError(); + } + return { + ...config, + hostname: workloadClientProxyHostname, + assignmentId: config.assignmentId === "" ? undefined : config.assignmentId, + }; +}); + +function proxyError(code: WorkloadClientProxyError["code"]) { + return WorkloadClientProxyError.make({ code }); +} + +export const readProjectedWorkloadToken = Effect.fn( + "agentos.aiGateway.workloadClient.readToken", +)(function*(path: string) { + const fileSystem = yield* FileSystem.FileSystem; + const bytes = yield* fileSystem.readFile(path).pipe( + Effect.mapError(() => proxyError("token_unavailable")), + ); + if (bytes.length === 0 || bytes.length > maximumTokenBytes) { + return yield* proxyError("token_unavailable"); + } + const token = yield* Effect.try({ + try: () => new TextDecoder("utf-8", { fatal: true }).decode(bytes), + catch: () => proxyError("token_unavailable"), + }); + if (token.trim() !== token || !jwtLike.test(token)) { + return yield* proxyError("token_unavailable"); + } + return token; +}); + +function forwardedHeaders( + input: Headers, + token: string, + assignmentId: string | undefined, +) { + const headers = new Headers(); + for (const [name, value] of input) { + const lower = name.toLowerCase(); + if (removedHeaders.has(lower) || lower.startsWith("x-agentos-")) continue; + headers.append(name, value); + } + headers.set("authorization", `Bearer ${token}`); + if (assignmentId !== undefined) { + headers.set("x-agentos-assignment-id", assignmentId); + } + return headers; +} + +export const workloadClientProxyReadinessResponse = Effect.fn( + "agentos.aiGateway.workloadClient.readiness", +)(function*(tokenPath: string) { + return yield* readProjectedWorkloadToken(tokenPath).pipe( + Effect.as(Response.json({ status: "ready" })), + Effect.catchTag("WorkloadClientProxyError", () => + Effect.succeed(Response.json({ status: "not_ready" }, { status: 503 }))), + ); +}); + +export const makeWorkloadClientProxyHandler = Effect.fn( + "agentos.aiGateway.workloadClient.makeHandler", +)(function*(options: WorkloadClientProxyOptions) { + if ( + options.upstreamBaseUrl.protocol !== "http:" && + options.upstreamBaseUrl.protocol !== "https:" + ) { + return yield* proxyError("invalid_request"); + } + const handle = Effect.fn("agentos.aiGateway.workloadClient.forward")( + function*(request: Request) { + const incoming = yield* Effect.try({ + try: () => new URL(request.url), + catch: () => proxyError("invalid_request"), + }); + if ( + request.method !== "POST" || + !allowedPaths.has(incoming.pathname) || + incoming.search !== "" || + incoming.hash !== "" + ) { + return yield* proxyError("invalid_request"); + } + const token = yield* readProjectedWorkloadToken(options.tokenPath); + const upstream = new URL(incoming.pathname, options.upstreamBaseUrl); + const forwarded = yield* Effect.try({ + try: () => { + const init: RequestInit & { readonly duplex: "half" } = { + method: "POST", + headers: forwardedHeaders( + request.headers, + token, + options.assignmentId, + ), + body: request.body, + redirect: "manual", + signal: request.signal, + duplex: "half", + }; + return new Request(upstream.toString(), init); + }, + catch: () => proxyError("invalid_request"), + }); + return yield* options.forward(forwarded); + }, + ); + return handle; +}); + +export function workloadClientProxyErrorResponse( + error: WorkloadClientProxyError, +) { + const status = error.code === "invalid_request" + ? 404 + : error.code === "token_unavailable" + ? 503 + : 502; + return Response.json({ error: error.code }, { status }); +} diff --git a/services/ai-gateway/tests/workload-client-proxy.effect.test.ts b/services/ai-gateway/tests/workload-client-proxy.effect.test.ts new file mode 100644 index 00000000..7682a131 --- /dev/null +++ b/services/ai-gateway/tests/workload-client-proxy.effect.test.ts @@ -0,0 +1,244 @@ +import * as BunServices from "@effect/platform-bun/BunServices"; +import { assert, layer } from "@effect/vitest"; +import { ConfigProvider, Effect, FileSystem, Ref, Schema } from "effect"; +import { parse } from "yaml"; + +import { + loadWorkloadClientProxyConfig, + makeWorkloadClientProxyHandler, + workloadClientProxyErrorResponse, + workloadClientProxyReadinessResponse, +} from "../src/workload-client-proxy.ts"; + +const suite = layer(BunServices.layer); +const token = "eyJhbGciOiJSUzI1NiJ9.eyJzdWIiOiJmaXh0dXJlIn0.signature"; +const assignmentId = "20000000-0000-4000-8000-000000000001"; + +const HermesConfiguration = Schema.Struct({ + model: Schema.Struct({ + provider: Schema.String, + default: Schema.String, + }), + providers: Schema.Struct({ + "agentos-gateway": Schema.Struct({ + name: Schema.String, + api: Schema.String, + api_key: Schema.String, + transport: Schema.String, + discover_models: Schema.Boolean, + models: Schema.Array(Schema.String), + }), + }), + agent: Schema.Struct({ api_max_retries: Schema.Number }), + fallback_providers: Schema.Array(Schema.String), + fallback_model: Schema.String, +}); + +function environment(values: Readonly>) { + return ConfigProvider.layer(ConfigProvider.fromEnv({ env: { ...values } })); +} + +suite("Hermes Responses workload client proxy", (it) => { + it.effect("resolves the pinned Hermes named provider fixture to an allowed Responses path", () => + Effect.gen(function*() { + const fileSystem = yield* FileSystem.FileSystem; + const fixtureUrl = new URL( + "../../../docs/integrations/hermes-ai-gateway.config.yaml", + import.meta.url, + ); + const fixture = yield* fileSystem.readFileString(fixtureUrl.pathname); + const config = yield* Schema.decodeUnknownEffect(HermesConfiguration)( + parse(fixture), + ); + + assert.strictEqual(config.model.provider, "agentos-gateway"); + const provider = config.providers["agentos-gateway"]; + assert.strictEqual(provider.name, "AgentOS Gateway"); + assert.strictEqual(provider.transport, "codex_responses"); + assert.strictEqual(provider.api, "http://127.0.0.1:8790/v1"); + assert.strictEqual( + provider.api_key, + "agentos-workload-identity-placeholder", + ); + assert.isFalse(provider.discover_models); + assert.deepStrictEqual(provider.models, [config.model.default]); + assert.strictEqual(config.agent.api_max_retries, 0); + assert.deepStrictEqual(config.fallback_providers, []); + assert.strictEqual(config.fallback_model, ""); + + const directory = yield* fileSystem.makeTempDirectoryScoped(); + const tokenPath = `${directory}/token`; + yield* fileSystem.writeFileString(tokenPath, token); + const forwarded = yield* Ref.make(null); + const handler = yield* makeWorkloadClientProxyHandler({ + upstreamBaseUrl: new URL("http://agentgateway-openai.agentos.svc.cluster.local:8788"), + tokenPath, + forward: (request) => + Ref.set(forwarded, request).pipe( + Effect.as(new Response(null, { status: 204 })), + ), + }); + + const hermesResponsesUrl = `${ + provider.api.replace(/\/$/, "") + }/responses`; + const response = yield* handler(new Request(hermesResponsesUrl, { + method: "POST", + headers: { authorization: `Bearer ${provider.api_key}` }, + body: JSON.stringify({ model: config.model.default }), + })); + const request = yield* Ref.get(forwarded); + + assert.strictEqual(response.status, 204); + assert.strictEqual( + request?.url, + "http://agentgateway-openai.agentos.svc.cluster.local:8788/v1/responses", + ); + })); + + it.effect("rereads identity, sanitizes authority headers, and injects only trusted assignment", () => + Effect.gen(function*() { + const fileSystem = yield* FileSystem.FileSystem; + const directory = yield* fileSystem.makeTempDirectoryScoped(); + const tokenPath = `${directory}/token`; + yield* fileSystem.writeFileString(tokenPath, token); + const requests = yield* Ref.make>([]); + const handler = yield* makeWorkloadClientProxyHandler({ + upstreamBaseUrl: new URL("http://agentgateway-openai.agentos.svc.cluster.local:8788"), + tokenPath, + assignmentId, + forward: (request) => + Ref.update(requests, (current) => [...current, request]).pipe( + Effect.as(new Response("provider failure", { + status: 429, + headers: { "x-upstream": "preserved" }, + })), + ), + }); + + const first = yield* handler(new Request("http://127.0.0.1:8790/v1/responses", { + method: "POST", + headers: { + authorization: "Bearer hermes-placeholder", + "content-type": "application/json", + traceparent: "00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01", + "x-agentos-assignment-id": "forged-assignment", + "x-agentos-decision": "forged-decision", + "x-agentos-grant": "forged-grant", + }, + body: "{\"model\":\"gpt-exact\"}", + })); + yield* fileSystem.writeFileString(tokenPath, `${token}a`); + const second = yield* handler(new Request("http://127.0.0.1:8790/v1/responses/compact", { + method: "POST", + body: "{}", + })); + + assert.strictEqual(first.status, 429); + assert.strictEqual(first.headers.get("x-upstream"), "preserved"); + assert.strictEqual(yield* Effect.promise(() => first.text()), "provider failure"); + assert.strictEqual(second.status, 429); + const forwarded = yield* Ref.get(requests); + assert.lengthOf(forwarded, 2); + assert.strictEqual(forwarded[0]?.headers.get("authorization"), `Bearer ${token}`); + assert.strictEqual(forwarded[0]?.headers.get("x-agentos-assignment-id"), assignmentId); + assert.isNull(forwarded[0]?.headers.get("x-agentos-decision")); + assert.isNull(forwarded[0]?.headers.get("x-agentos-grant")); + assert.strictEqual( + forwarded[0]?.headers.get("traceparent"), + "00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01", + ); + assert.strictEqual(forwarded[1]?.headers.get("authorization"), `Bearer ${token}a`); + assert.strictEqual( + yield* Effect.promise(() => forwarded[0]?.text() ?? Promise.resolve("")), + "{\"model\":\"gpt-exact\"}", + ); + })); + + it.effect("hard-codes loopback listening and rejects invalid trusted assignment configuration", () => + Effect.gen(function*() { + const config = yield* loadWorkloadClientProxyConfig().pipe( + Effect.provide(environment({ + AI_GATEWAY_URL: "http://agentgateway-openai.agentos.svc.cluster.local:8788", + AI_GATEWAY_WORKLOAD_PROXY_HOST: "0.0.0.0", + AGENTOS_ASSIGNMENT_ID: assignmentId, + })), + ); + assert.strictEqual(config.hostname, "127.0.0.1"); + assert.strictEqual(config.assignmentId, assignmentId); + + for (const invalidAssignmentId of [ + "caller-controlled-value", + "20000000-0000-1000-8000-000000000001", + "20000000-0000-4000-7000-000000000001", + ]) { + const invalid = yield* loadWorkloadClientProxyConfig().pipe( + Effect.provide(environment({ + AI_GATEWAY_URL: "http://agentgateway-openai.agentos.svc.cluster.local:8788", + AGENTOS_ASSIGNMENT_ID: invalidAssignmentId, + })), + Effect.flip, + ); + assert.strictEqual(invalid.code, "invalid_configuration"); + } + })); + + it.effect("reports readiness only for a bounded, trimmed, UTF-8 JWT-like token", () => + Effect.gen(function*() { + const fileSystem = yield* FileSystem.FileSystem; + const directory = yield* fileSystem.makeTempDirectoryScoped(); + const tokenPath = `${directory}/token`; + + const missing = yield* workloadClientProxyReadinessResponse(tokenPath); + assert.strictEqual(missing.status, 503); + + const invalidTokens: ReadonlyArray = [ + new Uint8Array(), + new TextEncoder().encode(`${token}\n`), + new TextEncoder().encode("not-a-jwt"), + new Uint8Array([0xff, 0xfe, 0xfd]), + new Uint8Array(16 * 1024 + 1).fill(97), + ]; + for (const bytes of invalidTokens) { + yield* fileSystem.writeFile(tokenPath, bytes); + const response = yield* workloadClientProxyReadinessResponse(tokenPath); + assert.strictEqual(response.status, 503); + } + + yield* fileSystem.writeFileString(tokenPath, token); + const ready = yield* workloadClientProxyReadinessResponse(tokenPath); + assert.strictEqual(ready.status, 200); + assert.deepStrictEqual(yield* Effect.promise(() => ready.json()), { + status: "ready", + }); + })); + + it.effect("fails closed without sending unsupported, unauthenticated, or readiness requests", () => + Effect.gen(function*() { + const fileSystem = yield* FileSystem.FileSystem; + const directory = yield* fileSystem.makeTempDirectoryScoped(); + const tokenPath = `${directory}/token`; + const calls = yield* Ref.make(0); + const handler = yield* makeWorkloadClientProxyHandler({ + upstreamBaseUrl: new URL("http://agentgateway-openai.agentos.svc.cluster.local:8788"), + tokenPath, + forward: () => Ref.update(calls, (value) => value + 1).pipe( + Effect.as(new Response(null, { status: 200 })), + ), + }); + + const unsupported = yield* handler(new Request("http://127.0.0.1:8790/v1/chat/completions", { + method: "POST", + })).pipe(Effect.flip); + assert.strictEqual(unsupported.code, "invalid_request"); + assert.strictEqual(workloadClientProxyErrorResponse(unsupported).status, 404); + + const missing = yield* handler(new Request("http://127.0.0.1:8790/v1/responses", { + method: "POST", + })).pipe(Effect.flip); + assert.strictEqual(missing.code, "token_unavailable"); + assert.strictEqual(workloadClientProxyErrorResponse(missing).status, 503); + yield* workloadClientProxyReadinessResponse(tokenPath); + assert.strictEqual(yield* Ref.get(calls), 0); + })); +}); diff --git a/tooling/effect-migration/exceptions.json b/tooling/effect-migration/exceptions.json index 8acac5e4..a248fc6c 100644 --- a/tooling/effect-migration/exceptions.json +++ b/tooling/effect-migration/exceptions.json @@ -433,6 +433,18 @@ "removalCondition": "Retain only while the AI Gateway remains a standalone Bun service and keep all domain behavior inside Effect.", "expiresOn": null }, + { + "kind": "outer-host-adapter", + "path": "services/ai-gateway/src/workload-client-proxy-main.ts", + "rule": "no-runtime-execution", + "match": "BunRuntime.runMain", + "maximumOccurrences": 1, + "reason": "The workload client proxy executable needs one reviewed Bun process entry boundary around its fully typed, scoped Effect Platform application.", + "ownerIssue": 105, + "test": "services/ai-gateway/tests/workload-client-proxy.effect.test.ts", + "removalCondition": "Retain only while the loopback workload client proxy remains a standalone Bun sidecar entrypoint and keep all request behavior inside Effect.", + "expiresOn": null + }, { "kind": "outer-host-adapter", "path": "services/egress-authz/src/main.ts", diff --git a/tooling/effect-migration/inventory.json b/tooling/effect-migration/inventory.json index 40864338..eff4b5f4 100644 --- a/tooling/effect-migration/inventory.json +++ b/tooling/effect-migration/inventory.json @@ -963,7 +963,7 @@ { "id": "ai-gateway-entrypoint", "issue": 105, - "pattern": "^services/ai-gateway/src/main\\.ts$", + "pattern": "^services/ai-gateway/src/(?:main|workload-client-proxy-main)\\.ts$", "runtime": "bun-entrypoint", "package": "@agentos/ai-gateway", "io": ["config", "filesystem", "http", "oauth", "process", "stream", "telemetry", "runtime-adapter"], From a82af0e1d6472c1aa3312796b24fee55ff363f54 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Robin=20Br=C3=A4mer?= <22003767+robinbraemer@users.noreply.github.com> Date: Mon, 10 Aug 2026 15:11:24 +0200 Subject: [PATCH 02/28] no-mistakes(review): Hardened Hermes proxy timeouts, probes, policy wiring, and redirects --- docs/integrations/hermes-ai-gateway.md | 27 ++++++++- services/ai-gateway/src/config.ts | 7 ++- services/ai-gateway/src/main.ts | 11 +++- services/ai-gateway/src/provider-http.ts | 6 ++ .../src/workload-client-proxy-main.ts | 9 ++- .../ai-gateway/src/workload-client-proxy.ts | 21 ++++++- .../tests/provider-http.effect.test.ts | 55 +++++++++++++++++++ .../workload-client-proxy.effect.test.ts | 46 ++++++++++++++++ 8 files changed, 170 insertions(+), 12 deletions(-) diff --git a/docs/integrations/hermes-ai-gateway.md b/docs/integrations/hermes-ai-gateway.md index 38f6849c..4329b606 100644 --- a/docs/integrations/hermes-ai-gateway.md +++ b/docs/integrations/hermes-ai-gateway.md @@ -86,11 +86,20 @@ binary and the governed backend topology, but does not own or apply a foreign Hermes StatefulSet. Add the selected-client label, projected identity volume, and loopback sidecar to the reviewed workload manifest: +The current Agentgateway NetworkPolicy admits client Pods from the `agentos` +namespace, or from an already approved namespace labeled +`agentos.akua.dev/managed-by: agentos-firstmate`, only when the Pod has an +`agentos.akua.dev/agent` label. Deploy Hermes in one of those approved +namespaces and preserve that reachability label. These labels only permit +network reachability; the projected ServiceAccount token and Agentgateway +authorizer provide authentication and authorization. + ```yaml spec: template: metadata: labels: + agentos.akua.dev/agent: "hermes" agentos.akua.dev/agentgateway-client: "true" spec: automountServiceAccountToken: false @@ -110,10 +119,22 @@ spec: ports: - name: workload-proxy containerPort: 8790 + livenessProbe: + exec: + command: + - /usr/bin/curl + - --fail + - --silent + - --show-error + - http://127.0.0.1:8790/livez readinessProbe: - httpGet: - path: /readyz - port: workload-proxy + exec: + command: + - /usr/bin/curl + - --fail + - --silent + - --show-error + - http://127.0.0.1:8790/readyz securityContext: allowPrivilegeEscalation: false capabilities: diff --git a/services/ai-gateway/src/config.ts b/services/ai-gateway/src/config.ts index 5d5f8d58..2a392bc0 100644 --- a/services/ai-gateway/src/config.ts +++ b/services/ai-gateway/src/config.ts @@ -9,6 +9,9 @@ import { Schema, } from "effect"; +export const defaultAIGatewayIdleTimeoutSeconds = 255; +export const defaultAIGatewayGracefulShutdownMillis = 20_000; + const RawConfig = Config.all({ home: Config.string("HOME").pipe(Config.withDefault(".")), stateDirectory: Config.string("AI_GATEWAY_STATE_DIR").pipe( @@ -21,11 +24,11 @@ const RawConfig = Config.all({ Config.withDefault(8787), ), idleTimeoutSeconds: Config.int("AI_GATEWAY_IDLE_TIMEOUT_SECONDS").pipe( - Config.withDefault(255), + Config.withDefault(defaultAIGatewayIdleTimeoutSeconds), ), gracefulShutdownMillis: Config.int( "AI_GATEWAY_GRACEFUL_SHUTDOWN_MILLIS", - ).pipe(Config.withDefault(20_000)), + ).pipe(Config.withDefault(defaultAIGatewayGracefulShutdownMillis)), clientAuthenticationMode: Config.string( "AI_GATEWAY_CLIENT_AUTH_MODE", ).pipe(Config.withDefault("shared_token")), diff --git a/services/ai-gateway/src/main.ts b/services/ai-gateway/src/main.ts index 21fe7656..d22c86c4 100755 --- a/services/ai-gateway/src/main.ts +++ b/services/ai-gateway/src/main.ts @@ -60,7 +60,11 @@ import { AIGatewayTelemetry, makeAIGatewayTelemetry, } from "./observability.ts"; -import { AIProviderHttp, AIProviderHttpLive } from "./provider-http.ts"; +import { + AIProviderHttp, + AIProviderHttpLive, + AIProviderHttpRequestInit, +} from "./provider-http.ts"; import { AIGatewayOtlpLive } from "./otlp.ts"; import { CodexQuota, makeCodexQuotaLayer } from "./quota.ts"; import { makeEffectManagedAccountVaultLayer } from "./managed-account-live.ts"; @@ -304,10 +308,13 @@ const startup = Effect.gen(function*() { }).pipe(Effect.scoped); if (import.meta.main) { + const aiProviderHttpClientLayer = BunHttpClient.layer.pipe( + Layer.provide(AIProviderHttpRequestInit), + ); const platform = Layer.mergeAll( BunCryptoLayer, BunFileSystem.layer, - BunHttpClient.layer, + aiProviderHttpClientLayer, BunPath.layer, ConfigProvider.layer(ConfigProvider.fromEnv()), ); diff --git a/services/ai-gateway/src/provider-http.ts b/services/ai-gateway/src/provider-http.ts index 97165ded..7fb28b52 100644 --- a/services/ai-gateway/src/provider-http.ts +++ b/services/ai-gateway/src/provider-http.ts @@ -1,6 +1,7 @@ import { classifyAIError } from "@akua-dev/agentos"; import { Context, Effect, Layer, Schema, Stream } from "effect"; import { + FetchHttpClient, HttpClient, HttpClientError, HttpClientRequest, @@ -69,6 +70,11 @@ export const AIProviderHttpLive = Layer.effect( }), ); +export const AIProviderHttpRequestInit = Layer.succeed( + FetchHttpClient.RequestInit, + { redirect: "manual" }, +); + function providerHttpError(code: AIProviderHttpError["code"]) { return AIProviderHttpError.make({ code }); } diff --git a/services/ai-gateway/src/workload-client-proxy-main.ts b/services/ai-gateway/src/workload-client-proxy-main.ts index b79184b2..97ac187f 100644 --- a/services/ai-gateway/src/workload-client-proxy-main.ts +++ b/services/ai-gateway/src/workload-client-proxy-main.ts @@ -20,6 +20,7 @@ import { import { AIProviderHttp, AIProviderHttpLive, + AIProviderHttpRequestInit, type AIProviderResponse, } from "./provider-http.ts"; import { @@ -85,6 +86,7 @@ const startup = Effect.gen(function*() { Layer.provide(BunHttpServer.layer({ hostname: config.hostname, port: config.port, + idleTimeout: config.idleTimeoutSeconds, gracefulShutdownTimeout: config.gracefulShutdownMillis, })), ); @@ -97,11 +99,14 @@ const startup = Effect.gen(function*() { }); if (import.meta.main) { + const aiProviderHttpClientLayer = BunHttpClient.layer.pipe( + Layer.provide(AIProviderHttpRequestInit), + ); const platform = Layer.mergeAll( BunFileSystem.layer, - BunHttpClient.layer, + aiProviderHttpClientLayer, ConfigProvider.layer(ConfigProvider.fromEnv()), - AIProviderHttpLive.pipe(Layer.provide(BunHttpClient.layer)), + AIProviderHttpLive.pipe(Layer.provide(aiProviderHttpClientLayer)), ); BunRuntime.runMain(startup.pipe( Effect.tapError(() => diff --git a/services/ai-gateway/src/workload-client-proxy.ts b/services/ai-gateway/src/workload-client-proxy.ts index 7874a2d8..5f93904d 100644 --- a/services/ai-gateway/src/workload-client-proxy.ts +++ b/services/ai-gateway/src/workload-client-proxy.ts @@ -1,5 +1,10 @@ import { Config, Effect, FileSystem, Schema } from "effect"; +import { + defaultAIGatewayGracefulShutdownMillis, + defaultAIGatewayIdleTimeoutSeconds, +} from "./config.ts"; + const maximumTokenBytes = 16 * 1024; const jwtLike = /^[A-Za-z0-9_-]+\.[A-Za-z0-9_-]+\.[A-Za-z0-9_-]+$/; const allowedPaths = new Set(["/v1/responses", "/v1/responses/compact"]); @@ -46,6 +51,13 @@ const Port = Schema.Number.pipe( ); const Configuration = Schema.Struct({ port: Port, + idleTimeoutSeconds: Schema.Number.pipe( + Schema.check( + Schema.isInt(), + Schema.isGreaterThanOrEqualTo(0), + Schema.isLessThanOrEqualTo(255), + ), + ), tokenPath: Schema.String.pipe(Schema.check(Schema.isMinLength(1))), upstreamBaseUrl: Schema.URL, assignmentId: Schema.Union([ @@ -75,6 +87,9 @@ export const loadWorkloadClientProxyConfig = Effect.fn( port: Config.int("AI_GATEWAY_WORKLOAD_PROXY_PORT").pipe( Config.withDefault(8_790), ), + idleTimeoutSeconds: Config.int("AI_GATEWAY_IDLE_TIMEOUT_SECONDS").pipe( + Config.withDefault(defaultAIGatewayIdleTimeoutSeconds), + ), tokenPath: Config.string("AGENTOS_EGRESS_TOKEN_FILE").pipe( Config.withDefault("/var/run/secrets/agentos-egress/token"), ), @@ -82,9 +97,9 @@ export const loadWorkloadClientProxyConfig = Effect.fn( assignmentId: Config.string("AGENTOS_ASSIGNMENT_ID").pipe( Config.withDefault(""), ), - gracefulShutdownMillis: Config.int( - "AI_GATEWAY_WORKLOAD_PROXY_GRACEFUL_SHUTDOWN_MILLIS", - ).pipe(Config.withDefault(10_000)), + gracefulShutdownMillis: Config.int("AI_GATEWAY_GRACEFUL_SHUTDOWN_MILLIS").pipe( + Config.withDefault(defaultAIGatewayGracefulShutdownMillis), + ), }).pipe(Effect.mapError(configurationError)); const config = yield* Schema.decodeUnknownEffect(Configuration)(raw).pipe( Effect.mapError(configurationError), diff --git a/services/ai-gateway/tests/provider-http.effect.test.ts b/services/ai-gateway/tests/provider-http.effect.test.ts index 7b70a049..711b6342 100644 --- a/services/ai-gateway/tests/provider-http.effect.test.ts +++ b/services/ai-gateway/tests/provider-http.effect.test.ts @@ -1,6 +1,7 @@ import { assert, describe, it } from "@effect/vitest"; import { Effect, Layer, Option, Ref, Stream } from "effect"; import { + FetchHttpClient, HttpClient, HttpClientError, HttpClientRequest, @@ -11,6 +12,7 @@ import { AIProviderHttp, AIProviderHttpError, AIProviderHttpLive, + AIProviderHttpRequestInit, } from "../src/provider-http.ts"; function providerLayer( @@ -96,6 +98,59 @@ describe("AI provider HTTP adapter", () => { }); })); + it.effect("keeps credential-bearing redirects manual at the fetch boundary", () => + Effect.gen(function*() { + const calls: Array<{ + readonly authorization: string | null; + readonly redirect: RequestInit["redirect"] | "default"; + readonly url: string; + }> = []; + const fetch: typeof globalThis.fetch = (input, init) => { + const request = input instanceof Request + ? input + : new Request(input, init); + calls.push({ + authorization: new Headers(init?.headers).get("authorization"), + redirect: init?.redirect ?? "default", + url: request.url, + }); + if (init?.redirect !== "manual") { + calls.push({ + authorization: new Headers(init?.headers).get("authorization"), + redirect: init?.redirect ?? "default", + url: "https://redirect-target.invalid/v1/responses", + }); + } + return Promise.resolve(new Response(null, { + status: 307, + headers: { location: "https://redirect-target.invalid/v1/responses" }, + })); + }; + const fetchLayer = FetchHttpClient.layer.pipe( + Layer.provide(AIProviderHttpRequestInit), + Layer.provide(Layer.succeed(FetchHttpClient.Fetch, fetch)), + ); + const layer = AIProviderHttpLive.pipe( + Layer.provide(fetchLayer), + ); + const provider = yield* AIProviderHttp.pipe(Effect.provide(layer)); + const response = yield* provider.execute(new Request( + "https://api.openai.test/v1/responses", + { + method: "POST", + headers: { authorization: "Bearer projected-workload-token" }, + body: "{}", + }, + )); + + assert.strictEqual(response.status, 307); + assert.deepStrictEqual(calls, [{ + authorization: "Bearer projected-workload-token", + redirect: "manual", + url: "https://api.openai.test/v1/responses", + }]); + })); + it.effect("maps request construction and transport failures to closed typed errors", () => Effect.forEach(transportCases, ({ cause, expected }) => Effect.gen(function*() { diff --git a/services/ai-gateway/tests/workload-client-proxy.effect.test.ts b/services/ai-gateway/tests/workload-client-proxy.effect.test.ts index 7682a131..cf5f2e5f 100644 --- a/services/ai-gateway/tests/workload-client-proxy.effect.test.ts +++ b/services/ai-gateway/tests/workload-client-proxy.effect.test.ts @@ -166,6 +166,18 @@ suite("Hermes Responses workload client proxy", (it) => { ); assert.strictEqual(config.hostname, "127.0.0.1"); assert.strictEqual(config.assignmentId, assignmentId); + assert.strictEqual(config.idleTimeoutSeconds, 255); + assert.strictEqual(config.gracefulShutdownMillis, 20_000); + + const configured = yield* loadWorkloadClientProxyConfig().pipe( + Effect.provide(environment({ + AI_GATEWAY_URL: "http://agentgateway-openai.agentos.svc.cluster.local:8788", + AI_GATEWAY_IDLE_TIMEOUT_SECONDS: "120", + AI_GATEWAY_GRACEFUL_SHUTDOWN_MILLIS: "45000", + })), + ); + assert.strictEqual(configured.idleTimeoutSeconds, 120); + assert.strictEqual(configured.gracefulShutdownMillis, 45_000); for (const invalidAssignmentId of [ "caller-controlled-value", @@ -183,6 +195,40 @@ suite("Hermes Responses workload client proxy", (it) => { } })); + it.effect("preserves upstream response streams across a long gap", () => + Effect.gen(function*() { + const fileSystem = yield* FileSystem.FileSystem; + const directory = yield* fileSystem.makeTempDirectoryScoped(); + const tokenPath = `${directory}/token`; + yield* fileSystem.writeFileString(tokenPath, token); + const handler = yield* makeWorkloadClientProxyHandler({ + upstreamBaseUrl: new URL("http://agentgateway-openai.agentos.svc.cluster.local:8788"), + tokenPath, + forward: () => Effect.succeed(new Response( + new ReadableStream({ + start(controller) { + controller.enqueue(new TextEncoder().encode("first")); + setTimeout(() => { + controller.enqueue(new TextEncoder().encode("second")); + controller.close(); + }, 1_100); + }, + }), + { status: 200, headers: { "content-type": "text/plain" } }, + )), + }); + + const response = yield* handler(new Request( + "http://127.0.0.1:8790/v1/responses", + { method: "POST" }, + )); + assert.strictEqual(response.status, 200); + assert.strictEqual( + yield* Effect.promise(() => response.text()), + "firstsecond", + ); + })); + it.effect("reports readiness only for a bounded, trimmed, UTF-8 JWT-like token", () => Effect.gen(function*() { const fileSystem = yield* FileSystem.FileSystem; From 9bd12cc31bf8064ca21b363f90a306ce60d54265 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Robin=20Br=C3=A4mer?= <22003767+robinbraemer@users.noreply.github.com> Date: Mon, 10 Aug 2026 15:27:02 +0200 Subject: [PATCH 03/28] no-mistakes(review): Hardened AI gateway proxy boundaries and cancellation --- services/ai-gateway/src/provider-http.ts | 17 ++++- .../src/workload-client-proxy-main.ts | 32 +++++++-- .../ai-gateway/src/workload-client-proxy.ts | 19 +++--- .../tests/provider-http.effect.test.ts | 34 +++++++++- .../workload-client-proxy.effect.test.ts | 67 ++++++++++++++++++- 5 files changed, 154 insertions(+), 15 deletions(-) diff --git a/services/ai-gateway/src/provider-http.ts b/services/ai-gateway/src/provider-http.ts index 7fb28b52..9e43fe61 100644 --- a/services/ai-gateway/src/provider-http.ts +++ b/services/ai-gateway/src/provider-http.ts @@ -47,7 +47,10 @@ export const AIProviderHttpLive = Layer.effect( try: () => HttpClientRequest.fromWeb(request), catch: () => providerHttpError("request_invalid"), }); - const response = yield* client.execute(clientRequest).pipe( + const response = yield* Effect.raceFirst( + client.execute(clientRequest), + abortOnSignal(request.signal), + ).pipe( Effect.mapError((error) => providerHttpError(httpErrorCode(error))), ); const hasNoBody = request.method === "HEAD" || @@ -75,6 +78,18 @@ export const AIProviderHttpRequestInit = Layer.succeed( { redirect: "manual" }, ); +function abortOnSignal(signal: AbortSignal) { + return Effect.callback((resume) => { + const onAbort = () => resume(Effect.interrupt); + if (signal.aborted) { + onAbort(); + return Effect.void; + } + signal.addEventListener("abort", onAbort, { once: true }); + return Effect.sync(() => signal.removeEventListener("abort", onAbort)); + }); +} + function providerHttpError(code: AIProviderHttpError["code"]) { return AIProviderHttpError.make({ code }); } diff --git a/services/ai-gateway/src/workload-client-proxy-main.ts b/services/ai-gateway/src/workload-client-proxy-main.ts index 97ac187f..4acd3a45 100644 --- a/services/ai-gateway/src/workload-client-proxy-main.ts +++ b/services/ai-gateway/src/workload-client-proxy-main.ts @@ -31,13 +31,37 @@ import { WorkloadClientProxyError, } from "./workload-client-proxy.ts"; -function responseFromUpstream( +const hopByHopResponseHeaders = new Set([ + "connection", + "keep-alive", + "proxy-authenticate", + "proxy-authorization", + "te", + "trailer", + "transfer-encoding", + "upgrade", +]); +const httpToken = /^[!#$%&'*+\-.^_`|~0-9A-Za-z]+$/; + +function responseHeaders(upstream: AIProviderResponse) { + const headers = new Headers(upstream.headers); + const connection = headers.get("connection"); + headers.delete("connection"); + for (const name of connection?.split(",") ?? []) { + const normalized = name.trim().toLowerCase(); + if (httpToken.test(normalized)) headers.delete(normalized); + } + for (const name of hopByHopResponseHeaders) headers.delete(name); + headers.delete("content-encoding"); + headers.delete("content-length"); + return headers; +} + +export function responseFromUpstream( upstream: AIProviderResponse, ) { return Effect.gen(function*() { - const headers = new Headers(upstream.headers); - headers.delete("content-encoding"); - headers.delete("content-length"); + const headers = responseHeaders(upstream); if (upstream.body === null) { return new Response(null, { status: upstream.status, headers }); } diff --git a/services/ai-gateway/src/workload-client-proxy.ts b/services/ai-gateway/src/workload-client-proxy.ts index 5f93904d..339c2eff 100644 --- a/services/ai-gateway/src/workload-client-proxy.ts +++ b/services/ai-gateway/src/workload-client-proxy.ts @@ -80,6 +80,15 @@ function configurationError() { }); } +function isValidUpstreamBaseUrl(url: URL) { + return (url.protocol === "http:" || url.protocol === "https:") && + url.username === "" && + url.password === "" && + url.pathname === "/" && + url.search === "" && + url.hash === ""; +} + export const loadWorkloadClientProxyConfig = Effect.fn( "agentos.aiGateway.workloadClient.loadConfig", )(function*() { @@ -104,10 +113,7 @@ export const loadWorkloadClientProxyConfig = Effect.fn( const config = yield* Schema.decodeUnknownEffect(Configuration)(raw).pipe( Effect.mapError(configurationError), ); - if ( - config.upstreamBaseUrl.protocol !== "http:" && - config.upstreamBaseUrl.protocol !== "https:" - ) { + if (!isValidUpstreamBaseUrl(config.upstreamBaseUrl)) { return yield* configurationError(); } return { @@ -172,10 +178,7 @@ export const workloadClientProxyReadinessResponse = Effect.fn( export const makeWorkloadClientProxyHandler = Effect.fn( "agentos.aiGateway.workloadClient.makeHandler", )(function*(options: WorkloadClientProxyOptions) { - if ( - options.upstreamBaseUrl.protocol !== "http:" && - options.upstreamBaseUrl.protocol !== "https:" - ) { + if (!isValidUpstreamBaseUrl(options.upstreamBaseUrl)) { return yield* proxyError("invalid_request"); } const handle = Effect.fn("agentos.aiGateway.workloadClient.forward")( diff --git a/services/ai-gateway/tests/provider-http.effect.test.ts b/services/ai-gateway/tests/provider-http.effect.test.ts index 711b6342..a1925da2 100644 --- a/services/ai-gateway/tests/provider-http.effect.test.ts +++ b/services/ai-gateway/tests/provider-http.effect.test.ts @@ -1,5 +1,14 @@ import { assert, describe, it } from "@effect/vitest"; -import { Effect, Layer, Option, Ref, Stream } from "effect"; +import { + Deferred, + Effect, + Exit, + Fiber, + Layer, + Option, + Ref, + Stream, +} from "effect"; import { FetchHttpClient, HttpClient, @@ -151,6 +160,29 @@ describe("AI provider HTTP adapter", () => { }]); })); + it.effect("aborts the transport when the caller aborts before upstream headers", () => + Effect.gen(function*() { + const transportSignal = yield* Deferred.make(); + const layer = providerLayer((_request, _url, signal) => + Deferred.succeed(transportSignal, signal).pipe( + Effect.andThen(Effect.never), + )); + const controller = new AbortController(); + const provider = yield* AIProviderHttp.pipe(Effect.provide(layer)); + const fiber = yield* Effect.forkChild(Effect.exit(provider.execute( + new Request("https://api.openai.test/v1/responses", { + method: "POST", + signal: controller.signal, + }), + ))); + const signal = yield* Deferred.await(transportSignal); + yield* Effect.sync(() => controller.abort()); + const exit = yield* Fiber.join(fiber); + + assert.isFalse(Exit.isSuccess(exit)); + assert.isTrue(signal.aborted); + })); + it.effect("maps request construction and transport failures to closed typed errors", () => Effect.forEach(transportCases, ({ cause, expected }) => Effect.gen(function*() { diff --git a/services/ai-gateway/tests/workload-client-proxy.effect.test.ts b/services/ai-gateway/tests/workload-client-proxy.effect.test.ts index cf5f2e5f..0ef203fd 100644 --- a/services/ai-gateway/tests/workload-client-proxy.effect.test.ts +++ b/services/ai-gateway/tests/workload-client-proxy.effect.test.ts @@ -1,6 +1,13 @@ import * as BunServices from "@effect/platform-bun/BunServices"; import { assert, layer } from "@effect/vitest"; -import { ConfigProvider, Effect, FileSystem, Ref, Schema } from "effect"; +import { + ConfigProvider, + Effect, + FileSystem, + Ref, + Schema, + Stream, +} from "effect"; import { parse } from "yaml"; import { @@ -9,6 +16,7 @@ import { workloadClientProxyErrorResponse, workloadClientProxyReadinessResponse, } from "../src/workload-client-proxy.ts"; +import { responseFromUpstream } from "../src/workload-client-proxy-main.ts"; const suite = layer(BunServices.layer); const token = "eyJhbGciOiJSUzI1NiJ9.eyJzdWIiOiJmaXh0dXJlIn0.signature"; @@ -179,6 +187,20 @@ suite("Hermes Responses workload client proxy", (it) => { assert.strictEqual(configured.idleTimeoutSeconds, 120); assert.strictEqual(configured.gracefulShutdownMillis, 45_000); + for (const invalidUrl of [ + "ftp://agentgateway-openai.agentos.svc.cluster.local:8788", + "http://user:password@agentgateway-openai.agentos.svc.cluster.local:8788", + "http://agentgateway-openai.agentos.svc.cluster.local:8788?trace=1", + "http://agentgateway-openai.agentos.svc.cluster.local:8788#fragment", + "http://agentgateway-openai.agentos.svc.cluster.local:8788/v1", + ]) { + const invalid = yield* loadWorkloadClientProxyConfig().pipe( + Effect.provide(environment({ AI_GATEWAY_URL: invalidUrl })), + Effect.flip, + ); + assert.strictEqual(invalid.code, "invalid_configuration"); + } + for (const invalidAssignmentId of [ "caller-controlled-value", "20000000-0000-1000-8000-000000000001", @@ -195,6 +217,49 @@ suite("Hermes Responses workload client proxy", (it) => { } })); + it.effect("strips hop-by-hop response headers while preserving provider data", () => + Effect.gen(function*() { + const response = yield* responseFromUpstream({ + status: 429, + headers: { + connection: "keep-alive, x-connection-hop", + "keep-alive": "timeout=5", + "proxy-authenticate": "Basic", + "proxy-authorization": "Basic secret", + te: "trailers", + trailer: "x-trailer", + "transfer-encoding": "chunked", + upgrade: "websocket", + "x-connection-hop": "private", + "x-provider-request-id": "provider-1", + }, + body: Stream.succeed(new TextEncoder().encode("provider-body")), + }); + + assert.strictEqual(response.status, 429); + for (const name of [ + "connection", + "keep-alive", + "proxy-authenticate", + "proxy-authorization", + "te", + "trailer", + "transfer-encoding", + "upgrade", + "x-connection-hop", + ]) { + assert.isNull(response.headers.get(name)); + } + assert.strictEqual( + response.headers.get("x-provider-request-id"), + "provider-1", + ); + assert.strictEqual( + yield* Effect.promise(() => response.text()), + "provider-body", + ); + })); + it.effect("preserves upstream response streams across a long gap", () => Effect.gen(function*() { const fileSystem = yield* FileSystem.FileSystem; From a16ce23a13591acf185399762786aa24473b7bf3 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Robin=20Br=C3=A4mer?= <22003767+robinbraemer@users.noreply.github.com> Date: Mon, 10 Aug 2026 15:40:14 +0200 Subject: [PATCH 04/28] no-mistakes(review): Scoped provider redirects and linked stream cancellation --- services/ai-gateway/src/main.ts | 12 ++-- services/ai-gateway/src/provider-http.ts | 9 +++ .../src/workload-client-proxy-main.ts | 9 +-- .../tests/provider-http.effect.test.ts | 65 +++++++++++++++++++ 4 files changed, 81 insertions(+), 14 deletions(-) diff --git a/services/ai-gateway/src/main.ts b/services/ai-gateway/src/main.ts index d22c86c4..5e7f7662 100755 --- a/services/ai-gateway/src/main.ts +++ b/services/ai-gateway/src/main.ts @@ -62,8 +62,7 @@ import { } from "./observability.ts"; import { AIProviderHttp, - AIProviderHttpLive, - AIProviderHttpRequestInit, + makeAIProviderHttpLive, } from "./provider-http.ts"; import { AIGatewayOtlpLive } from "./otlp.ts"; import { CodexQuota, makeCodexQuotaLayer } from "./quota.ts"; @@ -133,6 +132,8 @@ const AIGatewayStatusClientLive = Layer.effect( }), ); +const aiProviderHttpLive = makeAIProviderHttpLive(BunHttpClient.layer); + function acquireAIGatewayTelemetry() { return Effect.gen(function*() { const disabled = yield* Config.boolean("OTEL_SDK_DISABLED").pipe( @@ -167,7 +168,7 @@ function makeAIGatewayRuntimeLive( const crypto = yield* Crypto.Crypto; const fileSystem = yield* FileSystem.FileSystem; const provider = yield* AIProviderHttp.pipe( - Effect.provide(AIProviderHttpLive), + Effect.provide(aiProviderHttpLive), ); const quota = yield* CodexQuota.pipe( Effect.provide(makeCodexQuotaLayer(config.quotaTimeoutMillis)), @@ -308,13 +309,10 @@ const startup = Effect.gen(function*() { }).pipe(Effect.scoped); if (import.meta.main) { - const aiProviderHttpClientLayer = BunHttpClient.layer.pipe( - Layer.provide(AIProviderHttpRequestInit), - ); const platform = Layer.mergeAll( BunCryptoLayer, BunFileSystem.layer, - aiProviderHttpClientLayer, + BunHttpClient.layer, BunPath.layer, ConfigProvider.layer(ConfigProvider.fromEnv()), ); diff --git a/services/ai-gateway/src/provider-http.ts b/services/ai-gateway/src/provider-http.ts index 9e43fe61..5d41bc66 100644 --- a/services/ai-gateway/src/provider-http.ts +++ b/services/ai-gateway/src/provider-http.ts @@ -58,6 +58,7 @@ export const AIProviderHttpLive = Layer.effect( const body: AIProviderResponse["body"] = hasNoBody ? null : response.stream.pipe( + Stream.interruptWhen(abortOnSignal(request.signal)), Stream.mapError((error) => providerHttpError(streamErrorCode(error)) ), @@ -78,6 +79,14 @@ export const AIProviderHttpRequestInit = Layer.succeed( { redirect: "manual" }, ); +export function makeAIProviderHttpLive( + clientLayer: Layer.Layer, +) { + return AIProviderHttpLive.pipe( + Layer.provide(clientLayer.pipe(Layer.provide(AIProviderHttpRequestInit))), + ); +} + function abortOnSignal(signal: AbortSignal) { return Effect.callback((resume) => { const onAbort = () => resume(Effect.interrupt); diff --git a/services/ai-gateway/src/workload-client-proxy-main.ts b/services/ai-gateway/src/workload-client-proxy-main.ts index 4acd3a45..b4259c9c 100644 --- a/services/ai-gateway/src/workload-client-proxy-main.ts +++ b/services/ai-gateway/src/workload-client-proxy-main.ts @@ -19,8 +19,7 @@ import { import { AIProviderHttp, - AIProviderHttpLive, - AIProviderHttpRequestInit, + makeAIProviderHttpLive, type AIProviderResponse, } from "./provider-http.ts"; import { @@ -123,14 +122,10 @@ const startup = Effect.gen(function*() { }); if (import.meta.main) { - const aiProviderHttpClientLayer = BunHttpClient.layer.pipe( - Layer.provide(AIProviderHttpRequestInit), - ); const platform = Layer.mergeAll( BunFileSystem.layer, - aiProviderHttpClientLayer, + makeAIProviderHttpLive(BunHttpClient.layer), ConfigProvider.layer(ConfigProvider.fromEnv()), - AIProviderHttpLive.pipe(Layer.provide(aiProviderHttpClientLayer)), ); BunRuntime.runMain(startup.pipe( Effect.tapError(() => diff --git a/services/ai-gateway/tests/provider-http.effect.test.ts b/services/ai-gateway/tests/provider-http.effect.test.ts index a1925da2..8c10ea86 100644 --- a/services/ai-gateway/tests/provider-http.effect.test.ts +++ b/services/ai-gateway/tests/provider-http.effect.test.ts @@ -22,6 +22,7 @@ import { AIProviderHttpError, AIProviderHttpLive, AIProviderHttpRequestInit, + makeAIProviderHttpLive, } from "../src/provider-http.ts"; function providerLayer( @@ -160,6 +161,35 @@ describe("AI provider HTTP adapter", () => { }]); })); + it.effect("scopes manual redirects to the provider client", () => + Effect.gen(function*() { + const redirects: Array = []; + const fetch: typeof globalThis.fetch = (_input, init) => { + redirects.push(init?.redirect ?? "default"); + return Promise.resolve(new Response(null, { status: 204 })); + }; + const ordinaryClientLayer = FetchHttpClient.layer.pipe( + Layer.provide(Layer.succeed(FetchHttpClient.Fetch, fetch)), + ); + const provider = yield* AIProviderHttp.pipe( + Effect.provide(makeAIProviderHttpLive(ordinaryClientLayer)), + ); + const ordinaryClient = yield* HttpClient.HttpClient.pipe( + Effect.provide(ordinaryClientLayer), + ); + + yield* Effect.scoped( + HttpClient.withScope(ordinaryClient).execute( + HttpClientRequest.get("https://api.openai.test/status"), + ), + ); + yield* provider.execute(new Request( + "https://api.openai.test/v1/responses", + )); + + assert.deepStrictEqual(redirects, ["default", "manual"]); + })); + it.effect("aborts the transport when the caller aborts before upstream headers", () => Effect.gen(function*() { const transportSignal = yield* Deferred.make(); @@ -183,6 +213,41 @@ describe("AI provider HTTP adapter", () => { assert.isTrue(signal.aborted); })); + it.effect("interrupts an in-flight provider stream when the caller aborts", () => + Effect.gen(function*() { + const transportSignal = yield* Deferred.make(); + let bodyStarted = false; + const layer = providerLayer((request, _url, signal) => + Deferred.succeed(transportSignal, signal).pipe( + Effect.andThen(Effect.succeed(HttpClientResponse.fromWeb( + request, + new Response(new ReadableStream({ + pull(controller) { + bodyStarted = true; + controller.enqueue(new Uint8Array([1])); + return new Promise(() => {}); + }, + })), + ))), + )); + const controller = new AbortController(); + const provider = yield* AIProviderHttp.pipe(Effect.provide(layer)); + const response = yield* provider.execute(new Request( + "https://api.openai.test/v1/responses", + { signal: controller.signal }, + )); + const bodyFiber = yield* Effect.forkChild(Effect.exit( + Stream.runDrain(response.body ?? Stream.empty), + )); + while (!bodyStarted) yield* Effect.yieldNow; + const signal = yield* Deferred.await(transportSignal); + yield* Effect.sync(() => controller.abort()); + const exit = yield* Fiber.join(bodyFiber); + + assert.isFalse(Exit.isSuccess(exit)); + assert.isTrue(signal.aborted); + })); + it.effect("maps request construction and transport failures to closed typed errors", () => Effect.forEach(transportCases, ({ cause, expected }) => Effect.gen(function*() { From 3815bc12d13302f3d317bdb7963d1e9a95437830 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Robin=20Br=C3=A4mer?= <22003767+robinbraemer@users.noreply.github.com> Date: Mon, 10 Aug 2026 16:04:49 +0200 Subject: [PATCH 05/28] no-mistakes(review): Harden proxy credential stripping and centralize Hermes migration workflow --- docs/integrations/hermes-ai-gateway.md | 32 ++++--------------- .../skills/agentos-ai-gateway/SKILL.md | 28 ++++++++++++++++ .../ai-gateway/src/workload-client-proxy.ts | 1 + .../workload-client-proxy.effect.test.ts | 2 ++ 4 files changed, 37 insertions(+), 26 deletions(-) diff --git a/docs/integrations/hermes-ai-gateway.md b/docs/integrations/hermes-ai-gateway.md index 4329b606..78ab9661 100644 --- a/docs/integrations/hermes-ai-gateway.md +++ b/docs/integrations/hermes-ai-gateway.md @@ -163,29 +163,9 @@ and its approved access profile through the normal AgentOS access-plane procedure, and preserve the existing NetworkPolicy. The label grants only network reachability to Agentgateway; it is not authorization. -## Migration from the live v0.1.24 topology - -The observed pre-migration topology uses one `ai-gateway` replica with a shared -`AI_GATEWAY_TOKEN`, while Hermes uses direct in-Pod provider authentication and -has no projected egress identity. Do not mutate it in place. - -1. Keep Hermes on verified direct OAuth while deploying the reviewed - Agentgateway plus `agentos-egress-authz` topology and registering the Hermes - workload identity/access profile. -2. Build and publish the reviewed AgentOS revision, then update the Hermes - manifest with the exact immutable adapter image digest and projected token - wiring. Do not copy the legacy shared token or any `ai-gateway-client` Secret. -3. Render and review the StatefulSet and NetworkPolicies. Confirm the proxy is - loopback-only, Agentgateway is the only AI Gateway ingress, and ordinary - Internet egress is unchanged. -4. After explicit rollout approval, start a new Hermes session with the exact - selected model and authorize one short fixed no-tool response. Verify the - effective provider/model and failure fidelity without reading prompts, - responses, tokens, account IDs, or vault state. -5. Keep direct OAuth intact until that evidence is accepted. Roll back by - restoring the prior Hermes config/workload revision; do not delete provider - auth or retained homes as part of route rollback. -6. Retire the legacy shared-token request path only after every selected client - has either passed the workload-identity route or returned to verified direct - authentication. Secret removal, deployment, restart, and provider login are - separate human approval gates. +## Operator workflow + +Use `$agentos-ai-gateway` for the approval, migration, rollout, verification, +rollback, and retirement workflow, including migration from the live v0.1.24 +shared-token topology. This page owns only the Hermes client contract and Pod +wiring. diff --git a/packages/agentos/skills/agentos-ai-gateway/SKILL.md b/packages/agentos/skills/agentos-ai-gateway/SKILL.md index d4e80abb..6d1ca7ef 100644 --- a/packages/agentos/skills/agentos-ai-gateway/SKILL.md +++ b/packages/agentos/skills/agentos-ai-gateway/SKILL.md @@ -221,6 +221,34 @@ a real process/Pod restart; Pi `/reload` cannot change environment. Ask before interrupting a Mate and preserve its native session reference through the normal recovery procedure. +## Hermes workload-proxy migration + +For the live v0.1.24 Hermes topology, the observed pre-migration state uses one +`ai-gateway` replica with a shared `AI_GATEWAY_TOKEN`, while Hermes uses direct +in-Pod provider authentication and has no projected egress identity. Do not +mutate it in place. + +1. Keep Hermes on verified direct OAuth while deploying the reviewed + Agentgateway plus `agentos-egress-authz` topology and registering the Hermes + workload identity/access profile. +2. Build and publish the reviewed AgentOS revision, then update the Hermes + manifest with the exact immutable adapter image digest and projected token + wiring. Do not copy the legacy shared token or any `ai-gateway-client` Secret. +3. Render and review the StatefulSet and NetworkPolicies. Confirm the proxy is + loopback-only, Agentgateway is the only AI Gateway ingress, and ordinary + Internet egress is unchanged. +4. After explicit rollout approval, start a new Hermes session with the exact + selected model and authorize one short fixed no-tool response. Verify the + effective provider/model and failure fidelity without reading prompts, + responses, tokens, account IDs, or vault state. +5. Keep direct OAuth intact until that evidence is accepted. Roll back by + restoring the prior Hermes config/workload revision; do not delete provider + auth or retained homes as part of route rollback. +6. Retire the legacy shared-token request path only after every selected client + has either passed the workload-identity route or returned to verified direct + authentication. Secret removal, deployment, restart, and provider login are + separate human approval gates. + For First and Second Mate, the additive client patch sets `AGENTOS_PI_PROVIDER_MODE=ai-gateway` on `prepare-home`. Before Pi can start, AgentOS atomically reconciles only its marker-owned `openai-codex` transport diff --git a/services/ai-gateway/src/workload-client-proxy.ts b/services/ai-gateway/src/workload-client-proxy.ts index 339c2eff..8a106cb4 100644 --- a/services/ai-gateway/src/workload-client-proxy.ts +++ b/services/ai-gateway/src/workload-client-proxy.ts @@ -23,6 +23,7 @@ const removedHeaders = new Set([ "trailer", "transfer-encoding", "upgrade", + "x-ai-gateway-token", ]); export class WorkloadClientProxyError extends Schema.TaggedErrorClass()( diff --git a/services/ai-gateway/tests/workload-client-proxy.effect.test.ts b/services/ai-gateway/tests/workload-client-proxy.effect.test.ts index 0ef203fd..da562bc0 100644 --- a/services/ai-gateway/tests/workload-client-proxy.effect.test.ts +++ b/services/ai-gateway/tests/workload-client-proxy.effect.test.ts @@ -133,6 +133,7 @@ suite("Hermes Responses workload client proxy", (it) => { "x-agentos-assignment-id": "forged-assignment", "x-agentos-decision": "forged-decision", "x-agentos-grant": "forged-grant", + "x-ai-gateway-token": "legacy-shared-token", }, body: "{\"model\":\"gpt-exact\"}", })); @@ -152,6 +153,7 @@ suite("Hermes Responses workload client proxy", (it) => { assert.strictEqual(forwarded[0]?.headers.get("x-agentos-assignment-id"), assignmentId); assert.isNull(forwarded[0]?.headers.get("x-agentos-decision")); assert.isNull(forwarded[0]?.headers.get("x-agentos-grant")); + assert.isNull(forwarded[0]?.headers.get("x-ai-gateway-token")); assert.strictEqual( forwarded[0]?.headers.get("traceparent"), "00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01", From b1afc00df75d262e8aac44bac1a68df7fd33de67 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Robin=20Br=C3=A4mer?= <22003767+robinbraemer@users.noreply.github.com> Date: Mon, 10 Aug 2026 16:14:51 +0200 Subject: [PATCH 06/28] no-mistakes(review): Correct Hermes provider selection and disable stream reconnects --- docs/integrations/hermes-ai-gateway.config.yaml | 2 +- docs/integrations/hermes-ai-gateway.md | 17 +++++++++++------ .../tests/workload-client-proxy.effect.test.ts | 2 +- 3 files changed, 13 insertions(+), 8 deletions(-) diff --git a/docs/integrations/hermes-ai-gateway.config.yaml b/docs/integrations/hermes-ai-gateway.config.yaml index 99dd714f..3b0c9941 100644 --- a/docs/integrations/hermes-ai-gateway.config.yaml +++ b/docs/integrations/hermes-ai-gateway.config.yaml @@ -1,6 +1,6 @@ # Hermes Agent 0.20.0 (v2026.8.3) deterministic integration fixture. model: - provider: agentos-gateway + provider: custom:agentos-gateway default: gpt-exact-fixture providers: diff --git a/docs/integrations/hermes-ai-gateway.md b/docs/integrations/hermes-ai-gateway.md index 78ab9661..6f99ede2 100644 --- a/docs/integrations/hermes-ai-gateway.md +++ b/docs/integrations/hermes-ai-gateway.md @@ -41,7 +41,7 @@ in `config.yaml` or `.env`. ```yaml model: - provider: agentos-gateway + provider: custom:agentos-gateway default: providers: @@ -62,12 +62,12 @@ fallback_model: "" ``` These fields are load-bearing for Hermes 0.20's named-provider resolver. The -input must select `provider: agentos-gateway`; the matching provider entry's +input must select `provider: custom:agentos-gateway`; the matching provider entry's `transport: codex_responses` retains the Responses transport, and its `api` points that transport at the loopback `/v1` base. Hermes normalizes this named -input at runtime to `provider: custom`, `api_mode: codex_responses`, and -`base_url: http://127.0.0.1:8790/v1`. Putting `provider: custom` directly in the -input falls back to Chat Completions. The checked fixture +input at runtime to the named custom provider with `api_mode: codex_responses` +and `base_url: http://127.0.0.1:8790/v1`. Putting bare `provider: custom` +directly in the input falls back to Chat Completions. The checked fixture [`hermes-ai-gateway.config.yaml`](./hermes-ai-gateway.config.yaml) is exercised against the named input contract and the sidecar allowlist in the Gateway test suite. @@ -77,7 +77,9 @@ contract: after Hermes sends a request, it must surface the real `401`, `403`, `429`, timeout, or provider failure rather than replaying the turn through a route that may acquire another account. Keep auxiliary model slots direct or configure each approved slot explicitly through the same no-retry contract; -`auto` inherits the main route. +`auto` inherits the main route. Set `HERMES_STREAM_RETRIES=0` on the Hermes +container as shown below; this separately disables mid-stream reconnects after +partial output. ## Pod wiring @@ -105,6 +107,9 @@ spec: automountServiceAccountToken: false containers: - name: hermes + env: + - name: HERMES_STREAM_RETRIES + value: "0" - name: ai-gateway-workload-proxy image: command: ["ai-gateway-workload-proxy"] diff --git a/services/ai-gateway/tests/workload-client-proxy.effect.test.ts b/services/ai-gateway/tests/workload-client-proxy.effect.test.ts index da562bc0..fcc81551 100644 --- a/services/ai-gateway/tests/workload-client-proxy.effect.test.ts +++ b/services/ai-gateway/tests/workload-client-proxy.effect.test.ts @@ -59,7 +59,7 @@ suite("Hermes Responses workload client proxy", (it) => { parse(fixture), ); - assert.strictEqual(config.model.provider, "agentos-gateway"); + assert.strictEqual(config.model.provider, "custom:agentos-gateway"); const provider = config.providers["agentos-gateway"]; assert.strictEqual(provider.name, "AgentOS Gateway"); assert.strictEqual(provider.transport, "codex_responses"); From 3766e099223f56507c3ff6d5f863883ecb7dbcac Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Robin=20Br=C3=A4mer?= <22003767+robinbraemer@users.noreply.github.com> Date: Mon, 10 Aug 2026 16:30:39 +0200 Subject: [PATCH 07/28] no-mistakes(review): Block unsafe Hermes revision and bind auxiliary routes explicitly --- .../hermes-ai-gateway.config.yaml | 50 ++++++++++++++ docs/integrations/hermes-ai-gateway.md | 65 +++++++++++++++++-- .../workload-client-proxy.effect.test.ts | 27 ++++++++ 3 files changed, 136 insertions(+), 6 deletions(-) diff --git a/docs/integrations/hermes-ai-gateway.config.yaml b/docs/integrations/hermes-ai-gateway.config.yaml index 3b0c9941..97564c37 100644 --- a/docs/integrations/hermes-ai-gateway.config.yaml +++ b/docs/integrations/hermes-ai-gateway.config.yaml @@ -13,6 +13,56 @@ providers: models: - gpt-exact-fixture +auxiliary: + vision: + provider: custom + model: gpt-exact-fixture + base_url: http://127.0.0.1:8790/v1 + api_key: agentos-workload-identity-placeholder + fallback_chain: [] + web_extract: + provider: custom + model: gpt-exact-fixture + base_url: http://127.0.0.1:8790/v1 + api_key: agentos-workload-identity-placeholder + fallback_chain: [] + compression: + provider: custom + model: gpt-exact-fixture + base_url: http://127.0.0.1:8790/v1 + api_key: agentos-workload-identity-placeholder + fallback_chain: [] + skills_hub: + provider: custom + model: gpt-exact-fixture + base_url: http://127.0.0.1:8790/v1 + api_key: agentos-workload-identity-placeholder + fallback_chain: [] + mcp: + provider: custom + model: gpt-exact-fixture + base_url: http://127.0.0.1:8790/v1 + api_key: agentos-workload-identity-placeholder + fallback_chain: [] + approval: + provider: custom + model: gpt-exact-fixture + base_url: http://127.0.0.1:8790/v1 + api_key: agentos-workload-identity-placeholder + fallback_chain: [] + title_generation: + provider: custom + model: gpt-exact-fixture + base_url: http://127.0.0.1:8790/v1 + api_key: agentos-workload-identity-placeholder + fallback_chain: [] + triage_specifier: + provider: custom + model: gpt-exact-fixture + base_url: http://127.0.0.1:8790/v1 + api_key: agentos-workload-identity-placeholder + fallback_chain: [] + agent: api_max_retries: 0 diff --git a/docs/integrations/hermes-ai-gateway.md b/docs/integrations/hermes-ai-gateway.md index 6f99ede2..76efa66b 100644 --- a/docs/integrations/hermes-ai-gateway.md +++ b/docs/integrations/hermes-ai-gateway.md @@ -5,6 +5,13 @@ other approved clients that speak the OpenAI Responses API but cannot reread a Kubernetes projected token for every request. Direct per-agent OAuth remains the recovery path. +> Deployment gate: this Hermes revision is not eligible for this route. Its +> `codex_responses` runtime hardcodes one retry around `responses.create`, so a +> transport failure after partial output can replay the request; +> `HERMES_STREAM_RETRIES=0` does not change that behavior. Do not deploy the +> fixture or Pod wiring below until a pinned Hermes revision with a verified +> zero-retry Responses path is available. + ## Client contract The client keeps authority over the exact model and reasoning configuration. It @@ -59,6 +66,56 @@ agent: fallback_providers: [] fallback_model: "" + +auxiliary: + vision: + provider: custom + model: + base_url: http://127.0.0.1:8790/v1 + api_key: agentos-workload-identity-placeholder + fallback_chain: [] + web_extract: + provider: custom + model: + base_url: http://127.0.0.1:8790/v1 + api_key: agentos-workload-identity-placeholder + fallback_chain: [] + compression: + provider: custom + model: + base_url: http://127.0.0.1:8790/v1 + api_key: agentos-workload-identity-placeholder + fallback_chain: [] + skills_hub: + provider: custom + model: + base_url: http://127.0.0.1:8790/v1 + api_key: agentos-workload-identity-placeholder + fallback_chain: [] + mcp: + provider: custom + model: + base_url: http://127.0.0.1:8790/v1 + api_key: agentos-workload-identity-placeholder + fallback_chain: [] + approval: + provider: custom + model: + base_url: http://127.0.0.1:8790/v1 + api_key: agentos-workload-identity-placeholder + fallback_chain: [] + title_generation: + provider: custom + model: + base_url: http://127.0.0.1:8790/v1 + api_key: agentos-workload-identity-placeholder + fallback_chain: [] + triage_specifier: + provider: custom + model: + base_url: http://127.0.0.1:8790/v1 + api_key: agentos-workload-identity-placeholder + fallback_chain: [] ``` These fields are load-bearing for Hermes 0.20's named-provider resolver. The @@ -77,9 +134,8 @@ contract: after Hermes sends a request, it must surface the real `401`, `403`, `429`, timeout, or provider failure rather than replaying the turn through a route that may acquire another account. Keep auxiliary model slots direct or configure each approved slot explicitly through the same no-retry contract; -`auto` inherits the main route. Set `HERMES_STREAM_RETRIES=0` on the Hermes -container as shown below; this separately disables mid-stream reconnects after -partial output. +`auto` is not permitted for this integration. Each supported auxiliary slot is +bound to the same exact loopback endpoint, model, and empty fallback chain. ## Pod wiring @@ -107,9 +163,6 @@ spec: automountServiceAccountToken: false containers: - name: hermes - env: - - name: HERMES_STREAM_RETRIES - value: "0" - name: ai-gateway-workload-proxy image: command: ["ai-gateway-workload-proxy"] diff --git a/services/ai-gateway/tests/workload-client-proxy.effect.test.ts b/services/ai-gateway/tests/workload-client-proxy.effect.test.ts index fcc81551..7e6594eb 100644 --- a/services/ai-gateway/tests/workload-client-proxy.effect.test.ts +++ b/services/ai-gateway/tests/workload-client-proxy.effect.test.ts @@ -37,6 +37,13 @@ const HermesConfiguration = Schema.Struct({ models: Schema.Array(Schema.String), }), }), + auxiliary: Schema.Record(Schema.String, Schema.Struct({ + provider: Schema.String, + model: Schema.String, + base_url: Schema.String, + api_key: Schema.String, + fallback_chain: Schema.Array(Schema.Unknown), + })), agent: Schema.Struct({ api_max_retries: Schema.Number }), fallback_providers: Schema.Array(Schema.String), fallback_model: Schema.String, @@ -73,6 +80,26 @@ suite("Hermes Responses workload client proxy", (it) => { assert.strictEqual(config.agent.api_max_retries, 0); assert.deepStrictEqual(config.fallback_providers, []); assert.strictEqual(config.fallback_model, ""); + const auxiliaryNames = [ + "approval", + "compression", + "mcp", + "skills_hub", + "title_generation", + "triage_specifier", + "vision", + "web_extract", + ]; + assert.deepStrictEqual(Object.keys(config.auxiliary).sort(), auxiliaryNames); + for (const name of auxiliaryNames) { + assert.deepInclude(config.auxiliary[name], { + provider: "custom", + model: config.model.default, + base_url: "http://127.0.0.1:8790/v1", + api_key: "agentos-workload-identity-placeholder", + fallback_chain: [], + }); + } const directory = yield* fileSystem.makeTempDirectoryScoped(); const tokenPath = `${directory}/token`; From 891504b63f33da6d6722d7c49d2a9fbdbc3bea98 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Robin=20Br=C3=A4mer?= <22003767+robinbraemer@users.noreply.github.com> Date: Mon, 10 Aug 2026 16:42:56 +0200 Subject: [PATCH 08/28] no-mistakes(review): Enforce Hermes zero-retry gate and disable auxiliary routing --- .../hermes-ai-gateway.config.yaml | 50 ---------------- docs/integrations/hermes-ai-gateway.md | 58 ++----------------- .../skills/agentos-ai-gateway/SKILL.md | 7 +++ .../workload-client-proxy.effect.test.ts | 31 +--------- 4 files changed, 14 insertions(+), 132 deletions(-) diff --git a/docs/integrations/hermes-ai-gateway.config.yaml b/docs/integrations/hermes-ai-gateway.config.yaml index 97564c37..3b0c9941 100644 --- a/docs/integrations/hermes-ai-gateway.config.yaml +++ b/docs/integrations/hermes-ai-gateway.config.yaml @@ -13,56 +13,6 @@ providers: models: - gpt-exact-fixture -auxiliary: - vision: - provider: custom - model: gpt-exact-fixture - base_url: http://127.0.0.1:8790/v1 - api_key: agentos-workload-identity-placeholder - fallback_chain: [] - web_extract: - provider: custom - model: gpt-exact-fixture - base_url: http://127.0.0.1:8790/v1 - api_key: agentos-workload-identity-placeholder - fallback_chain: [] - compression: - provider: custom - model: gpt-exact-fixture - base_url: http://127.0.0.1:8790/v1 - api_key: agentos-workload-identity-placeholder - fallback_chain: [] - skills_hub: - provider: custom - model: gpt-exact-fixture - base_url: http://127.0.0.1:8790/v1 - api_key: agentos-workload-identity-placeholder - fallback_chain: [] - mcp: - provider: custom - model: gpt-exact-fixture - base_url: http://127.0.0.1:8790/v1 - api_key: agentos-workload-identity-placeholder - fallback_chain: [] - approval: - provider: custom - model: gpt-exact-fixture - base_url: http://127.0.0.1:8790/v1 - api_key: agentos-workload-identity-placeholder - fallback_chain: [] - title_generation: - provider: custom - model: gpt-exact-fixture - base_url: http://127.0.0.1:8790/v1 - api_key: agentos-workload-identity-placeholder - fallback_chain: [] - triage_specifier: - provider: custom - model: gpt-exact-fixture - base_url: http://127.0.0.1:8790/v1 - api_key: agentos-workload-identity-placeholder - fallback_chain: [] - agent: api_max_retries: 0 diff --git a/docs/integrations/hermes-ai-gateway.md b/docs/integrations/hermes-ai-gateway.md index 76efa66b..6f318e82 100644 --- a/docs/integrations/hermes-ai-gateway.md +++ b/docs/integrations/hermes-ai-gateway.md @@ -66,56 +66,6 @@ agent: fallback_providers: [] fallback_model: "" - -auxiliary: - vision: - provider: custom - model: - base_url: http://127.0.0.1:8790/v1 - api_key: agentos-workload-identity-placeholder - fallback_chain: [] - web_extract: - provider: custom - model: - base_url: http://127.0.0.1:8790/v1 - api_key: agentos-workload-identity-placeholder - fallback_chain: [] - compression: - provider: custom - model: - base_url: http://127.0.0.1:8790/v1 - api_key: agentos-workload-identity-placeholder - fallback_chain: [] - skills_hub: - provider: custom - model: - base_url: http://127.0.0.1:8790/v1 - api_key: agentos-workload-identity-placeholder - fallback_chain: [] - mcp: - provider: custom - model: - base_url: http://127.0.0.1:8790/v1 - api_key: agentos-workload-identity-placeholder - fallback_chain: [] - approval: - provider: custom - model: - base_url: http://127.0.0.1:8790/v1 - api_key: agentos-workload-identity-placeholder - fallback_chain: [] - title_generation: - provider: custom - model: - base_url: http://127.0.0.1:8790/v1 - api_key: agentos-workload-identity-placeholder - fallback_chain: [] - triage_specifier: - provider: custom - model: - base_url: http://127.0.0.1:8790/v1 - api_key: agentos-workload-identity-placeholder - fallback_chain: [] ``` These fields are load-bearing for Hermes 0.20's named-provider resolver. The @@ -132,10 +82,10 @@ suite. `api_max_retries: 0` and the empty fallback configuration are part of the contract: after Hermes sends a request, it must surface the real `401`, `403`, `429`, timeout, or provider failure rather than replaying the turn through a -route that may acquire another account. Keep auxiliary model slots direct or -configure each approved slot explicitly through the same no-retry contract; -`auto` is not permitted for this integration. Each supported auxiliary slot is -bound to the same exact loopback endpoint, model, and empty fallback chain. +route that may acquire another account. Auxiliary requests are out of scope for +this blocked Hermes revision. Do not enable them or configure `provider: auto`, +`provider: custom`, or a fallback chain until Hermes provides a verified native +Responses path with no automatic retry, redirect, or fallback. ## Pod wiring diff --git a/packages/agentos/skills/agentos-ai-gateway/SKILL.md b/packages/agentos/skills/agentos-ai-gateway/SKILL.md index 6d1ca7ef..130aca07 100644 --- a/packages/agentos/skills/agentos-ai-gateway/SKILL.md +++ b/packages/agentos/skills/agentos-ai-gateway/SKILL.md @@ -223,6 +223,13 @@ normal recovery procedure. ## Hermes workload-proxy migration +Before any Hermes migration, verify that the selected pinned revision's +`codex_responses` runtime has a zero-retry Responses path. Do not deploy or +update this integration while that gate is unmet: Hermes `0.20.0` / `v2026.8.3` +hardcodes one retry around `responses.create`. Auxiliary requests are also out +of scope until Hermes provides a native Responses path with no automatic retry, +redirect, or fallback; do not enable them during migration. + For the live v0.1.24 Hermes topology, the observed pre-migration state uses one `ai-gateway` replica with a shared `AI_GATEWAY_TOKEN`, while Hermes uses direct in-Pod provider authentication and has no projected egress identity. Do not diff --git a/services/ai-gateway/tests/workload-client-proxy.effect.test.ts b/services/ai-gateway/tests/workload-client-proxy.effect.test.ts index 7e6594eb..684ffc68 100644 --- a/services/ai-gateway/tests/workload-client-proxy.effect.test.ts +++ b/services/ai-gateway/tests/workload-client-proxy.effect.test.ts @@ -37,13 +37,6 @@ const HermesConfiguration = Schema.Struct({ models: Schema.Array(Schema.String), }), }), - auxiliary: Schema.Record(Schema.String, Schema.Struct({ - provider: Schema.String, - model: Schema.String, - base_url: Schema.String, - api_key: Schema.String, - fallback_chain: Schema.Array(Schema.Unknown), - })), agent: Schema.Struct({ api_max_retries: Schema.Number }), fallback_providers: Schema.Array(Schema.String), fallback_model: Schema.String, @@ -62,8 +55,9 @@ suite("Hermes Responses workload client proxy", (it) => { import.meta.url, ); const fixture = yield* fileSystem.readFileString(fixtureUrl.pathname); + const parsedFixture = parse(fixture) as Record; const config = yield* Schema.decodeUnknownEffect(HermesConfiguration)( - parse(fixture), + parsedFixture, ); assert.strictEqual(config.model.provider, "custom:agentos-gateway"); @@ -80,26 +74,7 @@ suite("Hermes Responses workload client proxy", (it) => { assert.strictEqual(config.agent.api_max_retries, 0); assert.deepStrictEqual(config.fallback_providers, []); assert.strictEqual(config.fallback_model, ""); - const auxiliaryNames = [ - "approval", - "compression", - "mcp", - "skills_hub", - "title_generation", - "triage_specifier", - "vision", - "web_extract", - ]; - assert.deepStrictEqual(Object.keys(config.auxiliary).sort(), auxiliaryNames); - for (const name of auxiliaryNames) { - assert.deepInclude(config.auxiliary[name], { - provider: "custom", - model: config.model.default, - base_url: "http://127.0.0.1:8790/v1", - api_key: "agentos-workload-identity-placeholder", - fallback_chain: [], - }); - } + assert.isFalse(Object.prototype.hasOwnProperty.call(parsedFixture, "auxiliary")); const directory = yield* fileSystem.makeTempDirectoryScoped(); const tokenPath = `${directory}/token`; From bb9998338796b6ee751b13e329d1673a7d648730 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Robin=20Br=C3=A4mer?= <22003767+robinbraemer@users.noreply.github.com> Date: Mon, 10 Aug 2026 16:54:13 +0200 Subject: [PATCH 09/28] no-mistakes(review): Protect leases on abort and normalize 205 responses --- services/ai-gateway/src/forward.ts | 4 +++- services/ai-gateway/src/provider-http.ts | 3 ++- .../src/workload-client-proxy-main.ts | 2 +- .../ai-gateway/tests/forward.effect.test.ts | 23 ++++++++++++++++++- .../workload-client-proxy.effect.test.ts | 16 +++++++++++++ 5 files changed, 44 insertions(+), 4 deletions(-) diff --git a/services/ai-gateway/src/forward.ts b/services/ai-gateway/src/forward.ts index 7cbc7e86..039cb8cf 100644 --- a/services/ai-gateway/src/forward.ts +++ b/services/ai-gateway/src/forward.ts @@ -229,7 +229,9 @@ export const makeAIForwardHandler = Effect.fn( requestTelemetry.upstreamStarted(upstreamRequest.headers), ); const upstreamResult = yield* Effect.result( - options.provider.execute(upstreamRequest), + options.provider.execute(upstreamRequest).pipe( + Effect.onInterrupt(() => releaseLease(lease, requestTelemetry)), + ), ); if (Result.isFailure(upstreamResult)) { yield* diagnostic( diff --git a/services/ai-gateway/src/provider-http.ts b/services/ai-gateway/src/provider-http.ts index 5d41bc66..176561f9 100644 --- a/services/ai-gateway/src/provider-http.ts +++ b/services/ai-gateway/src/provider-http.ts @@ -54,7 +54,8 @@ export const AIProviderHttpLive = Layer.effect( Effect.mapError((error) => providerHttpError(httpErrorCode(error))), ); const hasNoBody = request.method === "HEAD" || - response.status === 204 || response.status === 304; + response.status === 204 || response.status === 205 || + response.status === 304; const body: AIProviderResponse["body"] = hasNoBody ? null : response.stream.pipe( diff --git a/services/ai-gateway/src/workload-client-proxy-main.ts b/services/ai-gateway/src/workload-client-proxy-main.ts index b4259c9c..c5ecaaf0 100644 --- a/services/ai-gateway/src/workload-client-proxy-main.ts +++ b/services/ai-gateway/src/workload-client-proxy-main.ts @@ -61,7 +61,7 @@ export function responseFromUpstream( ) { return Effect.gen(function*() { const headers = responseHeaders(upstream); - if (upstream.body === null) { + if (upstream.body === null || upstream.status === 205) { return new Response(null, { status: upstream.status, headers }); } const body = yield* Stream.toReadableStreamEffect(upstream.body); diff --git a/services/ai-gateway/tests/forward.effect.test.ts b/services/ai-gateway/tests/forward.effect.test.ts index c752b402..4ea18423 100644 --- a/services/ai-gateway/tests/forward.effect.test.ts +++ b/services/ai-gateway/tests/forward.effect.test.ts @@ -8,7 +8,7 @@ import { type ProviderBudgetSettlementReceiptV1, type ProviderBudgetSettlementReportV1, } from "@akua-dev/agentos"; -import { Effect, Fiber, Layer, Metric, Ref, Stream, Tracer } from "effect"; +import { Effect, Exit, Fiber, Layer, Metric, Ref, Stream, Tracer } from "effect"; import { TestClock } from "effect/testing"; import { @@ -821,6 +821,27 @@ describe("Effect AI Gateway forwarding", () => { ]); })); + it.effect("releases an acquired route when the provider is interrupted before headers", () => + Effect.gen(function*() { + const route = yield* makeLease(); + const settlement = yield* makeSettlementRecorder(); + const handler = yield* makeAIForwardHandler({ + authentication: { kind: "workload_identity" }, + acquire: () => Effect.succeed(route.lease), + provider: AIProviderHttp.of({ + execute: () => Effect.interrupt, + }), + settlements: settlement.settlements, + now: Effect.succeed(now), + heartbeatMillis: 40_000, + maximumUsageEventBytes: 4_096, + }); + const exit = yield* Effect.exit(handler(gatewayRequest())); + + assert.isFalse(Exit.isSuccess(exit)); + assert.strictEqual(yield* Ref.get(route.releases), 1); + })); + it.effect("ends telemetry when a finite provider response cannot be constructed", () => Effect.gen(function*() { const route = yield* makeLease(); diff --git a/services/ai-gateway/tests/workload-client-proxy.effect.test.ts b/services/ai-gateway/tests/workload-client-proxy.effect.test.ts index 684ffc68..509bdc1b 100644 --- a/services/ai-gateway/tests/workload-client-proxy.effect.test.ts +++ b/services/ai-gateway/tests/workload-client-proxy.effect.test.ts @@ -264,6 +264,22 @@ suite("Hermes Responses workload client proxy", (it) => { ); })); + it.effect("preserves a 205 response without constructing a body", () => + Effect.gen(function*() { + const response = yield* responseFromUpstream({ + status: 205, + headers: { "content-type": "text/plain" }, + body: Stream.succeed(new TextEncoder().encode("invalid-body")), + }); + + assert.strictEqual(response.status, 205); + assert.isNull(response.body); + assert.strictEqual( + yield* Effect.promise(() => response.text()), + "", + ); + })); + it.effect("preserves upstream response streams across a long gap", () => Effect.gen(function*() { const fileSystem = yield* FileSystem.FileSystem; From b40f5afa8e92f9631a211233cbb14bfb0aa8e8e1 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Robin=20Br=C3=A4mer?= <22003767+robinbraemer@users.noreply.github.com> Date: Mon, 10 Aug 2026 17:05:30 +0200 Subject: [PATCH 10/28] no-mistakes(review): Bracket post-acquisition lease cleanup --- services/ai-gateway/src/forward.ts | 296 +++++++++--------- .../ai-gateway/tests/forward.effect.test.ts | 29 ++ 2 files changed, 185 insertions(+), 140 deletions(-) diff --git a/services/ai-gateway/src/forward.ts b/services/ai-gateway/src/forward.ts index 039cb8cf..c7cc7d1d 100644 --- a/services/ai-gateway/src/forward.ts +++ b/services/ai-gateway/src/forward.ts @@ -18,6 +18,7 @@ import { Effect, Exit, Option, + Ref, Result, Schema, Stream, @@ -188,6 +189,7 @@ export const makeAIForwardHandler = Effect.fn( } const sessionKey = attributed.success; yield* diagnostic(requestTelemetry.routeStarted); + const leaseReleaseState = yield* Ref.make(false); const acquired = yield* Effect.result(options.acquire( sessionKey, request.signal, @@ -214,156 +216,170 @@ export const makeAIForwardHandler = Effect.fn( })); return jsonResponse(503, "no_eligible_account"); } - yield* diagnostic(requestTelemetry.routeEnded("acquired")); + const releaseLeaseOnce = Effect.gen(function*() { + const shouldRelease = yield* Ref.modify( + leaseReleaseState, + (released): readonly [boolean, boolean] => [!released, true], + ); + if (shouldRelease) yield* releaseLease(lease, requestTelemetry); + }); - const upstreamRequest = yield* makeUpstreamRequest(request, url, lease); - if (upstreamRequest === undefined) { - yield* releaseLease(lease, requestTelemetry); - yield* diagnostic(requestTelemetry.end({ - status: 400, - streamOutcome: "not_streamed", - })); - return jsonResponse(400, "invalid_request"); - } - yield* diagnostic( - requestTelemetry.upstreamStarted(upstreamRequest.headers), - ); - const upstreamResult = yield* Effect.result( - options.provider.execute(upstreamRequest).pipe( - Effect.onInterrupt(() => releaseLease(lease, requestTelemetry)), - ), - ); - if (Result.isFailure(upstreamResult)) { + const postAcquisition = Effect.gen(function*() { + yield* diagnostic(requestTelemetry.routeEnded("acquired")); + const upstreamRequest = yield* makeUpstreamRequest(request, url, lease); + if (upstreamRequest === undefined) { + yield* releaseLeaseOnce; + yield* diagnostic(requestTelemetry.end({ + status: 400, + streamOutcome: "not_streamed", + })); + return jsonResponse(400, "invalid_request"); + } yield* diagnostic( - requestTelemetry.upstreamFailed(upstreamResult.failure), + requestTelemetry.upstreamStarted(upstreamRequest.headers), ); - yield* releaseLease(lease, requestTelemetry); - yield* diagnostic(requestTelemetry.end({ - status: 502, - error: upstreamResult.failure, - streamOutcome: request.signal.aborted ? "aborted" : "upstream_error", - })); - return jsonResponse(502, providerErrorCode(upstreamResult.failure)); - } - const upstream = upstreamResult.success; - const headersResult = yield* Effect.result(Effect.try({ - try: () => new Headers(upstream.headers), - catch: () => AIForwardConfigurationError.make({ - code: "invalid_configuration", - }), - })); - if (Result.isFailure(headersResult)) { - yield* diagnostic( - requestTelemetry.upstreamFailed(headersResult.failure), + const upstreamResult = yield* Effect.result( + options.provider.execute(upstreamRequest), ); - yield* releaseLease(lease, requestTelemetry); - yield* diagnostic(requestTelemetry.end({ - status: 502, - error: headersResult.failure, - streamOutcome: "upstream_error", + if (Result.isFailure(upstreamResult)) { + yield* diagnostic( + requestTelemetry.upstreamFailed(upstreamResult.failure), + ); + yield* releaseLeaseOnce; + yield* diagnostic(requestTelemetry.end({ + status: 502, + error: upstreamResult.failure, + streamOutcome: request.signal.aborted ? "aborted" : "upstream_error", + })); + return jsonResponse(502, providerErrorCode(upstreamResult.failure)); + } + const upstream = upstreamResult.success; + const headersResult = yield* Effect.result(Effect.try({ + try: () => new Headers(upstream.headers), + catch: () => AIForwardConfigurationError.make({ + code: "invalid_configuration", + }), })); - return jsonResponse(502, "invalid_provider_response"); - } - const upstreamHeaders = headersResult.success; - yield* diagnostic( - requestTelemetry.upstreamHeaders(upstream.status, upstreamHeaders), - ); - const responseHeaders = sanitizeResponseHeaders(upstreamHeaders); - const responseMetadata = yield* Effect.result(finiteResponse( - null, - upstream.status, - responseHeaders, - )); - if (Result.isFailure(responseMetadata)) { + if (Result.isFailure(headersResult)) { + yield* diagnostic( + requestTelemetry.upstreamFailed(headersResult.failure), + ); + yield* releaseLeaseOnce; + yield* diagnostic(requestTelemetry.end({ + status: 502, + error: headersResult.failure, + streamOutcome: "upstream_error", + })); + return jsonResponse(502, "invalid_provider_response"); + } + const upstreamHeaders = headersResult.success; yield* diagnostic( - requestTelemetry.upstreamFailed(responseMetadata.failure), + requestTelemetry.upstreamHeaders(upstream.status, upstreamHeaders), ); - yield* releaseLease(lease, requestTelemetry); - yield* diagnostic(requestTelemetry.end({ - status: 502, - error: responseMetadata.failure, - streamOutcome: "upstream_error", - })); - return jsonResponse(502, "invalid_provider_response"); - } - yield* recordResponse(lease, upstream.status, upstreamHeaders); - if (upstream.body === null) { - yield* settleWithoutBody( - options.settlements, - authentication.authorization, + const responseHeaders = sanitizeResponseHeaders(upstreamHeaders); + const responseMetadata = yield* Effect.result(finiteResponse( + null, upstream.status, - ); - yield* releaseLease(lease, requestTelemetry); - yield* diagnostic(requestTelemetry.end({ - status: upstream.status, - streamOutcome: "not_streamed", - })); - return responseMetadata.success; - } - if (!responseStatusAllowsBody(upstream.status)) { - const failure = AIForwardConfigurationError.make({ - code: "invalid_configuration", - }); - yield* diagnostic(requestTelemetry.upstreamFailed(failure)); - yield* releaseLease(lease, requestTelemetry); - yield* diagnostic(requestTelemetry.end({ - status: 502, - error: failure, - streamOutcome: "upstream_error", - })); - return jsonResponse(502, "invalid_provider_response"); - } - - const observerResult = upstream.status < 400 && - isEventStream(upstreamHeaders) - ? yield* Effect.result(makeOpenAITerminalUsageObserver({ - maximumEventBytes: options.maximumUsageEventBytes, - })) - : undefined; - if (observerResult !== undefined && Result.isFailure(observerResult)) { - yield* releaseLease(lease, requestTelemetry); - return jsonResponse(503, "accounting_unavailable"); - } - const observer = observerResult === undefined - ? undefined - : observerResult.success; - const monitored = upstream.body.pipe( - Stream.tap((chunk) => - diagnostic(requestTelemetry.streamChunk(chunk.byteLength)) - ), - observer === undefined - ? (stream) => stream - : Stream.tap((chunk) => observer.observe(chunk)), - Stream.onExit((exit) => - finalizeStream( - exit, - lease, + responseHeaders, + )); + if (Result.isFailure(responseMetadata)) { + yield* diagnostic( + requestTelemetry.upstreamFailed(responseMetadata.failure), + ); + yield* releaseLeaseOnce; + yield* diagnostic(requestTelemetry.end({ + status: 502, + error: responseMetadata.failure, + streamOutcome: "upstream_error", + })); + return jsonResponse(502, "invalid_provider_response"); + } + yield* recordResponse(lease, upstream.status, upstreamHeaders); + if (upstream.body === null) { + yield* settleWithoutBody( options.settlements, authentication.authorization, upstream.status, - observer, - requestTelemetry, - request.signal, - ) + ); + yield* releaseLeaseOnce; + yield* diagnostic(requestTelemetry.end({ + status: upstream.status, + streamOutcome: "not_streamed", + })); + return responseMetadata.success; + } + if (!responseStatusAllowsBody(upstream.status)) { + const failure = AIForwardConfigurationError.make({ + code: "invalid_configuration", + }); + yield* diagnostic(requestTelemetry.upstreamFailed(failure)); + yield* releaseLeaseOnce; + yield* diagnostic(requestTelemetry.end({ + status: 502, + error: failure, + streamOutcome: "upstream_error", + })); + return jsonResponse(502, "invalid_provider_response"); + } + + const observerResult = upstream.status < 400 && + isEventStream(upstreamHeaders) + ? yield* Effect.result(makeOpenAITerminalUsageObserver({ + maximumEventBytes: options.maximumUsageEventBytes, + })) + : undefined; + if (observerResult !== undefined && Result.isFailure(observerResult)) { + yield* releaseLeaseOnce; + return jsonResponse(503, "accounting_unavailable"); + } + const observer = observerResult === undefined + ? undefined + : observerResult.success; + const monitored = upstream.body.pipe( + Stream.tap((chunk) => + diagnostic(requestTelemetry.streamChunk(chunk.byteLength)) + ), + observer === undefined + ? (stream) => stream + : Stream.tap((chunk) => observer.observe(chunk)), + Stream.onExit((exit) => + finalizeStream( + exit, + releaseLeaseOnce, + options.settlements, + authentication.authorization, + upstream.status, + observer, + requestTelemetry, + request.signal, + ) + ), + ); + const scoped = Stream.unwrap(Effect.gen(function*() { + yield* heartbeat(lease, options.heartbeatMillis).pipe( + Effect.forkScoped({ startImmediately: true }), + ); + return monitored; + })); + const body = yield* Stream.toReadableStreamEffect(scoped); + const responseResult = yield* Effect.result(finiteResponse( + body, + upstream.status, + responseHeaders, + )); + if (Result.isFailure(responseResult)) { + yield* releaseLeaseOnce; + return jsonResponse(502, "invalid_provider_response"); + } + return responseResult.success; + }); + return yield* postAcquisition.pipe( + Effect.onExit((exit) => + Exit.isSuccess(exit) + ? Effect.void + : Effect.uninterruptible(releaseLeaseOnce) ), ); - const scoped = Stream.unwrap(Effect.gen(function*() { - yield* heartbeat(lease, options.heartbeatMillis).pipe( - Effect.forkScoped({ startImmediately: true }), - ); - return monitored; - })); - const body = yield* Stream.toReadableStreamEffect(scoped); - const responseResult = yield* Effect.result(finiteResponse( - body, - upstream.status, - responseHeaders, - )); - if (Result.isFailure(responseResult)) { - yield* releaseLease(lease, requestTelemetry); - return jsonResponse(502, "invalid_provider_response"); - } - return responseResult.success; }); return handler; }); @@ -495,7 +511,7 @@ function heartbeat(lease: AIForwardLease, heartbeatMillis: number) { function finalizeStream( exit: Exit.Exit, - lease: AIForwardLease, + releaseLeaseOnce: Effect.Effect, settlements: ProviderBudgetSettlementReporter["Service"], authorization: ProviderAuthorizationGrantV1 | undefined, status: number, @@ -522,7 +538,7 @@ function finalizeStream( } } } - yield* releaseLease(lease, telemetry); + yield* releaseLeaseOnce; const outcome = telemetryStreamOutcome(exit, signal); const failure = Exit.isFailure(exit) ? Option.getOrUndefined(Cause.findErrorOption(exit.cause)) @@ -533,7 +549,7 @@ function finalizeStream( ...(failure === undefined ? {} : { error: failure }), })); }).pipe( - Effect.catchCause(() => releaseLease(lease, telemetry)), + Effect.catchCause(() => releaseLeaseOnce), Effect.catchCause(() => Effect.void), Effect.uninterruptible, ); diff --git a/services/ai-gateway/tests/forward.effect.test.ts b/services/ai-gateway/tests/forward.effect.test.ts index 4ea18423..70ff0574 100644 --- a/services/ai-gateway/tests/forward.effect.test.ts +++ b/services/ai-gateway/tests/forward.effect.test.ts @@ -842,6 +842,35 @@ describe("Effect AI Gateway forwarding", () => { assert.strictEqual(yield* Ref.get(route.releases), 1); })); + it.effect("releases an acquired route when post-acquisition work defects", () => + Effect.gen(function*() { + const route = yield* makeLease(); + const settlement = yield* makeSettlementRecorder(); + const brokenLease: AIForwardLease = { + kind: route.lease.kind, + get accessToken(): string { + throw new Error("invalid lease"); + }, + renew: route.lease.renew, + release: route.lease.release, + }; + const handler = yield* makeAIForwardHandler({ + authentication: { kind: "workload_identity" }, + acquire: () => Effect.succeed(brokenLease), + provider: AIProviderHttp.of({ + execute: () => Effect.die("provider must not execute"), + }), + settlements: settlement.settlements, + now: Effect.succeed(now), + heartbeatMillis: 40_000, + maximumUsageEventBytes: 4_096, + }); + const exit = yield* Effect.exit(handler(gatewayRequest())); + + assert.isFalse(Exit.isSuccess(exit)); + assert.strictEqual(yield* Ref.get(route.releases), 1); + })); + it.effect("ends telemetry when a finite provider response cannot be constructed", () => Effect.gen(function*() { const route = yield* makeLease(); From 83786fbf5bf890433eac3525affe45b569d763c9 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Robin=20Br=C3=A4mer?= <22003767+robinbraemer@users.noreply.github.com> Date: Mon, 10 Aug 2026 17:14:36 +0200 Subject: [PATCH 11/28] no-mistakes(review): Release routing reservations on acquisition cancellation --- services/ai-gateway/src/gateway-service.ts | 89 +++++++++++-------- .../tests/gateway-service.effect.test.ts | 26 +++++- 2 files changed, 75 insertions(+), 40 deletions(-) diff --git a/services/ai-gateway/src/gateway-service.ts b/services/ai-gateway/src/gateway-service.ts index e9fb860a..10f38b50 100644 --- a/services/ai-gateway/src/gateway-service.ts +++ b/services/ai-gateway/src/gateway-service.ts @@ -7,6 +7,7 @@ import { import { Clock, Effect, + Exit, Ref, Result, Schema, @@ -151,50 +152,60 @@ export const makeAIGatewayApplication = Effect.fn( ? fallbackLease(fallbackKey) : undefined; } + const releaseReservation = routing.release(reservation.leaseToken).pipe( + Effect.asVoid, + Effect.catchCause(() => Effect.void), + Effect.uninterruptible, + ); const credentialResult = yield* Effect.result( - vault.getFreshCredential(reservation.accountId), + Effect.gen(function*() { + const credential = yield* vault.getFreshCredential( + reservation.accountId, + ); + const lease: AIForwardLease = { + kind: "codex_oauth", + accessToken: credential.accessToken, + providerAccountId: credential.providerAccountId, + renew: Clock.currentTimeMillis.pipe( + Effect.flatMap((renewedAt) => + routing.renew(reservation.leaseToken, renewedAt) + ), + Effect.mapError(() => routeError("state_unavailable")), + ), + release: routing.release(reservation.leaseToken).pipe( + Effect.asVoid, + Effect.mapError(() => routeError("state_unavailable")), + ), + recordResponse: (status, headers) => + Effect.gen(function*() { + const responseAt = yield* Clock.currentTimeMillis; + if (status === 401) { + yield* vault.markNeedsReauth( + reservation.accountId, + credential.accessToken, + ).pipe(Effect.catchCause(() => Effect.succeed(false))); + } + yield* routing.recordResponse( + reservation.accountId, + status, + headers, + responseAt, + ).pipe( + Effect.mapError(() => routeError("state_unavailable")), + ); + }), + }; + return lease; + }).pipe( + Effect.onExit((exit) => + Exit.isSuccess(exit) ? Effect.void : releaseReservation + ), + ), ); if (Result.isFailure(credentialResult)) { - yield* routing.release(reservation.leaseToken).pipe( - Effect.catchCause(() => Effect.void), - ); return yield* routeError(routeCodeForAccount(credentialResult.failure)); } - const credential = credentialResult.success; - const lease: AIForwardLease = { - kind: "codex_oauth", - accessToken: credential.accessToken, - providerAccountId: credential.providerAccountId, - renew: Clock.currentTimeMillis.pipe( - Effect.flatMap((renewedAt) => - routing.renew(reservation.leaseToken, renewedAt) - ), - Effect.mapError(() => routeError("state_unavailable")), - ), - release: routing.release(reservation.leaseToken).pipe( - Effect.asVoid, - Effect.mapError(() => routeError("state_unavailable")), - ), - recordResponse: (status, headers) => - Effect.gen(function*() { - const responseAt = yield* Clock.currentTimeMillis; - if (status === 401) { - yield* vault.markNeedsReauth( - reservation.accountId, - credential.accessToken, - ).pipe(Effect.catchCause(() => Effect.succeed(false))); - } - yield* routing.recordResponse( - reservation.accountId, - status, - headers, - responseAt, - ).pipe( - Effect.mapError(() => routeError("state_unavailable")), - ); - }), - }; - return lease; + return credentialResult.success; }, ); diff --git a/services/ai-gateway/tests/gateway-service.effect.test.ts b/services/ai-gateway/tests/gateway-service.effect.test.ts index a9ef4b1b..c71d8a28 100644 --- a/services/ai-gateway/tests/gateway-service.effect.test.ts +++ b/services/ai-gateway/tests/gateway-service.effect.test.ts @@ -10,7 +10,7 @@ import { type ProviderBudgetSettlementReceiptV1, type ProviderBudgetSettlementReportV1, } from "@akua-dev/agentos"; -import { Effect, Ref, Stream } from "effect"; +import { Deferred, Effect, Fiber, Ref, Stream } from "effect"; import { TestClock } from "effect/testing"; import { @@ -379,6 +379,30 @@ describe("Effect AI Gateway application", () => { }]); })); + it.effect("releases a reservation when credential acquisition is interrupted", () => + Effect.gen(function*() { + yield* TestClock.setTime(now); + const services = yield* makeTestServices(true); + const credentialStarted = yield* Deferred.make(); + const application = yield* makeApplication({ + ...services, + vault: ManagedAccountVault.of({ + ...services.vault, + getFreshCredential: () => + Deferred.succeed(credentialStarted, undefined).pipe( + Effect.andThen(Effect.never), + ), + }), + }); + const requestFiber = yield* Effect.forkChild(Effect.exit( + application.handle(providerRequest()), + )); + yield* Deferred.await(credentialStarted); + yield* Fiber.interrupt(requestFiber); + + assert.strictEqual(yield* Ref.get(services.released), 1); + })); + it.effect("preserves serving behavior with native telemetry enabled and disabled", () => Effect.forEach([true, false], (enabled) => Effect.gen(function*() { From 3276627faf75e8a65f6a107297bfc1042cf43425 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Robin=20Br=C3=A4mer?= <22003767+robinbraemer@users.noreply.github.com> Date: Mon, 10 Aug 2026 17:23:27 +0200 Subject: [PATCH 12/28] no-mistakes(review): Protect routing leases across acquisition bookkeeping cancellation --- .../ai-gateway/src/effect-routing-state.ts | 44 +++++++++++++------ 1 file changed, 31 insertions(+), 13 deletions(-) diff --git a/services/ai-gateway/src/effect-routing-state.ts b/services/ai-gateway/src/effect-routing-state.ts index 6b078d6e..e2cfff12 100644 --- a/services/ai-gateway/src/effect-routing-state.ts +++ b/services/ai-gateway/src/effect-routing-state.ts @@ -8,6 +8,7 @@ import { } from "@akua-dev/codex-router/core"; import { Effect, + Exit, Layer, Option, Ref, @@ -112,19 +113,36 @@ export function makeEffectAIRoutingStateLayer( : acquired.accountId === decision.accountId ? decision.reason : "current_account_hysteresis"; - yield* Ref.set(lastSelection, Option.some({ - observedAt: input.now, - reason: decisionReason, - candidates: decision.candidates, - })); - return acquired === undefined - ? undefined - : { - accountId: acquired.accountId, - leaseToken: acquired.leaseToken, - expiresAt: acquired.expiresAt, - decisionReason, - }; + if (acquired === undefined) { + yield* Ref.set(lastSelection, Option.some({ + observedAt: input.now, + reason: decisionReason, + candidates: decision.candidates, + })); + return undefined; + } + const releaseLease = routing.release(acquired.leaseToken).pipe( + Effect.asVoid, + Effect.catchCause(() => Effect.void), + Effect.uninterruptible, + ); + return yield* Effect.gen(function*() { + yield* Ref.set(lastSelection, Option.some({ + observedAt: input.now, + reason: decisionReason, + candidates: decision.candidates, + })); + return { + accountId: acquired.accountId, + leaseToken: acquired.leaseToken, + expiresAt: acquired.expiresAt, + decisionReason, + }; + }).pipe( + Effect.onExit((exit) => + Exit.isSuccess(exit) ? Effect.void : releaseLease + ), + ); })); const evaluate: AIRoutingState["Service"]["evaluate"] = (input) => From 52a75621694f4264052c05a691bc1698a8844274 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Robin=20Br=C3=A4mer?= <22003767+robinbraemer@users.noreply.github.com> Date: Mon, 10 Aug 2026 17:39:59 +0200 Subject: [PATCH 13/28] no-mistakes(review): Make routing lease handoffs cancellation-safe --- .../ai-gateway/src/effect-routing-state.ts | 86 ++++++++------- services/ai-gateway/src/gateway-service.ts | 103 ++++++++++-------- 2 files changed, 104 insertions(+), 85 deletions(-) diff --git a/services/ai-gateway/src/effect-routing-state.ts b/services/ai-gateway/src/effect-routing-state.ts index e2cfff12..104a14bb 100644 --- a/services/ai-gateway/src/effect-routing-state.ts +++ b/services/ai-gateway/src/effect-routing-state.ts @@ -100,49 +100,51 @@ export function makeEffectAIRoutingStateLayer( config, input.now, ); - const lease = yield* routing.acquire({ - candidates: input.candidates.map(toRouterCandidate), - now: input.now, - ...(input.sessionKey === undefined - ? {} - : { sessionKey: SessionKey.make(input.sessionKey) }), - }); - const acquired = Option.getOrUndefined(lease); - const decisionReason = acquired === undefined - ? decision.reason - : acquired.accountId === decision.accountId + return yield* Effect.uninterruptible(Effect.gen(function*() { + const lease = yield* routing.acquire({ + candidates: input.candidates.map(toRouterCandidate), + now: input.now, + ...(input.sessionKey === undefined + ? {} + : { sessionKey: SessionKey.make(input.sessionKey) }), + }); + const acquired = Option.getOrUndefined(lease); + const decisionReason = acquired === undefined ? decision.reason - : "current_account_hysteresis"; - if (acquired === undefined) { - yield* Ref.set(lastSelection, Option.some({ - observedAt: input.now, - reason: decisionReason, - candidates: decision.candidates, - })); - return undefined; - } - const releaseLease = routing.release(acquired.leaseToken).pipe( - Effect.asVoid, - Effect.catchCause(() => Effect.void), - Effect.uninterruptible, - ); - return yield* Effect.gen(function*() { - yield* Ref.set(lastSelection, Option.some({ - observedAt: input.now, - reason: decisionReason, - candidates: decision.candidates, - })); - return { - accountId: acquired.accountId, - leaseToken: acquired.leaseToken, - expiresAt: acquired.expiresAt, - decisionReason, - }; - }).pipe( - Effect.onExit((exit) => - Exit.isSuccess(exit) ? Effect.void : releaseLease - ), - ); + : acquired.accountId === decision.accountId + ? decision.reason + : "current_account_hysteresis"; + if (acquired === undefined) { + yield* Ref.set(lastSelection, Option.some({ + observedAt: input.now, + reason: decisionReason, + candidates: decision.candidates, + })); + return undefined; + } + const releaseLease = routing.release(acquired.leaseToken).pipe( + Effect.asVoid, + Effect.catchCause(() => Effect.void), + Effect.uninterruptible, + ); + return yield* Effect.gen(function*() { + yield* Ref.set(lastSelection, Option.some({ + observedAt: input.now, + reason: decisionReason, + candidates: decision.candidates, + })); + return { + accountId: acquired.accountId, + leaseToken: acquired.leaseToken, + expiresAt: acquired.expiresAt, + decisionReason, + }; + }).pipe( + Effect.onExit((exit) => + Exit.isSuccess(exit) ? Effect.void : releaseLease + ), + ); + })); })); const evaluate: AIRoutingState["Service"]["evaluate"] = (input) => diff --git a/services/ai-gateway/src/gateway-service.ts b/services/ai-gateway/src/gateway-service.ts index 10f38b50..3bbb2dc1 100644 --- a/services/ai-gateway/src/gateway-service.ts +++ b/services/ai-gateway/src/gateway-service.ts @@ -142,6 +142,7 @@ export const makeAIGatewayApplication = Effect.fn( ).pipe(Effect.catchCause(() => Effect.void)), { discard: true }, ); + const releaseReservationState = yield* Ref.make(false); const reservation = yield* routing.acquire({ candidates: set.candidates, now: currentTime, @@ -157,55 +158,71 @@ export const makeAIGatewayApplication = Effect.fn( Effect.catchCause(() => Effect.void), Effect.uninterruptible, ); - const credentialResult = yield* Effect.result( - Effect.gen(function*() { - const credential = yield* vault.getFreshCredential( - reservation.accountId, - ); - const lease: AIForwardLease = { - kind: "codex_oauth", - accessToken: credential.accessToken, - providerAccountId: credential.providerAccountId, - renew: Clock.currentTimeMillis.pipe( - Effect.flatMap((renewedAt) => - routing.renew(reservation.leaseToken, renewedAt) + const releaseReservationOnce = Effect.gen(function*() { + const shouldRelease = yield* Ref.modify( + releaseReservationState, + (released): readonly [boolean, boolean] => [!released, true], + ); + if (shouldRelease) yield* releaseReservation; + }).pipe(Effect.uninterruptible); + return yield* Effect.gen(function*() { + const credentialResult = yield* Effect.result( + Effect.gen(function*() { + const credential = yield* vault.getFreshCredential( + reservation.accountId, + ); + const lease: AIForwardLease = { + kind: "codex_oauth", + accessToken: credential.accessToken, + providerAccountId: credential.providerAccountId, + renew: Clock.currentTimeMillis.pipe( + Effect.flatMap((renewedAt) => + routing.renew(reservation.leaseToken, renewedAt) + ), + Effect.mapError(() => routeError("state_unavailable")), ), - Effect.mapError(() => routeError("state_unavailable")), - ), - release: routing.release(reservation.leaseToken).pipe( - Effect.asVoid, - Effect.mapError(() => routeError("state_unavailable")), - ), - recordResponse: (status, headers) => - Effect.gen(function*() { - const responseAt = yield* Clock.currentTimeMillis; - if (status === 401) { - yield* vault.markNeedsReauth( + release: routing.release(reservation.leaseToken).pipe( + Effect.asVoid, + Effect.mapError(() => routeError("state_unavailable")), + ), + recordResponse: (status, headers) => + Effect.gen(function*() { + const responseAt = yield* Clock.currentTimeMillis; + if (status === 401) { + yield* vault.markNeedsReauth( + reservation.accountId, + credential.accessToken, + ).pipe(Effect.catchCause(() => Effect.succeed(false))); + } + yield* routing.recordResponse( reservation.accountId, - credential.accessToken, - ).pipe(Effect.catchCause(() => Effect.succeed(false))); - } - yield* routing.recordResponse( - reservation.accountId, - status, - headers, - responseAt, - ).pipe( - Effect.mapError(() => routeError("state_unavailable")), - ); - }), - }; - return lease; - }).pipe( + status, + headers, + responseAt, + ).pipe( + Effect.mapError(() => routeError("state_unavailable")), + ); + }), + }; + return lease; + }), + ).pipe( Effect.onExit((exit) => - Exit.isSuccess(exit) ? Effect.void : releaseReservation + Exit.isSuccess(exit) ? Effect.void : releaseReservationOnce ), + ); + if (Result.isFailure(credentialResult)) { + yield* releaseReservationOnce; + return yield* routeError( + routeCodeForAccount(credentialResult.failure), + ); + } + return credentialResult.success; + }).pipe( + Effect.onExit((exit) => + Exit.isSuccess(exit) ? Effect.void : releaseReservationOnce ), ); - if (Result.isFailure(credentialResult)) { - return yield* routeError(routeCodeForAccount(credentialResult.failure)); - } - return credentialResult.success; }, ); From 27ed5111b5b73dfc11d55997f892c96701c14b04 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Robin=20Br=C3=A4mer?= <22003767+robinbraemer@users.noreply.github.com> Date: Mon, 10 Aug 2026 17:51:44 +0200 Subject: [PATCH 14/28] no-mistakes(review): Make routing ownership handoff interruptible and safe --- .../ai-gateway/src/effect-routing-state.ts | 90 ++++++++++--------- 1 file changed, 46 insertions(+), 44 deletions(-) diff --git a/services/ai-gateway/src/effect-routing-state.ts b/services/ai-gateway/src/effect-routing-state.ts index 104a14bb..ebc12e54 100644 --- a/services/ai-gateway/src/effect-routing-state.ts +++ b/services/ai-gateway/src/effect-routing-state.ts @@ -100,51 +100,53 @@ export function makeEffectAIRoutingStateLayer( config, input.now, ); - return yield* Effect.uninterruptible(Effect.gen(function*() { - const lease = yield* routing.acquire({ - candidates: input.candidates.map(toRouterCandidate), - now: input.now, - ...(input.sessionKey === undefined - ? {} - : { sessionKey: SessionKey.make(input.sessionKey) }), - }); - const acquired = Option.getOrUndefined(lease); - const decisionReason = acquired === undefined - ? decision.reason - : acquired.accountId === decision.accountId + return yield* Effect.uninterruptibleMask((restore) => + Effect.gen(function*() { + const lease = yield* routing.acquire({ + candidates: input.candidates.map(toRouterCandidate), + now: input.now, + ...(input.sessionKey === undefined + ? {} + : { sessionKey: SessionKey.make(input.sessionKey) }), + }); + const acquired = Option.getOrUndefined(lease); + const decisionReason = acquired === undefined ? decision.reason - : "current_account_hysteresis"; - if (acquired === undefined) { - yield* Ref.set(lastSelection, Option.some({ - observedAt: input.now, - reason: decisionReason, - candidates: decision.candidates, - })); - return undefined; - } - const releaseLease = routing.release(acquired.leaseToken).pipe( - Effect.asVoid, - Effect.catchCause(() => Effect.void), - Effect.uninterruptible, - ); - return yield* Effect.gen(function*() { - yield* Ref.set(lastSelection, Option.some({ - observedAt: input.now, - reason: decisionReason, - candidates: decision.candidates, - })); - return { - accountId: acquired.accountId, - leaseToken: acquired.leaseToken, - expiresAt: acquired.expiresAt, - decisionReason, - }; - }).pipe( - Effect.onExit((exit) => - Exit.isSuccess(exit) ? Effect.void : releaseLease - ), - ); - })); + : acquired.accountId === decision.accountId + ? decision.reason + : "current_account_hysteresis"; + if (acquired === undefined) { + yield* Ref.set(lastSelection, Option.some({ + observedAt: input.now, + reason: decisionReason, + candidates: decision.candidates, + })); + return undefined; + } + const releaseLease = routing.release(acquired.leaseToken).pipe( + Effect.asVoid, + Effect.catchCause(() => Effect.void), + Effect.uninterruptible, + ); + return yield* restore(Effect.gen(function*() { + yield* Ref.set(lastSelection, Option.some({ + observedAt: input.now, + reason: decisionReason, + candidates: decision.candidates, + })); + return { + accountId: acquired.accountId, + leaseToken: acquired.leaseToken, + expiresAt: acquired.expiresAt, + decisionReason, + }; + }).pipe( + Effect.onExit((exit) => + Exit.isSuccess(exit) ? Effect.void : releaseLease + ), + )); + }), + ); })); const evaluate: AIRoutingState["Service"]["evaluate"] = (input) => From 1d81c4b99fb50f3afd50d8551f70fd76350cb5ea Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Robin=20Br=C3=A4mer?= <22003767+robinbraemer@users.noreply.github.com> Date: Mon, 10 Aug 2026 18:05:28 +0200 Subject: [PATCH 15/28] no-mistakes(review): Use interruption-safe resource ownership across gateway leases --- .../ai-gateway/src/effect-routing-state.ts | 64 ++++---- services/ai-gateway/src/forward.ts | 4 +- services/ai-gateway/src/gateway-service.ts | 149 +++++++++--------- 3 files changed, 103 insertions(+), 114 deletions(-) diff --git a/services/ai-gateway/src/effect-routing-state.ts b/services/ai-gateway/src/effect-routing-state.ts index ebc12e54..f3bb6836 100644 --- a/services/ai-gateway/src/effect-routing-state.ts +++ b/services/ai-gateway/src/effect-routing-state.ts @@ -100,52 +100,46 @@ export function makeEffectAIRoutingStateLayer( config, input.now, ); - return yield* Effect.uninterruptibleMask((restore) => - Effect.gen(function*() { - const lease = yield* routing.acquire({ - candidates: input.candidates.map(toRouterCandidate), - now: input.now, - ...(input.sessionKey === undefined - ? {} - : { sessionKey: SessionKey.make(input.sessionKey) }), - }); + return yield* Effect.acquireUseRelease( + routing.acquire({ + candidates: input.candidates.map(toRouterCandidate), + now: input.now, + ...(input.sessionKey === undefined + ? {} + : { sessionKey: SessionKey.make(input.sessionKey) }), + }), + (lease) => Effect.gen(function*() { const acquired = Option.getOrUndefined(lease); const decisionReason = acquired === undefined ? decision.reason : acquired.accountId === decision.accountId ? decision.reason : "current_account_hysteresis"; - if (acquired === undefined) { - yield* Ref.set(lastSelection, Option.some({ - observedAt: input.now, - reason: decisionReason, - candidates: decision.candidates, - })); - return undefined; + yield* Ref.set(lastSelection, Option.some({ + observedAt: input.now, + reason: decisionReason, + candidates: decision.candidates, + })); + return acquired === undefined + ? undefined + : { + accountId: acquired.accountId, + leaseToken: acquired.leaseToken, + expiresAt: acquired.expiresAt, + decisionReason, + }; + }), + (lease, exit) => { + const acquired = Option.getOrUndefined(lease); + if (acquired === undefined || Exit.isSuccess(exit)) { + return Effect.void; } - const releaseLease = routing.release(acquired.leaseToken).pipe( + return routing.release(acquired.leaseToken).pipe( Effect.asVoid, Effect.catchCause(() => Effect.void), Effect.uninterruptible, ); - return yield* restore(Effect.gen(function*() { - yield* Ref.set(lastSelection, Option.some({ - observedAt: input.now, - reason: decisionReason, - candidates: decision.candidates, - })); - return { - accountId: acquired.accountId, - leaseToken: acquired.leaseToken, - expiresAt: acquired.expiresAt, - decisionReason, - }; - }).pipe( - Effect.onExit((exit) => - Exit.isSuccess(exit) ? Effect.void : releaseLease - ), - )); - }), + }, ); })); diff --git a/services/ai-gateway/src/forward.ts b/services/ai-gateway/src/forward.ts index c7cc7d1d..860ebca8 100644 --- a/services/ai-gateway/src/forward.ts +++ b/services/ai-gateway/src/forward.ts @@ -216,13 +216,13 @@ export const makeAIForwardHandler = Effect.fn( })); return jsonResponse(503, "no_eligible_account"); } - const releaseLeaseOnce = Effect.gen(function*() { + const releaseLeaseOnce = Effect.uninterruptible(Effect.gen(function*() { const shouldRelease = yield* Ref.modify( leaseReleaseState, (released): readonly [boolean, boolean] => [!released, true], ); if (shouldRelease) yield* releaseLease(lease, requestTelemetry); - }); + })); const postAcquisition = Effect.gen(function*() { yield* diagnostic(requestTelemetry.routeEnded("acquired")); diff --git a/services/ai-gateway/src/gateway-service.ts b/services/ai-gateway/src/gateway-service.ts index 3bbb2dc1..d4a855a0 100644 --- a/services/ai-gateway/src/gateway-service.ts +++ b/services/ai-gateway/src/gateway-service.ts @@ -142,87 +142,82 @@ export const makeAIGatewayApplication = Effect.fn( ).pipe(Effect.catchCause(() => Effect.void)), { discard: true }, ); - const releaseReservationState = yield* Ref.make(false); - const reservation = yield* routing.acquire({ - candidates: set.candidates, - now: currentTime, - ...(sessionKey === undefined ? {} : { sessionKey }), - }).pipe(Effect.mapError(() => routeError("routing_unavailable"))); - if (reservation === undefined) { - return fallbackAvailable && fallbackKey !== undefined - ? fallbackLease(fallbackKey) - : undefined; - } - const releaseReservation = routing.release(reservation.leaseToken).pipe( - Effect.asVoid, - Effect.catchCause(() => Effect.void), - Effect.uninterruptible, - ); - const releaseReservationOnce = Effect.gen(function*() { - const shouldRelease = yield* Ref.modify( - releaseReservationState, - (released): readonly [boolean, boolean] => [!released, true], - ); - if (shouldRelease) yield* releaseReservation; - }).pipe(Effect.uninterruptible); - return yield* Effect.gen(function*() { - const credentialResult = yield* Effect.result( - Effect.gen(function*() { - const credential = yield* vault.getFreshCredential( - reservation.accountId, - ); - const lease: AIForwardLease = { - kind: "codex_oauth", - accessToken: credential.accessToken, - providerAccountId: credential.providerAccountId, - renew: Clock.currentTimeMillis.pipe( - Effect.flatMap((renewedAt) => - routing.renew(reservation.leaseToken, renewedAt) + const credentialResult = yield* Effect.result( + Effect.acquireUseRelease( + routing.acquire({ + candidates: set.candidates, + now: currentTime, + ...(sessionKey === undefined ? {} : { sessionKey }), + }).pipe(Effect.mapError(() => routeError("routing_unavailable"))), + (reservation) => { + if (reservation === undefined) { + return Effect.succeed( + fallbackAvailable && fallbackKey !== undefined + ? fallbackLease(fallbackKey) + : undefined, + ); + } + return Effect.gen(function*() { + const credential = yield* vault.getFreshCredential( + reservation.accountId, + ); + const lease: AIForwardLease = { + kind: "codex_oauth", + accessToken: credential.accessToken, + providerAccountId: credential.providerAccountId, + renew: Clock.currentTimeMillis.pipe( + Effect.flatMap((renewedAt) => + routing.renew(reservation.leaseToken, renewedAt) + ), + Effect.mapError(() => routeError("state_unavailable")), ), - Effect.mapError(() => routeError("state_unavailable")), - ), - release: routing.release(reservation.leaseToken).pipe( - Effect.asVoid, - Effect.mapError(() => routeError("state_unavailable")), - ), - recordResponse: (status, headers) => - Effect.gen(function*() { - const responseAt = yield* Clock.currentTimeMillis; - if (status === 401) { - yield* vault.markNeedsReauth( + release: routing.release(reservation.leaseToken).pipe( + Effect.asVoid, + Effect.mapError(() => routeError("state_unavailable")), + ), + recordResponse: (status, headers) => + Effect.gen(function*() { + const responseAt = yield* Clock.currentTimeMillis; + if (status === 401) { + yield* vault.markNeedsReauth( + reservation.accountId, + credential.accessToken, + ).pipe(Effect.catchCause(() => Effect.succeed(false))); + } + yield* routing.recordResponse( reservation.accountId, - credential.accessToken, - ).pipe(Effect.catchCause(() => Effect.succeed(false))); - } - yield* routing.recordResponse( - reservation.accountId, - status, - headers, - responseAt, - ).pipe( - Effect.mapError(() => routeError("state_unavailable")), - ); - }), - }; - return lease; - }), - ).pipe( - Effect.onExit((exit) => - Exit.isSuccess(exit) ? Effect.void : releaseReservationOnce - ), - ); - if (Result.isFailure(credentialResult)) { - yield* releaseReservationOnce; - return yield* routeError( - routeCodeForAccount(credentialResult.failure), - ); - } - return credentialResult.success; - }).pipe( - Effect.onExit((exit) => - Exit.isSuccess(exit) ? Effect.void : releaseReservationOnce + status, + headers, + responseAt, + ).pipe( + Effect.mapError(() => routeError("state_unavailable")), + ); + }), + }; + return lease; + }); + }, + (reservation, exit) => { + if (reservation === undefined || Exit.isSuccess(exit)) { + return Effect.void; + } + return routing.release(reservation.leaseToken).pipe( + Effect.asVoid, + Effect.catchCause(() => Effect.void), + Effect.uninterruptible, + ); + }, ), ); + if (Result.isFailure(credentialResult)) { + if (credentialResult.failure._tag === "AIForwardRouteError") { + return yield* credentialResult.failure; + } + return yield* routeError( + routeCodeForAccount(credentialResult.failure), + ); + } + return credentialResult.success; }, ); From 92c3336cdef344b4de751d73b8e317848287a271 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Robin=20Br=C3=A4mer?= <22003767+robinbraemer@users.noreply.github.com> Date: Mon, 10 Aug 2026 18:26:55 +0200 Subject: [PATCH 16/28] no-mistakes(review): Make gateway lease handoffs cancellation-safe --- .../ai-gateway/src/effect-routing-state.ts | 61 +-- services/ai-gateway/src/forward.ts | 378 +++++++++--------- services/ai-gateway/src/gateway-service.ts | 152 +++---- services/ai-gateway/src/routing-service.ts | 8 +- .../ai-gateway/tests/forward.effect.test.ts | 72 ++-- .../tests/gateway-service.effect.test.ts | 14 +- .../tests/routing-state.effect.test.ts | 110 ++--- 7 files changed, 420 insertions(+), 375 deletions(-) diff --git a/services/ai-gateway/src/effect-routing-state.ts b/services/ai-gateway/src/effect-routing-state.ts index f3bb6836..5246f9d9 100644 --- a/services/ai-gateway/src/effect-routing-state.ts +++ b/services/ai-gateway/src/effect-routing-state.ts @@ -8,7 +8,6 @@ import { } from "@akua-dev/codex-router/core"; import { Effect, - Exit, Layer, Option, Ref, @@ -68,9 +67,9 @@ export function makeEffectAIRoutingStateLayer( }; })); - const acquire: AIRoutingState["Service"]["acquire"] = (input) => - routeEffect(Effect.gen(function*() { - let durable = yield* routing.summary(input.now); + const acquire: AIRoutingState["Service"]["acquire"] = (input, use) => + Effect.gen(function*() { + let durable = yield* routeEffect(routing.summary(input.now)); const recovered = durable.accounts.filter((account) => account.requiresReauthentication && input.candidates.some((candidate) => @@ -78,7 +77,7 @@ export function makeEffectAIRoutingStateLayer( !candidate.needsReauth ) ); - yield* Effect.forEach( + yield* routeEffect(Effect.forEach( recovered, (account) => routing.recordResponse( @@ -87,19 +86,20 @@ export function makeEffectAIRoutingStateLayer( input.now, ), { discard: true }, - ); + )); if (recovered.length > 0) { - durable = yield* routing.summary(input.now); + durable = yield* routeEffect(routing.summary(input.now)); } const diagnosticCandidates = overlayRoutingSummary( input.candidates, durable, ); - const decision = yield* selectEffect( + const decision = yield* routeEffect(selectEffect( diagnosticCandidates, config, input.now, - ); + )); + const transferred = yield* Ref.make(false); return yield* Effect.acquireUseRelease( routing.acquire({ candidates: input.candidates.map(toRouterCandidate), @@ -107,7 +107,11 @@ export function makeEffectAIRoutingStateLayer( ...(input.sessionKey === undefined ? {} : { sessionKey: SessionKey.make(input.sessionKey) }), - }), + }).pipe( + Effect.mapError(() => + AIRoutingStateError.make({ code: "state_unavailable" }) + ), + ), (lease) => Effect.gen(function*() { const acquired = Option.getOrUndefined(lease); const decisionReason = acquired === undefined @@ -120,28 +124,33 @@ export function makeEffectAIRoutingStateLayer( reason: decisionReason, candidates: decision.candidates, })); - return acquired === undefined - ? undefined - : { - accountId: acquired.accountId, - leaseToken: acquired.leaseToken, - expiresAt: acquired.expiresAt, - decisionReason, - }; + const transfer = Effect.uninterruptible( + Ref.set(transferred, true), + ); + return yield* use( + acquired === undefined + ? undefined + : { + accountId: acquired.accountId, + leaseToken: acquired.leaseToken, + expiresAt: acquired.expiresAt, + decisionReason, + }, + transfer, + ); }), - (lease, exit) => { + (lease, exit) => Effect.uninterruptible(Effect.gen(function*() { const acquired = Option.getOrUndefined(lease); - if (acquired === undefined || Exit.isSuccess(exit)) { - return Effect.void; - } - return routing.release(acquired.leaseToken).pipe( + if (acquired === undefined) return; + const wasTransferred = yield* Ref.get(transferred); + if (wasTransferred && exit._tag === "Success") return; + yield* routing.release(acquired.leaseToken).pipe( Effect.asVoid, Effect.catchCause(() => Effect.void), - Effect.uninterruptible, ); - }, + })), ); - })); + }); const evaluate: AIRoutingState["Service"]["evaluate"] = (input) => routeEffect(Effect.gen(function*() { diff --git a/services/ai-gateway/src/forward.ts b/services/ai-gateway/src/forward.ts index 860ebca8..628d1e53 100644 --- a/services/ai-gateway/src/forward.ts +++ b/services/ai-gateway/src/forward.ts @@ -82,12 +82,16 @@ export type AIForwardClientAuthentication = export interface AIForwardOptions { readonly authentication: AIForwardClientAuthentication; - readonly acquire: ( + readonly acquire: ( sessionKey: string | undefined, signal: AbortSignal, authorization: ProviderAuthorizationGrantV1 | undefined, telemetry: AIGatewayRequestTelemetry, - ) => Effect.Effect; + use: ( + lease: AIForwardLease | undefined, + transfer: Effect.Effect, + ) => Effect.Effect, + ) => Effect.Effect; readonly provider: AIProviderHttp["Service"]; readonly settlements: ProviderBudgetSettlementReporter["Service"]; readonly now: Effect.Effect; @@ -189,195 +193,197 @@ export const makeAIForwardHandler = Effect.fn( } const sessionKey = attributed.success; yield* diagnostic(requestTelemetry.routeStarted); - const leaseReleaseState = yield* Ref.make(false); - const acquired = yield* Effect.result(options.acquire( + return yield* options.acquire( sessionKey, request.signal, authentication.authorization, requestTelemetry, - )); - if (Result.isFailure(acquired)) { - yield* diagnostic( - requestTelemetry.routeEnded("error", acquired.failure), - ); - yield* diagnostic(requestTelemetry.end({ - status: 503, - error: acquired.failure, - streamOutcome: "not_streamed", - })); - return jsonResponse(503, "route_unavailable"); - } - const lease = acquired.success; - if (lease === undefined) { - yield* diagnostic(requestTelemetry.routeEnded("unavailable")); - yield* diagnostic(requestTelemetry.end({ - status: 503, - streamOutcome: "not_streamed", - })); - return jsonResponse(503, "no_eligible_account"); - } - const releaseLeaseOnce = Effect.uninterruptible(Effect.gen(function*() { - const shouldRelease = yield* Ref.modify( - leaseReleaseState, - (released): readonly [boolean, boolean] => [!released, true], - ); - if (shouldRelease) yield* releaseLease(lease, requestTelemetry); - })); - - const postAcquisition = Effect.gen(function*() { - yield* diagnostic(requestTelemetry.routeEnded("acquired")); - const upstreamRequest = yield* makeUpstreamRequest(request, url, lease); - if (upstreamRequest === undefined) { - yield* releaseLeaseOnce; - yield* diagnostic(requestTelemetry.end({ - status: 400, - streamOutcome: "not_streamed", - })); - return jsonResponse(400, "invalid_request"); - } - yield* diagnostic( - requestTelemetry.upstreamStarted(upstreamRequest.headers), - ); - const upstreamResult = yield* Effect.result( - options.provider.execute(upstreamRequest), - ); - if (Result.isFailure(upstreamResult)) { - yield* diagnostic( - requestTelemetry.upstreamFailed(upstreamResult.failure), - ); - yield* releaseLeaseOnce; - yield* diagnostic(requestTelemetry.end({ - status: 502, - error: upstreamResult.failure, - streamOutcome: request.signal.aborted ? "aborted" : "upstream_error", - })); - return jsonResponse(502, providerErrorCode(upstreamResult.failure)); - } - const upstream = upstreamResult.success; - const headersResult = yield* Effect.result(Effect.try({ - try: () => new Headers(upstream.headers), - catch: () => AIForwardConfigurationError.make({ - code: "invalid_configuration", - }), - })); - if (Result.isFailure(headersResult)) { - yield* diagnostic( - requestTelemetry.upstreamFailed(headersResult.failure), - ); - yield* releaseLeaseOnce; - yield* diagnostic(requestTelemetry.end({ - status: 502, - error: headersResult.failure, - streamOutcome: "upstream_error", - })); - return jsonResponse(502, "invalid_provider_response"); - } - const upstreamHeaders = headersResult.success; - yield* diagnostic( - requestTelemetry.upstreamHeaders(upstream.status, upstreamHeaders), - ); - const responseHeaders = sanitizeResponseHeaders(upstreamHeaders); - const responseMetadata = yield* Effect.result(finiteResponse( - null, - upstream.status, - responseHeaders, - )); - if (Result.isFailure(responseMetadata)) { - yield* diagnostic( - requestTelemetry.upstreamFailed(responseMetadata.failure), - ); - yield* releaseLeaseOnce; - yield* diagnostic(requestTelemetry.end({ - status: 502, - error: responseMetadata.failure, - streamOutcome: "upstream_error", - })); - return jsonResponse(502, "invalid_provider_response"); - } - yield* recordResponse(lease, upstream.status, upstreamHeaders); - if (upstream.body === null) { - yield* settleWithoutBody( - options.settlements, - authentication.authorization, - upstream.status, - ); - yield* releaseLeaseOnce; - yield* diagnostic(requestTelemetry.end({ - status: upstream.status, - streamOutcome: "not_streamed", - })); - return responseMetadata.success; - } - if (!responseStatusAllowsBody(upstream.status)) { - const failure = AIForwardConfigurationError.make({ - code: "invalid_configuration", - }); - yield* diagnostic(requestTelemetry.upstreamFailed(failure)); - yield* releaseLeaseOnce; - yield* diagnostic(requestTelemetry.end({ - status: 502, - error: failure, - streamOutcome: "upstream_error", - })); - return jsonResponse(502, "invalid_provider_response"); - } - - const observerResult = upstream.status < 400 && - isEventStream(upstreamHeaders) - ? yield* Effect.result(makeOpenAITerminalUsageObserver({ - maximumEventBytes: options.maximumUsageEventBytes, - })) - : undefined; - if (observerResult !== undefined && Result.isFailure(observerResult)) { - yield* releaseLeaseOnce; - return jsonResponse(503, "accounting_unavailable"); - } - const observer = observerResult === undefined - ? undefined - : observerResult.success; - const monitored = upstream.body.pipe( - Stream.tap((chunk) => - diagnostic(requestTelemetry.streamChunk(chunk.byteLength)) - ), - observer === undefined - ? (stream) => stream - : Stream.tap((chunk) => observer.observe(chunk)), - Stream.onExit((exit) => - finalizeStream( - exit, - releaseLeaseOnce, - options.settlements, - authentication.authorization, + (lease, transfer) => { + if (lease === undefined) { + return Effect.gen(function*() { + yield* diagnostic(requestTelemetry.routeEnded("unavailable")); + yield* diagnostic(requestTelemetry.end({ + status: 503, + streamOutcome: "not_streamed", + })); + return jsonResponse(503, "no_eligible_account"); + }); + } + return Effect.gen(function*() { + const leaseReleaseState = yield* Ref.make(false); + const transferLease = Effect.uninterruptible(transfer); + const releaseLeaseOnce = Effect.uninterruptible(Effect.gen(function*() { + yield* transferLease; + const shouldRelease = yield* Ref.modify( + leaseReleaseState, + (released): readonly [boolean, boolean] => [!released, true], + ); + if (shouldRelease) yield* releaseLease(lease, requestTelemetry); + })); + + yield* diagnostic(requestTelemetry.routeEnded("acquired")); + const upstreamRequest = yield* makeUpstreamRequest(request, url, lease); + if (upstreamRequest === undefined) { + yield* releaseLeaseOnce; + yield* diagnostic(requestTelemetry.end({ + status: 400, + streamOutcome: "not_streamed", + })); + return jsonResponse(400, "invalid_request"); + } + yield* diagnostic( + requestTelemetry.upstreamStarted(upstreamRequest.headers), + ); + const upstreamResult = yield* Effect.result( + options.provider.execute(upstreamRequest), + ); + if (Result.isFailure(upstreamResult)) { + yield* diagnostic( + requestTelemetry.upstreamFailed(upstreamResult.failure), + ); + yield* releaseLeaseOnce; + yield* diagnostic(requestTelemetry.end({ + status: 502, + error: upstreamResult.failure, + streamOutcome: request.signal.aborted ? "aborted" : "upstream_error", + })); + return jsonResponse(502, providerErrorCode(upstreamResult.failure)); + } + const upstream = upstreamResult.success; + const headersResult = yield* Effect.result(Effect.try({ + try: () => new Headers(upstream.headers), + catch: () => AIForwardConfigurationError.make({ + code: "invalid_configuration", + }), + })); + if (Result.isFailure(headersResult)) { + yield* diagnostic( + requestTelemetry.upstreamFailed(headersResult.failure), + ); + yield* releaseLeaseOnce; + yield* diagnostic(requestTelemetry.end({ + status: 502, + error: headersResult.failure, + streamOutcome: "upstream_error", + })); + return jsonResponse(502, "invalid_provider_response"); + } + const upstreamHeaders = headersResult.success; + yield* diagnostic( + requestTelemetry.upstreamHeaders(upstream.status, upstreamHeaders), + ); + const responseHeaders = sanitizeResponseHeaders(upstreamHeaders); + const responseMetadata = yield* Effect.result(finiteResponse( + null, upstream.status, - observer, - requestTelemetry, - request.signal, - ) - ), - ); - const scoped = Stream.unwrap(Effect.gen(function*() { - yield* heartbeat(lease, options.heartbeatMillis).pipe( - Effect.forkScoped({ startImmediately: true }), - ); - return monitored; - })); - const body = yield* Stream.toReadableStreamEffect(scoped); - const responseResult = yield* Effect.result(finiteResponse( - body, - upstream.status, - responseHeaders, - )); - if (Result.isFailure(responseResult)) { - yield* releaseLeaseOnce; - return jsonResponse(502, "invalid_provider_response"); - } - return responseResult.success; - }); - return yield* postAcquisition.pipe( - Effect.onExit((exit) => - Exit.isSuccess(exit) - ? Effect.void - : Effect.uninterruptible(releaseLeaseOnce) + responseHeaders, + )); + if (Result.isFailure(responseMetadata)) { + yield* diagnostic( + requestTelemetry.upstreamFailed(responseMetadata.failure), + ); + yield* releaseLeaseOnce; + yield* diagnostic(requestTelemetry.end({ + status: 502, + error: responseMetadata.failure, + streamOutcome: "upstream_error", + })); + return jsonResponse(502, "invalid_provider_response"); + } + yield* recordResponse(lease, upstream.status, upstreamHeaders); + if (upstream.body === null) { + yield* settleWithoutBody( + options.settlements, + authentication.authorization, + upstream.status, + ); + yield* releaseLeaseOnce; + yield* diagnostic(requestTelemetry.end({ + status: upstream.status, + streamOutcome: "not_streamed", + })); + return responseMetadata.success; + } + if (!responseStatusAllowsBody(upstream.status)) { + const failure = AIForwardConfigurationError.make({ + code: "invalid_configuration", + }); + yield* diagnostic(requestTelemetry.upstreamFailed(failure)); + yield* releaseLeaseOnce; + yield* diagnostic(requestTelemetry.end({ + status: 502, + error: failure, + streamOutcome: "upstream_error", + })); + return jsonResponse(502, "invalid_provider_response"); + } + + const observerResult = upstream.status < 400 && + isEventStream(upstreamHeaders) + ? yield* Effect.result(makeOpenAITerminalUsageObserver({ + maximumEventBytes: options.maximumUsageEventBytes, + })) + : undefined; + if (observerResult !== undefined && Result.isFailure(observerResult)) { + yield* releaseLeaseOnce; + return jsonResponse(503, "accounting_unavailable"); + } + const observer = observerResult === undefined + ? undefined + : observerResult.success; + const monitored = upstream.body.pipe( + Stream.tap((chunk) => + diagnostic(requestTelemetry.streamChunk(chunk.byteLength)) + ), + observer === undefined + ? (stream) => stream + : Stream.tap((chunk) => observer.observe(chunk)), + Stream.onExit((exit) => + finalizeStream( + exit, + releaseLeaseOnce, + options.settlements, + authentication.authorization, + upstream.status, + observer, + requestTelemetry, + request.signal, + ) + ), + ); + const scoped = Stream.unwrap(Effect.gen(function*() { + yield* heartbeat(lease, options.heartbeatMillis).pipe( + Effect.forkScoped({ startImmediately: true }), + ); + return monitored; + })); + const body = yield* Stream.toReadableStreamEffect(scoped); + const responseResult = yield* Effect.result(finiteResponse( + body, + upstream.status, + responseHeaders, + )); + if (Result.isFailure(responseResult)) { + yield* releaseLeaseOnce; + return jsonResponse(502, "invalid_provider_response"); + } + return yield* Effect.uninterruptible(Effect.gen(function*() { + yield* transferLease; + return responseResult.success; + })); + }); + }, + ).pipe( + Effect.catchTag("AIForwardRouteError", (failure) => + Effect.gen(function*() { + yield* diagnostic(requestTelemetry.routeEnded("error", failure)); + yield* diagnostic(requestTelemetry.end({ + status: 503, + error: failure, + streamOutcome: "not_streamed", + })); + return jsonResponse(503, "route_unavailable"); + }) ), ); }); diff --git a/services/ai-gateway/src/gateway-service.ts b/services/ai-gateway/src/gateway-service.ts index d4a855a0..04ee0a5b 100644 --- a/services/ai-gateway/src/gateway-service.ts +++ b/services/ai-gateway/src/gateway-service.ts @@ -7,7 +7,6 @@ import { import { Clock, Effect, - Exit, Ref, Result, Schema, @@ -120,12 +119,16 @@ export const makeAIGatewayApplication = Effect.fn( ); const acquire = Effect.fn("agentos.aiGateway.acquireRoute")( - function*( + function*( sessionKey: string | undefined, _signal: AbortSignal, _authorization: ProviderAuthorizationGrantV1 | undefined, telemetry: AIGatewayRequestTelemetry, - ): Effect.fn.Return { + use: ( + lease: AIForwardLease | undefined, + transfer: Effect.Effect, + ) => Effect.Effect, + ): Effect.fn.Return { const set = yield* candidateSet(telemetry); const currentTime = yield* Clock.currentTimeMillis; yield* Effect.forEach( @@ -142,89 +145,86 @@ export const makeAIGatewayApplication = Effect.fn( ).pipe(Effect.catchCause(() => Effect.void)), { discard: true }, ); - const credentialResult = yield* Effect.result( - Effect.acquireUseRelease( - routing.acquire({ - candidates: set.candidates, - now: currentTime, - ...(sessionKey === undefined ? {} : { sessionKey }), - }).pipe(Effect.mapError(() => routeError("routing_unavailable"))), - (reservation) => { - if (reservation === undefined) { - return Effect.succeed( - fallbackAvailable && fallbackKey !== undefined - ? fallbackLease(fallbackKey) - : undefined, + return yield* routing.acquire({ + candidates: set.candidates, + now: currentTime, + ...(sessionKey === undefined ? {} : { sessionKey }), + }, (reservation, transfer) => { + if (reservation === undefined) { + return use( + fallbackAvailable && fallbackKey !== undefined + ? fallbackLease(fallbackKey) + : undefined, + Effect.void, + ); + } + return Effect.gen(function*() { + const credential = yield* vault.getFreshCredential( + reservation.accountId, + ).pipe( + Effect.mapError((error) => + routeError(routeCodeForAccount(error)) + ), + ); + const releaseState = yield* Ref.make(false); + const release = Effect.uninterruptible(Effect.gen(function*() { + const shouldRelease = yield* Ref.modify( + releaseState, + (released): readonly [boolean, boolean] => [!released, true], + ); + if (shouldRelease) { + yield* routing.release(reservation.leaseToken).pipe( + Effect.asVoid, + Effect.mapError(() => routeError("state_unavailable")), ); } - return Effect.gen(function*() { - const credential = yield* vault.getFreshCredential( - reservation.accountId, - ); - const lease: AIForwardLease = { - kind: "codex_oauth", - accessToken: credential.accessToken, - providerAccountId: credential.providerAccountId, - renew: Clock.currentTimeMillis.pipe( - Effect.flatMap((renewedAt) => - routing.renew(reservation.leaseToken, renewedAt) - ), - Effect.mapError(() => routeError("state_unavailable")), - ), - release: routing.release(reservation.leaseToken).pipe( - Effect.asVoid, + })); + const lease: AIForwardLease = { + kind: "codex_oauth", + accessToken: credential.accessToken, + providerAccountId: credential.providerAccountId, + renew: Clock.currentTimeMillis.pipe( + Effect.flatMap((renewedAt) => + routing.renew(reservation.leaseToken, renewedAt) + ), + Effect.mapError(() => routeError("state_unavailable")), + ), + release, + recordResponse: (status, headers) => + Effect.gen(function*() { + const responseAt = yield* Clock.currentTimeMillis; + if (status === 401) { + yield* vault.markNeedsReauth( + reservation.accountId, + credential.accessToken, + ).pipe(Effect.catchCause(() => Effect.succeed(false))); + } + yield* routing.recordResponse( + reservation.accountId, + status, + headers, + responseAt, + ).pipe( Effect.mapError(() => routeError("state_unavailable")), - ), - recordResponse: (status, headers) => - Effect.gen(function*() { - const responseAt = yield* Clock.currentTimeMillis; - if (status === 401) { - yield* vault.markNeedsReauth( - reservation.accountId, - credential.accessToken, - ).pipe(Effect.catchCause(() => Effect.succeed(false))); - } - yield* routing.recordResponse( - reservation.accountId, - status, - headers, - responseAt, - ).pipe( - Effect.mapError(() => routeError("state_unavailable")), - ); - }), - }; - return lease; - }); - }, - (reservation, exit) => { - if (reservation === undefined || Exit.isSuccess(exit)) { - return Effect.void; - } - return routing.release(reservation.leaseToken).pipe( - Effect.asVoid, - Effect.catchCause(() => Effect.void), - Effect.uninterruptible, - ); - }, + ); + }), + }; + return yield* use(lease, transfer); + }); + }).pipe( + Effect.mapError((error) => + error._tag === "AIForwardRouteError" + ? error + : routeError("routing_unavailable") ), ); - if (Result.isFailure(credentialResult)) { - if (credentialResult.failure._tag === "AIForwardRouteError") { - return yield* credentialResult.failure; - } - return yield* routeError( - routeCodeForAccount(credentialResult.failure), - ); - } - return credentialResult.success; }, ); const forward = yield* makeAIForwardHandler({ authentication: options.authentication, - acquire: (sessionKey, signal, authorization, telemetry) => - acquire(sessionKey, signal, authorization, telemetry), + acquire: (sessionKey, signal, authorization, telemetry, use) => + acquire(sessionKey, signal, authorization, telemetry, use), provider, settlements, now: Clock.currentTimeMillis, diff --git a/services/ai-gateway/src/routing-service.ts b/services/ai-gateway/src/routing-service.ts index 0ed43fe4..8da726d2 100644 --- a/services/ai-gateway/src/routing-service.ts +++ b/services/ai-gateway/src/routing-service.ts @@ -49,9 +49,13 @@ export class AIRoutingState extends Context.Service< readonly summary: ( now: number, ) => Effect.Effect; - readonly acquire: ( + readonly acquire: ( input: AIRoutingAcquireInput, - ) => Effect.Effect; + use: ( + reservation: AIAcquiredReservation | undefined, + transfer: Effect.Effect, + ) => Effect.Effect, + ) => Effect.Effect; readonly evaluate: ( input: AIRoutingEvaluateInput, ) => Effect.Effect; diff --git a/services/ai-gateway/tests/forward.effect.test.ts b/services/ai-gateway/tests/forward.effect.test.ts index 70ff0574..8287a98b 100644 --- a/services/ai-gateway/tests/forward.effect.test.ts +++ b/services/ai-gateway/tests/forward.effect.test.ts @@ -109,6 +109,19 @@ const makeLease = Effect.fn("test.aiForward.makeLease")(function*() { return { lease, releases, renewals }; }); +function acquireLease(lease: AIForwardLease | undefined) { + return ( + _session: string | undefined, + _signal: AbortSignal, + _authorization: ProviderAuthorizationGrantV1 | undefined, + _telemetry: AIGatewayRequestTelemetry, + use: ( + lease: AIForwardLease | undefined, + transfer: Effect.Effect, + ) => Effect.Effect, + ) => use(lease, Effect.void); +} + const makeSettlementRecorder = Effect.fn( "test.aiForward.makeSettlementRecorder", )(function*() { @@ -195,7 +208,7 @@ describe("Effect AI Gateway forwarding", () => { const telemetry = yield* makeTelemetryRecorder(); const handler = yield* makeAIForwardHandler({ authentication: { kind: "workload_identity" }, - acquire: () => Effect.succeed(route.lease), + acquire: acquireLease(route.lease), provider: AIProviderHttp.of({ execute: () => Effect.succeed({ status: 200, @@ -253,7 +266,7 @@ describe("Effect AI Gateway forwarding", () => { const telemetry = yield* makeAIGatewayTelemetry(); const handler = yield* makeAIForwardHandler({ authentication: { kind: "workload_identity" }, - acquire: () => Effect.succeed(route.lease), + acquire: acquireLease(route.lease), provider: AIProviderHttp.of({ execute: () => Effect.succeed({ status: 200, @@ -300,7 +313,7 @@ describe("Effect AI Gateway forwarding", () => { const telemetry = yield* makeTelemetryRecorder(); const handler = yield* makeAIForwardHandler({ authentication: { kind: "workload_identity" }, - acquire: () => Effect.succeed(route.lease), + acquire: acquireLease(route.lease), provider: AIProviderHttp.of({ execute: () => Effect.succeed({ status: 500, headers: {}, body: null }), }), @@ -341,7 +354,7 @@ describe("Effect AI Gateway forwarding", () => { const telemetry = yield* makeTelemetryRecorder(); const handler = yield* makeAIForwardHandler({ authentication: { kind: "shared_token", token: "client-secret" }, - acquire: () => Effect.succeed(undefined), + acquire: acquireLease(undefined), provider: AIProviderHttp.of({ execute: () => Effect.succeed({ status: 500, headers: {}, body: null }), }), @@ -385,7 +398,7 @@ describe("Effect AI Gateway forwarding", () => { const telemetry = yield* makeTelemetryRecorder(); const handler = yield* makeAIForwardHandler({ authentication: { kind: "workload_identity" }, - acquire: () => Effect.succeed(undefined), + acquire: acquireLease(undefined), provider: AIProviderHttp.of({ execute: () => Effect.succeed({ status: 500, headers: {}, body: null }), }), @@ -467,7 +480,7 @@ describe("Effect AI Gateway forwarding", () => { const telemetry = yield* makeTelemetryRecorder(); const handler = yield* makeAIForwardHandler({ authentication: { kind: "workload_identity" }, - acquire: () => Effect.succeed(undefined), + acquire: acquireLease(undefined), provider: AIProviderHttp.of({ execute: () => Effect.succeed({ status: 500, headers: {}, body: null }), }), @@ -514,7 +527,7 @@ describe("Effect AI Gateway forwarding", () => { }; const handler = yield* makeAIForwardHandler({ authentication: { kind: "workload_identity" }, - acquire: () => Effect.succeed(lease), + acquire: acquireLease(lease), provider: AIProviderHttp.of({ execute: () => Effect.succeed({ status: 500, headers: {}, body: null }), }), @@ -572,8 +585,10 @@ describe("Effect AI Gateway forwarding", () => { }); const handler = yield* makeAIForwardHandler({ authentication: { kind: "workload_identity" }, - acquire: (_session, _signal, authorization) => - Ref.set(attribution, authorization).pipe(Effect.as(route.lease)), + acquire: (_session, _signal, authorization, _telemetry, use) => + Ref.set(attribution, authorization).pipe( + Effect.flatMap(() => use(route.lease, Effect.void)), + ), provider, settlements: settlement.settlements, now: Effect.succeed(now), @@ -624,7 +639,7 @@ describe("Effect AI Gateway forwarding", () => { const settlement = yield* makeSettlementRecorder(); const handler = yield* makeAIForwardHandler({ authentication: { kind: "workload_identity" }, - acquire: () => Effect.succeed(route.lease), + acquire: acquireLease(route.lease), provider: AIProviderHttp.of({ execute: () => Effect.succeed({ status: 200, @@ -651,7 +666,7 @@ describe("Effect AI Gateway forwarding", () => { const settlement = yield* makeSettlementRecorder(); const handler = yield* makeAIForwardHandler({ authentication: { kind: "workload_identity" }, - acquire: () => Effect.succeed(route.lease), + acquire: acquireLease(route.lease), provider: AIProviderHttp.of({ execute: () => Effect.succeed({ status: 429, @@ -712,7 +727,7 @@ describe("Effect AI Gateway forwarding", () => { }; const handler = yield* makeAIForwardHandler({ authentication: { kind: "workload_identity" }, - acquire: () => Effect.succeed(route.lease), + acquire: acquireLease(route.lease), provider: AIProviderHttp.of({ execute: () => Effect.succeed({ status, @@ -779,7 +794,7 @@ describe("Effect AI Gateway forwarding", () => { const telemetry = yield* makeTelemetryRecorder(); const handler = yield* makeAIForwardHandler({ authentication: { kind: "workload_identity" }, - acquire: () => Effect.succeed(route.lease), + acquire: acquireLease(route.lease), provider: AIProviderHttp.of({ execute: () => Effect.succeed({ status: 204, @@ -827,7 +842,7 @@ describe("Effect AI Gateway forwarding", () => { const settlement = yield* makeSettlementRecorder(); const handler = yield* makeAIForwardHandler({ authentication: { kind: "workload_identity" }, - acquire: () => Effect.succeed(route.lease), + acquire: acquireLease(route.lease), provider: AIProviderHttp.of({ execute: () => Effect.interrupt, }), @@ -856,7 +871,7 @@ describe("Effect AI Gateway forwarding", () => { }; const handler = yield* makeAIForwardHandler({ authentication: { kind: "workload_identity" }, - acquire: () => Effect.succeed(brokenLease), + acquire: acquireLease(brokenLease), provider: AIProviderHttp.of({ execute: () => Effect.die("provider must not execute"), }), @@ -878,7 +893,7 @@ describe("Effect AI Gateway forwarding", () => { const telemetry = yield* makeTelemetryRecorder(); const handler = yield* makeAIForwardHandler({ authentication: { kind: "workload_identity" }, - acquire: () => Effect.succeed(route.lease), + acquire: acquireLease(route.lease), provider: AIProviderHttp.of({ execute: () => Effect.succeed({ status: 700, headers: {}, body: null }), }), @@ -925,7 +940,7 @@ describe("Effect AI Gateway forwarding", () => { const telemetry = yield* makeTelemetryRecorder(); const handler = yield* makeAIForwardHandler({ authentication: { kind: "workload_identity" }, - acquire: () => Effect.succeed(route.lease), + acquire: acquireLease(route.lease), provider: AIProviderHttp.of({ execute: () => Effect.succeed({ status: 700, @@ -976,7 +991,7 @@ describe("Effect AI Gateway forwarding", () => { const telemetry = yield* makeTelemetryRecorder(); const handler = yield* makeAIForwardHandler({ authentication: { kind: "workload_identity" }, - acquire: () => Effect.succeed(route.lease), + acquire: acquireLease(route.lease), provider: AIProviderHttp.of({ execute: () => Effect.succeed({ status: 204, @@ -1027,9 +1042,10 @@ describe("Effect AI Gateway forwarding", () => { const settlement = yield* makeSettlementRecorder(); const handler = yield* makeAIForwardHandler({ authentication: { kind: "workload_identity" }, - acquire: () => Ref.update(acquireCalls, (count) => count + 1).pipe( - Effect.as(undefined), - ), + acquire: (_session, _signal, _authorization, _telemetry, use) => + Ref.update(acquireCalls, (count) => count + 1).pipe( + Effect.flatMap(() => use(undefined, Effect.void)), + ), provider: AIProviderHttp.of({ execute: () => Ref.update(providerCalls, (count) => count + 1).pipe( Effect.as({ status: 500, headers: {}, body: null }), @@ -1054,7 +1070,7 @@ describe("Effect AI Gateway forwarding", () => { const telemetry = yield* makeTelemetryRecorder(); const handler = yield* makeAIForwardHandler({ authentication: { kind: "workload_identity" }, - acquire: () => Effect.succeed(route.lease), + acquire: acquireLease(route.lease), provider: AIProviderHttp.of({ execute: () => Effect.fail(AIProviderHttpError.make({ code: "provider_unavailable", @@ -1105,7 +1121,7 @@ describe("Effect AI Gateway forwarding", () => { headers.revoke(); const handler = yield* makeAIForwardHandler({ authentication: { kind: "workload_identity" }, - acquire: () => Effect.succeed(route.lease), + acquire: acquireLease(route.lease), provider: AIProviderHttp.of({ execute: () => Effect.succeed({ status: 200, @@ -1155,7 +1171,7 @@ describe("Effect AI Gateway forwarding", () => { const telemetry = yield* makeTelemetryRecorder(); const handler = yield* makeAIForwardHandler({ authentication: { kind: "workload_identity" }, - acquire: () => Effect.succeed(route.lease), + acquire: acquireLease(route.lease), provider: AIProviderHttp.of({ execute: () => Effect.succeed({ status: 200, @@ -1210,7 +1226,7 @@ describe("Effect AI Gateway forwarding", () => { const telemetry = yield* makeTelemetryRecorder(); const handler = yield* makeAIForwardHandler({ authentication: { kind: "workload_identity" }, - acquire: () => Effect.succeed(route.lease), + acquire: acquireLease(route.lease), provider: AIProviderHttp.of({ execute: () => Effect.succeed({ status: 200, @@ -1275,7 +1291,7 @@ describe("Effect AI Gateway forwarding", () => { }); const handler = yield* makeAIForwardHandler({ authentication: { kind: "workload_identity" }, - acquire: () => Effect.succeed(route.lease), + acquire: acquireLease(route.lease), provider: AIProviderHttp.of({ execute: () => Effect.succeed({ status: 200, @@ -1304,7 +1320,7 @@ describe("Effect AI Gateway forwarding", () => { const telemetry = yield* makeTelemetryRecorder("upstream_headers"); const handler = yield* makeAIForwardHandler({ authentication: { kind: "workload_identity" }, - acquire: () => Effect.succeed(route.lease), + acquire: acquireLease(route.lease), provider: AIProviderHttp.of({ execute: () => Effect.succeed({ status: 200, @@ -1334,7 +1350,7 @@ describe("Effect AI Gateway forwarding", () => { const settlement = yield* makeSettlementRecorder(); const handler = yield* makeAIForwardHandler({ authentication: { kind: "workload_identity" }, - acquire: () => Effect.succeed(route.lease), + acquire: acquireLease(route.lease), provider: AIProviderHttp.of({ execute: () => Effect.succeed({ status: 200, diff --git a/services/ai-gateway/tests/gateway-service.effect.test.ts b/services/ai-gateway/tests/gateway-service.effect.test.ts index c71d8a28..eebd1a4a 100644 --- a/services/ai-gateway/tests/gateway-service.effect.test.ts +++ b/services/ai-gateway/tests/gateway-service.effect.test.ts @@ -140,14 +140,14 @@ const makeTestServices = Effect.fn("test.aiGateway.makeServices")( activeReservations: 0, reservationsByAccount: {}, }), - acquire: () => Effect.succeed(withAccounts + acquire: (_input, use) => use(withAccounts ? { - accountId: "managed-a", - leaseToken: "lease-a", - expiresAt: now + 60_000, - decisionReason: "best_candidate", - } - : undefined), + accountId: "managed-a", + leaseToken: "lease-a", + expiresAt: now + 60_000, + decisionReason: "best_candidate", + } + : undefined, Effect.void), evaluate: () => Effect.succeed(withAccounts ? { accountId: "managed-a", diff --git a/services/ai-gateway/tests/routing-state.effect.test.ts b/services/ai-gateway/tests/routing-state.effect.test.ts index dcb5b6b4..39a6d874 100644 --- a/services/ai-gateway/tests/routing-state.effect.test.ts +++ b/services/ai-gateway/tests/routing-state.effect.test.ts @@ -68,33 +68,30 @@ describe("Effect canonical durable routing state", () => { defaultRoutingConfig, (routing) => Effect.gen(function*() { - const acquired = yield* routing.acquire({ + return yield* routing.acquire({ candidates, now, sessionKey: "session-1", - }); - assert.isDefined(acquired); - if (acquired === undefined) return undefined; - assert.strictEqual(acquired.accountId, "a"); - assert.strictEqual( - yield* routing.renew(acquired.leaseToken, now + 30_000), - true, - ); - const summary = yield* routing.summary(now + 1); - assert.strictEqual(summary.activeReservations, 1); - assert.deepStrictEqual(summary.reservationsByAccount, { a: 1 }); - assert.strictEqual(summary.lastSelection?.reason, "best_candidate"); - assert.strictEqual( - summary.lastSelection?.candidates.some((candidate) => - candidate.accountId === "a" && candidate.eligible - ), - true, - ); - assert.strictEqual( - yield* routing.release(acquired.leaseToken), - true, - ); - return acquired; + }, (acquired) => Effect.gen(function*() { + assert.isDefined(acquired); + if (acquired === undefined) return undefined; + assert.strictEqual(acquired.accountId, "a"); + assert.strictEqual( + yield* routing.renew(acquired.leaseToken, now + 30_000), + true, + ); + const summary = yield* routing.summary(now + 1); + assert.strictEqual(summary.activeReservations, 1); + assert.deepStrictEqual(summary.reservationsByAccount, { a: 1 }); + assert.strictEqual(summary.lastSelection?.reason, "best_candidate"); + assert.strictEqual( + summary.lastSelection?.candidates.some((candidate) => + candidate.accountId === "a" && candidate.eligible + ), + true, + ); + return acquired; + })); }), ); assert.isDefined(first); @@ -107,7 +104,7 @@ describe("Effect canonical durable routing state", () => { candidates, now: now + 30_001, sessionKey: "session-1", - }), + }, (acquired) => Effect.succeed(acquired)), ); assert.strictEqual(sticky?.accountId, "a"); }).pipe(Effect.provide(BunFileSystem.layer)))); @@ -123,11 +120,19 @@ describe("Effect canonical durable routing state", () => { { ...defaultRoutingConfig, reservationTtlMs: 10 }, (routing) => Effect.gen(function*() { - assert.isDefined(yield* routing.acquire({ candidates, now })); - yield* routing.acquire({ candidates, now: now + 11 }); - assert.strictEqual( - (yield* routing.summary(now + 11)).activeReservations, - 1, + assert.isDefined(yield* routing.acquire( + { candidates, now }, + (acquired) => Effect.succeed(acquired), + )); + yield* routing.acquire( + { candidates, now: now + 11 }, + (acquired) => Effect.gen(function*() { + assert.isDefined(acquired); + assert.strictEqual( + (yield* routing.summary(now + 11)).activeReservations, + 1, + ); + }), ); }), ); @@ -156,15 +161,17 @@ describe("Effect canonical durable routing state", () => { defaultRoutingConfig, (routing) => Effect.gen(function*() { - const first = yield* routing.acquire({ candidates: tied, now }); - const second = yield* routing.acquire({ - candidates: tied, - now: now + 1, - }); - assert.strictEqual(first?.accountId, "a"); - assert.strictEqual(first?.decisionReason, "best_candidate"); - assert.strictEqual(second?.accountId, "b"); - assert.strictEqual(second?.decisionReason, "best_candidate"); + yield* routing.acquire({ candidates: tied, now }, (first) => + routing.acquire({ + candidates: tied, + now: now + 1, + }, (second) => Effect.sync(() => { + assert.strictEqual(first?.accountId, "a"); + assert.strictEqual(first?.decisionReason, "best_candidate"); + assert.strictEqual(second?.accountId, "b"); + assert.strictEqual(second?.decisionReason, "best_candidate"); + })) + ); }), ); }).pipe(Effect.provide(BunFileSystem.layer)))); @@ -187,18 +194,21 @@ describe("Effect canonical durable routing state", () => { needsReauth: true, }], now, + }, (acquired) => Effect.gen(function*() { + assert.isUndefined(acquired); + const summary = yield* routing.summary(now); + assert.strictEqual( + summary.lastSelection?.reason, + "no_eligible_accounts", + ); + assert.deepStrictEqual(summary.lastSelection?.candidates, [{ + accountId: "reauth", + eligible: false, + freshness: "unknown", + rejectionCode: "reauthentication_required", + }]); + return acquired; })); - const summary = yield* routing.summary(now); - assert.strictEqual( - summary.lastSelection?.reason, - "no_eligible_accounts", - ); - assert.deepStrictEqual(summary.lastSelection?.candidates, [{ - accountId: "reauth", - eligible: false, - freshness: "unknown", - rejectionCode: "reauthentication_required", - }]); }), ); }).pipe(Effect.provide(BunFileSystem.layer)))); From 61efd0fa3ca65b673dd6139b1c1b45583801f362 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Robin=20Br=C3=A4mer?= <22003767+robinbraemer@users.noreply.github.com> Date: Mon, 10 Aug 2026 18:40:55 +0200 Subject: [PATCH 17/28] no-mistakes(review): Use canonical router header sanitization --- .../ai-gateway/src/workload-client-proxy.ts | 26 +++++-------------- .../workload-client-proxy.effect.test.ts | 14 ++++++++++ 2 files changed, 20 insertions(+), 20 deletions(-) diff --git a/services/ai-gateway/src/workload-client-proxy.ts b/services/ai-gateway/src/workload-client-proxy.ts index 8a106cb4..a747fab9 100644 --- a/services/ai-gateway/src/workload-client-proxy.ts +++ b/services/ai-gateway/src/workload-client-proxy.ts @@ -1,3 +1,4 @@ +import { sanitizeRequestHeaders } from "@akua-dev/codex-router/codex"; import { Config, Effect, FileSystem, Schema } from "effect"; import { @@ -11,21 +12,6 @@ const allowedPaths = new Set(["/v1/responses", "/v1/responses/compact"]); const assignmentIdPattern = /^[0-9a-f]{8}-[0-9a-f]{4}-4[0-9a-f]{3}-[89ab][0-9a-f]{3}-[0-9a-f]{12}$/i; export const workloadClientProxyHostname = "127.0.0.1"; -const removedHeaders = new Set([ - "authorization", - "connection", - "content-length", - "host", - "keep-alive", - "proxy-authenticate", - "proxy-authorization", - "te", - "trailer", - "transfer-encoding", - "upgrade", - "x-ai-gateway-token", -]); - export class WorkloadClientProxyError extends Schema.TaggedErrorClass()( "WorkloadClientProxyError", { @@ -153,11 +139,11 @@ function forwardedHeaders( token: string, assignmentId: string | undefined, ) { - const headers = new Headers(); - for (const [name, value] of input) { - const lower = name.toLowerCase(); - if (removedHeaders.has(lower) || lower.startsWith("x-agentos-")) continue; - headers.append(name, value); + const headers = sanitizeRequestHeaders(input); + for (const name of Array.from(headers.keys())) { + if (name.toLowerCase().startsWith("x-agentos-")) { + headers.delete(name); + } } headers.set("authorization", `Bearer ${token}`); if (assignmentId !== undefined) { diff --git a/services/ai-gateway/tests/workload-client-proxy.effect.test.ts b/services/ai-gateway/tests/workload-client-proxy.effect.test.ts index 509bdc1b..99519517 100644 --- a/services/ai-gateway/tests/workload-client-proxy.effect.test.ts +++ b/services/ai-gateway/tests/workload-client-proxy.effect.test.ts @@ -131,6 +131,13 @@ suite("Hermes Responses workload client proxy", (it) => { headers: { authorization: "Bearer hermes-placeholder", "content-type": "application/json", + "api-key": "caller-provider-secret", + "x-api-key": "caller-provider-secret-2", + "chatgpt-account-id": "caller-account", + baggage: "caller-baggage", + "x-ai-router-token": "caller-router-token", + "x-codex-router-session": "caller-router-session", + "x-ai-gateway-session": "caller-gateway-session", traceparent: "00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01", "x-agentos-assignment-id": "forged-assignment", "x-agentos-decision": "forged-decision", @@ -156,6 +163,13 @@ suite("Hermes Responses workload client proxy", (it) => { assert.isNull(forwarded[0]?.headers.get("x-agentos-decision")); assert.isNull(forwarded[0]?.headers.get("x-agentos-grant")); assert.isNull(forwarded[0]?.headers.get("x-ai-gateway-token")); + assert.isNull(forwarded[0]?.headers.get("api-key")); + assert.isNull(forwarded[0]?.headers.get("x-api-key")); + assert.isNull(forwarded[0]?.headers.get("chatgpt-account-id")); + assert.isNull(forwarded[0]?.headers.get("baggage")); + assert.isNull(forwarded[0]?.headers.get("x-ai-router-token")); + assert.isNull(forwarded[0]?.headers.get("x-codex-router-session")); + assert.isNull(forwarded[0]?.headers.get("x-ai-gateway-session")); assert.strictEqual( forwarded[0]?.headers.get("traceparent"), "00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01", From d0756d7943429d6d570f1e537ad239540fd60b05 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Robin=20Br=C3=A4mer?= <22003767+robinbraemer@users.noreply.github.com> Date: Mon, 10 Aug 2026 18:59:13 +0200 Subject: [PATCH 18/28] no-mistakes(review): Fix bounded token reads and preserve lease-release retries --- services/ai-gateway/src/forward.ts | 30 ++++++++++++++----- services/ai-gateway/src/gateway-service.ts | 26 ++++++++++++---- .../ai-gateway/src/workload-client-proxy.ts | 8 +++-- 3 files changed, 49 insertions(+), 15 deletions(-) diff --git a/services/ai-gateway/src/forward.ts b/services/ai-gateway/src/forward.ts index 628d1e53..2f021965 100644 --- a/services/ai-gateway/src/forward.ts +++ b/services/ai-gateway/src/forward.ts @@ -210,15 +210,29 @@ export const makeAIForwardHandler = Effect.fn( }); } return Effect.gen(function*() { - const leaseReleaseState = yield* Ref.make(false); + type LeaseReleaseState = "available" | "releasing" | "released"; + const leaseReleaseState = yield* Ref.make("available"); const transferLease = Effect.uninterruptible(transfer); const releaseLeaseOnce = Effect.uninterruptible(Effect.gen(function*() { - yield* transferLease; - const shouldRelease = yield* Ref.modify( + const action = yield* Ref.modify( leaseReleaseState, - (released): readonly [boolean, boolean] => [!released, true], + (state): readonly ["attempt" | "done" | "wait", LeaseReleaseState] => + state === "available" + ? ["attempt", "releasing"] + : state === "released" + ? ["done", state] + : ["wait", state], ); - if (shouldRelease) yield* releaseLease(lease, requestTelemetry); + if (action === "done") return true; + if (action === "wait") return false; + const released = yield* releaseLease(lease, requestTelemetry); + if (released) { + yield* Ref.set(leaseReleaseState, "released"); + yield* transferLease; + return true; + } + yield* Ref.set(leaseReleaseState, "available"); + return false; })); yield* diagnostic(requestTelemetry.routeEnded("acquired")); @@ -517,7 +531,7 @@ function heartbeat(lease: AIForwardLease, heartbeatMillis: number) { function finalizeStream( exit: Exit.Exit, - releaseLeaseOnce: Effect.Effect, + releaseLeaseOnce: Effect.Effect, settlements: ProviderBudgetSettlementReporter["Service"], authorization: ProviderAuthorizationGrantV1 | undefined, status: number, @@ -544,7 +558,8 @@ function finalizeStream( } } } - yield* releaseLeaseOnce; + const released = yield* releaseLeaseOnce; + if (!released) yield* releaseLeaseOnce; const outcome = telemetryStreamOutcome(exit, signal); const failure = Exit.isFailure(exit) ? Option.getOrUndefined(Cause.findErrorOption(exit.cause)) @@ -612,6 +627,7 @@ function releaseLease( ? telemetry.routeReleased : telemetry.routeReleaseFailed, ); + return Exit.isSuccess(release); }).pipe( Effect.uninterruptible, ); diff --git a/services/ai-gateway/src/gateway-service.ts b/services/ai-gateway/src/gateway-service.ts index 04ee0a5b..b1e4eec3 100644 --- a/services/ai-gateway/src/gateway-service.ts +++ b/services/ai-gateway/src/gateway-service.ts @@ -7,6 +7,7 @@ import { import { Clock, Effect, + Exit, Ref, Result, Schema, @@ -166,18 +167,31 @@ export const makeAIGatewayApplication = Effect.fn( routeError(routeCodeForAccount(error)) ), ); - const releaseState = yield* Ref.make(false); + type ReleaseState = "available" | "releasing" | "released"; + const releaseState = yield* Ref.make("available"); const release = Effect.uninterruptible(Effect.gen(function*() { - const shouldRelease = yield* Ref.modify( + const action = yield* Ref.modify( releaseState, - (released): readonly [boolean, boolean] => [!released, true], + (state): readonly ["attempt" | "done" | "wait", ReleaseState] => + state === "available" + ? ["attempt", "releasing"] + : state === "released" + ? ["done", state] + : ["wait", state], ); - if (shouldRelease) { - yield* routing.release(reservation.leaseToken).pipe( + if (action !== "attempt") return; + const released = yield* Effect.exit( + routing.release(reservation.leaseToken).pipe( Effect.asVoid, Effect.mapError(() => routeError("state_unavailable")), - ); + ), + ); + if (Exit.isSuccess(released)) { + yield* Ref.set(releaseState, "released"); + return; } + yield* Ref.set(releaseState, "available"); + return yield* Effect.failCause(released.cause); })); const lease: AIForwardLease = { kind: "codex_oauth", diff --git a/services/ai-gateway/src/workload-client-proxy.ts b/services/ai-gateway/src/workload-client-proxy.ts index a747fab9..0dd298dc 100644 --- a/services/ai-gateway/src/workload-client-proxy.ts +++ b/services/ai-gateway/src/workload-client-proxy.ts @@ -1,5 +1,5 @@ import { sanitizeRequestHeaders } from "@akua-dev/codex-router/codex"; -import { Config, Effect, FileSystem, Schema } from "effect"; +import { Config, Effect, FileSystem, Option, Schema } from "effect"; import { defaultAIGatewayGracefulShutdownMillis, @@ -118,7 +118,11 @@ export const readProjectedWorkloadToken = Effect.fn( "agentos.aiGateway.workloadClient.readToken", )(function*(path: string) { const fileSystem = yield* FileSystem.FileSystem; - const bytes = yield* fileSystem.readFile(path).pipe( + const bytes = yield* Effect.scoped(Effect.gen(function*() { + const file = yield* fileSystem.open(path, { flag: "r" }); + const read = yield* file.readAlloc(maximumTokenBytes + 1); + return Option.getOrElse(read, () => new Uint8Array()); + })).pipe( Effect.mapError(() => proxyError("token_unavailable")), ); if (bytes.length === 0 || bytes.length > maximumTokenBytes) { From 50a94bb65d04c03402f2f99ac071166c77b65b66 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Robin=20Br=C3=A4mer?= <22003767+robinbraemer@users.noreply.github.com> Date: Mon, 10 Aug 2026 19:11:13 +0200 Subject: [PATCH 19/28] no-mistakes(review): Restrict gateway destination and retry stream lease cleanup --- services/ai-gateway/src/forward.ts | 19 ++++++++++++++++--- .../ai-gateway/src/workload-client-proxy.ts | 4 +++- .../workload-client-proxy.effect.test.ts | 2 ++ 3 files changed, 21 insertions(+), 4 deletions(-) diff --git a/services/ai-gateway/src/forward.ts b/services/ai-gateway/src/forward.ts index 2f021965..35a421dc 100644 --- a/services/ai-gateway/src/forward.ts +++ b/services/ai-gateway/src/forward.ts @@ -539,6 +539,20 @@ function finalizeStream( telemetry: AIGatewayRequestTelemetry, signal: AbortSignal, ): Effect.Effect { + let released = false; + const releaseStreamLease = Effect.whileLoop({ + while: () => !released, + body: () => releaseLeaseOnce.pipe( + Effect.flatMap((attempted) => + attempted + ? Effect.succeed(true) + : Effect.sleep(1_000).pipe(Effect.as(false)) + ), + ), + step: (attempted) => { + released = attempted; + }, + }); return Effect.gen(function*() { if (authorization !== undefined) { if (status >= 400) { @@ -558,8 +572,7 @@ function finalizeStream( } } } - const released = yield* releaseLeaseOnce; - if (!released) yield* releaseLeaseOnce; + yield* releaseStreamLease; const outcome = telemetryStreamOutcome(exit, signal); const failure = Exit.isFailure(exit) ? Option.getOrUndefined(Cause.findErrorOption(exit.cause)) @@ -570,7 +583,7 @@ function finalizeStream( ...(failure === undefined ? {} : { error: failure }), })); }).pipe( - Effect.catchCause(() => releaseLeaseOnce), + Effect.catchCause(() => releaseStreamLease), Effect.catchCause(() => Effect.void), Effect.uninterruptible, ); diff --git a/services/ai-gateway/src/workload-client-proxy.ts b/services/ai-gateway/src/workload-client-proxy.ts index 0dd298dc..b7a2dd6f 100644 --- a/services/ai-gateway/src/workload-client-proxy.ts +++ b/services/ai-gateway/src/workload-client-proxy.ts @@ -9,6 +9,8 @@ import { const maximumTokenBytes = 16 * 1024; const jwtLike = /^[A-Za-z0-9_-]+\.[A-Za-z0-9_-]+\.[A-Za-z0-9_-]+$/; const allowedPaths = new Set(["/v1/responses", "/v1/responses/compact"]); +const reviewedAgentgatewayOrigin = + "http://agentgateway-openai.agentos.svc.cluster.local:8788"; const assignmentIdPattern = /^[0-9a-f]{8}-[0-9a-f]{4}-4[0-9a-f]{3}-[89ab][0-9a-f]{3}-[0-9a-f]{12}$/i; export const workloadClientProxyHostname = "127.0.0.1"; @@ -68,7 +70,7 @@ function configurationError() { } function isValidUpstreamBaseUrl(url: URL) { - return (url.protocol === "http:" || url.protocol === "https:") && + return url.origin === reviewedAgentgatewayOrigin && url.username === "" && url.password === "" && url.pathname === "/" && diff --git a/services/ai-gateway/tests/workload-client-proxy.effect.test.ts b/services/ai-gateway/tests/workload-client-proxy.effect.test.ts index 99519517..02009ed8 100644 --- a/services/ai-gateway/tests/workload-client-proxy.effect.test.ts +++ b/services/ai-gateway/tests/workload-client-proxy.effect.test.ts @@ -207,6 +207,8 @@ suite("Hermes Responses workload client proxy", (it) => { for (const invalidUrl of [ "ftp://agentgateway-openai.agentos.svc.cluster.local:8788", + "http://agentgateway-github.agentos.svc.cluster.local:8788", + "http://example.invalid:8788", "http://user:password@agentgateway-openai.agentos.svc.cluster.local:8788", "http://agentgateway-openai.agentos.svc.cluster.local:8788?trace=1", "http://agentgateway-openai.agentos.svc.cluster.local:8788#fragment", From d9fd77e25e46cae83a462bb4c51c03da4cb09965 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Robin=20Br=C3=A4mer?= <22003767+robinbraemer@users.noreply.github.com> Date: Mon, 10 Aug 2026 19:20:04 +0200 Subject: [PATCH 20/28] no-mistakes(review): Bound stream cleanup and retain routing release ownership --- services/ai-gateway/src/effect-routing-state.ts | 13 ++++++++----- services/ai-gateway/src/forward.ts | 9 ++++++--- 2 files changed, 14 insertions(+), 8 deletions(-) diff --git a/services/ai-gateway/src/effect-routing-state.ts b/services/ai-gateway/src/effect-routing-state.ts index 5246f9d9..f04f1942 100644 --- a/services/ai-gateway/src/effect-routing-state.ts +++ b/services/ai-gateway/src/effect-routing-state.ts @@ -11,6 +11,7 @@ import { Layer, Option, Ref, + Schedule, } from "effect"; import { @@ -46,6 +47,11 @@ export function makeEffectAIRoutingStateLayer( const lastSelection = yield* Ref.make>( Option.none(), ); + const releaseRoutingLease = (leaseToken: LeaseToken) => + routing.release(leaseToken).pipe( + Effect.asVoid, + Effect.retry(Schedule.spaced("1 second")), + ); const summary: AIRoutingState["Service"]["summary"] = (now) => routeEffect(Effect.gen(function*() { @@ -144,10 +150,7 @@ export function makeEffectAIRoutingStateLayer( if (acquired === undefined) return; const wasTransferred = yield* Ref.get(transferred); if (wasTransferred && exit._tag === "Success") return; - yield* routing.release(acquired.leaseToken).pipe( - Effect.asVoid, - Effect.catchCause(() => Effect.void), - ); + yield* releaseRoutingLease(acquired.leaseToken); })), ); }); @@ -173,7 +176,7 @@ export function makeEffectAIRoutingStateLayer( renew: (leaseToken, now) => routeEffect(routing.renew(LeaseToken.make(leaseToken), now)), release: (leaseToken) => - routeEffect(routing.release(LeaseToken.make(leaseToken))).pipe( + routeEffect(releaseRoutingLease(LeaseToken.make(leaseToken))).pipe( Effect.as(true), ), recordResponse: (accountId, status, headers, now) => diff --git a/services/ai-gateway/src/forward.ts b/services/ai-gateway/src/forward.ts index 35a421dc..24889a29 100644 --- a/services/ai-gateway/src/forward.ts +++ b/services/ai-gateway/src/forward.ts @@ -539,17 +539,20 @@ function finalizeStream( telemetry: AIGatewayRequestTelemetry, signal: AbortSignal, ): Effect.Effect { + const maximumReleaseAttempts = 3; + let attempts = 0; let released = false; const releaseStreamLease = Effect.whileLoop({ - while: () => !released, + while: () => !released && attempts < maximumReleaseAttempts, body: () => releaseLeaseOnce.pipe( Effect.flatMap((attempted) => - attempted - ? Effect.succeed(true) + attempted || attempts + 1 >= maximumReleaseAttempts + ? Effect.succeed(attempted) : Effect.sleep(1_000).pipe(Effect.as(false)) ), ), step: (attempted) => { + attempts += 1; released = attempted; }, }); From 0682e2ce3318a7802913c92bc808418e81f14546 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Robin=20Br=C3=A4mer?= <22003767+robinbraemer@users.noreply.github.com> Date: Mon, 10 Aug 2026 19:27:29 +0200 Subject: [PATCH 21/28] no-mistakes(review): Bound shared routing release retries --- services/ai-gateway/src/effect-routing-state.ts | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/services/ai-gateway/src/effect-routing-state.ts b/services/ai-gateway/src/effect-routing-state.ts index f04f1942..19317a1e 100644 --- a/services/ai-gateway/src/effect-routing-state.ts +++ b/services/ai-gateway/src/effect-routing-state.ts @@ -50,7 +50,7 @@ export function makeEffectAIRoutingStateLayer( const releaseRoutingLease = (leaseToken: LeaseToken) => routing.release(leaseToken).pipe( Effect.asVoid, - Effect.retry(Schedule.spaced("1 second")), + Effect.retry({ schedule: Schedule.spaced("1 second"), times: 3 }), ); const summary: AIRoutingState["Service"]["summary"] = (now) => From 2db7bc2b42681c61a1fb761f0e5eb57a25be8183 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Robin=20Br=C3=A4mer?= <22003767+robinbraemer@users.noreply.github.com> Date: Mon, 10 Aug 2026 19:47:11 +0200 Subject: [PATCH 22/28] no-mistakes(review): Model routing ownership cleanup in cancellation test --- .../tests/gateway-service.effect.test.ts | 35 ++++++++++++++----- 1 file changed, 26 insertions(+), 9 deletions(-) diff --git a/services/ai-gateway/tests/gateway-service.effect.test.ts b/services/ai-gateway/tests/gateway-service.effect.test.ts index eebd1a4a..09e6a1bf 100644 --- a/services/ai-gateway/tests/gateway-service.effect.test.ts +++ b/services/ai-gateway/tests/gateway-service.effect.test.ts @@ -117,6 +117,15 @@ const makeTestServices = Effect.fn("test.aiGateway.makeServices")( >>([]); const released = yield* Ref.make(0); const quotaCalls = yield* Ref.make(0); + const reservation = withAccounts + ? { + accountId: "managed-a", + leaseToken: "lease-a", + expiresAt: now + 60_000, + decisionReason: "best_candidate", + } + : undefined; + const releaseReservation = Ref.update(released, (count) => count + 1); const vault = ManagedAccountVault.of({ list: Effect.succeed(withAccounts ? [{ @@ -140,14 +149,22 @@ const makeTestServices = Effect.fn("test.aiGateway.makeServices")( activeReservations: 0, reservationsByAccount: {}, }), - acquire: (_input, use) => use(withAccounts - ? { - accountId: "managed-a", - leaseToken: "lease-a", - expiresAt: now + 60_000, - decisionReason: "best_candidate", - } - : undefined, Effect.void), + acquire: (_input, use) => Effect.gen(function*() { + const transferred = yield* Ref.make(false); + return yield* Effect.acquireUseRelease( + Effect.succeed(reservation), + (acquired) => use( + acquired, + Effect.uninterruptible(Ref.set(transferred, true)), + ), + (acquired, exit) => Effect.gen(function*() { + if (acquired === undefined) return; + const wasTransferred = yield* Ref.get(transferred); + if (wasTransferred && exit._tag === "Success") return; + yield* releaseReservation; + }), + ); + }), evaluate: () => Effect.succeed(withAccounts ? { accountId: "managed-a", @@ -159,7 +176,7 @@ const makeTestServices = Effect.fn("test.aiGateway.makeServices")( candidates: [], }), renew: () => Effect.succeed(true), - release: () => Ref.update(released, (count) => count + 1).pipe( + release: () => releaseReservation.pipe( Effect.as(true), ), recordResponse: () => Effect.void, From c3c4c50cdeea21acc3700cab13fa282ac810a5a0 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Robin=20Br=C3=A4mer?= <22003767+robinbraemer@users.noreply.github.com> Date: Mon, 10 Aug 2026 20:19:22 +0200 Subject: [PATCH 23/28] no-mistakes(document): Document Hermes proxy contract and clean Effect lint --- ARCHITECTURE.md | 17 ++++-- docs/integrations/hermes-ai-gateway.md | 18 +++++- services/ai-gateway/README.md | 11 ++-- .../ai-gateway/src/effect-routing-state.ts | 2 +- .../ai-gateway/tests/forward.effect.test.ts | 47 ++++++++++++++-- .../tests/gateway-service.effect.test.ts | 20 +++++-- .../tests/provider-http.effect.test.ts | 55 ++++++++++--------- .../tests/routing-state.effect.test.ts | 2 +- .../workload-client-proxy.effect.test.ts | 34 ++++++------ 9 files changed, 140 insertions(+), 66 deletions(-) diff --git a/ARCHITECTURE.md b/ARCHITECTURE.md index c1833e32..931cce20 100644 --- a/ARCHITECTURE.md +++ b/ARCHITECTURE.md @@ -475,15 +475,20 @@ never selects or changes a provider or model. `quota-axi` remains an observation-only tool and has no routing, login or mutation authority. An approved Responses client that cannot supply command-backed authentication -uses the AgentOS `ai-gateway-workload-proxy` as a loopback-only sidecar. The -sidecar rereads projected workload identity for each request, removes the +uses the AgentOS `ai-gateway-workload-proxy` as a loopback-only sidecar; the +exact Hermes contract and Pod wiring live in +[`docs/integrations/hermes-ai-gateway.md`](./docs/integrations/hermes-ai-gateway.md). +The sidecar rereads projected workload identity for each request, removes the client's placeholder authorization and all caller-supplied AgentOS authority headers, optionally injects a validated Assignment ID from its own trusted configuration, and forwards only the two reviewed Responses paths once to -Agentgateway. W3C trace correlation passes through unchanged. Its process-only -liveness and local projected-token readiness never contact an upstream. It -owns no credential, selection, retry, fallback, prompt, response, or deployment -authority. The +Agentgateway. It preserves the upstream status, end-to-end headers after +required hop-by-hop and stale `Content-Encoding`/`Content-Length` metadata +removal, and stream without following redirects. W3C trace correlation passes +through unchanged. Its +process-only liveness and local projected-token readiness never contact an +upstream. It owns no credential, +selection, retry, fallback, prompt, response, or deployment authority. The client keeps exact model selection and disables request replay and provider fallback for this route; its workload owner composes the sidecar and projected token into that client's reviewed native manifest. diff --git a/docs/integrations/hermes-ai-gateway.md b/docs/integrations/hermes-ai-gateway.md index 6f318e82..1e39ff31 100644 --- a/docs/integrations/hermes-ai-gateway.md +++ b/docs/integrations/hermes-ai-gateway.md @@ -27,8 +27,11 @@ sends only `POST /v1/responses` or `POST /v1/responses/compact` to a loopback `agentgateway-openai`; W3C `traceparent` and `tracestate` correlation remain unchanged; - forwards the request once, without inspecting or storing its body; and -- returns the actual upstream status, headers, and stream without retrying or - selecting another model or account. +- returns the actual upstream status, end-to-end headers after required + hop-by-hop and stale `Content-Encoding`/`Content-Length` metadata removal, + and stream without retrying, following redirects, or selecting another model + or account. A transport failure returns the adapter's stable `502`; it never + becomes apparent success. Agentgateway and `agentos-egress-authz`, not the sidecar, authenticate the Pod and authorize its registered Agent or Assignment. Do not expose the loopback @@ -119,6 +122,10 @@ spec: env: - name: AI_GATEWAY_URL value: http://agentgateway-openai.agentos.svc.cluster.local:8788 + - name: AI_GATEWAY_IDLE_TIMEOUT_SECONDS + value: "255" + - name: AI_GATEWAY_GRACEFUL_SHUTDOWN_MILLIS + value: "20000" - name: AGENTOS_EGRESS_TOKEN_FILE value: /var/run/secrets/agentos-egress/token # Optional: set only from the workload owner's trusted Assignment. @@ -171,6 +178,13 @@ and its approved access profile through the normal AgentOS access-plane procedure, and preserve the existing NetworkPolicy. The label grants only network reachability to Agentgateway; it is not authorization. +The proxy's idle timeout defaults to the shown maximum of 255 seconds and its +graceful-shutdown timeout defaults to 20,000 milliseconds. The idle timeout is +an inactivity limit, not a total response limit; retain the 255-second value +when the approved provider stream can have gaps that long. The proxy uses +manual redirect handling, so a `3xx` response is returned to Hermes rather than +followed. + ## Operator workflow Use `$agentos-ai-gateway` for the approval, migration, rollout, verification, diff --git a/services/ai-gateway/README.md b/services/ai-gateway/README.md index 2347633f..330289e4 100644 --- a/services/ai-gateway/README.md +++ b/services/ai-gateway/README.md @@ -60,13 +60,10 @@ the multi-subscription pool; `$agentos-ai-gateway` owns the exact lifecycle and native Pi/Codex configuration. Responses clients that cannot reread a projected Kubernetes token on every -request use the loopback-only `ai-gateway-workload-proxy` sidecar. It replaces -the client's non-secret transport placeholder with fresh workload identity, -forwards only `/v1/responses` and `/v1/responses/compact` once, and preserves -the actual upstream response. Its `/livez` is process-only and `/readyz` -validates projected identity locally without upstream traffic. The pinned -Hermes 0.20 contract, Pod wiring, and migration from the legacy shared-token -topology are documented in +request use the loopback-only `ai-gateway-workload-proxy` sidecar. Its exact +Responses paths, identity and failure contract, timeout settings, health +checks, Pod wiring, and migration from the legacy shared-token topology are +documented in [`docs/integrations/hermes-ai-gateway.md`](../../docs/integrations/hermes-ai-gateway.md). `GET /healthz` proves only that the process is serving. `GET /readyz` requires diff --git a/services/ai-gateway/src/effect-routing-state.ts b/services/ai-gateway/src/effect-routing-state.ts index 19317a1e..b28e8867 100644 --- a/services/ai-gateway/src/effect-routing-state.ts +++ b/services/ai-gateway/src/effect-routing-state.ts @@ -150,7 +150,7 @@ export function makeEffectAIRoutingStateLayer( if (acquired === undefined) return; const wasTransferred = yield* Ref.get(transferred); if (wasTransferred && exit._tag === "Success") return; - yield* releaseRoutingLease(acquired.leaseToken); + yield* routeEffect(releaseRoutingLease(acquired.leaseToken)); })), ); }); diff --git a/services/ai-gateway/tests/forward.effect.test.ts b/services/ai-gateway/tests/forward.effect.test.ts index 8287a98b..614c818f 100644 --- a/services/ai-gateway/tests/forward.effect.test.ts +++ b/services/ai-gateway/tests/forward.effect.test.ts @@ -8,7 +8,17 @@ import { type ProviderBudgetSettlementReceiptV1, type ProviderBudgetSettlementReportV1, } from "@akua-dev/agentos"; -import { Effect, Exit, Fiber, Layer, Metric, Ref, Stream, Tracer } from "effect"; +import { + Effect, + Exit, + Fiber, + Layer, + Metric, + Option, + Ref, + Stream, + Tracer, +} from "effect"; import { TestClock } from "effect/testing"; import { @@ -96,6 +106,7 @@ const encoder = new TextEncoder(); const makeLease = Effect.fn("test.aiForward.makeLease")(function*() { const releases = yield* Ref.make(0); + const released = yield* Ref.make(false); const renewals = yield* Ref.make(0); const lease: AIForwardLease = { kind: "openai_api_key", @@ -103,7 +114,13 @@ const makeLease = Effect.fn("test.aiForward.makeLease")(function*() { renew: Ref.updateAndGet(renewals, (count) => count + 1).pipe( Effect.as(true), ), - release: Ref.update(releases, (count) => count + 1), + release: Effect.gen(function*() { + const shouldRelease = yield* Ref.modify( + released, + (state): readonly [boolean, boolean] => [!state, true], + ); + if (shouldRelease) yield* Ref.update(releases, (count) => count + 1); + }), recordResponse: () => Effect.void, }; return { lease, releases, renewals }; @@ -119,7 +136,29 @@ function acquireLease(lease: AIForwardLease | undefined) { lease: AIForwardLease | undefined, transfer: Effect.Effect, ) => Effect.Effect, - ) => use(lease, Effect.void); + ) => { + if (lease === undefined) return use(undefined, Effect.void); + return Effect.gen(function*() { + const transferred = yield* Ref.make(false); + return yield* Effect.acquireUseRelease( + Effect.succeed(lease), + (acquired) => use( + acquired, + Effect.uninterruptible(Ref.set(transferred, true)), + ), + (acquired, exit) => + exit._tag === "Success" && acquired !== undefined + ? Ref.get(transferred).pipe( + Effect.flatMap((wasTransferred) => + wasTransferred ? Effect.void : acquired.release + ), + ) + : acquired === undefined + ? Effect.void + : acquired.release, + ); + }); + }; } const makeSettlementRecorder = Effect.fn( @@ -864,7 +903,7 @@ describe("Effect AI Gateway forwarding", () => { const brokenLease: AIForwardLease = { kind: route.lease.kind, get accessToken(): string { - throw new Error("invalid lease"); + return Option.getOrThrow(Option.none()); }, renew: route.lease.renew, release: route.lease.release, diff --git a/services/ai-gateway/tests/gateway-service.effect.test.ts b/services/ai-gateway/tests/gateway-service.effect.test.ts index 09e6a1bf..5239b220 100644 --- a/services/ai-gateway/tests/gateway-service.effect.test.ts +++ b/services/ai-gateway/tests/gateway-service.effect.test.ts @@ -401,14 +401,26 @@ describe("Effect AI Gateway application", () => { yield* TestClock.setTime(now); const services = yield* makeTestServices(true); const credentialStarted = yield* Deferred.make(); + const credentialCalls = yield* Ref.make(0); const application = yield* makeApplication({ ...services, vault: ManagedAccountVault.of({ ...services.vault, - getFreshCredential: () => - Deferred.succeed(credentialStarted, undefined).pipe( - Effect.andThen(Effect.never), - ), + getFreshCredential: () => Effect.gen(function*() { + const call = yield* Ref.modify( + credentialCalls, + (count): readonly [number, number] => [count, count + 1], + ); + if (call === 0) { + return { + providerAccountId: "provider-a", + accessToken: "oauth-provider-secret", + expiresAt: now + 60_000, + }; + } + yield* Deferred.succeed(credentialStarted, undefined); + return yield* Effect.never; + }), }), }); const requestFiber = yield* Effect.forkChild(Effect.exit( diff --git a/services/ai-gateway/tests/provider-http.effect.test.ts b/services/ai-gateway/tests/provider-http.effect.test.ts index 8c10ea86..34278106 100644 --- a/services/ai-gateway/tests/provider-http.effect.test.ts +++ b/services/ai-gateway/tests/provider-http.effect.test.ts @@ -115,30 +115,33 @@ describe("AI provider HTTP adapter", () => { readonly redirect: RequestInit["redirect"] | "default"; readonly url: string; }> = []; - const fetch: typeof globalThis.fetch = (input, init) => { - const request = input instanceof Request - ? input - : new Request(input, init); - calls.push({ - authorization: new Headers(init?.headers).get("authorization"), - redirect: init?.redirect ?? "default", - url: request.url, - }); - if (init?.redirect !== "manual") { + const fetchImpl = Object.assign( + (input: string | Request | URL, init?: RequestInit) => { + const request = input instanceof Request + ? input + : new Request(input.toString(), init); calls.push({ authorization: new Headers(init?.headers).get("authorization"), redirect: init?.redirect ?? "default", - url: "https://redirect-target.invalid/v1/responses", + url: request.url, }); - } - return Promise.resolve(new Response(null, { - status: 307, - headers: { location: "https://redirect-target.invalid/v1/responses" }, - })); - }; + if (init?.redirect !== "manual") { + calls.push({ + authorization: new Headers(init?.headers).get("authorization"), + redirect: init?.redirect ?? "default", + url: "https://redirect-target.invalid/v1/responses", + }); + } + return Promise.resolve(new Response(null, { + status: 307, + headers: { location: "https://redirect-target.invalid/v1/responses" }, + })); + }, + { preconnect: globalThis["fetch"].preconnect }, + ); const fetchLayer = FetchHttpClient.layer.pipe( Layer.provide(AIProviderHttpRequestInit), - Layer.provide(Layer.succeed(FetchHttpClient.Fetch, fetch)), + Layer.provide(Layer.succeed(FetchHttpClient.Fetch, fetchImpl)), ); const layer = AIProviderHttpLive.pipe( Layer.provide(fetchLayer), @@ -164,12 +167,15 @@ describe("AI provider HTTP adapter", () => { it.effect("scopes manual redirects to the provider client", () => Effect.gen(function*() { const redirects: Array = []; - const fetch: typeof globalThis.fetch = (_input, init) => { - redirects.push(init?.redirect ?? "default"); - return Promise.resolve(new Response(null, { status: 204 })); - }; + const fetchImpl = Object.assign( + (_input: string | Request | URL, init?: RequestInit) => { + redirects.push(init?.redirect ?? "default"); + return Promise.resolve(new Response(null, { status: 204 })); + }, + { preconnect: globalThis["fetch"].preconnect }, + ); const ordinaryClientLayer = FetchHttpClient.layer.pipe( - Layer.provide(Layer.succeed(FetchHttpClient.Fetch, fetch)), + Layer.provide(Layer.succeed(FetchHttpClient.Fetch, fetchImpl)), ); const provider = yield* AIProviderHttp.pipe( Effect.provide(makeAIProviderHttpLive(ordinaryClientLayer)), @@ -222,10 +228,9 @@ describe("AI provider HTTP adapter", () => { Effect.andThen(Effect.succeed(HttpClientResponse.fromWeb( request, new Response(new ReadableStream({ - pull(controller) { + start(controller) { bodyStarted = true; controller.enqueue(new Uint8Array([1])); - return new Promise(() => {}); }, })), ))), diff --git a/services/ai-gateway/tests/routing-state.effect.test.ts b/services/ai-gateway/tests/routing-state.effect.test.ts index 39a6d874..4a9421fb 100644 --- a/services/ai-gateway/tests/routing-state.effect.test.ts +++ b/services/ai-gateway/tests/routing-state.effect.test.ts @@ -208,7 +208,7 @@ describe("Effect canonical durable routing state", () => { rejectionCode: "reauthentication_required", }]); return acquired; - })); + }))); }), ); }).pipe(Effect.provide(BunFileSystem.layer)))); diff --git a/services/ai-gateway/tests/workload-client-proxy.effect.test.ts b/services/ai-gateway/tests/workload-client-proxy.effect.test.ts index 02009ed8..96d560f6 100644 --- a/services/ai-gateway/tests/workload-client-proxy.effect.test.ts +++ b/services/ai-gateway/tests/workload-client-proxy.effect.test.ts @@ -8,6 +8,7 @@ import { Schema, Stream, } from "effect"; +import { TestClock } from "effect/testing"; import { parse } from "yaml"; import { @@ -55,7 +56,7 @@ suite("Hermes Responses workload client proxy", (it) => { import.meta.url, ); const fixture = yield* fileSystem.readFileString(fixtureUrl.pathname); - const parsedFixture = parse(fixture) as Record; + const parsedFixture = parse(fixture); const config = yield* Schema.decodeUnknownEffect(HermesConfiguration)( parsedFixture, ); @@ -305,27 +306,28 @@ suite("Hermes Responses workload client proxy", (it) => { const handler = yield* makeWorkloadClientProxyHandler({ upstreamBaseUrl: new URL("http://agentgateway-openai.agentos.svc.cluster.local:8788"), tokenPath, - forward: () => Effect.succeed(new Response( - new ReadableStream({ - start(controller) { - controller.enqueue(new TextEncoder().encode("first")); - setTimeout(() => { - controller.enqueue(new TextEncoder().encode("second")); - controller.close(); - }, 1_100); - }, - }), - { status: 200, headers: { "content-type": "text/plain" } }, - )), + forward: () => Stream.toReadableStreamEffect(Stream.concat( + Stream.succeed(new TextEncoder().encode("first")), + Stream.fromEffect( + Effect.sleep("1.1 seconds").pipe( + Effect.andThen(Effect.succeed(new TextEncoder().encode("second"))), + ), + ), + )).pipe( + Effect.map((body) => new Response(body, { + status: 200, + headers: { "content-type": "text/plain" }, + })), + ), }); - const response = yield* handler(new Request( + const response = yield* TestClock.withLive(handler(new Request( "http://127.0.0.1:8790/v1/responses", { method: "POST" }, - )); + ))); assert.strictEqual(response.status, 200); assert.strictEqual( - yield* Effect.promise(() => response.text()), + yield* TestClock.withLive(Effect.promise(() => response.text())), "firstsecond", ); })); From faa3b5cbf6ddd1628ddbadb9bb7a7b7ea346fc4a Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Robin=20Br=C3=A4mer?= <22003767+robinbraemer@users.noreply.github.com> Date: Mon, 10 Aug 2026 21:28:35 +0200 Subject: [PATCH 24/28] no-mistakes(review): Preserve responses through lease cleanup failures --- .../ai-gateway/src/effect-routing-state.ts | 12 ++-- .../tests/routing-state.effect.test.ts | 63 ++++++++++++++++++- 2 files changed, 69 insertions(+), 6 deletions(-) diff --git a/services/ai-gateway/src/effect-routing-state.ts b/services/ai-gateway/src/effect-routing-state.ts index b28e8867..a5310d1f 100644 --- a/services/ai-gateway/src/effect-routing-state.ts +++ b/services/ai-gateway/src/effect-routing-state.ts @@ -35,11 +35,11 @@ type LastSelection = NonNullable; export function makeEffectAIRoutingStateLayer( path: string, config: RoutingConfig, -) { - const infrastructure = sqliteRoutingStateLayer( + routingLayer = sqliteRoutingStateLayer( path, toRouterConfig(config), - ); + ), +) { return Layer.effect( AIRoutingState, Effect.gen(function*() { @@ -150,7 +150,9 @@ export function makeEffectAIRoutingStateLayer( if (acquired === undefined) return; const wasTransferred = yield* Ref.get(transferred); if (wasTransferred && exit._tag === "Success") return; - yield* routeEffect(releaseRoutingLease(acquired.leaseToken)); + yield* routeEffect(releaseRoutingLease(acquired.leaseToken)).pipe( + Effect.catchCause(() => Effect.void), + ); })), ); }); @@ -187,7 +189,7 @@ export function makeEffectAIRoutingStateLayer( )), }); }), - ).pipe(Layer.provide(infrastructure)); + ).pipe(Layer.provide(routingLayer)); } function overlayRoutingSummary( diff --git a/services/ai-gateway/tests/routing-state.effect.test.ts b/services/ai-gateway/tests/routing-state.effect.test.ts index 4a9421fb..07235881 100644 --- a/services/ai-gateway/tests/routing-state.effect.test.ts +++ b/services/ai-gateway/tests/routing-state.effect.test.ts @@ -1,9 +1,21 @@ import * as BunFileSystem from "@effect/platform-bun/BunFileSystem"; +import { + AccountId, + LeaseToken, + RouteLease, + RoutingState as CodexRoutingState, + RoutingStateError, + RoutingSummary, +} from "@akua-dev/codex-router/core"; import { assert, describe, it } from "@effect/vitest"; import { Effect, + Fiber, FileSystem, + Layer, + Option, } from "effect"; +import { TestClock } from "effect/testing"; import { makeEffectAIRoutingStateLayer } from "../src/effect-routing-state.ts"; import { defaultRoutingConfig } from "../src/selection.ts"; @@ -48,11 +60,16 @@ function useRouting( path: string, config: RoutingConfig, operation: (routing: AIRoutingState["Service"]) => Effect.Effect, + routingLayer?: Layer.Layer, ) { return Effect.scoped(Effect.gen(function*() { const routing = yield* AIRoutingState; return yield* operation(routing); - }).pipe(Effect.provide(makeEffectAIRoutingStateLayer(path, config)))); + }).pipe(Effect.provide(makeEffectAIRoutingStateLayer( + path, + config, + routingLayer, + )))); } describe("Effect canonical durable routing state", () => { @@ -109,6 +126,50 @@ describe("Effect canonical durable routing state", () => { assert.strictEqual(sticky?.accountId, "a"); }).pipe(Effect.provide(BunFileSystem.layer)))); + it.effect("preserves successful use when lease cleanup fails", () => + Effect.gen(function*() { + const fiber = yield* Effect.forkChild(useRouting( + "unused", + defaultRoutingConfig, + (routing) => + routing.acquire({ candidates, now }, (acquired) => + Effect.gen(function*() { + assert.isDefined(acquired); + if (acquired === undefined) { + return new Response(null, { status: 503 }); + } + return new Response(null, { + status: 429, + headers: { "retry-after": "7" }, + }); + }) + ), + Layer.succeed(CodexRoutingState, { + acquire: () => Effect.succeed(Option.some(RouteLease.make({ + accountId: AccountId.make("a"), + expiresAt: now + 60_000, + leaseToken: LeaseToken.make("lease-token"), + sessionKey: Option.none(), + }))), + recordResponse: () => Effect.void, + release: () => Effect.fail(RoutingStateError.make({ + message: "cleanup failed", + })), + renew: () => Effect.succeed(true), + summary: () => Effect.succeed(RoutingSummary.make({ + activeReservations: 0, + accounts: [], + assignments: 0, + })), + }), + )); + yield* Effect.yieldNow; + yield* TestClock.adjust(5_001); + const response = yield* Fiber.join(fiber); + assert.strictEqual(response.status, 429); + assert.strictEqual(response.headers.get("retry-after"), "7"); + })); + it.effect("expires leases without inventing implicit stickiness", () => Effect.scoped(Effect.gen(function*() { const fileSystem = yield* FileSystem.FileSystem; From 456d885f2c2a0bbbe76e187e437d15e89e2380b7 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Robin=20Br=C3=A4mer?= <22003767+robinbraemer@users.noreply.github.com> Date: Mon, 10 Aug 2026 21:51:07 +0200 Subject: [PATCH 25/28] no-mistakes(document): Refresh workload identity guidance and routing typing --- docs/security/workload-identity.md | 2 +- services/ai-gateway/src/effect-routing-state.ts | 3 ++- 2 files changed, 3 insertions(+), 2 deletions(-) diff --git a/docs/security/workload-identity.md b/docs/security/workload-identity.md index ef42da14..0ef0172f 100644 --- a/docs/security/workload-identity.md +++ b/docs/security/workload-identity.md @@ -11,7 +11,7 @@ AgentOS workloads receive a second, explicit projected ServiceAccount token for - requested lifetime: 600 seconds; and - volume and mount: `agentos-egress-identity`, read-only with mode `0440`. -This is not the workload's Kubernetes API token. Persistent Second Mates retain their separately managed Kubernetes supervision identity. Crewmates keep `automountServiceAccountToken: false` and receive only the explicit audience-scoped projection. Only the long-running Agent container mounts this token; tool-install and home-preparation init containers do not, and domain admission rejects any Crewmate init-container mount. The kubelet rotates projected tokens, so clients must read the token file for each request rather than retaining its contents for the process lifetime. +This is not the workload's Kubernetes API token. Persistent Second Mates retain their separately managed Kubernetes supervision identity. Crewmates keep `automountServiceAccountToken: false` and receive only the explicit audience-scoped projection. Only the long-running provider client or a reviewed provider sidecar/adapter mounts this token; tool-install and home-preparation init containers do not, and domain admission rejects any Crewmate init-container mount. The kubelet rotates projected tokens, so the component that presents the identity must read the token file for each request rather than retaining its contents for the process lifetime. The authorization service built by #90 supplies the raw token only to `KubernetesTokenReviewer`. It requests exactly the dedicated audience and accepts a result only when `authenticated` is true and the same audience appears in `status.audiences`. Kubernetes documents that a TokenReview client setting audiences must check the returned intersection; accepting an empty result would fall back to API-server audience semantics and is forbidden here. See the Kubernetes [TokenReview API](https://kubernetes.io/docs/reference/kubernetes-api/definitions/token-review-v1-authentication/). diff --git a/services/ai-gateway/src/effect-routing-state.ts b/services/ai-gateway/src/effect-routing-state.ts index a5310d1f..d793a3aa 100644 --- a/services/ai-gateway/src/effect-routing-state.ts +++ b/services/ai-gateway/src/effect-routing-state.ts @@ -35,7 +35,8 @@ type LastSelection = NonNullable; export function makeEffectAIRoutingStateLayer( path: string, config: RoutingConfig, - routingLayer = sqliteRoutingStateLayer( + routingLayer: Layer.Layer = + sqliteRoutingStateLayer( path, toRouterConfig(config), ), From 7feccabf148c7532152ab0eb3431d36ab0c4e3cc Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Robin=20Br=C3=A4mer?= <22003767+robinbraemer@users.noreply.github.com> Date: Mon, 10 Aug 2026 22:36:37 +0200 Subject: [PATCH 26/28] test(ai-gateway): keep provider layers scoped --- .../tests/provider-http.effect.test.ts | 47 ++++++++++--------- 1 file changed, 25 insertions(+), 22 deletions(-) diff --git a/services/ai-gateway/tests/provider-http.effect.test.ts b/services/ai-gateway/tests/provider-http.effect.test.ts index 34278106..03da4fdc 100644 --- a/services/ai-gateway/tests/provider-http.effect.test.ts +++ b/services/ai-gateway/tests/provider-http.effect.test.ts @@ -146,15 +146,17 @@ describe("AI provider HTTP adapter", () => { const layer = AIProviderHttpLive.pipe( Layer.provide(fetchLayer), ); - const provider = yield* AIProviderHttp.pipe(Effect.provide(layer)); - const response = yield* provider.execute(new Request( - "https://api.openai.test/v1/responses", - { - method: "POST", - headers: { authorization: "Bearer projected-workload-token" }, - body: "{}", - }, - )); + const response = yield* Effect.gen(function*() { + const provider = yield* AIProviderHttp; + return yield* provider.execute(new Request( + "https://api.openai.test/v1/responses", + { + method: "POST", + headers: { authorization: "Bearer projected-workload-token" }, + body: "{}", + }, + )); + }).pipe(Effect.provide(layer)); assert.strictEqual(response.status, 307); assert.deepStrictEqual(calls, [{ @@ -177,21 +179,22 @@ describe("AI provider HTTP adapter", () => { const ordinaryClientLayer = FetchHttpClient.layer.pipe( Layer.provide(Layer.succeed(FetchHttpClient.Fetch, fetchImpl)), ); - const provider = yield* AIProviderHttp.pipe( + yield* Effect.gen(function*() { + const ordinaryClient = yield* HttpClient.HttpClient; + yield* Effect.scoped( + HttpClient.withScope(ordinaryClient).execute( + HttpClientRequest.get("https://api.openai.test/status"), + ), + ); + }).pipe(Effect.provide(ordinaryClientLayer)); + yield* Effect.gen(function*() { + const provider = yield* AIProviderHttp; + yield* provider.execute(new Request( + "https://api.openai.test/v1/responses", + )); + }).pipe( Effect.provide(makeAIProviderHttpLive(ordinaryClientLayer)), ); - const ordinaryClient = yield* HttpClient.HttpClient.pipe( - Effect.provide(ordinaryClientLayer), - ); - - yield* Effect.scoped( - HttpClient.withScope(ordinaryClient).execute( - HttpClientRequest.get("https://api.openai.test/status"), - ), - ); - yield* provider.execute(new Request( - "https://api.openai.test/v1/responses", - )); assert.deepStrictEqual(redirects, ["default", "manual"]); })); From 043d25db786860b5187b9c1ab4c90dcc7beb437d Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Robin=20Br=C3=A4mer?= <22003767+robinbraemer@users.noreply.github.com> Date: Tue, 11 Aug 2026 09:16:26 +0200 Subject: [PATCH 27/28] test(ai-gateway): observe fetch URLs without rebuilding requests --- services/ai-gateway/tests/provider-http.effect.test.ts | 6 ++---- 1 file changed, 2 insertions(+), 4 deletions(-) diff --git a/services/ai-gateway/tests/provider-http.effect.test.ts b/services/ai-gateway/tests/provider-http.effect.test.ts index 03da4fdc..5fe1013f 100644 --- a/services/ai-gateway/tests/provider-http.effect.test.ts +++ b/services/ai-gateway/tests/provider-http.effect.test.ts @@ -117,13 +117,11 @@ describe("AI provider HTTP adapter", () => { }> = []; const fetchImpl = Object.assign( (input: string | Request | URL, init?: RequestInit) => { - const request = input instanceof Request - ? input - : new Request(input.toString(), init); + const url = input instanceof Request ? input.url : input.toString(); calls.push({ authorization: new Headers(init?.headers).get("authorization"), redirect: init?.redirect ?? "default", - url: request.url, + url, }); if (init?.redirect !== "manual") { calls.push({ From 2d3348ed519c5255ccd8837c3743af25243e9554 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Robin=20Br=C3=A4mer?= <22003767+robinbraemer@users.noreply.github.com> Date: Tue, 11 Aug 2026 09:33:36 +0200 Subject: [PATCH 28/28] no-mistakes(document): Clarify Fleet AI Gateway client documentation --- .../agentos/skills/agentos-ai-gateway/SKILL.md | 12 ++++++------ .../docs/content/docs/operate/ai-gateway.mdx | 16 ++++++++++------ 2 files changed, 16 insertions(+), 12 deletions(-) diff --git a/packages/agentos/skills/agentos-ai-gateway/SKILL.md b/packages/agentos/skills/agentos-ai-gateway/SKILL.md index 130aca07..632d626c 100644 --- a/packages/agentos/skills/agentos-ai-gateway/SKILL.md +++ b/packages/agentos/skills/agentos-ai-gateway/SKILL.md @@ -223,12 +223,12 @@ normal recovery procedure. ## Hermes workload-proxy migration -Before any Hermes migration, verify that the selected pinned revision's -`codex_responses` runtime has a zero-retry Responses path. Do not deploy or -update this integration while that gate is unmet: Hermes `0.20.0` / `v2026.8.3` -hardcodes one retry around `responses.create`. Auxiliary requests are also out -of scope until Hermes provides a native Responses path with no automatic retry, -redirect, or fallback; do not enable them during migration. +Before any Hermes migration, read the [Hermes integration +contract](../../../../docs/integrations/hermes-ai-gateway.md) and verify that +the selected pinned revision passes its deployment gate. Do not deploy or +update this integration while that gate is unmet; auxiliary requests remain out +of scope until the selected revision satisfies the contract's no-retry, +no-redirect and no-fallback requirements. For the live v0.1.24 Hermes topology, the observed pre-migration state uses one `ai-gateway` replica with a shared `AI_GATEWAY_TOKEN`, while Hermes uses direct diff --git a/website/apps/docs/content/docs/operate/ai-gateway.mdx b/website/apps/docs/content/docs/operate/ai-gateway.mdx index a67eb515..561b2e2c 100644 --- a/website/apps/docs/content/docs/operate/ai-gateway.mdx +++ b/website/apps/docs/content/docs/operate/ai-gateway.mdx @@ -6,6 +6,8 @@ canonical: path: packages/agentos/skills/agentos-ai-gateway/SKILL.md - label: AI Gateway service path: services/ai-gateway + - label: Hermes integration contract + path: docs/integrations/hermes-ai-gateway.md --- The Fleet AI Gateway is optional pooled capacity for reviewed Codex subscription accounts with an @@ -17,13 +19,15 @@ retiring the gateway. It keeps server-owned provider OAuth and quota-aware routi deliberate service boundary. Client Agents receive only the scoped connection material intended for them; they never receive a provider credential or Fleet-shared inference token. -Clients may live in `agentos` or a Second-Mate domain namespace. They call +Native AgentOS clients may live in `agentos` or a Second-Mate domain namespace. They call `agentgateway-openai.agentos.svc.cluster.local` and authenticate with their kubelet-rotated, -audience-bound ServiceAccount token. Agentgateway asks the TokenReview/OpenFGA authorizer for a -closed grant lasting at most 15 seconds. Only Agentgateway can reach the AI Gateway through its -NetworkPolicy; provider OAuth, the operator Secret and the Gateway vault remain in the core -namespace. This lets a Second Mate manage Crewmates in its namespace without gaining another -domain’s identity or provider credentials, while unrelated Internet traffic stays direct. +audience-bound ServiceAccount token. A Responses client that cannot reread that token for every +request uses the loopback-only workload proxy documented in the Hermes integration contract. +Agentgateway asks the TokenReview/OpenFGA authorizer for a closed grant lasting at most 15 seconds. +Only Agentgateway can reach the AI Gateway through its NetworkPolicy; provider OAuth, the operator +Secret and the Gateway vault remain in the core namespace. This lets a Second Mate manage +Crewmates in its namespace without gaining another domain’s identity or provider credentials, +while unrelated Internet traffic stays direct. ## What it does not own