feat(auth): make the provider access plane operable and recoverable

[robinbraemer]

Outcome

Make the provider access plane deployable, observable, recoverable, budget-aware, and safely upgradeable without coupling ordinary Agent Internet access to its availability.

Scope

• Add Kustomize integration, least-privilege identities, Services, probes, PodDisruptionBudgets, sizing/topology, and service-isolation NetworkPolicy.
• Define semantic readiness for agentgateway, authorizer, OpenFGA, provider adapters, credentials, policy/profile versions, budgets, and revocation.
• Integrate native gateway OTel with feat(observability): end-to-end OpenTelemetry for AgentOS Fleets #56, including safe identity correlation, access outcomes, MCP/HTTP latency, token/cost, rate classes, and pipeline health.
• Keep prompt logging and content-bearing request logs disabled; use the Fleet Collector and remote backend for supported observability.
• Add operation-journal reconciliation where configuration/profile/budget rollout crosses authorities.
• Document install, upgrade, rollback, backup/restore, adapter compromise, surgical kill switch, and direct-provider break glass.

Acceptance criteria

• Dependency failure has a typed attributable result and never leaks credentials.
• Ordinary outbound Internet works during complete access-plane outage.
• Rolling upgrade/rollback does not acknowledge unapplied policy, profile, credential, or budget changes.
• Readiness detects stale policy, unavailable identity checks, database failure, unusable credentials, and unenforced budget state.
• Telemetry correlates Fleet, Mate, Assignment, workload, policy, route, adapter, and provider outcome without content or unbounded metric labels.
• Kill switches affect only selected identities/profiles and survive Pod restarts.
• Runbooks are exercised in a disposable Fleet.

[robinbraemer]

Provider-access progress landed in #151 (7c6834cb92ea10e61e186ff60618d82cbee939f7):

• the OpenFGA management transport and Kubernetes model publisher now use Effect HttpClient, scoped requests, bounded Stream decoding, typed errors, and Layer-based tests
• raw fetch, Promise callbacks, Web-stream readers, and internal runtime execution were removed from those boundaries
• the repository-wide Effect gate now prevents those escapes from returning
• the implementation plan sequences live TokenReview/Pod/ServiceAccount identity, PostgreSQL policy snapshots, OpenFGA PDP, authorizer HTTP service, readiness/revocation, Kustomize, telemetry, and disposable-Fleet conformance

Next dependency slice: live Kubernetes workload identity adapters. The issue remains open until its complete acceptance matrix is proven.

[robinbraemer]

Live Kubernetes identity landed in #152 (c03a7dcc295622de6601e0567fb931068395d76a):

• concrete Effect Layers now implement TokenReview plus live Pod and ServiceAccount lookup
• the authorizer client token is reread for every request; the subject token is sent only in the bounded TokenReview body
• the mounted Kubernetes CA configures an isolated Effect HTTP client
• the full request/response stream is scoped, interruptible, size-bounded, and covered by one timeout
• only object 404 is normalized to absence; 401/403/409/429/5xx, malformed data, trust, credentials, network, timeout, and size failures remain distinct and content-free
• the Effect gate covers both new paths with zero exceptions

Required GitHub check and the full local suite passed. Next dependency slice: the least-privilege PostgreSQL identity/profile/ceiling snapshot store.

[robinbraemer]

PR #153 merged the PostgreSQL authorizer boundary: migration 0019 exposes only three schema-owner-protected SECURITY DEFINER readers, strips broad role grants, creates no login/credential, and the Effect SqlClient stores reject ambiguous, pending, expired, future, stale, scope-mismatched, inactive, and unreconciled snapshots. PGlite proves real role denial/allowed paths; pool starvation, database failure/recovery, interruption, and redaction are covered. Required CI passed with 361 Effect-assigned paths and zero violations. This advances the typed dependency/readiness foundation; deployment, semantic readiness, revocation, telemetry, and disposable-Fleet runbooks remain open.

[robinbraemer]

Landed PR #154 at 544527b: the concrete Effect-native provider PDP now reads the immutable PostgreSQL binding/profile/ceiling snapshot, revalidates freshness and references, matches one exact server-configured environment permission at both policy layers, and requires profile, ceiling, and effective-subject OpenFGA checks against pinned store/model IDs with HIGHER_CONSISTENCY. It returns only a cryptographically generated opaque reference with a maximum 15-second TTL. Typed failures remain content-free and distinguish invalid route, identity, database, stale policy, three denial layers, rate class, future rate/budget results, OpenFGA, and reference generation. Denial/interruption tests prove forwarding is never reached. Verification: full bun run check; 539 core pass / 4 intentional skip, 137 Effect pass, 114 site pass, all typechecks/lint, Effect gate 363 paths with zero violations. Remaining here: Effect HTTP service, semantic readiness/revocation, Kustomize deployment, adapter integration, telemetry/runbooks, and disposable-Fleet conformance.

[robinbraemer]

Progress landed in #155: the production agentos-egress-authz Effect Platform service now has two-replica Kustomize deployment, scoped PostgreSQL/OpenFGA/Kubernetes identity Layers, semantic dependency readiness, bounded admission/timeouts, least-privilege RBAC, private ingress-only networking, mounted-file secrets, and image wiring. Required CI check passed. This intentionally leaves ordinary Internet egress available and does not claim the later budget/rate/notification acceptance criteria.