oauth: CH-side JWT verifier sidecar; drop cluster_secret + ClaimsToHeaders

[BorisTyshkevich]

Summary

Shrink MCP's trust radius: MCP no longer impersonates users to ClickHouse via the shared cluster_secret. Instead, each query carries Authorization: Basic base64(email:JWT) and ClickHouse's <http_authentication> delegates to a new colocated ch-jwt-verify sidecar that validates the JWT against the upstream IdP's JWKS. A compromised MCP pod can no longer forge identity to CH.

E2E-verified in production: gating-mode MCP behind Auth0 with the sidecar colocated in the ClickHouse pod, returning currentUser() as the OAuth user's verified email.

What changes

Gating mode wire format. Forward mode keeps Authorization: Bearer <jwt> (Antalya token_processors re-validates). Gating mode now sends Authorization: Basic base64(email:JWT):

• MCP unverified-decodes the JWT's email claim (or the namespaced */email fallback) and sets Auth.Username = email, Auth.Password = JWT. The CH go-driver assembles the Basic header.
• MCP forces Protocol = HTTP under OAuth-enabled mode (TCP native has no <http_authentication> equivalent).
• MCP performs no cryptographic check per request — the sidecar is the gate.

New binary: cmd/ch-jwt-verify/ (sidecar).

• POST /verify with Authorization: Basic … → 200 + optional {"settings": …} on success, any non-200 on failure.
• Shares pkg/oauth/jwt.go + jwks.go for JWKS fetch/cache and signature verification.
• Enforces signature, iss, aud byte-equality (RFC 8707), exp/nbf/iat with clock skew, required scopes, identity policy (verified-email, domain allow-listing), and user-vs-email match (configurable lowercase_equal / exact).
• Maps OAuth scopes to ClickHouse session settings via settings_from_scope: config.
• In-process verification cache keyed by SHA256(JWT) with positive/negative TTLs.

Removed from MCP (~1751 LOC):

• cluster_secret + cluster_name (pkg/clickhouse/client.go, pkg/config/config.go, cmd/altinity-mcp/main.go)
• ClaimsToHeaders claim → header mapping + ClickHouseHeaderName custom-header option (pkg/oauth/forward.go, pkg/oauth/config.go)
• MCP-side identity policy: AllowedEmailDomains, AllowedHostedDomains, AllowUnverifiedEmail (moved to sidecar config)
• Per-request MCP-side JWT validation (pkg/server/server_client.go, cmd/altinity-mcp/oauth_server.go) — sidecar is the gate
• validateClusterSecretConfig startup check
• Files: pkg/oauth/identity.go, pkg/clickhouse/cluster_secret_test.go, pkg/server/oauth_gating_embedded_test.go, docs/oauth_next_refactor.md

Added (~489 LOC + new packages):

• cmd/ch-jwt-verify/ — main + verify + settings + config + tests
• pkg/oauth/policy.go — EmailDomain / ContainsDomain / HasRequiredScopes (extracted for sidecar reuse)
• helm/ch-jwt-verify/ — ConfigMap chart (sidecar YAML config + CH <http_authentication_servers> XML drop-in + reusable container fragment template). Deploys colocated in the CH StatefulSet pod (loopback trust model).
• Dockerfile.ch-jwt-verify + scripts/build-ch-jwt-verify-image.sh
• docs/ch-jwt-verify.md — full reference doc for the sidecar
• docs/oauth_authorization.md rewritten to be generic and current

Startup-validation tightening (cmd/altinity-mcp/main.go):

• OAuth-enabled now requires HTTP CH protocol unconditionally (both modes route through CH's HTTP interface).
• Startup ping skipped under any OAuth mode (creds are per-request).

Per-cluster migration plan

1. Deploy the sidecar — colocated container in the CH pod (production trust model, loopback only) OR standalone Deployment in the same namespace (test/shadow shape).
2. Push <http_authentication_servers> to CH config.d/.
3. For each OAuth user: CREATE USER OR REPLACE <email> ON CLUSTER ... IDENTIFIED WITH http SERVER 'ch_jwt_verify' SCHEME 'BASIC' DEFAULT ROLE ... (grammar token is http, not http_authenticator — CH rejects the latter with SYNTAX_ERROR).
4. Drop cluster_secret/cluster_name/CLICKHOUSE_CLUSTER_SECRET from helm values, switch protocol: http / port: 8123, bump image tag.
5. Helm upgrade, e2e-verify.

Test plan

• go test ./... — all packages pass (cmd/altinity-mcp, cmd/ch-jwt-verify, pkg/oauth, pkg/server, pkg/clickhouse, pkg/config)
• go vet ./... — clean
• Sidecar unit tests cover: signature verification, aud/exp/nbf rejection, lowercase-equal user-vs-email matching, verified-email enforcement, allowed-email-domains, required-scopes, scope→settings mapping, negative cache, missing/malformed Basic header
• MCP-side tests: per-request validation removed; gating mode requires JWT with email claim; non-JWT bearer rejected with clear error
• End-to-end in production: gating-mode MCP behind Auth0, sidecar colocated in CH pod, claude.ai → MCP → CH returns currentUser() as the OAuth user's verified email
• Helm chart helm/ch-jwt-verify/ lints cleanly (helm lint)

Caveats / follow-ups (not in this PR)

• clickhouse-operator quirk: when injecting the sidecar into a CHI podTemplate, the operator auto-injects volumeMount{name: default, mountPath: /var/lib/clickhouse} on every container; the actual data PVC volume is named with the volumeClaimTemplate name (e.g. default-1-1), so the injected mount fails validation. Workaround: declare an emptyDir volume named default in the podTemplate.
• CHIT (ClickHouseInstallationTemplate) for the sidecar. Direct kubectl patch chi/<name> edits are wiped by any operator-side reconcile push. A proper CHIT that the operator preserves on reconcile is the long-term fix.
• Per-cluster user migration is not automated. Each OAuth user must be re-CREATEd with the new IDENTIFIED WITH http clause before they can reach the upgraded MCP. Until that's done they cannot query through the new path.

Image artifacts

• ghcr.io/altinity/altinity-mcp:sidecar-<sha>-arm64 (public)
• ghcr.io/altinity/ch-jwt-verify:sidecar-<sha>-arm64 (private by default — flip to public via GitHub UI, or use an ImagePullSecret in the deploying namespace)

🤖 Generated with Claude Code