deploy: Helm chart for the api and engine roles (#61, #62)

[richardmhope]

Stacked on #102 (which is stacked on #101). Review the last commit.

deploy/helm/icebergsst/ — api Deployment/Service/Ingress, engine Deployment + HPA, a
pre-upgrade migration Job, a ConfigMap, two Secrets, two ServiceAccounts, and an
optional NetworkPolicy pair.

Two Secrets, not one

This is the shape of the whole chart. The api's Secret holds the database URL, master key, session
secret and OIDC client secret. The engine's holds a broker URL and its own token — no database
credential, no master key, because there is nothing an engine could correctly do with either.

The engine Deployment does not reference the api's Secret at all, so ADR 0002 is enforced by what
is mounted rather than by everyone remembering. Separate ServiceAccounts mean a cluster RBAC
policy can express the same boundary.

Production should create both out of band and set secrets.existingApiSecret /
existingEngineSecret — values passed to Helm end up in the release's stored manifest, which is
not where a master key belongs. The inline values exist so a first install works; the docs say
plainly that they are not the production path.

Other decisions worth a look

• Postgres and Redis are not in the chart. A bundled database is a development convenience that
  becomes a production liability: no backups, no failover, and a helm uninstall that takes the
  findings with it.
• Migrations are a pre-install,pre-upgrade hook running the api image and
  python -m iceberg_api migrate — the same entry point compose and the migration tests use, so
  production runs what has been exercised. hook-delete-policy leaves a failed Job behind for
  its logs.
• Engine autoscaling on, api autoscaling off. Engines hold leases rather than state and a
  lapsed lease is reclaimed (ADR 0009); the api's useful replica count is availability, not CPU.
  The HPA has a 5-minute scale-down window so the quiet moment after a scheduled sweep does not
  kill engines mid-lease and cause re-done work.
• Restricted Pod Security Standard everywhere: non-root uid 10001, no privilege escalation,
  read-only root filesystem, all capabilities dropped, RuntimeDefault seccomp, resource requests.
• NOTES.txt warns about the two settings that are wrong by default for a real cluster —
  trustedProxyHops at 0 behind an ingress, and secrets supplied via values.

Verification

deploy/helm/verify-chart.sh (make helm-verify, plus a CI job) lints, renders with
example-values.yaml, renders again against externally-managed Secrets, then asserts twelve
properties of the rendered manifests that helm lint cannot see — the engine mounting only its
own Secret and carrying no DB config, migrations as a pre-upgrade hook on the api image, nothing
migrating on start, every workload hardened with requests set, the engine having an HPA and no
Service. The cheap textual half runs in the ordinary suite.

Run against helm 4.2.3: lint clean, both renders succeed, all twelve checks pass.

helm lint                    ok
helm template (example)      ok
helm template (ext secrets)  ok
engine: no db config         ok
engine: own Secret only      ok
migrations: pre-upgrade hook ok
migrations: packaged entry   ok
migrations: api image        ok
workloads hardened           ok
engine HPA                   ok
engine has no Service        ok

check-yaml now skips the chart templates: Go templates only become YAML once rendered, which is
exactly what helm-verify checks.

Closes #61
Closes #62

🤖 Generated with Claude Code