feat(k8s): migrate pipelines to prod EKS as a private service

[sumit-badsara]

Migrates the Open WebUI Pipelines filter service off Porter onto the self-managed prod EKS cluster (hub-and-spoke ArgoCD), mirroring the skillshield backend pattern.

Private by design

Pipelines is called only in-cluster by ai-gateway / ai-gateway-data at /v1/perform_filters, and it is an arbitrary-code plugin runtime — so it deploys PRIVATE: ClusterIP service, no ingress, no DNS.

What's here

• k8s/chart/ — Deployment + Service (ClusterIP:8080, targetPort http) + HPA (min1/max2, CPU/mem 75) + ExternalSecret (ESO prod/pipelines → <release>-secrets, injected via envFrom). Ingress template kept but gated enabled: false.
  • Stateless — no DB/redis/migrations, so no migrate Job (unlike skillshield).
  • Probes hit GET / (get_status → unauthenticated 200 {"status": true}); generous startup window for cold model load.
  • PIPELINES_API_KEY comes from Secrets Manager, never the app's insecure 0p3n-w3bu! default.
  • image: …/k8s/pipelines:prod, pullPolicy: Always; strategy: RollingUpdate maxSurge 1 maxUnavailable 0; revisionHistoryLimit: 3.
• k8s/prod/values.yaml — prod overlay (tag prod, ingress disabled, prod/pipelines, autoscaling min1/max2, env PORT/HOST/PIPELINES_DIR).
• .github/workflows/deploy-eks.yml — SHA-pinned actions. PR = build-only; push main = push :sha only; dispatch = advance :prod + rollout (fail-fast). Migrate step removed (stateless).
• .github/workflows/chart-validate.yml — helm lint + template on k8s/**.

Notable deltas from the existing chart

• ExternalSecret target renamed <release>-env → <release>-secrets (skillshield/envFrom convention).
• Dropped the securityContext (mirrors skillshield; the image runs as root and writes to PIPELINES_DIR at runtime — non-root uid 1000 would break pip installs / plugin writes).
• Dropped the unused configMap template and the hardcoded DD_AGENT_HOST env (neither in skillshield; pipelines ships no gateway plugin config).

Validation

• helm lint k8s/chart -f k8s/prod/values.yaml ✅
• helm template pipelines-prod k8s/chart -f k8s/prod/values.yaml → Deployment, Service (ClusterIP:8080), ExternalSecret, HPA (min1/max2); no Ingress ✅
• Workflows parse as valid YAML ✅

Companion unbound-infra PR wires ArgoCD + the repo secret + OIDC trust. The gateway keeps calling Porter pipelines until the later cutover.

🤖 Generated with Claude Code

---

Note

Medium Risk
Privileged CI (OIDC → ECR/EKS) and dispatch-only :prod tagging affect prod rollout; chart changes harden secrets and network exposure but the workload remains a root plugin runtime executing arbitrary code.

Overview
Adds prod EKS packaging for Pipelines (private ClusterIP, ArgoCD-managed manifests) plus CI that separates image build from prod rollout.

Helm chart is reshaped for the skillshield-style deploy: drops ConfigMap/DD_AGENT_HOST, renames ExternalSecret output to <release>-secrets, switches ESO from dataFrom extract to pinned secretKeys (so missing PIPELINES_API_KEY fails sync instead of falling back to the app default), moves non-secret config into env, and uses values-driven seccomp/cap-drop hardening without runAsNonRoot. Adds k8s/prod/values.yaml (tag :prod, prod/pipelines, ingress off, HPA 1–2); dev overlay scales to 0 with HPA off.

.github/workflows/deploy-eks.yml builds/pushes immutable :sha on main; :prod advances only on workflow_dispatch (then rollout restart with 15m fail-fast). .github/workflows/chart-validate.yml runs helm lint + helm template on k8s/** so chart-only changes still get CI. k8s/README.md documents private in-cluster cutover prerequisites.

^{Reviewed by Cursor Bugbot for commit 0fed160. Bugbot is set up for automated code reviews on this repo. Configure here.}

Greptile Summary

This PR moves the Pipelines service onto the private prod EKS path. The main changes are:

• Adds Helm chart validation for Kubernetes changes.
• Adds an EKS build and deploy workflow with ECR image promotion.
• Updates the Helm chart for a private ClusterIP service with ExternalSecret-backed configuration.
• Adds prod and dev values for the new deployment shape.
• Documents EKS prerequisites, soak checks, and gateway cutover.

Confidence Score: 5/5

This looks safe to merge.

• No blocking issues found in the changed code.

Important Files Changed

Filename	Overview
.github/workflows/chart-validate.yml	Adds Helm lint and template checks for Kubernetes chart changes.
.github/workflows/deploy-eks.yml	Adds the EKS image build, ECR promotion, and rollout workflow.
k8s/chart/templates/deployment.yaml	Updates the Deployment for secret injection, image validation, probes, and container hardening.
k8s/chart/templates/external-secret.yaml	Renames the generated secret and pins the required secret property.
k8s/chart/values.yaml	Sets the base chart defaults for image, ExternalSecret, env, autoscaling, probes, and hardening.
k8s/prod/values.yaml	Adds the production values for the private Pipelines service.
k8s/dev/values.yaml	Keeps dev scaled down while configuring the dev secret source.

_{Reviews (3): Last reviewed commit: "fix(chart,ci): round 2 review — enforce ..." | Re-trigger Greptile}

Context used:

• Rule used - Ensure that the confidence score is always within ... (source)

Learned From
websentry-ai/ai-gateway-data#448

[cursor]

Cursor Bugbot has reviewed your changes using high effort and found 2 potential issues.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit ee59818. Configure here.}

[cursor]

Stale comment