Chart: Helm Refurbish

[bugraoz93]

Body

As part of our ongoing Helm chart refurbish work in Apache Airflow, we will follow a structured approach to ensure stability for existing users while preparing for a clean, well-defined next major release.

https://cwiki.apache.org/confluence/display/AIRFLOW/Helm+Refurbish
https://cwiki.apache.org/confluence/display/AIRFLOW/Direction+and+Plan+for+Chart+2.0?src=contextnavpagetreemode

Single tracker for the Helm chart 2.0 refurbishment. Each section below is a category of work; the items inside are the concrete action items. Tick as they land. Larger items may spawn dedicated PRs and issues — link them from the relevant action item rather than removing the checkbox.

Contributor guide: Developing the Helm Chart (coming in #66659). Canonical chart-vs-Kustomize criteria: chart/kustomize-overlays/CONTRIBUTING.rst. The chart never removes a component without a working overlay already in place.

---

Deprecate and remove legacy keys

Deprecation warning ships on 1.2x for at least one minor before removal on main.

• Deprecate apiSecretKeySecretName on 1.2x; remove in 2.0
• Deprecate apiServer.apiServerConfig on 1.2x; remove in 2.0
• Remove misplaced ingress keys: registry.secretNames, securityContext, ingress.apiServer.host, ingress.apiServer.tls.enabled, ingress.apiServer.tls.secretName, ingress.enabled
• Stage all webserver-related deprecations on main
• Audit remaining Airflow-below-3.0 compatibility keys; deprecate then remove

Relocate parameters to their owning sections

No functional change — parameters work today but live under the wrong parent. Canonical name changes.

• Move Redis parameters under the celery section
• Move Jobs parameters under the kubernetes section
• Consolidate data.celery.* under a celery-specific heading
• Align podDisruptionBudget shape with pod_template_file (Kubernetes executor pattern)
• Audit remaining parameters under kubernetes.* that belong elsewhere; relocate

Restructure persistence model around bundles

Persistence becomes a bundle configuration. Log and Dag folders split into two distinct types: Log and Bundles.

• Split persistence configuration into Log and Bundles types
• Implement single DagBundle per deployment
• Support multiple DagProcessor instances per bundle
• Design multi-team support against the bundle shape (not retrofitted onto the current model)

Migrate components to Kustomize overlays

Components delivered as overlays rather than chart-native templates. Each item may spawn its own PR/issue.

• Kerberos overlay (sidecar-injection pattern)
• Document GitDagBundle migration; remove gitSync from chart afterwards (overlay only if community demand persists)
• Elasticsearch / OpenSearch overlay
• PostgreSQL StatefulSet overlay (chart stops shipping a database)
• StatsD overlay (OpenTelemetry default in chart)
• HPA / KEDA overlay (standalone-addition pattern)
• General telemetry / monitoring overlays beyond the OpenTelemetry default

Decouple airflow.cfg from the chart

All configuration expressed through environment variables. Chart keeps a minimal cfg surface; complex per-component config moves to overlays.

• Express all chart configuration via environment variables
• Reduce chart cfg surface to basics
• Move complex per-component configuration to overlays
• Investigate per-component cfg (separate cfg per Airflow component) for the major bump

Consolidate duplicate configuration paths

A given setting is reachable from one section. A given image is defined in one place.

• Normalise Images section so each image is defined in one place rather than two
• Reshuffle image-related parameters into the components they belong to
• Audit remaining settings reachable from multiple sections; consolidate

Tighten chart defaults to a least-privilege baseline

• Enforce security context at the component and container level
• Apply least-privilege defaults consistently to roles and role bindings
• Provide sensible serviceAccount defaults; allow finer shaping via overlays
• Readiness / liveness probes cover the common case (~80%) in the chart; remaining shapes via overlays

Open scoping investigations

Spikes that inform later decisions and do not directly produce chart changes. Outcomes feed back into the relevant section above.

• PgBouncer for Airflow v3 and higher: load test or equivalent verification to decide whether it still adds value
• Migration image and waitForMigrations: revisit against newer Kubernetes features to decide whether the two-image arrangement and explicit wait step are still required
• Multi-image solution placement (chart vs overlay): pending component-isolation work

Committer

• I acknowledge that I am a maintainer/committer of the Apache Airflow project.