From 225947271961dca373b3383f41c8019c563b8996 Mon Sep 17 00:00:00 2001 From: Aaryan Mahajan Date: Wed, 22 Jul 2026 18:43:51 +0400 Subject: [PATCH 1/3] Document how to rotate the Fernet key with the Helm chart The Helm chart docs explained how to point at a custom Fernet key Secret but never how to safely rotate one. The chart also generates the Fernet key Secret via a pre-install hook, which means it cannot be updated through helm upgrade, so a self-managed Secret plus the standard prepend/rotate/trim procedure is required. Document that gap so operators aren't left guessing or reverse-engineering the chart templates. closes: #18509 --- chart/docs/production-guide.rst | 53 +++++++++++++++++++++++++++++++ chart/newsfragments/18509.doc.rst | 1 + 2 files changed, 54 insertions(+) create mode 100644 chart/newsfragments/18509.doc.rst diff --git a/chart/docs/production-guide.rst b/chart/docs/production-guide.rst index 077fd0a2b973b..ec06d893e4b11 100644 --- a/chart/docs/production-guide.rst +++ b/chart/docs/production-guide.rst @@ -260,6 +260,59 @@ generated using the secret key has a short expiry time though. Make sure that ti that you run Airflow components on is synchronized (for example using ntpd). You might get "forbidden" errors when the logs are accessed otherwise. +Fernet Key +---------- + +Airflow uses a Fernet key to encrypt sensitive data, such as connection passwords, stored in the metadata database. +See :doc:`Fernet ` for background on how this works. If you do not provide a +key, the chart generates one and stores it in the ``-fernet-key`` Kubernetes Secret the first time you +run ``helm install``. + +.. warning:: + + The chart only creates that Secret on ``helm install`` -- it is not re-created or otherwise updated by + ``helm upgrade``. This applies both to the auto-generated key and to a key you pass through ``fernetKey`` in the + values file, so neither can be changed by re-running ``helm upgrade``. To rotate the Fernet key you need to manage + the Secret yourself, as described below. + +To provide your own key, either set ``fernetKey`` in the values file: + +.. code-block:: yaml + :caption: values.yaml + + fernetKey: + +.. warning:: + + Due to security concerns, it is advised to use a Kubernetes Secret instead of setting the Fernet key directly in the values file. + +or create your own Kubernetes Secret containing a ``fernet-key`` key with a base64-encoded value, and point the chart at it with ``fernetKeySecretName``: + +.. code-block:: bash + + kubectl create secret generic my-fernet-key --from-literal="fernet-key=" + +.. code-block:: yaml + :caption: values.yaml + + fernetKeySecretName: my-fernet-key + +Rotating the Fernet key +^^^^^^^^^^^^^^^^^^^^^^^^ + +Once connections, variables and triggers have been encrypted with a Fernet key, changing the key outright makes the +existing encrypted values unreadable. To rotate the key without losing access to them, follow the same procedure as +:doc:`Rotating encryption keys `, applied to the Kubernetes Secret that backs +``AIRFLOW__CORE__FERNET_KEY``: + +#. Make sure you are using a self-managed Secret through ``fernetKeySecretName`` -- as noted above, the + chart-generated Secret cannot be updated through ``helm upgrade``. +#. Update the ``fernet-key`` value in that Secret to a comma-separated list ``new_fernet_key,old_fernet_key``, then + restart the Airflow components (for example with ``kubectl rollout restart``) so they pick up the new value -- + Kubernetes does not refresh environment variables sourced from a Secret in already-running pods. +#. Run ``airflow rotate-fernet-key`` to re-encrypt existing connections, variables and triggers with the new key. +#. Update the Secret again so it only contains ``new_fernet_key``, and restart the components once more. + JWT Secret ---------- diff --git a/chart/newsfragments/18509.doc.rst b/chart/newsfragments/18509.doc.rst new file mode 100644 index 0000000000000..e29633397fa68 --- /dev/null +++ b/chart/newsfragments/18509.doc.rst @@ -0,0 +1 @@ +Document how to rotate the Fernet key when deploying with the Helm chart. From d78e7dd2b0788defe60d471efd5ea13939dcfb0c Mon Sep 17 00:00:00 2001 From: Aaryan Mahajan Date: Wed, 22 Jul 2026 21:40:54 +0400 Subject: [PATCH 2/3] Address review feedback on Fernet key rotation docs Reviewers pointed out three gaps: the plaintext-key warning didn't explicitly say not to commit values.yaml to version control, the kubectl-created Secret example didn't mention that it needs to be managed as infrastructure so it survives a cluster migration or redeploy, and the rotation steps described updating the Secret's value without ever showing the command to do it. Rename the newsfragment to the PR number (70242) instead of the issue number (18509) per towncrier convention. --- chart/docs/production-guide.rst | 31 ++++++++++++++++--- .../{18509.doc.rst => 70242.doc.rst} | 0 2 files changed, 27 insertions(+), 4 deletions(-) rename chart/newsfragments/{18509.doc.rst => 70242.doc.rst} (100%) diff --git a/chart/docs/production-guide.rst b/chart/docs/production-guide.rst index ec06d893e4b11..e9c74ea5e6c75 100644 --- a/chart/docs/production-guide.rst +++ b/chart/docs/production-guide.rst @@ -284,7 +284,8 @@ To provide your own key, either set ``fernetKey`` in the values file: .. warning:: - Due to security concerns, it is advised to use a Kubernetes Secret instead of setting the Fernet key directly in the values file. + Due to security concerns, it is advised to use a Kubernetes Secret instead of setting the Fernet key directly in + the values file. Never commit a ``values.yaml`` containing a plaintext Fernet key to version control. or create your own Kubernetes Secret containing a ``fernet-key`` key with a base64-encoded value, and point the chart at it with ``fernetKeySecretName``: @@ -297,6 +298,12 @@ or create your own Kubernetes Secret containing a ``fernet-key`` key with a base fernetKeySecretName: my-fernet-key +.. note:: + + ``kubectl create secret`` only creates an object in the cluster -- it is not tracked anywhere else. Manage this + Secret through your normal Infrastructure-as-Code process (for example a GitOps repository, a Sealed Secret, or + an External Secrets Operator resource) so it is not lost on a cluster migration or a full redeploy. + Rotating the Fernet key ^^^^^^^^^^^^^^^^^^^^^^^^ @@ -307,11 +314,27 @@ existing encrypted values unreadable. To rotate the key without losing access to #. Make sure you are using a self-managed Secret through ``fernetKeySecretName`` -- as noted above, the chart-generated Secret cannot be updated through ``helm upgrade``. -#. Update the ``fernet-key`` value in that Secret to a comma-separated list ``new_fernet_key,old_fernet_key``, then - restart the Airflow components (for example with ``kubectl rollout restart``) so they pick up the new value -- +#. Update the ``fernet-key`` value in that Secret to a comma-separated list ``new_fernet_key,old_fernet_key``, using + the same ``kubectl create secret`` command as above with ``--dry-run=client -o yaml`` piped into ``kubectl apply`` + so it updates the existing Secret in place instead of failing because it already exists: + + .. code-block:: bash + + kubectl create secret generic my-fernet-key \ + --from-literal="fernet-key=new_fernet_key,old_fernet_key" \ + --dry-run=client -o yaml | kubectl apply -f - + + Then restart the Airflow components (for example with ``kubectl rollout restart``) so they pick up the new value -- Kubernetes does not refresh environment variables sourced from a Secret in already-running pods. #. Run ``airflow rotate-fernet-key`` to re-encrypt existing connections, variables and triggers with the new key. -#. Update the Secret again so it only contains ``new_fernet_key``, and restart the components once more. +#. Update the Secret again, the same way, so it only contains ``new_fernet_key``, and restart the components once + more: + + .. code-block:: bash + + kubectl create secret generic my-fernet-key \ + --from-literal="fernet-key=new_fernet_key" \ + --dry-run=client -o yaml | kubectl apply -f - JWT Secret ---------- diff --git a/chart/newsfragments/18509.doc.rst b/chart/newsfragments/70242.doc.rst similarity index 100% rename from chart/newsfragments/18509.doc.rst rename to chart/newsfragments/70242.doc.rst From f3a3cf5247775e136089cdb3cf16ecc724205ad4 Mon Sep 17 00:00:00 2001 From: Aaryan Mahajan Date: Fri, 24 Jul 2026 12:43:25 +0400 Subject: [PATCH 3/3] Address review feedback on Fernet key rotation docs Tighten the wording per reviewer suggestions and drop the newsfragment, since a docs-only addition does not need one. --- chart/docs/production-guide.rst | 17 ++++++----------- chart/newsfragments/70242.doc.rst | 1 - 2 files changed, 6 insertions(+), 12 deletions(-) delete mode 100644 chart/newsfragments/70242.doc.rst diff --git a/chart/docs/production-guide.rst b/chart/docs/production-guide.rst index e9c74ea5e6c75..5d6695fd44c6b 100644 --- a/chart/docs/production-guide.rst +++ b/chart/docs/production-guide.rst @@ -271,9 +271,7 @@ run ``helm install``. .. warning:: The chart only creates that Secret on ``helm install`` -- it is not re-created or otherwise updated by - ``helm upgrade``. This applies both to the auto-generated key and to a key you pass through ``fernetKey`` in the - values file, so neither can be changed by re-running ``helm upgrade``. To rotate the Fernet key you need to manage - the Secret yourself, as described below. + ``helm upgrade``. This applies both to the auto-generated key and to a key you pass through ``fernetKey`` in the values file. To rotate the Fernet key, you need to manage the Secret yourself, as described below. To provide your own key, either set ``fernetKey`` in the values file: @@ -301,7 +299,7 @@ or create your own Kubernetes Secret containing a ``fernet-key`` key with a base .. note:: ``kubectl create secret`` only creates an object in the cluster -- it is not tracked anywhere else. Manage this - Secret through your normal Infrastructure-as-Code process (for example a GitOps repository, a Sealed Secret, or + Secret through your Infrastructure-as-a-Code process (for example, a GitOps repository, a Sealed Secret, or an External Secrets Operator resource) so it is not lost on a cluster migration or a full redeploy. Rotating the Fernet key @@ -312,8 +310,7 @@ existing encrypted values unreadable. To rotate the key without losing access to :doc:`Rotating encryption keys `, applied to the Kubernetes Secret that backs ``AIRFLOW__CORE__FERNET_KEY``: -#. Make sure you are using a self-managed Secret through ``fernetKeySecretName`` -- as noted above, the - chart-generated Secret cannot be updated through ``helm upgrade``. +#. Make sure you are using a self-managed Secret through ``fernetKeySecretName`` #. Update the ``fernet-key`` value in that Secret to a comma-separated list ``new_fernet_key,old_fernet_key``, using the same ``kubectl create secret`` command as above with ``--dry-run=client -o yaml`` piped into ``kubectl apply`` so it updates the existing Secret in place instead of failing because it already exists: @@ -324,11 +321,9 @@ existing encrypted values unreadable. To rotate the key without losing access to --from-literal="fernet-key=new_fernet_key,old_fernet_key" \ --dry-run=client -o yaml | kubectl apply -f - - Then restart the Airflow components (for example with ``kubectl rollout restart``) so they pick up the new value -- - Kubernetes does not refresh environment variables sourced from a Secret in already-running pods. -#. Run ``airflow rotate-fernet-key`` to re-encrypt existing connections, variables and triggers with the new key. -#. Update the Secret again, the same way, so it only contains ``new_fernet_key``, and restart the components once - more: + Then restart the Airflow components (for example with ``kubectl rollout restart``) so they pick up the new value +#. Run ``airflow rotate-fernet-key``, in one of the Airflow components pods, to re-encrypt existing connections, variables and triggers with the new key. +#. Update the Secret, so it only contains ``new_fernet_key``, and restart the components once more: .. code-block:: bash diff --git a/chart/newsfragments/70242.doc.rst b/chart/newsfragments/70242.doc.rst deleted file mode 100644 index e29633397fa68..0000000000000 --- a/chart/newsfragments/70242.doc.rst +++ /dev/null @@ -1 +0,0 @@ -Document how to rotate the Fernet key when deploying with the Helm chart.