[Task]:Upgrade SPS to use Apache Airflow 3.0

[LucaCinquini]

[LucaCinquini]

See official upgrade documentation: https://airflow.apache.org/docs/apache-airflow/stable/installation/upgrading_to_airflow3.html

But unfortunately it looks like there isn't yet a Helm chart that supports it? See https://airflow.apache.org/docs/helm-chart/stable/index.html

[LucaCinquini]

This upgrade guide is also well written and might be a good starting point: https://www.astronomer.io/docs/learn/airflow-upgrade-2-3/

[nikki-t]

Notes from documentation

No Direct Database Access: Task code can no longer directly import and use Airflow database sessions or models.

• Does any of the Unity-Py code do this? I don’t think it does but may be worth checking.

1.Pre-requisites:

1. Airflow 2.7 or later
2. Python version 3.9-3.12
3. Review breaking changes

Breaking changes: https://airflow.apache.org/docs/apache-airflow/stable/installation/upgrading_to_airflow3.html#breaking-changes

• Simple Auth is now default auth_manager. To continue using FAB as the Auth Manager, please install the FAB provider and set auth_manager to FabAuthManager
• Not quite sure how we want to support Cognito but with the convergence work Cognito may not be necessary.

2.Clean and back up existing Airflow instance

1. Back up the Airflow database instance. I think we do this with the database snapshots?
2. Clean the database with: airflow db clean. Not sure how important this is for us to do since I think we regularly deploy a whole new install of the Airflow components across AWS accounts.
3. Make sure there are not errors related to DAG processing: airflow dags reserialize

3.Check Airflow DAGs for compatibility

1. Check compatibility with ruff (>=0.11.6): ruff check dag/ --select AIR301 --fix --preview
2. Can we add this to our pre-commit?

4.Install the standard providers

1. Common Operators (BashOperator and PythonOperator) have been split to a separate package: apache-airflow-providers-standard
2. This can also be installed on Airflow 2.x for testing.

5.Review custom operators for db direct access

1. I don't think that has any impact on us.

6.Deployment Managers – Upgrade your Airflow Instance

1. Run configuration check: airflow config update
   a. airflow config update --fix
2. Migrate the database: airflow db migrate

7.Change your start up scripts

1. Run the Webserver: airflow api-server
2. Run the DAG processor: airflow dag-processor
3. I am not sure how necessary this will be for our deployments.

8.Helm chart for Airflow 3

1. It does look like the most recent release of the Helm supports Airflow 3.
2. Review the Helm chart and the results of running a check on the configuration file to see what settings might need to be changed.
3. Could also try deploying our template file with the new Airflow 3 to narrow down issues.

Steps to upgrade

1. Check Airflow and Python versions, review breaking changes.
2. Retrieve existing config file from a running deployment and run a configuration check to track what settings might need to be changed and how. I think the Helm chart will deal with actual updated settings so no need to update anything quite yet.
   a. airflow config lint
3. Run the ruff command to check for existing DAG combability and plan required updates to DAGs.
4. Understand required Helm chart changes making sure to install newly required providers.
5. Back up and clean the database.
6. Upgrade Airflow via Helm deployment.

[LucaCinquini]

@nikki-t : thanks for the analysis. I would suggest to try the latest Helm Chart first with the current Airflow 2.X since we will need a working Chart later to test the upgrade.