Create Airflow DAG to Run Transform Pipeline Using a Dataset Parameter

[lakshmi-kovvuri1]

Overview
We need an Airflow DAG that can run the transform pipeline for any dataset by passing the dataset name as a parameter before the DAG starts. This removes the need for separate DAGs per dataset and makes the transform step more flexible and easier to operate. The DAG should accept the dataset name, pass it to the transform task, and follow our existing Airflow patterns.
As per our discussion with Owen on 16-feb-2026

Tech Approach

• Add a new Airflow DAG with a dataset name defined as an Airflow Param.
• Use a small “configure” task to push the dataset parameter into XCom.
• Update the ECS overrides so the container receives DATASET_NAME through environment variables.
• Reuse existing patterns from collection DAGs (e.g., configure-dag, EcsRunTaskOperator).
• Document how to trigger the DAG and which dataset names are supported.

Acceptance Criteria / Tests

• DAG appears in Airflow UI and allows users to select a dataset name before execution.
• The transform step runs successfully for different dataset values.
• ECS task receives the correct dataset name via environment variables.
• XCom values for parameters are visible and correct.
• Tests confirm the DAG works in dev with multiple datasets without code changes.

Resourcing & Dependencies
No prerequisite tickets.
Any team member familiar with Airflow and ECS tasks can complete it.
Only internal QA needed before deploying to higher environments.

[mattsan-dev]

Although we have constructed this DAG as a workaround for the local development issues, the logging currently available within the DAG does not provide sufficient insight into the existing transformation logic. This may be the result of the transformation steps lacking appropriate print statements. We intend to validate this as part of testing our newly refactored phases.