This project demonstrates two patterns for orchestrating secure file transfers in enterprise environments where direct access to on-premises systems is restricted. It is designed as a reference implementation for teams migrating from legacy job schedulers (e.g., Tidal) to Apache Airflow on Astronomer.
1. SFTP via DMZ Jump Host (dags/sftp_via_dmz_ssh.py)
Transfers files through a Windows DMZ jump host using SSH tunneling.
Workflow:
- Check for matching files at the source path
- Transfer files via SFTP batch through the DMZ host (PowerShell + OpenSSH)
- Archive transferred files
When to use: Your organization already has a DMZ with SSH access and wants to minimize new cloud dependencies.
2. SFTP via Azure Data Factory (dags/sftp_via_adf.py)
Triggers an Azure Data Factory pipeline to handle the file transfer, replacing a legacy on-premises job. The task is deferrable, so it releases the Airflow worker slot while ADF runs.
When to use: Your organization is cloud-first or already uses ADF for data movement, and wants to eliminate on-premises job scheduler dependencies entirely.