Why
#67056 decoupled remote logging from the hardcoded if/elif chain in
airflow-core/src/airflow/config_templates/airflow_local_settings.py. Core and the Task SDK now
share one resolver (airflow_shared.logging.factory.resolve_remote_task_log) that dispatches on the
URL scheme of [logging] remote_base_log_folder via ProvidersManager and instantiates the
provider class through a no-arg from_config() classmethod.
The legacy per-scheme chain in airflow_local_settings.py is transitional: it stays as a fallback
until every remote logging backend has been migrated to the new mechanism. This issue tracks that
migration, one sub-issue per backend.
The pattern
Each migration is a small, self-contained PR with three parts (see the merged amazon examples:
s3 — #69817, cloudwatch — #69816):
-
Add from_config() — a no-arg classmethod on the provider's RemoteLogIO class that mirrors
the corresponding legacy branch in airflow_local_settings.py, including the
[logging] remote_task_handler_kwargs IO-kwargs merge (where the legacy branch applies it) and
expanduser on base_log_folder, so behavior is unchanged for existing configs.
-
Register the scheme — add a remote-logging: section to the provider's provider.yaml and
mirror it in get_provider_info.py:
remote-logging:
- classpath: airflow.providers.amazon.aws.log.s3_task_handler.S3RemoteLogIO
scheme: s3
-
Tests — mirror TestS3RemoteLogIOFromConfig in
providers/amazon/tests/unit/amazon/aws/log/test_s3_task_handler.py: from_config field mapping,
IO-kwargs filtering, scheme registration, and end-to-end resolve_remote_task_log dispatch.
-
Real system test — set up the actual backend yourself (a real service or a local
equivalent), run a task with remote logging enabled, and verify logs are uploaded and read back
through the new dispatch path. Include the setup and verification results in the PR description.
If from_config() raises on a bad config, the shared factory falls back to the legacy path — so
these migrations are non-breaking by construction.
Backends
Follow-up once backends are migrated
How to contribute
Comment on a sub-issue to claim it — one PR per scheme, following the merged amazon examples.
The elasticsearch and opensearch sub-issues include a small design decision (those backends are
currently selected by host config rather than URL scheme); design feedback is welcome there.
Why
#67056 decoupled remote logging from the hardcoded if/elif chain in
airflow-core/src/airflow/config_templates/airflow_local_settings.py. Core and the Task SDK nowshare one resolver (
airflow_shared.logging.factory.resolve_remote_task_log) that dispatches on theURL scheme of
[logging] remote_base_log_folderviaProvidersManagerand instantiates theprovider class through a no-arg
from_config()classmethod.The legacy per-scheme chain in
airflow_local_settings.pyis transitional: it stays as a fallbackuntil every remote logging backend has been migrated to the new mechanism. This issue tracks that
migration, one sub-issue per backend.
The pattern
Each migration is a small, self-contained PR with three parts (see the merged amazon examples:
s3 — #69817, cloudwatch — #69816):
Add
from_config()— a no-arg classmethod on the provider'sRemoteLogIOclass that mirrorsthe corresponding legacy branch in
airflow_local_settings.py, including the[logging] remote_task_handler_kwargsIO-kwargs merge (where the legacy branch applies it) andexpanduseronbase_log_folder, so behavior is unchanged for existing configs.Register the scheme — add a
remote-logging:section to the provider'sprovider.yamlandmirror it in
get_provider_info.py:Tests — mirror
TestS3RemoteLogIOFromConfiginproviders/amazon/tests/unit/amazon/aws/log/test_s3_task_handler.py:from_configfield mapping,IO-kwargs filtering, scheme registration, and end-to-end
resolve_remote_task_logdispatch.Real system test — set up the actual backend yourself (a real service or a local
equivalent), run a task with remote logging enabled, and verify logs are uploaded and read back
through the new dispatch path. Include the setup and verification results in the PR description.
If
from_config()raises on a bad config, the shared factory falls back to the legacy path — sothese migrations are non-breaking by construction.
Backends
s3(amazon) — Add S3RemoteLogIO.from_config and register s3 remote logging scheme #69817cloudwatch(amazon) — Add CloudWatchRemoteLogIO.from_config and register cloudwatch scheme #69816gs(google) — Add GCSRemoteLogIO.from_config and register gs remote logging scheme #70266stackdriver(google) — Add StackdriverRemoteLogIO.from_config and register stackdriver scheme #70267wasb(microsoft.azure) — Add WasbRemoteLogIO.from_config and register wasb remote logging scheme #70268oss(alibaba) — Add OSSRemoteLogIO.from_config and register oss remote logging scheme #70269hdfs(apache.hdfs) — Add HdfsRemoteLogIO.from_config and register hdfs remote logging scheme #70270elasticsearch— Add ElasticsearchRemoteLogIO.from_config and register elasticsearch scheme #70271opensearch— Add OpensearchRemoteLogIO.from_config and register opensearch scheme #70272Follow-up once backends are migrated
legacy branches in
airflow_local_settings.py, gated on the provider versions that shipfrom_config(step 2 of the plan in Decouple remote logging config from core #67056).whichever the community prefers.
How to contribute
Comment on a sub-issue to claim it — one PR per scheme, following the merged amazon examples.
The
elasticsearchandopensearchsub-issues include a small design decision (those backends arecurrently selected by host config rather than URL scheme); design feedback is welcome there.