Skip to content

Provider migration for decoupling remote logging from core #70265

Description

@jason810496

Why

#67056 decoupled remote logging from the hardcoded if/elif chain in
airflow-core/src/airflow/config_templates/airflow_local_settings.py. Core and the Task SDK now
share one resolver (airflow_shared.logging.factory.resolve_remote_task_log) that dispatches on the
URL scheme of [logging] remote_base_log_folder via ProvidersManager and instantiates the
provider class through a no-arg from_config() classmethod.

The legacy per-scheme chain in airflow_local_settings.py is transitional: it stays as a fallback
until every remote logging backend has been migrated to the new mechanism. This issue tracks that
migration, one sub-issue per backend.

The pattern

Each migration is a small, self-contained PR with three parts (see the merged amazon examples:
s3 — #69817, cloudwatch — #69816):

  1. Add from_config() — a no-arg classmethod on the provider's RemoteLogIO class that mirrors
    the corresponding legacy branch in airflow_local_settings.py, including the
    [logging] remote_task_handler_kwargs IO-kwargs merge (where the legacy branch applies it) and
    expanduser on base_log_folder, so behavior is unchanged for existing configs.

  2. Register the scheme — add a remote-logging: section to the provider's provider.yaml and
    mirror it in get_provider_info.py:

    remote-logging:
      - classpath: airflow.providers.amazon.aws.log.s3_task_handler.S3RemoteLogIO
        scheme: s3
  3. Tests — mirror TestS3RemoteLogIOFromConfig in
    providers/amazon/tests/unit/amazon/aws/log/test_s3_task_handler.py: from_config field mapping,
    IO-kwargs filtering, scheme registration, and end-to-end resolve_remote_task_log dispatch.

  4. Real system test — set up the actual backend yourself (a real service or a local
    equivalent), run a task with remote logging enabled, and verify logs are uploaded and read back
    through the new dispatch path. Include the setup and verification results in the PR description.

If from_config() raises on a bad config, the shared factory falls back to the legacy path — so
these migrations are non-breaking by construction.

Backends

Follow-up once backends are migrated

  • Once all provider migrations are merged and released, add a deprecation warning to the
    legacy branches in airflow_local_settings.py, gated on the provider versions that ship
    from_config (step 2 of the plan in Decouple remote logging config from core #67056).
  • Remove the hardcoded remote logging config from core in a future minor or major release,
    whichever the community prefers.

How to contribute

Comment on a sub-issue to claim it — one PR per scheme, following the merged amazon examples.
The elasticsearch and opensearch sub-issues include a small design decision (those backends are
currently selected by host config rather than URL scheme); design feedback is welcome there.

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions