Skip to content

Add WasbRemoteLogIO.from_config and register wasb remote logging scheme #70268

Description

@jason810496

Part of #70265 (related: #67056).

Why

#67056 decoupled remote logging from the hardcoded branches in airflow_local_settings.py: core and
the Task SDK now resolve the handler via ProvidersManager dispatch on the
[logging] remote_base_log_folder URL scheme, instantiating the provider class through a no-arg
from_config() classmethod. This issue migrates the wasb scheme.

What

  • Add WasbRemoteLogIO.from_config() in
    providers/microsoft/azure/src/airflow/providers/microsoft/azure/log/wasb_task_handler.py,
    mirroring the legacy branch:
    elif remote_base_log_folder.startswith("wasb"):
    from airflow.providers.microsoft.azure.log.wasb_task_handler import WasbRemoteLogIO
    _default_conn_name_from("airflow.providers.microsoft.azure.hooks.wasb", "WasbHook")
    wasb_log_container = conf.get_mandatory_value(
    "azure_remote_logging", "remote_wasb_log_container", fallback="airflow-logs"
    )
    # Handle both URI format (wasb://logs) and plain path (e.g., wasb-logs)
    wasb_remote_base = remote_base_log_folder.removeprefix("wasb://")
    REMOTE_TASK_LOG = WasbRemoteLogIO(
    **cast(
    "dict[str, Any]",
    {
    "base_log_folder": BASE_LOG_FOLDER,
    "remote_base": wasb_remote_base,
    "delete_local_copy": delete_local_copy,
    "wasb_container": wasb_log_container,
    }
    | _io_kwargs,
    )
    )

    — including the [logging] remote_task_handler_kwargs IO-kwargs merge, expanduser on
    base_log_folder, reading [azure_remote_logging] remote_wasb_log_container (fallback
    airflow-logs) into wasb_container, and stripping the wasb:// prefix from
    remote_base_log_folder, so behavior is unchanged for existing configs.
  • Register the wasb scheme under a remote-logging: section in
    providers/microsoft/azure/provider.yaml and mirror it in
    providers/microsoft/azure/src/airflow/providers/microsoft/azure/get_provider_info.py.
  • Add tests mirroring TestS3RemoteLogIOFromConfig in
    providers/amazon/tests/unit/amazon/aws/log/test_s3_task_handler.py.
  • Verify end to end with a real system test: set up the actual backend yourself (a real
    service or a local equivalent), run a task with remote logging enabled, and confirm logs are
    uploaded and read back through the new dispatch path. Include the setup and verification
    results in the PR description.

Notes

The legacy branch matches any value starting with wasb — including non-URI values like
wasb-logs. Scheme dispatch only fires for proper wasb://... URIs (a value without :// has no
URL scheme), so non-URI values keep resolving through the legacy fallback until the chain is
removed. The wasb:// URI form should be documented as the supported form going forward; migrating
non-URI configs can be handled in the deprecation step tracked by the meta issue.

Reference

Merged examples to follow: #69817 (s3), #69816 (cloudwatch). If from_config raises on a bad
config, the shared factory falls back to the legacy path, so this is not a breaking change.

The legacy branch derives a default connection id from WasbHook.default_conn_name; handle the
connection id the same way the merged amazon PRs do (the hook reads
[logging] remote_log_conn_id itself).

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions