Skip to content

Add StackdriverRemoteLogIO.from_config and register stackdriver scheme #70267

Description

@jason810496

Part of #70265 (related: #67056).

Why

#67056 decoupled remote logging from the hardcoded branches in airflow_local_settings.py: core and
the Task SDK now resolve the handler via ProvidersManager dispatch on the
[logging] remote_base_log_folder URL scheme, instantiating the provider class through a no-arg
from_config() classmethod. This issue migrates the stackdriver scheme.

What

  • Add StackdriverRemoteLogIO.from_config() in
    providers/google/src/airflow/providers/google/cloud/log/stackdriver_task_handler.py,
    mirroring the legacy branch:
    elif remote_base_log_folder.startswith("stackdriver://"):
    from airflow.providers.google.cloud.log.stackdriver_task_handler import StackdriverRemoteLogIO
    key_path = conf.get_mandatory_value("logging", "GOOGLE_KEY_PATH", fallback=None)
    # stackdriver:///airflow-tasks => airflow-tasks
    log_name = urlsplit(remote_base_log_folder).path[1:]
    REMOTE_TASK_LOG = StackdriverRemoteLogIO(
    **cast(
    "dict[str, Any]",
    {
    "base_log_folder": BASE_LOG_FOLDER,
    "gcp_log_name": log_name,
    "gcp_key_path": key_path,
    "delete_local_copy": delete_local_copy,
    }
    | _io_kwargs,
    )
    )

    — including the [logging] remote_task_handler_kwargs IO-kwargs merge and expanduser on
    base_log_folder, so behavior is unchanged for existing configs.
  • Register the stackdriver scheme under a remote-logging: section in
    providers/google/provider.yaml and mirror it in
    providers/google/src/airflow/providers/google/get_provider_info.py.
  • Add tests mirroring TestS3RemoteLogIOFromConfig in
    providers/amazon/tests/unit/amazon/aws/log/test_s3_task_handler.py.
  • Verify end to end with a real system test: set up the actual backend yourself (a real
    service or a local equivalent), run a task with remote logging enabled, and confirm logs are
    uploaded and read back through the new dispatch path. Include the setup and verification
    results in the PR description.

Notes

  • The legacy branch parses the log name from the URL path: stackdriver:///airflow-tasks
    gcp_log_name="airflow-tasks" (urlsplit(...).path[1:]). Mirror this in from_config, with a
    clear ValueError when no log name can be derived (see how Add CloudWatchRemoteLogIO.from_config and register cloudwatch scheme #69816 handles log_group_arn
    parsing for cloudwatch).
  • It also reads [logging] google_key_path (fallback None) into gcp_key_path, and passes no
    remote_base — unlike the other object-storage backends.

Reference

Merged examples to follow: #69817 (s3), #69816 (cloudwatch). If from_config raises on a bad
config, the shared factory falls back to the legacy path, so this is not a breaking change.

Metadata

Metadata

Assignees

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions