Skip to content

Add GCSRemoteLogIO.from_config and register gs remote logging scheme #70266

Description

@jason810496

Part of #70265 (related: #67056).

Why

#67056 decoupled remote logging from the hardcoded branches in airflow_local_settings.py: core and
the Task SDK now resolve the handler via ProvidersManager dispatch on the
[logging] remote_base_log_folder URL scheme, instantiating the provider class through a no-arg
from_config() classmethod. This issue migrates the gs scheme.

What

  • Add GCSRemoteLogIO.from_config() in
    providers/google/src/airflow/providers/google/cloud/log/gcs_task_handler.py, mirroring the
    legacy branch:
    elif remote_base_log_folder.startswith("gs://"):
    from airflow.providers.google.cloud.log.gcs_task_handler import GCSRemoteLogIO
    _default_conn_name_from("airflow.providers.google.cloud.hooks.gcs", "GCSHook")
    key_path = conf.get_mandatory_value("logging", "google_key_path", fallback=None)
    REMOTE_TASK_LOG = GCSRemoteLogIO(
    **cast(
    "dict[str, Any]",
    {
    "base_log_folder": BASE_LOG_FOLDER,
    "remote_base": remote_base_log_folder,
    "delete_local_copy": delete_local_copy,
    "gcp_key_path": key_path,
    }
    | _io_kwargs,
    )
    )

    — including the [logging] remote_task_handler_kwargs IO-kwargs merge, expanduser on
    base_log_folder, and reading [logging] google_key_path (fallback None) into
    gcp_key_path, so behavior is unchanged for existing configs.
  • Register the gs scheme under a remote-logging: section in
    providers/google/provider.yaml and mirror it in
    providers/google/src/airflow/providers/google/get_provider_info.py.
  • Add tests mirroring TestS3RemoteLogIOFromConfig in
    providers/amazon/tests/unit/amazon/aws/log/test_s3_task_handler.py.
  • Verify end to end with a real system test: set up the actual backend yourself (a real
    service or a local equivalent), run a task with remote logging enabled, and confirm logs are
    uploaded and read back through the new dispatch path. Include the setup and verification
    results in the PR description.

Reference

Merged examples to follow: #69817 (s3), #69816 (cloudwatch). If from_config raises on a bad
config, the shared factory falls back to the legacy path, so this is not a breaking change.

The legacy branch derives a default connection id from GCSHook.default_conn_name; handle the
connection id the same way the merged amazon PRs do (the hook reads
[logging] remote_log_conn_id itself).

Metadata

Metadata

Assignees

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions