You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
#67056 decoupled remote logging from the hardcoded branches in airflow_local_settings.py: core and
the Task SDK now resolve the handler via ProvidersManager dispatch on the [logging] remote_base_log_folder URL scheme, instantiating the provider class through a no-arg from_config() classmethod. This issue migrates the gs scheme.
What
Add GCSRemoteLogIO.from_config() in providers/google/src/airflow/providers/google/cloud/log/gcs_task_handler.py, mirroring the
legacy branch:
— including the [logging] remote_task_handler_kwargs IO-kwargs merge, expanduser on base_log_folder, and reading [logging] google_key_path (fallback None) into gcp_key_path, so behavior is unchanged for existing configs.
Register the gs scheme under a remote-logging: section in providers/google/provider.yaml and mirror it in providers/google/src/airflow/providers/google/get_provider_info.py.
Add tests mirroring TestS3RemoteLogIOFromConfig in providers/amazon/tests/unit/amazon/aws/log/test_s3_task_handler.py.
Verify end to end with a real system test: set up the actual backend yourself (a real
service or a local equivalent), run a task with remote logging enabled, and confirm logs are
uploaded and read back through the new dispatch path. Include the setup and verification
results in the PR description.
Reference
Merged examples to follow: #69817 (s3), #69816 (cloudwatch). If from_config raises on a bad
config, the shared factory falls back to the legacy path, so this is not a breaking change.
The legacy branch derives a default connection id from GCSHook.default_conn_name; handle the
connection id the same way the merged amazon PRs do (the hook reads [logging] remote_log_conn_id itself).
Part of #70265 (related: #67056).
Why
#67056 decoupled remote logging from the hardcoded branches in
airflow_local_settings.py: core andthe Task SDK now resolve the handler via
ProvidersManagerdispatch on the[logging] remote_base_log_folderURL scheme, instantiating the provider class through a no-argfrom_config()classmethod. This issue migrates thegsscheme.What
GCSRemoteLogIO.from_config()inproviders/google/src/airflow/providers/google/cloud/log/gcs_task_handler.py, mirroring thelegacy branch:
airflow/airflow-core/src/airflow/config_templates/airflow_local_settings.py
Lines 207 to 224 in 104ad12
— including the
[logging] remote_task_handler_kwargsIO-kwargs merge,expanduseronbase_log_folder, and reading[logging] google_key_path(fallbackNone) intogcp_key_path, so behavior is unchanged for existing configs.gsscheme under aremote-logging:section inproviders/google/provider.yamland mirror it inproviders/google/src/airflow/providers/google/get_provider_info.py.TestS3RemoteLogIOFromConfiginproviders/amazon/tests/unit/amazon/aws/log/test_s3_task_handler.py.service or a local equivalent), run a task with remote logging enabled, and confirm logs are
uploaded and read back through the new dispatch path. Include the setup and verification
results in the PR description.
Reference
Merged examples to follow: #69817 (s3), #69816 (cloudwatch). If
from_configraises on a badconfig, the shared factory falls back to the legacy path, so this is not a breaking change.
The legacy branch derives a default connection id from
GCSHook.default_conn_name; handle theconnection id the same way the merged amazon PRs do (the hook reads
[logging] remote_log_conn_iditself).