You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
#67056 decoupled remote logging from the hardcoded branches in airflow_local_settings.py: core and
the Task SDK now resolve the handler via ProvidersManager dispatch on the [logging] remote_base_log_folder URL scheme, instantiating the provider class through a no-arg from_config() classmethod. This issue migrates the OpenSearch backend.
Design note (feedback welcome)
Unlike the object-storage backends, OpenSearch is currently selected by [opensearch] host being
set — remote_base_log_folder typically has no scheme for these deployments, so pure scheme
dispatch cannot reach it. The proposal (same as the sibling Elasticsearch issue #70271):
Register an opensearch scheme and document setting [logging] remote_base_log_folder = opensearch:// as the forward-looking configuration.
Keep the legacy host-based selection working via the transitional fallback in airflow_local_settings.py until the chain is removed (tracked by the meta issue), so existing
configs are unaffected.
What
Add OpensearchRemoteLogIO.from_config() in providers/opensearch/src/airflow/providers/opensearch/log/os_task_handler.py, mirroring the
legacy branch:
— reading the [opensearch] options the branch reads (host, port, username, password, target_index, write_stdout, write_to_os, json_format, host_field, offset_field, and log_id_template with its {dag_id}-{task_id}-{run_id}-{map_index}-{try_number} fallback) plus [logging] base_log_folder (with expanduser) and [logging] delete_local_logs, so
behavior is unchanged for existing configs. Note the legacy branch does not merge remote_task_handler_kwargs IO-kwargs for this backend — mirror that too.
Register the opensearch scheme under a remote-logging: section in providers/opensearch/provider.yaml and mirror it in providers/opensearch/src/airflow/providers/opensearch/get_provider_info.py.
Document the remote_base_log_folder = opensearch:// form in the provider's logging docs.
Add tests mirroring TestS3RemoteLogIOFromConfig in providers/amazon/tests/unit/amazon/aws/log/test_s3_task_handler.py.
Verify end to end with a real system test: set up the actual backend yourself (a real
OpenSearch cluster or a local one, e.g. via Docker), run a task with remote logging enabled,
and confirm logs are written and read back through the new dispatch path. Include the setup
and verification results in the PR description.
Reference
Merged examples to follow: #69817 (s3), #69816 (cloudwatch). If from_config raises on a bad
config, the shared factory falls back to the legacy path, so this is not a breaking change.
Part of #70265 (related: #67056).
Why
#67056 decoupled remote logging from the hardcoded branches in
airflow_local_settings.py: core andthe Task SDK now resolve the handler via
ProvidersManagerdispatch on the[logging] remote_base_log_folderURL scheme, instantiating the provider class through a no-argfrom_config()classmethod. This issue migrates the OpenSearch backend.Design note (feedback welcome)
Unlike the object-storage backends, OpenSearch is currently selected by
[opensearch] hostbeingset —
remote_base_log_foldertypically has no scheme for these deployments, so pure schemedispatch cannot reach it. The proposal (same as the sibling Elasticsearch issue #70271):
opensearchscheme and document setting[logging] remote_base_log_folder = opensearch://as the forward-looking configuration.airflow_local_settings.pyuntil the chain is removed (tracked by the meta issue), so existingconfigs are unaffected.
What
OpensearchRemoteLogIO.from_config()inproviders/opensearch/src/airflow/providers/opensearch/log/os_task_handler.py, mirroring thelegacy branch:
airflow/airflow-core/src/airflow/config_templates/airflow_local_settings.py
Lines 349 to 379 in 104ad12
— reading the
[opensearch]options the branch reads (host,port,username,password,target_index,write_stdout,write_to_os,json_format,host_field,offset_field, andlog_id_templatewith its{dag_id}-{task_id}-{run_id}-{map_index}-{try_number}fallback) plus[logging] base_log_folder(withexpanduser) and[logging] delete_local_logs, sobehavior is unchanged for existing configs. Note the legacy branch does not merge
remote_task_handler_kwargsIO-kwargs for this backend — mirror that too.opensearchscheme under aremote-logging:section inproviders/opensearch/provider.yamland mirror it inproviders/opensearch/src/airflow/providers/opensearch/get_provider_info.py.remote_base_log_folder = opensearch://form in the provider's logging docs.TestS3RemoteLogIOFromConfiginproviders/amazon/tests/unit/amazon/aws/log/test_s3_task_handler.py.OpenSearch cluster or a local one, e.g. via Docker), run a task with remote logging enabled,
and confirm logs are written and read back through the new dispatch path. Include the setup
and verification results in the PR description.
Reference
Merged examples to follow: #69817 (s3), #69816 (cloudwatch). If
from_configraises on a badconfig, the shared factory falls back to the legacy path, so this is not a breaking change.