Fix custom stats metrics not being sent from plugins/listeners#69270
Open
Aaryan123456679 wants to merge 3 commits into
Open
Fix custom stats metrics not being sent from plugins/listeners#69270Aaryan123456679 wants to merge 3 commits into
Aaryan123456679 wants to merge 3 commits into
Conversation
Aaryan123456679
requested review from
XD-DENG,
ashb,
bugraoz93,
choo121600,
dheerajturaga,
dstandish,
ephraimbuddy,
henry3260,
hussein-awala,
jason810496,
jedcunningham,
o-nikolas,
pierrejeambrun,
rawwar and
shubhamraj-git
as code owners
July 2, 2026 17:43
boring-cyborg
Bot
added
area:API
Contributor
Author
|
potiuk
added
the
ready for maintainer review
Contributor
Author
|
Friendly ping! This PR is ready for review whenever someone has time. Thank you! |
pierrejeambrun
left a comment
pierrejeambrun
left a comment
Member
There was a problem hiding this comment.
Thanks for the PR.
Having calls to initialize_sdk_stats_backend all over the core code base really feels off. We're trying to split server / client and limit the dependencies there. We probably should use a _shared lib.
pierrejeambrun
removed
the
ready for maintainer review
Airflow keeps two independent Stats singletons: one internal to airflow-core, and the task-sdk copy that airflow.sdk.observability.stats (and the deprecated airflow.stats shim) expose to plugins and listener hooks. Long-running components only initialized the internal copy at startup, so a plugin's DAG-run listener hook running in the scheduler or API server silently got NoStatsLogger regardless of StatsD config, even though native Airflow metrics worked fine. closes: apache#69172
Plugins and listeners typically reach Stats through the task-sdk import path, a separate singleton from the one each component initializes for its own internal use, so plugin code kept using an unconfigured NoStatsLogger even after the component's own Stats was set up. Propagate the configuration to any sibling copy of the stats module already loaded under a different distribution path instead of having core import from airflow.sdk directly to reach it. Metrics initialization failures no longer prevent the API server, scheduler, Dag processor, triggerer, or executor from starting.
Aaryan123456679
force-pushed
the
fix/plugin-statsd-metrics-scheduler-init-69172
branch
from
July 24, 2026 12:12
0de55d0 to
396cb29
Compare
Contributor
Author
|
Hi @pierrejeambrun , I have done the necessary changes, kindly let me know in case any other change is required. Thanks. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Airflow keeps two independent
Statssingletons because of the shared/symlinked library architecture: one internal copy that airflow-core initializes at startup (airflow._shared.observability.metrics.stats), and the task-sdk copy exposed by the officially recommendedairflow.sdk.observability.statsimport (and the deprecatedairflow.statsshim, which re-exports it). Long-running components — scheduler, DAG file processor, API server, executors, triggerer — only initialized the internal copy at startup. The task-sdk copy was only ever initialized inside the task-execution subprocess.Plugin/listener hooks such as
on_dag_run_success/on_dag_run_failedrun in the scheduler or API server process, never in the task subprocess, so a plugin callingStats.gauge(...)via the recommended import path silently fell back toNoStatsLogger— no error, no custom metrics — regardless of StatsD/Datadog/OTel configuration, even though Airflow's own native metrics worked fine.This adds
initialize_sdk_stats_backend()toairflow.observability.metrics.stats_utils, which configures the task-sdkStatssingleton with the same factory/config each process already uses for its own internal singleton, and calls it alongside the existingstats.initialize(...)call in the scheduler, DAG processor, API server, base executor, and triggerer.closes: #69172
Test plan
TestInitializeSdkStatsBackendinairflow-core/tests/unit/observability/metrics/test_stats.pyverifying the sdk singleton is configured with this process's backend and the same factory/legacy-names flag.test_app.py,test_manager.py,test_base_executor.py,test_scheduler_job.py, andtest_triggerer_job.pyto assert the new call happens alongside the existing one.Stats.gauge(...)silently resolved toNoStatsLogger; after callinginitialize_sdk_stats_backend(), it resolved toSafeStatsdLogger.Was generative AI tooling used to co-author this PR?
Generated-by: Claude Code (Sonnet 5) following the guidelines