metrics: fix stale Prometheus gauge labels after label change - #1561
Draft
RamavathChanti wants to merge 1 commit into
Draft
metrics: fix stale Prometheus gauge labels after label change#1561RamavathChanti wants to merge 1 commit into
RamavathChanti wants to merge 1 commit into
Conversation
Prometheus gauges kept old label combinations indefinitely when an application was renamed, a group's channel changed, or instances updated away from a version. The stale labels froze at their last value, causing incorrect sum() aggregations and misleading dashboards. Add GaugeVec.Reset() before each metrics update loop to clear stale label combinations. This is the standard Prometheus Go client pattern for gauges with dynamic labels. Applied to: - nebraska_application_instances_per_channel - nebraska_failed_updates - All 8 nebraska_group_* rollout progress metrics Fixes flatcar#1482 Signed-off-by: Chanti <chantib107@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Nebraska's
calculateMetrics()sets gauge values each tick but neverclears old label combinations. When an application is renamed, a group's
channel changes, or all instances update away from a version, the old
label series freeze at their last value forever until process restarts.
This causes
sum(nebraska_application_instances_per_channel)to reportdouble the real count after a rename, and stale per-version rows that
never disappear from dashboards.
Fix
Add
GaugeVec.Reset()before each gauge update loop. Drops all existingseries, then the loop recreates only what the current DB query returns.
Applied to:
nebraska_application_instances_per_channelnebraska_failed_updatesnebraska_group_*rollout progress metricsChanges
backend/pkg/metrics/metrics.go- added Reset() before each gauge update loopbackend/pkg/metrics/metrics_test.go(new) - 4 unit tests covering the fixHow to use
cd backend && go test -v ./pkg/metrics/
Fixes #1482