Skip to content

fix(metrics): make tracker swap atomic and Enable idempotent for test isolation - #106

Open
mhenrixon wants to merge 1 commit into
dashfrom
fix/metrics-test-isolation
Open

fix(metrics): make tracker swap atomic and Enable idempotent for test isolation#106
mhenrixon wants to merge 1 commit into
dashfrom
fix/metrics-test-isolation

Conversation

@mhenrixon

@mhenrixon mhenrixon commented Aug 13, 2026

Copy link
Copy Markdown
Collaborator

Summary

  • internal/metrics/metrics.go: Tracker is now a delegating tracker whose delegate swaps atomically via a new SetTracker(EventTracker) EventTracker; Enable() registers the Prometheus collectors under a sync.Once and never clobbers a tracker installed after it.
  • internal/metrics/metrics_test.go: pins idempotent Enable, swap/restore semantics, fake survival across Enable calls, and race-safety of swapping while emitting.
  • internal/server/cert_metrics_test.go: deletes the test-local switchableTracker (it duplicated exactly this mechanism); installFakeTracker now uses metrics.SetTracker with a t.Cleanup restore.

Root cause: Enable() did Tracker = NewPrometheusTracker() on every call — the second call in one process re-ran prometheus.MustRegister against the default registry (the -count=2 panic), and the first call orphaned the test suite's switchable fake by replacing the whole Tracker var (the cache-metrics flake, plus an unsynchronized write racing background emitters under -race).

Closes #104

Test plan

  • go test -race -count=2 ./internal/server/ — 3660 passed (previously deterministic panic)
  • go test -race -count=3 ./internal/server/ -run 'TestServer_AppliesConfiguredTimeoutsToEveryListener|TestCacheMetrics' — cache-metric deltas isolated from a metrics-enabled server in the same process
  • make test, go test -race ./... (2151 passed, 8 packages)
  • gofmt -l internal/ cmd/ clean, go vet ./... clean, make lint 0 issues

Deviations & judgment calls

  • Discovery: internal/server/cert_metrics_test.go already carried a switchableTracker (atomic delegate installed as metrics.Tracker in init()) precisely to avoid the tracker-swap data race — the bug was Enable() orphaning it. The fix promotes that same pattern into the metrics package and deletes the test-local copy rather than adding a second layer of indirection.
  • Judgment call: fixed the duplicate-registration panic with sync.Once (the issue's first option) instead of threading an injectable prometheus.Registerer through Config/Server (the second). Once is sufficient for both acceptance criteria; per-server registries would also drop the default registry's go_*/process_* collectors from /metrics unless re-added — a behavior change the issue didn't ask for.
  • Judgment call: exported the tracker interface as metrics.EventTracker so SetTracker doesn't take/return an unexported type (revive/staticcheck flag that).

Summary by cubic

Make the process-wide metrics tracker swap atomic and make metrics.Enable() idempotent to prevent duplicate Prometheus registration and fix test isolation races. Previously Enable() replaced metrics.Tracker and re-registered collectors on each call (panic under -count=2) and orphaned test fakes; now a delegating tracker forwards to an atomically swapped delegate and Enable() registers once and does not overwrite a tracker installed after it.

  • Introduces metrics.EventTracker and a delegating metrics.Tracker; adds metrics.SetTracker(EventTracker) EventTracker to atomically swap and restore the previous tracker.
  • metrics.Enable() is guarded by sync.Once; returns the Prometheus handler without re-registration and without clobbering a post-Enable() tracker.
  • Replaces the local switchableTracker in internal/server/cert_metrics_test.go with metrics.SetTracker and t.Cleanup.
  • Adds tests for delegation, idempotent Enable(), and race-safety while swapping.
  • Migration: tests that rewired metrics.Tracker directly should call metrics.SetTracker(fake) and restore with the returned previous tracker.

Written for commit 09a5abd. Summary will update on new commits.

Review in cubic

… isolation

## Summary
metrics.Enable() overwrote the package-global Tracker and re-ran
prometheus.MustRegister against the default registry on every call, so a
second metrics-enabled server in the same process (go test -count=2)
panicked on duplicate registration, and any Enable call orphaned the test
suite's switchable fake tracker -- making cache/cert metric assertions
depend on which tests ran before them.

Tracker is now a delegating tracker whose delegate swaps atomically via
SetTracker, and Enable registers the Prometheus collectors exactly once
per process without clobbering a tracker installed after it. The server
test suite's local switchableTracker duplicated exactly this mechanism
and is deleted in favor of metrics.SetTracker.

## Test Coverage
- TestEnable_IsIdempotent: two Enable calls, no duplicate-registration panic
- TestEnable_DoesNotReplaceATrackerInstalledAfterIt: fakes survive later Enable calls
- TestSetTracker_DelegatesEventsAndRestores: swap semantics + restore
- TestSetTracker_IsSafeUnderConcurrentEmission: swap under -race while emitting
- go test -race -count=2 ./internal/server/: 3660 passed (previously panicked)

## Verification
- [x] gofmt -l internal/ cmd/ clean
- [x] make test passes
- [x] go vet ./... and make lint (golangci-lint) clean
- [x] go test -race ./... clean

Closes #104

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No issues found across 3 files

Confidence score: 5/5

  • Automated review surfaced no issues in the provided summaries.
  • No files require special attention.

Re-trigger cubic

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Global Prometheus registry breaks test isolation: -count=2 panics, cache-metrics assertions flake

1 participant