Skip to content

Global Prometheus registry breaks test isolation: -count=2 panics, cache-metrics assertions flake #104

Description

@mhenrixon

Problem

The test suite carries process-global Prometheus state that breaks test isolation. Two symptoms, one root cause:

  1. go test -race -count=2 ./internal/server/ panics deterministically — reproduced on a clean dash worktree:
panic: duplicate metrics collector registration attempted [recovered, repanicked]
  prometheus.(*Registry).MustRegister
  internal/metrics.NewPrometheusTracker   metrics.go:235
  internal/metrics.Enable                 metrics.go:33
  internal/server.(*Server).startMetricsServer   server.go:365
  internal/server.testServerWithConfig    testing.go:160
  TestServer_AppliesConfiguredTimeoutsToEveryListener   server_timeouts_test.go:72

metrics.Enable() calls prometheus.MustRegister against the default global registry; the second -count pass re-registers the same collectors in the same process and panics.

  1. TestCacheMetrics_OneClientResultPerRequest fails intermittently (expected 1, actual 0 on hit/miss/store deltas) — the same global counters accumulate across tests/runs, so delta assertions depend on what ran before. This is the likely identity of the rare single-run -race flake seen during PR fix(san-cert): serve the held certificate while its replacement issues #103 verification (single reruns pass).

Any test that starts a metrics-enabled Server poisons the process for repeat runs; CI's single-count runs mostly dodge it, which is why it survives.

Proposed fix

Make metrics.Enable() idempotent (a sync.Once around tracker construction), and/or register against an injectable prometheus.Registerer so tests get a fresh registry per server instead of the process global. Cache-metrics tests should then read deltas from their own registry, killing the cross-test counter pollution too.

Acceptance criteria

  • GIVEN go test -race -count=2 ./internal/server/ THEN the suite passes with no duplicate-registration panic
  • GIVEN TestCacheMetrics_OneClientResultPerRequest run alongside any server-starting test THEN its delta assertions are isolated from other tests' metric emissions

Found while chasing a -race flake during PR #103 verification (2026-08-13); reproduced on dash unmodified.

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workingproxydash-proxy (Go) work

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions