Skip to content

fix: fail health checks for inconsistent metrics#833

Open
immanuwell wants to merge 1 commit into
prometheus:masterfrom
immanuwell:fix-health-check-inconsistent-metrics
Open

fix: fail health checks for inconsistent metrics#833
immanuwell wants to merge 1 commit into
prometheus:masterfrom
immanuwell:fix-health-check-inconsistent-metrics

Conversation

@immanuwell

Copy link
Copy Markdown

Fixes #544

Right now /-/healthy can still say 200 OK when /metrics is already busted after unchecked inconsistent pushes. This is easy to hit in practice, it only takes two tiny pushes when --push.disable-consistency-check is on.

This patch makes the health check fail when the gathered metrics are inconsistent, so k8s and friends can see the box is not ok.

Repro:

go run . --push.disable-consistency-check --web.listen-address=127.0.0.1:19091
curl -X POST --data-binary $'# TYPE some_metric counter\nsome_metric 1\n' http://127.0.0.1:19091/metrics/job/some_job/tag/val1
curl -X POST --data-binary $'# TYPE some_metric counter\nsome_metric{tag="val1"} 42\n' http://127.0.0.1:19091/metrics/job/some_job
curl -i http://127.0.0.1:19091/metrics
curl -i http://127.0.0.1:19091/-/healthy

Before this change /metrics returns 500 but /-/healthy still returns 200. After this change both return 500, nice and simple.

cc @machine424

Signed-off-by: immanuwell <pchpr.00@list.ru>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

/-/healthy should fail if metrics are inconsistent.

1 participant