feat(truenas): scrape + alerts for docker app fleet (VPN death, app health)#574
Open
igou-dev[bot] wants to merge 2 commits into
Open
feat(truenas): scrape + alerts for docker app fleet (VPN death, app health)#574igou-dev[bot] wants to merge 2 commits into
igou-dev[bot] wants to merge 2 commits into
Conversation
…ealth)
Companion to igou-inventory feat/truenas-app-monitoring, which deploys
docker-health-exporter (:9000) and cadvisor (:9338) on the NAS.
- truenas-docker-health.yaml / truenas-cadvisor.yaml: selectorless
Service + Endpoints + ServiceMonitor, mirroring truenas-netdata.
cadvisor scrape keeps only lifecycle + cpu/mem/oom series and drops
empty-name cgroup rows for cardinality.
- truenas.apps alert group:
- TrueNASDockerDaemonDown (critical)
- TrueNASVPNStackUnhealthy (critical): gluetun stack unhealthy 5m+ =
tunnel dead and deunhealth not recovering; kill-switch means nothing
else pages for this
- TrueNASContainerUnhealthy (warning), TrueNASContainerStopped
(critical, scoped to expected fleet — the 2026-07-16 20h-outage class)
- TrueNASVPNTunnelRestarted (warning, event-style): gluetun restart =
VPN died and was auto-healed, visible even when the unhealthy window
falls between scrapes
- TrueNASContainerFlapping + TrueNASContainerOOM (warning)
- Existing TrueNASExporterDown (job=~"truenas-.*") covers the two new
scrape targets automatically.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017P9CPrKJFTuYjnK1geLbbj
The exporter passes the Docker API container name through verbatim
("/gluetun") - found auditing the upstream source (e2e fixtures confirm).
Without this relabel every truenas.apps alert regex silently never
matches. Normalized at scrape time so rules and cadvisor's slash-less
name label line up.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017P9CPrKJFTuYjnK1geLbbj
Contributor
Author
|
Source audit of calum4/docker-prometheus-exporter v2.0.0 (pinned digest, tag 36fc6b7) — verdict: proceed.
🤖 Generated with Claude Code |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
Companion to igou-io/igou-inventory#215 (deploys
docker-health-exporter:9000 +cadvisor:9338 on the NAS). Closes the two blind spots: silent VPN death behind gluetun's kill-switch, and silent app outages (the 2026-07-16 qbittorrent 20h-down class). The existing*.biscuit.igou.systemsblackbox probes cover HTTP availability but cannot see a dead VPN (kill-switch keeps the web UIs green) or distinguish which container in a stack died.What
truenas-docker-health.yaml+truenas-cadvisor.yaml— selectorless Service + Endpoints + ServiceMonitor at 10.10.9.243, mirroringtruenas-netdata.yaml. The cadvisor scrape drops empty-name cgroup rows and keeps onlycontainer_start_time_seconds|container_last_seen|container_oom_events_total|container_memory_working_set_bytes|container_cpu_usage_seconds_totalfor cardinality.truenas.appsalert group (routes to EDA GitHub-issue pipeline + Gotify/Slack like the rest of the file):docker_up == 05mchanges(container_start_time_seconds{name=\"gluetun\"}[15m]) > 0— a VPN death that deunhealth healed between scrapes still moves the start timestampTrueNASExporterDown(job=~\"truenas-.*\") picks up both new scrape targets automatically.make testgreen.Ordering
Merge with igou-io/igou-inventory#215; either order is safe — until the containers converge on the NAS the new targets just report down (TrueNASExporterDown fires after 5m, so ideally converge the inventory side first or within the same window).
🤖 Generated with Claude Code
https://claude.ai/code/session_017P9CPrKJFTuYjnK1geLbbj