Skip to content

feat(truenas): scrape + alerts for docker app fleet (VPN death, app health)#574

Open
igou-dev[bot] wants to merge 2 commits into
mainfrom
feat/truenas-app-monitoring
Open

feat(truenas): scrape + alerts for docker app fleet (VPN death, app health)#574
igou-dev[bot] wants to merge 2 commits into
mainfrom
feat/truenas-app-monitoring

Conversation

@igou-dev

@igou-dev igou-dev Bot commented Jul 24, 2026

Copy link
Copy Markdown
Contributor

Why

Companion to igou-io/igou-inventory#215 (deploys docker-health-exporter :9000 + cadvisor :9338 on the NAS). Closes the two blind spots: silent VPN death behind gluetun's kill-switch, and silent app outages (the 2026-07-16 qbittorrent 20h-down class). The existing *.biscuit.igou.systems blackbox probes cover HTTP availability but cannot see a dead VPN (kill-switch keeps the web UIs green) or distinguish which container in a stack died.

What

  • truenas-docker-health.yaml + truenas-cadvisor.yaml — selectorless Service + Endpoints + ServiceMonitor at 10.10.9.243, mirroring truenas-netdata.yaml. The cadvisor scrape drops empty-name cgroup rows and keeps only container_start_time_seconds|container_last_seen|container_oom_events_total|container_memory_working_set_bytes|container_cpu_usage_seconds_total for cardinality.
  • New truenas.apps alert group (routes to EDA GitHub-issue pipeline + Gotify/Slack like the rest of the file):
alert severity signal
TrueNASDockerDaemonDown critical docker_up == 0 5m
TrueNASVPNStackUnhealthy critical gluetun/qbittorrent/prowlarr/flaresolverr unhealthy 5m+ = tunnel dead and deunhealth not recovering
TrueNASContainerUnhealthy warning any other container failing its healthcheck 5m+
TrueNASContainerStopped critical expected-fleet container stopped 10m (scoped regex so stray one-off containers don't page)
TrueNASVPNTunnelRestarted warning event-style: changes(container_start_time_seconds{name=\"gluetun\"}[15m]) > 0 — a VPN death that deunhealth healed between scrapes still moves the start timestamp
TrueNASContainerFlapping warning >3 restarts in 2h (crash loop / deunhealth restart storm)
TrueNASContainerOOM warning OOM kill in the last hour
  • The existing TrueNASExporterDown (job=~\"truenas-.*\") picks up both new scrape targets automatically.
  • make test green.

Ordering

Merge with igou-io/igou-inventory#215; either order is safe — until the containers converge on the NAS the new targets just report down (TrueNASExporterDown fires after 5m, so ideally converge the inventory side first or within the same window).

🤖 Generated with Claude Code

https://claude.ai/code/session_017P9CPrKJFTuYjnK1geLbbj

david-igou and others added 2 commits July 24, 2026 12:34
…ealth)

Companion to igou-inventory feat/truenas-app-monitoring, which deploys
docker-health-exporter (:9000) and cadvisor (:9338) on the NAS.

- truenas-docker-health.yaml / truenas-cadvisor.yaml: selectorless
  Service + Endpoints + ServiceMonitor, mirroring truenas-netdata.
  cadvisor scrape keeps only lifecycle + cpu/mem/oom series and drops
  empty-name cgroup rows for cardinality.
- truenas.apps alert group:
  - TrueNASDockerDaemonDown (critical)
  - TrueNASVPNStackUnhealthy (critical): gluetun stack unhealthy 5m+ =
    tunnel dead and deunhealth not recovering; kill-switch means nothing
    else pages for this
  - TrueNASContainerUnhealthy (warning), TrueNASContainerStopped
    (critical, scoped to expected fleet — the 2026-07-16 20h-outage class)
  - TrueNASVPNTunnelRestarted (warning, event-style): gluetun restart =
    VPN died and was auto-healed, visible even when the unhealthy window
    falls between scrapes
  - TrueNASContainerFlapping + TrueNASContainerOOM (warning)
- Existing TrueNASExporterDown (job=~"truenas-.*") covers the two new
  scrape targets automatically.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017P9CPrKJFTuYjnK1geLbbj
The exporter passes the Docker API container name through verbatim
("/gluetun") - found auditing the upstream source (e2e fixtures confirm).
Without this relabel every truenas.apps alert regex silently never
matches. Normalized at scrape time so rules and cadvisor's slash-less
name label line up.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017P9CPrKJFTuYjnK1geLbbj
@igou-dev

igou-dev Bot commented Jul 24, 2026

Copy link
Copy Markdown
Contributor Author

Source audit of calum4/docker-prometheus-exporter v2.0.0 (pinned digest, tag 36fc6b7) — verdict: proceed.

  • ~540 lines non-test Rust, read in full. Exactly 3 Docker API calls: ping, list_containers(all=true), inspect_container — and only .state is read from inspect, so container env vars (which hold secrets on this host) are never touched, logged, or exported. No outbound network beyond the socket + its own listener. No unsafe. No telemetry.
  • Supply chain: image is GitHub-Actions-built from tag on push (buildx, semver, ghcr mirror), digest-pinned here; scheduled cargo-audit workflow is active.
  • Honest caveats: maintainer inactive since ~Sept 2025 (6 open RUSTSEC advisories + 13 dependabot PRs, all bot-filed, unaddressed). The advisories are in unreachable code paths for this deployment (TLS/webpki — the exporter makes no TLS connections) or DoS-only on the LAN-scraped listener, which TrueNASExporterDown would catch. If a real patch is ever needed, the codebase is small enough to fork into igou-containers (zfs-exporter precedent) at that time.
  • Bug found and fixed in this PR (commit above): the exporter emits Docker's verbatim container name with leading slash (/gluetun, confirmed in upstream e2e fixtures) — alert regexes would never have matched. Now normalized via scrape-time metricRelabeling.

🤖 Generated with Claude Code

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant