user-workload-monitoring: pool-level TrueNAS alerts (saturation, capacity, health) + zfs_exporter scrape

[david-igou]

Background

During the 2026-07-10 TrueNAS API/UI timeout incident (#421), the ssd pool sat at multi-second write waits for hours with no alert. The existing truenas PrometheusRule (components/user-workload-monitoring/exporters/truenas/truenas-prometheusrule.yaml) covers per-disk hardware well (temps, SMART/SAS errors, NVMe wear, IPMI, exporter-down) but has two structural gaps:

1. Whole-pool saturation is invisible. The only latency rule, ColdDiskLatencyOutlier, is (a) scoped to the cold pool serials and (b) an outlier-vs-median test — when every disk in a pool is slow together (exactly what happened on ssd), the median rises with the disks and the rule can never fire.
2. No pool-level ZFS signals at all. zfs_exporter is live on the NAS (10.10.9.243:9134, igou-io/igou-inventory#151) exposing zfs_pool_health / zfs_pool_free_bytes / zfs_pool_size_bytes / zfs_pool_fragmentation_ratio and per-dataset gauges — but nothing scrapes it yet; there is no Service/Endpoints/ServiceMonitor for it in exporters/truenas/.

Alerts route to the existing EDA Alertmanager→GitHub-issue pipeline, so a saturation alert would have opened an issue about an hour before the UI became unusable.

Changes

1. Scrape zfs_exporter

Add truenas-zfs.yaml to components/user-workload-monitoring/exporters/truenas/ following the exact pattern of truenas-netdata.yaml (selectorless Service + manual Endpoints at 10.10.9.243:9134 + ServiceMonitor, instance relabeled to truenas.igou.systems). The existing TrueNASExporterDown rule (job=~"truenas-.*") covers the new target automatically if the job is named truenas-zfs.

Depends on igou-io/igou-inventory#151 (exporter deployment PR) merging; the endpoint is already live.

2. Pool saturation alerts (netdata busy-time, per-pool serial groups)

Serial→pool mapping (from lsblk/smartctl, 2026-07-10):

• ssd: S753NS0X206644B, S753NS0X206645D (870 EVO, mirror-0) + 2515E9B3E0E1, 2515E9B3E121 (BX500, mirror-1)
• cold: existing regex in ColdDiskLatencyOutlier (unchanged)
• fast: NVMe serials — map during implementation (lsblk -o NAME,SERIAL; note 2 of the 6 NVMe are unassigned 500GB P3s and must be excluded)

Proposed rules (thresholds to be tuned against a week of data; incident values were >800ms sustained):

- alert: SSDPoolSaturated
  expr: >
    avg(netdata_Disk_Busy_Time_Milliseconds_average{disk=~"S753NS0X206644B|S753NS0X206645D|2515E9B3E0E1|2515E9B3E121"}) > 400
  for: 10m
  labels:
    severity: warning
  annotations:
    summary: ssd pool disks averaging >40% busy for 10m
    description: >
      Sustained saturation of the ssd pool. Middleware state (system
      dataset), CSI provisioning, and the TrueNAS UI all degrade when this
      pool stalls - see igou-io/igou-openshift#421.

- alert: SSDPoolSaturatedCritical
  expr: >
    avg(netdata_Disk_Busy_Time_Milliseconds_average{disk=~"S753NS0X206644B|S753NS0X206645D|2515E9B3E0E1|2515E9B3E121"}) > 800
  for: 10m
  labels:
    severity: critical

Mirror the pair for fast once its serials are mapped. Consider extracting the per-pool disk lists into recording rules (e.g. truenas:pool_disk_busy_ms:avg{pool="ssd"}) so the dashboard (igou-io/igou-inventory issue, forthcoming) and alerts share one source of truth for the serial groups.

3. Pool health/capacity alerts (zfs_exporter, after step 1)

- alert: ZFSPoolNotHealthy
  expr: zfs_pool_health > 0        # 0 = ONLINE
  for: 5m
  labels:
    severity: critical

- alert: ZFSPoolLowCapacity
  expr: zfs_pool_free_bytes / zfs_pool_size_bytes < 0.15
  for: 30m
  labels:
    severity: warning

(Optionally a critical tier at <8%, and a fragmentation-trend warning if zfs_pool_fragmentation_ratio becomes a problem on ssd.)

4. (Optional) Direct middleware-responsiveness probe

The real incident symptom was API latency, and disk busy-time is a proxy. The blackbox-exporter component already exists (exporters/blackbox-exporter/): add a Probe against the TrueNAS web UI/API endpoint and alert on probe_duration_seconds > 5 or probe failure for 5m. This catches middleware stalls regardless of cause (pool saturation, middleware bug, etc.).

Validation

• up{job="truenas-zfs"} == 1 in UWM Prometheus; zfs_pool_health queryable in thanos-querier
• make test in igou-openshift; ArgoCD app Synced
• Threshold sanity: replay the alert exprs over the incident window (2026-07-09 16:00–22:00 local) via thanos — SSDPoolSaturated should have fired; verify it does not fire during normal daily load
• Confirm EDA pipeline opens a GitHub issue on a test alert (existing flow)

Non-goals

• Dashboard work (specced separately in igou-inventory)
• Changing ColdDiskLatencyOutlier (still useful for single-ailing-drive detection; the new rules cover the all-disks-slow case)

[david-igou]

Heads-up from igou-io/igou-inventory#154: the ssd pool's BX500 vdev (mirror-1) is being evicted via device removal rather than replaced. When implementing the SSDPoolSaturated rules and recording rules here, the ssd serial group must be only the two 870 EVOs (S753NS0X206644B|S753NS0X206645D) — including the freed BX500 serials (2515E9B3E0E1|2515E9B3E121) would average two idle disks into the pool metric and halve its sensitivity. Threshold guidance updates accordingly: pure-EVO pool baseline is 2–5 ms/op with flat latency; the 400ms/800ms avg-busy thresholds remain reasonable as past-the-knee signals.

[david-igou]

Groundwork landed in #423 (dashboard PR for igou-io/igou-inventory#154): the zfs_exporter ServiceMonitor (job="truenas-zfs", :9134) and the per-pool recording rules are now in truenas-prometheusrule.yaml — disk:truenas_disk_{busy_ms,throughput_bytes,iops}{pool,disk} and pool:truenas_disk_busy_ms:avg / pool:truenas_disk_throughput_bytes:sum / pool:truenas_disk_iops:sum. Serial→pool membership lives ONLY there (2026-07-10 post-BX500-eviction map).

Remaining for this issue:

• Pool saturation alerts on pool:truenas_disk_busy_ms:avg (400 ms warn / 800 ms crit — dashboard threshold lines already match).
• Pool capacity/health alerts on zfs_pool_health / zfs_pool_allocated_bytes÷size_bytes (scrape is live).
• Refactor ColdDiskLatencyOutlier's inline serial regex onto disk:truenas_disk_busy_ms{pool="cold"} — deliberately not touched in feat(monitoring): TrueNAS ZFS & Storage Health dashboard + zfs_exporter scrape + per-pool recording rules #423 to keep alert behavior changes out of the dashboard PR.

[igou-dev]

Status 2026-07-16: the scrape + dashboard + recording rules shipped via #423-#425 (pool:truenas_disk_busy_ms:avg et al. in truenas-prometheusrule.yaml). Remaining scope is the alerts themselves — no pool saturation/capacity/health alert rules exist yet (current alert list is ColdDiskLatencyOutlier + disk temp/SMART/IPMI only), and ColdDiskLatencyOutlier still carries its inline serial regex instead of using the recording rules.