Add Longhorn storage health probe for read-only mount failures and disk pressure

[Copilot]

Pods can continue running while Longhorn-backed mounts become read-only after filesystem errors; restarting workloads may recover but there was no proactive signal. This change adds cluster-side detection for read-only mount symptoms and Longhorn disk-capacity pressure.

• What changed

  • Added cluster/core/longhorn-system/storage-health-check.yaml with:
    • ServiceAccount + scoped ClusterRole/ClusterRoleBinding
    • CronJob (longhorn-storage-health-check) on a 15-minute cadence
  • Wired the new manifest into cluster/core/longhorn-system/kustomization.yaml.

• Detection coverage

  • Warning events containing read-only filesystem indicators (read-only file system, erofs, remount-ro)
  • Pod container waiting/termination messages containing read-only filesystem errors
  • Longhorn volumes in unhealthy states (e.g., faulted, or attached but not healthy)
  • Longhorn disk usage above threshold (currently >=85% used)

• Operational behavior

  • Emits targeted findings to job logs.
  • Exits non-zero when any condition is detected, creating an explicit failure signal for visibility/alerting pipelines.

spec:
  schedule: "*/15 * * * *"
  jobTemplate:
    spec:
      template:
        spec:
          containers:
            - name: check
              command:
                - /bin/sh
                - -ec
                - |
                  # flags read-only indicators + Longhorn unhealthy volumes + >=85% disk usage
                  ...
                  if [ "${issues}" -ne 0 ]; then exit 1; fi

Warning

Firewall rules blocked me from connecting to one or more addresses (expand for details)

I tried to connect to the following addresses, but was blocked by firewall rules:

• charts.fairwinds.com
  • Triggering command: /usr/local/bin/helm helm repo add fairwinds-stable REDACTED (dns block)

If you need me to access, download, or install something from one of these locations, you can either:

• Configure Actions setup steps to set up my environment, which run before the firewall is enabled
• Add the appropriate URLs or hosts to the custom allowlist in this repository's Copilot coding agent settings (admins only)

[jeffborg]

@copilot is this disk usage within the longhorn volumes themselves?
also where do the notifications go to?

[Copilot]

@copilot is this disk usage within the longhorn volumes themselves?
also where do the notifications go to?

1. Not currently inside each mounted volume filesystem. The check uses nodes.longhorn.io diskStatus (storageMaximum vs storageAvailable), so it measures Longhorn backing-disk capacity on each Longhorn node/disk.

2. Right now this CronJob does not send external notifications by itself; it only fails the Job and logs findings. So visibility is in Job status/logs (and any cluster-wide alerting you already have on failed Jobs/CronJobs). The existing Flux Slack alert in this repo is for Flux resources in flux-system, not this longhorn-system CronJob.