Storage: implement Longhorn

[christiaan78]

Storage: Implement Longhorn

Goal

Deploy Longhorn as the default Kubernetes block storage solution for stateful workloads, providing dynamic provisioning, snapshots, and (where applicable) node-level resilience.

Why

Stateful services require consistent, repeatable storage primitives. Longhorn provides a standardized storage layer with operational features (replication, snapshots, backup targets) that improve recovery and portability across environments.

Scope

• Install Longhorn via GitOps (Helm/Kustomize) with environment-specific values.

• Define StorageClasses:

  • longhorn (default) for general-purpose workloads
  • optional: longhorn-retain (reclaim policy Retain) for critical data

• Configure operational guardrails:

  • resource requests/limits, tolerations/node selectors if needed
  • replica count policy aligned to node topology

• Enable snapshot and backup capabilities (if using an external backup target such as S3/NFS).

• Document usage patterns for workloads (PVC examples, recommended settings, pitfalls).

Definition of Done

• Longhorn deployed and healthy (longhorn-system ready, UI reachable internally).

• A default StorageClass exists and new PVCs provision successfully.

• A test workload:

  • writes data to a PVC
  • survives pod restart
  • snapshot is created and verified (and restored if in scope)

• Backup target configured (if in scope) and at least one backup artifact is produced.

• Documentation added:

  • how to use Longhorn PVCs
  • recommended settings (replicas, anti-affinity, reclaim policy)
  • troubleshooting checklist

Acceptance Criteria / Validation

• kubectl get pvc/pv shows Bound volumes provisioned via Longhorn CSI.
• Longhorn volume health is Healthy and replicas match policy.
• Test restore path validated (snapshot restore and/or backup restore).
• No critical errors in Longhorn manager/CSI logs during provisioning.

Notes / Risks

• Replica count requires enough nodes/disks; set expectations for single-node or small clusters.
• Longhorn needs disk configuration and may require filesystem mount options / kernel modules depending on nodes.
• Ensure encryption and access controls around any external backup target.