Feature: persist + reattach user home volumes across destroy/recreate (Option A)

[espg]

Currently tofu destroy (and the scripts/pre-destroy-drain.sh step) do not touch user home directories.

Current behavior

• pre-destroy-drain.sh deletes only LoadBalancer Services and Ingresses (to clear ELB ENIs that block VPC deletion).
• User home volumes are dynamically-provisioned PVCs created by KubeSpawner per user (EBS via the CSI driver). They are not owned by the daskhub Helm release.
• tofu destroy deletes the EKS cluster/nodes directly. With the CSI controller gone, PVCs are never gracefully deleted, so the backing EBS volumes are left orphaned (each ~$0.08/GB-month) with no evacuation/backup of user data first.
• This is invisible today only because cae-testing uses dummy auth and no one spawned a server. On cae-dev/prod with real users, a destroy would strand (and silently keep billing) every user home volume.

Gaps

1. No pre-destroy evacuation/snapshot of user home data.
2. No deletion of user PVCs, so EBS volumes orphan.
3. verify-clean-teardown.sh has no authoritative EBS-volume check (name-based checks cover NAT/VPC/EIP/EKS/LB only).

Proposed fixes

• Add an opt-in PVC-cleanup phase to pre-destroy-drain.sh (destructive to user data — guard behind an explicit flag / confirmation).
• Optionally snapshot user PVCs to S3/EBS-snapshot before deletion (evacuation).
• Add an authoritative EBS check to verify-clean-teardown.sh:
  aws ec2 describe-volumes --filters Name=tag:kubernetes.io/cluster/<cluster>,Values=owned → fail on any remaining available/in-use volume.

Context

Surfaced during teardown hardening (drain + verify) — the verify tag-index check was made advisory because resourcegroupstaggingapi lags ~1h; name-based checks are authoritative but do not yet cover EBS.

[espg]

Root cause found: the default gp3 StorageClass sets reclaim_policy = "Delete" (modules/kubernetes/main.tf:62). So the design intent is already to delete user EBS on PVC deletion — the orphaning is a reliability bug (PVCs are never explicitly deleted before the CSI driver/cluster is torn down, so the Delete policy never runs), not a deliberate persistence choice.

Two coherent resolutions:

Option B — make the existing Delete intent reliable (recommended, small):

• pre-destroy-drain.sh: delete all PVCs in daskhub and wait for the backing EBS volumes to disappear (CSI controller must still be alive), then tofu destroy.
• verify-clean-teardown.sh: add authoritative EBS check (describe-volumes tagged kubernetes.io/cluster/<cluster>=owned, fail on any available/in-use).
• Result: reliable zero-cost teardown.

Option A — persist + reattach (deliberate feature, larger):

• Switch reclaim to Retain; add static-PV rebinding on redeploy (capture per-user EBS volume IDs, recreate PVs/PVCs bound to them), or move home dirs to EFS (persists independently, per-user access points, reattaches naturally).

Decision: pursue B to fix the bug now; revisit A only if durable user homes become a wanted feature.

[espg]

Option B — IMPLEMENTED (closes the orphan bug)

The neither-persist-nor-delete orphan bug is fixed; teardown now reliably deletes user EBS volumes:

• pre-destroy-drain.sh: after LB/ingress drain, it stops singleuser servers, deletes all PVCs cluster-wide, and waits for the backing EBS volumes to actually delete (captured from PV volumeHandles) while the CSI controller is still alive. Gated by KEEP_USER_VOLUMES (default = delete).
• modules/kubernetes/main.tf: the gp3 StorageClass now tags provisioned volumes KubernetesCluster=<cluster> (via tagSpecification_*) for attribution.
• verify-clean-teardown.sh: authoritative EBS check fails the job on any volume tagged KubernetesCluster=<cluster> left after destroy.

This issue now tracks Option A (durable homes) for the future.

Option A — persist + reattach (future feature)

Goal: user /home survives tofu destroy and reattaches on the next standup.

Approach 1 — EBS Retain + static-PV rebind (keep existing EBS):

• Flip gp3 reclaim_policy to Retain; run destroy with KEEP_USER_VOLUMES=1 (already supported) so volumes are not reclaimed.
• On redeploy, dynamic provisioning will NOT auto-rebind old volumes. Need reconciliation that: enumerates retained volumes tagged KubernetesCluster=<cluster> + kubernetes.io/created-for/pvc/name=claim-<user> (CSI adds created-for tags when extra-create-metadata is on), recreates a static PV per volume (pointing at the volumeHandle) and a pre-bound PVC claim-<user> in daskhub before spawn.
• Caveats: per-user machinery (script/controller or generated kubernetes_persistent_volume resources); EBS is AZ-bound so user nodes must schedule in each volume`s AZ.

Approach 2 — EFS for home dirs (recommended for true persistence):

• Single EFS filesystem + EFS CSI driver + per-user access points. Homes live on EFS, independent of cluster lifecycle — survives destroy, reattaches naturally, multi-AZ (no affinity headaches).
• Trade-offs: pricier per-GB, higher small-file latency (notebooks/git), needs EFS CSI addon + mount targets/SGs per subnet.

Cross-cutting requirements (either approach):

• Put the persistent store in a separate, non-destroyable stack (like the route53 zone / sops keys) so tofu destroy never removes it.
• Explicit opt-in — Option A changes destroy semantics (data survives), so it must be deliberate.
• pre-destroy-drain.sh already exposes KEEP_USER_VOLUMES as the hook to switch B->A behavior.