Skip to content

node-autoscaling.md

Latest commit

 

History

History
164 lines (128 loc) · 5.93 KB

node-autoscaling.md

File metadata and controls

164 lines (128 loc) · 5.93 KB
Raw

Node Autoscaling

Automatic horizontal and vertical scaling of Hetzner worker nodes via the Kubernetes Cluster Autoscaler with the Hetzner Cloud provider.

KSail natively installs and manages the Cluster Autoscaler when spec.cluster.autoscaler.node.enabled: true is set in the ksail config.

Architecture

KSail (static baseline)
├── 3 control planes (cx33, 4 vCPU / 8 GB, never autoscaled)
└── 3 static workers (cx33, 4 vCPU / 8 GB, guaranteed minimum, Longhorn storage nodes)

Cluster Autoscaler (dynamic workers, managed by KSail)
├── Pool: autoscale-small  → 0-1 × CX23 (2 vCPU, 4 GB)
├── Pool: autoscale-medium → 0-2 × CX33 (4 vCPU, 8 GB)
├── maxNodesTotal: 10 (3 CPs + 3 workers + headroom for autoscaler nodes)
└── Expander: LeastWaste
  • Horizontal scaling — autoscaler adds workers when pods are Pending due to insufficient resources, and removes underutilized workers after a configurable cooldown.
  • Vertical scaling — multiple node pools with different server types. The LeastWaste expander picks the pool left with the least idle CPU and memory after scheduling the pending pod (Price, which biases toward the cheapest pool, is also available). See cluster-autoscaler FAQ.
  • KSail integration — KSail installs the Cluster Autoscaler Helm chart, generates the worker config secret (cluster-autoscaler-config), and manages the Talos snapshot lifecycle. Node pool configuration lives in ksail.prod.yaml, not in Flux manifests.
  • Storage architecture — autoscaler nodes are compute-only (no Hetzner volume, no Longhorn storage). Static KSail workers have dedicated Hetzner volumes and serve as Longhorn storage nodes. Pods on autoscaler nodes access Longhorn PVCs via the CSI driver (network). The Hetzner Cluster Autoscaler does not support volume attachment.

How new nodes join

  1. Cluster Autoscaler detects Pending pods with unmet resource requests.
  2. It calls the Hetzner API to create a new server using:
    • HCLOUD_IMAGE — a Talos snapshot managed by KSail.
    • HCLOUD_CLOUD_INIT — base64-encoded Talos worker machine config generated by KSail.
  3. The server boots Talos, applies the machine config, and joins the cluster.
  4. Once the node is Ready, pending pods are scheduled.

Configuration

All autoscaler configuration lives in ksail.prod.yaml under spec.cluster.autoscaler.node:

spec:
  cluster:
    autoscaler:
      node:
        enabled: true
        expander: LeastWaste
        maxNodesTotal: 10
        scaleDownUnneededTime: "10m"
        pools:
          - name: autoscale-small
            serverType: cx23
            location: fsn1
            min: 0
            max: 1
          - name: autoscale-medium
            serverType: cx33
            location: fsn1
            min: 0
            max: 2
Field Default Description
enabled false Enable/disable node autoscaling
expander LeastWaste Node selection strategy: Price, LeastWaste, LeastNodes, Random
maxNodesTotal 0 (unlimited) Hard ceiling on total cluster nodes (all types)
scaleDownUnneededTime 10m Time before an underutilized node is eligible for removal
pools[].name — DNS-1123 pool identifier
pools[].serverType — Hetzner server type (e.g., cx23, cx33)
pools[].location — Hetzner datacenter (e.g., fsn1)
pools[].min — Minimum nodes in pool
pools[].max — Maximum nodes in pool

Cost guardrails

  • Hard max per pool — pools[].max caps each pool independently.
  • Hard max total — maxNodesTotal caps the total cluster node count (CPs + static workers + autoscaler workers). Set to 10 (3 CPs + 3 workers = 6 base, plus up to 3 autoscaler nodes from the current pool caps of 1 small + 2 medium).
  • Expander — LeastWaste (current) picks the node group left with the least idle capacity after scheduling; Price instead picks the cheapest eligible group.
  • Scale-down — underutilized nodes are removed after 10 minutes.

Adding more pools

Add a new entry to the pools list in ksail.prod.yaml and run ksail cluster update. KSail updates the Helm release automatically.

Troubleshooting

Autoscaler not scaling up

# Check autoscaler logs
kubectl -n kube-system logs -l app.kubernetes.io/name=cluster-autoscaler --tail=100

# Check for unschedulable pods
kubectl get pods -A --field-selector=status.phase=Pending

# Check autoscaler status ConfigMap
kubectl -n kube-system get cm cluster-autoscaler-status -o yaml

Autoscaler nodes not joining

# Check if the Hetzner server was created
hcloud server list --selector cluster.autoscaler.nodeGroupLabel

# Check Talos bootstrap status (if server IP is known)
talosctl -n <node-ip> health

# Verify the machine config is valid
talosctl validate --config worker.yaml --mode cloud

Cluster rebuild

After a full cluster rebuild (ksail cluster delete + create):

  1. KSail regenerates the worker config secret automatically.
  2. KSail manages the Talos snapshot lifecycle — no manual snapshot creation is needed.

Maintenance

Talos version upgrades

When bumping the Talos version in ksail.prod.yaml:

  1. KSail's snapshot lifecycle manager creates or updates the Talos snapshot automatically during cluster update.
  2. The worker machine config is regenerated to match.

Hetzner server type changes

Hetzner periodically renames or retires server types. Check the Hetzner Cloud changelog and update the pools[].serverType values in ksail.prod.yaml.