[AK-278] Karpenter auto-scaling for clusters

[robinbraemer]

Migrated from Linear: AK-278

GitHub is the canonical source of truth. The Linear issue is marked as a duplicate.

---

Why

Manual cluster scaling is tedious. Karpenter provides intelligent auto-scaling (both up and down), but direct cloud provider integration requires updating drivers on every managed control plane when adding new providers. CNAP API as a centralized broker solves this.

Architecture

Custom Karpenter CloudProvider implementation that proxies to CNAP API instead of calling cloud providers directly.

Karpenter Driver (Go binary):

• Implements cloudprovider.CloudProvider interface
• Runs as sidecar to K0s container in same pod
• Authenticated via injected secret (ServiceAccount token)
• Translates Karpenter CRDs → CNAP API calls
• Handles both upscaling (Create) and downscaling (Delete)

CNAP API (centralized control plane):

• Upscaling: POST /api/v1/clusters/{clusterId}/nodes
• Downscaling: DELETE /api/v1/clusters/{clusterId}/nodes/{nodeID}
• Uses Temporal workflows for idempotent operations
• Handles cloud provider integrations (Hetzner, AWS, etc.)

Benefits:

• Add cloud providers in CNAP API without updating drivers
• Centralized credential management
• Idempotent via Temporal (duplicate requests → same result)
• Reuse existing K0s token generation and worker.sh bootstrap logic

Upscaling Flow

Timeline: Create() → Instance Boots → Node Self-Registers

T+0s:    Karpenter creates NodeClaim (pending pods detected)
         └─> Driver's Create() called
         
T+0s:    CNAP API provisions server:
         1. Generate K0s worker join token (30min expiry)
         2. Build cloud-init with embedded token
         3. Call cloud provider API (Hetzner/AWS/etc)
         4. Create() RETURNS immediately with providerID
         
T+30s:   Server boots, cloud-init executes:
         - curl -sSLf https://get.k0s.sh | sh
         - k0s install worker --token-file /tmp/join-token
         - k0s start
         
T+60s:   K0s worker kubelet registers with API server
         └─> Node appears in cluster automatically
         
T+90s:   Node reports Ready status
         └─> Available for pod scheduling

Key: Nodes self-register via cloud-init - Karpenter watches for them to appear

Cloud-Init Script (CNAP generates)

#!/bin/bash -xe
exec > >(tee /var/log/user-data.log) 2>&1

# Install k0s
curl -sSLf https://get.k0s.sh | sh

# Write join token
cat > /tmp/join-token <<'EOF'
<base64-encoded-k0s-join-token>
EOF

# Join cluster
k0s install worker --token-file /tmp/join-token
k0s start

# Cleanup token
rm -f /tmp/join-token

Lifecycle Controller (Driver watches for nodes)

Karpenter's lifecycle controller watches for nodes to appear and matches them to NodeClaims:

Registration Phase:

• Watches for new Node objects via Kubernetes API
• Matches node to NodeClaim using providerID or instance ID labels
• Updates node labels/taints to match NodePool spec
• Removes karpenter.sh/unregistered taint

Timeout: If node doesn't register within 15 minutes, driver deletes instance and removes NodeClaim

Downscaling Flow

Timeline: Consolidation Decision → Pod Eviction → Instance Termination

T+0s:    Karpenter consolidation detects underutilized node
         └─> Scheduling simulation: pods fit on existing capacity
         
T+0s:    Disruption initiated:
         1. Cordon node (mark unschedulable)
         2. Apply taint: karpenter.sh/disrupted:NoSchedule
         3. Add finalizer: karpenter.sh/termination
         
T+1s:    Pod eviction begins:
         - Group pods by PodDisruptionBudget
         - Evict serially per PDB (respects disruptionsAllowed)
         - Skip: static pods, do-not-evict annotation
         
T+30s:   Pods gracefully terminate (terminationGracePeriodSeconds)
         └─> Controllers reschedule on other nodes
         
T+45s:   All pods evicted, node drained
         └─> Driver's Delete() called
         
T+45s:   CNAP API terminates server:
         - Call cloud provider API (DELETE instance)
         - Mark server status: TERMINATING
         - Return success (async termination)
         
T+60s:   Server terminating (cloud provider side)
         └─> Karpenter retries Delete()
         
T+90s:   Server terminated, no longer exists
         └─> Delete() returns NodeClaimNotFoundError
         
T+90s:   Finalizer removed, NodeClaim deleted
         └─> Cost savings realized

Consolidation Triggers

Karpenter reduces cluster cost through consolidation by detecting:

1. Empty Nodes (highest priority)

• No running non-daemon pods
• Deleted in parallel for faster cost reduction

2. Underutilized Nodes

• Pods can be rescheduled to existing capacity
• Node has been underutilized for consolidateAfter duration (default: 0s)

3. Multi-Node Consolidation

• 2+ nodes can be replaced with single cheaper instance
• Pre-spins replacement before deleting originals

Configuration (NodePool spec):

spec:
  disruption:
    consolidationPolicy: WhenEmptyOrUnderutilized
    consolidateAfter: 5m  # Wait 5min after pod change

Blockers preventing consolidation:

• Pods without controller owners
• Pods with karpenter.sh/do-not-evict annotation
• PodDisruptionBudgets with disruptionsAllowed = 0
• Pods unable to reschedule (affinity/topology constraints)

Pod Eviction Process

Karpenter uses Kubernetes-native Eviction API (not forceful deletion):

1. Grouping: Pods grouped by PodDisruptionBudget

2. Serial eviction per PDB:

• Prevents simultaneous violations of same PDB
• Respects disruptionsAllowed count
• Exponential backoff on 429 (PDB violation)

3. Pod exclusions:

• Static pods (not evicted)
• Succeeded/failed pods
• Pods tolerating karpenter.sh/disrupted taint
• Pods with karpenter.sh/do-not-evict (wait for natural termination)

4. Grace periods:

• Each pod gets terminationGracePeriodSeconds to shut down
• NodeClaim terminationGracePeriod overrides if shorter
• Setting to 0 forces immediate eviction (bypasses PDBs)

Design principle: Karpenter avoids force deletion - users must manually intervene if pods won't terminate.

Termination Timeline

Phase	Duration	Forced By terminationGracePeriod?
Minimum node lifetime	5 minutes	No (ineligible for consolidation)
Consolidation decision	Milliseconds	N/A
Cordoning + taint	Instant	No
Pod eviction	Variable (PDB-dependent)	Yes
Pod termination	Per-pod terminationGracePeriodSeconds	Yes
CloudProvider.Delete()	Cloud-dependent (30-120s)	No
Stabilization window	0-5 minutes before next consolidation	No

Total typical downscaling time: 45-120 seconds

Driver Delete() Implementation

func (c *CNAPCloudProvider) Delete(ctx context.Context, nc *v1.NodeClaim) error {
    serverID := extractServerID(nc.Status.ProviderID)
    
    // Check server state via CNAP API
    server, err := c.client.GetServer(ctx, serverID)
    if err != nil {
        if errors.Is(err, cnap.ErrServerNotFound) {
            // Server already gone — signal success to Karpenter
            return cloudprovider.NewNodeClaimNotFoundError(err)
        }
        return fmt.Errorf("get server: %w", err)
    }
    
    // Check if already terminated
    if server.Status == "TERMINATED" {
        return cloudprovider.NewNodeClaimNotFoundError(
            fmt.Errorf("server %s terminated", serverID)
        )
    }
    
    // If terminating, let it continue (Karpenter will retry)
    if server.Status == "TERMINATING" {
        return nil  // Retry later
    }
    
    // Trigger termination
    err = c.client.DeleteServer(ctx, serverID)
    if err != nil {
        return fmt.Errorf("delete server: %w", err)
    }
    
    // Return success (termination is async)
    // Karpenter will retry until server is gone
    return nil
}

Key behavior:

• First call: Triggers termination, returns nil
• Subsequent retries: Checks state, returns nil while terminating
• Final retry: Server not found, returns NodeClaimNotFoundError
• Karpenter removes finalizer only after NodeClaimNotFoundError

CNAP API Server Termination (Temporal Workflow)

async function terminateServerWorkflow(
  clusterID: string,
  serverID: string
): Promise<void> {
  // Idempotency check
  const server = await ctx.run(() => convex.getServer(serverID));
  if (!server || server.status === 'TERMINATED') {
    return; // Already gone
  }
  
  // Mark as terminating
  await ctx.run(() => convex.updateServer(serverID, { 
    status: 'TERMINATING' 
  }));
  
  // Call cloud provider API to delete instance
  await ctx.run(() => cloudProvider.terminateInstance(server.instanceID));
  
  // Poll for termination completion (with timeout)
  await ctx.run(() => waitForTermination(server.instanceID, {
    timeout: '5m',
    pollInterval: '10s',
  }));
  
  // Mark as terminated and clean up
  await ctx.run(() => convex.updateServer(serverID, { 
    status: 'TERMINATED' 
  }));
  
  // Delete from database after confirmation
  await ctx.run(() => convex.deleteServer(serverID));
}

Finalizer-Based Cleanup

Karpenter uses finalizers to prevent instance leaks:

Finalizer: karpenter.sh/termination

Workflow:

1. Delete request → Webhook adds finalizer
2. Cordon + drain node
3. Driver.Delete() called repeatedly
4. Returns NodeClaimNotFoundError when server gone
5. Finalizer removed
6. Kubernetes deletes node object

Purpose:

• Ensures cloud provider cleanup before Kubernetes deletion
• Prevents orphaned instances if Karpenter crashes
• Users can kubectl delete node instead of manual cloud operations

Error Handling

When Delete() fails:

• Karpenter logs error and retries with exponential backoff
• NodeClaim status annotated with error details
• Finalizer prevents node deletion until cleanup succeeds

When server termination hangs:

• CNAP API timeout after 5 minutes
• Manual intervention required
• Workflow can be canceled/retried via Temporal UI

When Karpenter unavailable during termination:

• Finalizer blocks node deletion
• Re-deploying Karpenter resumes termination
• Or manually remove finalizer (may leak instance - requires manual cleanup)

Implementation Details

CRDs Used

NodePool (karpenter.sh/v1) - Scheduling + consolidation rules:

apiVersion: karpenter.sh/v1
kind: NodePool
spec:
  template:
    spec:
      nodeClassRef:
        group: karpenter.cnap.tech
        kind: CNAPNodeClass
        name: default
      requirements:
        - key: node.kubernetes.io/instance-type
          operator: In
          values: ["m5.large", "m5.xlarge"]
      taints:
        - key: workload
          value: batch
          effect: NoSchedule
      terminationGracePeriod: 1h  # Max time before forced eviction
  disruption:
    consolidationPolicy: WhenEmptyOrUnderutilized
    consolidateAfter: 5m
  limits:
    cpu: "1000"
    memory: "1000Gi"

NodeClaim (karpenter.sh/v1) - Managed by Karpenter:

apiVersion: karpenter.sh/v1
kind: NodeClaim
metadata:
  finalizers:
    - karpenter.sh/termination  # Prevents deletion until cleanup done
status:
  providerID: "cnap:///cluster-abc123/node-xyz789"
  allocatable:
    cpu: "4"
    memory: "16Gi"
  capacity:
    cpu: "4"
    memory: "16Gi"
  nodeName: "ip-10-0-1-42.ec2.internal"

CNAPNodeClass (karpenter.cnap.tech/v1):

apiVersion: karpenter.cnap.tech/v1
kind: CNAPNodeClass
metadata:
  name: default
spec:
  apiEndpoint: https://api.cnap.tech
  secretRef:
    name: cnap-credentials
    key: api-token
  clusterId: cluster-abc123
  region: us-west-2

Scaling Detection

Karpenter detects scaling needs by:

1. Watching Kubernetes API for pods with Unschedulable=True condition
2. Immediate reaction (seconds, not minutes) when kube-scheduler marks pods unschedulable
3. Reading pod constraints: resource requests, selectors, affinity, tolerations
4. Simulated scheduling using kube-scheduler logic to check if pods fit existing nodes
5. Provisioning trigger if no existing nodes match requirements

Authentication

Driver → CNAP API:

• ServiceAccount token mounted at /var/run/secrets/kubernetes.io/serviceaccount/token
• Sent as Authorization: Bearer <token> header
• CNAP API validates token via Convex auth

CNAP API → Cloud Providers:

• Users configure cloud provider credentials in CNAP dashboard
• Stored in Convex, retrieved by Temporal workflow
• Per-cluster or per-workspace configuration

CloudProvider Interface

Must implement (from sigs.k8s.io/karpenter/pkg/cloudprovider):

type CloudProvider interface {
    Create(context.Context, *v1.NodeClaim) (*v1.NodeClaim, error)
    Delete(context.Context, *v1.NodeClaim) error
    Get(context.Context, string) (*v1.NodeClaim, error)
    List(context.Context) ([]*v1.NodeClaim, error)
    GetInstanceTypes(context.Context, *v1.NodePool) ([]*InstanceType, error)
    IsDrifted(context.Context, *v1.NodeClaim) (DriftReason, error)
    Name() string
    GetSupportedNodeClasses() []status.Object
}

CNAP API Endpoints

Create Node (Upscaling):

• POST /api/v1/clusters/{clusterId}/nodes
• Body: { uniqueID, cpu, memory, region, labels, taints }
• Returns immediately after starting provisioning
• Response: { nodeID, providerID, allocatable, capacity }
• Temporal workflow: token generation + cloud-init + cloud API

Delete Node (Downscaling):

• DELETE /api/v1/clusters/{clusterId}/nodes/{nodeID}
• Triggers server termination via cloud provider API
• Returns 200 if termination started
• Returns 404 if already deleted (idempotent)
• Temporal workflow: terminate + poll + cleanup

Get Node:

• GET /api/v1/clusters/{clusterId}/nodes/{nodeID}
• Returns: { nodeID, status, instanceID, providerID }
• Status: RUNNING | TERMINATING | TERMINATED

List Nodes:

• GET /api/v1/clusters/{clusterId}/nodes
• Returns all nodes for cluster with current status

Acceptance Criteria

• Given Karpenter deployed, when pods unschedulable, then driver calls CNAP API
• Given CNAP API receives create request, when workflow runs, then server provisioned with cloud-init
• Given server boots, when cloud-init executes, then node self-registers via K0s token
• Given node appears, when lifecycle controller sees it, then matches to NodeClaim
• Given node underutilized, when consolidation triggered, then pods evicted respecting PDBs
• Given pods evicted, when Delete() called, then CNAP terminates server
• Given server terminating, when Delete() retried, then returns nil until complete
• Given server terminated, when Delete() called, then returns NodeClaimNotFoundError

[robinbraemer]

Migrated from Linear — Robin, 2026-04-08T18:30:16.206Z

Source: https://linear.app/akua/issue/AK-278

but i don't see the message 'hello' to my feed-robin channel 🙁 pls tell me if u haven't seen that in linear comments in this issue.

[robinbraemer]

Migrated from Linear — Robin, 2026-04-08T18:28:49.596Z

Source: https://linear.app/akua/issue/AK-278

can u also see previous comments above? and do what i said?

[robinbraemer]

Migrated from Linear — Robin, 2026-04-08T18:27:49.921Z

Source: https://linear.app/akua/issue/AK-278

please i need an action to test notification ai is working in our discord

[robinbraemer]

Migrated from Linear — Robin, 2026-04-08T18:26:42.189Z

Source: https://linear.app/akua/issue/AK-278

another test u can see this comment. also look at previous comments above pls

[robinbraemer]

Migrated from Linear — Robin, 2026-04-08T10:23:29.304Z

Source: https://linear.app/akua/issue/AK-278

can you see me cortex? if u see this write to feed-robin channel hello

[robinbraemer]

Migrated from Linear — Robin, 2026-04-08T09:58:47.254Z

Source: https://linear.app/akua/issue/AK-278

Test cortex sees this

[robinbraemer]

Migrated from Linear — Robin, 2026-04-08T08:49:22.889Z

Source: https://linear.app/akua/issue/AK-278

Test cortex sees this

[robinbraemer]

Migrated from Linear — Robin, 2026-04-04T20:24:11.027Z

Source: https://linear.app/akua/issue/AK-278

Completed in feat/karpenter-provider PR (cnap#107). Full Karpenter CloudProvider implementation: Go thin proxy → CNAP Compute API → Temporal workflows → HCloud. Includes SA JWT auth, config resolution (CEP-0007), quota enforcement, Go-side backoff cache, default NodePool/NodeClass CRDs, and Helm chart with ApplicationSet.

[robinbraemer]

Migrated from Linear — Robin, 2026-03-16T12:00:00.497Z

Source: https://linear.app/akua/issue/AK-278

Karpenter CloudProvider Contract + Hetzner SDK Analysis

Deep-dived into the actual Karpenter source and Hetzner SDKs to understand exactly what our CNAP Karpenter driver + Temporal activities need to implement.

---

Karpenter CloudProvider Contract — What CNAP Must Implement

Create(ctx, *NodeClaim) (*NodeClaim, error)

Must return a NodeClaim with:

• Status.ProviderID — Critical. This is the ONLY way Karpenter matches NodeClaims to Nodes. The kubelet must register with the same providerID. For CNAP: cnap:///<clusterId>/<serverId> or hcloud:///<serverId>.
• Status.Capacity — cpu, memory, pods, ephemeral-storage
• Status.Allocatable — capacity minus overhead
• Labels: node.kubernetes.io/instance-type, topology.kubernetes.io/zone, karpenter.sh/capacity-type, kubernetes.io/arch, kubernetes.io/os

Timeouts if node never registers:

• 5 min launch timeout (Create → Launched condition)
• 15 min registration timeout (Launched → Node appears with matching providerID)
• If either expires, Karpenter deletes the NodeClaim and calls Delete() to clean up

Delete(ctx, *NodeClaim) error

• Return nil while termination in progress (Karpenter retries)
• Return NodeClaimNotFoundError when server gone (finalizer removed, cleanup complete)
• Return error on transient failure (retried with backoff)

GetInstanceTypes(ctx, *NodePool) ([]*InstanceType, error)

Each InstanceType needs:

• Name — e.g., "cpx21", "cx42", "cax11"
• Requirements — must define ALL well-known labels (arch, os, zone, capacity-type, instance-type)
• Offerings — per-zone availability + hourly price (USD). Karpenter sorts by price to pick cheapest.
• Capacity — cpu, memory, pods, ephemeral-storage, GPUs
• Overhead — KubeReserved + SystemReserved + EvictionThreshold

Source for this data: hcloud-go SDK provides client.ServerType.All() with Cores, Memory, Disk, CPUType, Architecture, and per-location Pricings (hourly/monthly prices). client.Pricing.Get() has full pricing. This maps directly to Karpenter's InstanceType struct.

IsDrifted(ctx, *NodeClaim) (DriftReason, error)

Called by the NodeClaim disruption controller. We'd use this for:

• Server type changed in CNAPNodeClass → drift all nodes
• k0s version changed → drift (triggers rolling replacement — this is the upgrade mechanism!)
• Image/OS changed → drift

List(ctx) ([]*NodeClaim, error)

Called every 2 minutes by garbage collection. Must query the CNAP API (or Hetzner directly) to return all managed instances. Used to detect orphaned cloud instances (NodeClaim exists but instance doesn't → cleanup).

Key architectural insight: Karpenter never calls the cloud during scheduling. It uses cached GetInstanceTypes() data. Only Create() and Delete() make cloud API calls, outside the scheduling loop.

---

hcloud-go SDK — Key Patterns for Temporal Activities

Critical: Nearly every mutating operation is async. The API returns an Action (or []*Action) immediately. You must poll via client.Action.WaitFor(ctx, actions...) until completion. Default poll: 500ms intervals.

Server creation returns MULTIPLE actions:

result.Action       — main creation
result.NextActions  — follow-up (network attachment, etc.)

Must wait for ALL. Server starts in initializing status.

Recommended Temporal activity pattern:

activity calls SDK → gets Action IDs → calls WaitFor(ctx) → returns result

The WaitFor respects context.Context, so Temporal's heartbeat timeout and cancellation propagate naturally. For server creation (~30-60s), use WaitForFunc with heartbeat callback.

Built-in retries in SDK: Automatically retries on rate limit, conflict, timeout, 502/504 — up to 5 times with exponential backoff + jitter (capped at 60s). So Temporal retry policies handle persistent failures, SDK handles transient ones.

Key Temporal activities to build:

Activity	SDK Call	Sync/Async	Notes
CreateServer	client.Server.Create() + WaitFor()	Async (30-60s)	Heartbeat during wait
DeleteServer	client.Server.DeleteWithResult() + WaitFor()	Async
ShutdownServer	client.Server.Shutdown() + WaitFor()	Async	Before delete
RebootServer	client.Server.Reboot() + WaitFor()	Async
CreateNetwork	client.Network.Create()	Sync	No Action
DeleteNetwork	client.Network.Delete()	Sync
CreateLoadBalancer	client.LoadBalancer.Create() + WaitFor()	Async
DeleteLoadBalancer	client.LoadBalancer.Delete()	Sync
AddLBTarget	client.LoadBalancer.AddServerTarget() + WaitFor()	Async
CreatePlacementGroup	client.PlacementGroup.Create()	Optional Action
ListServerTypes	client.ServerType.All()	Sync	Cache for GetInstanceTypes()
GetPricing	client.Pricing.Get()	Sync	Cache for GetInstanceTypes()
CreateFirewall	client.Firewall.Create() + WaitFor()	Async	CAPH doesn't do this, but we should
SetFirewallRules	client.Firewall.SetRules() + WaitFor()	Async

Idempotency patterns from CAPH to copy:

• Label-based ownership: cnap-cluster-<id>=owned + cnap-machine-name=<name> on all resources
• Two-tier server lookup: first by ID (from providerID), fallback to label selector
• Treat "already attached" as success (network, LB target)
• Treat "not found" on delete as success
• Graceful server deletion: remove from LB → shutdown → wait for off → delete

---

Robot API for Bare Metal — What We'd Build

No official TypeScript client exists. The API is simple enough to build our own:

• Base URL: https://robot-ws.your-server.de
• Auth: HTTP Basic (separate credentials from hcloud token, created in Robot panel)
• POST bodies: application/x-www-form-urlencoded (NOT JSON)
• Responses: JSON
• Rate limit: HTTP 403 with RATE_LIMIT_EXCEEDED error code

Only ~11 endpoints needed (what CAPH uses):

Endpoint	Method	Purpose
/server/{id}	GET	Get server details + IPs
/server/{id}	POST	Set server name
/boot/{id}/rescue	GET	Check rescue mode status
/boot/{id}/rescue	POST	Activate rescue (pass SSH key fingerprint)
/boot/{id}/rescue	DELETE	Deactivate rescue
/reset/{id}	GET	Get available reset types
/reset/{id}	POST	Trigger reset (type: sw/hw/power/man)
/key	GET	List SSH keys
/key	POST	Upload SSH key

All calls are synchronous HTTP. The physical effects (reboots) are fire-and-forget — you trigger the reboot, it returns immediately, and you must poll via SSH to verify the server actually rebooted.

installimage IS pre-installed in rescue. CAPH ships its own copy anyway (as .tgz uploaded via SSH), but we could use the built-in one.

SSH key fingerprints: Robot API uses MD5 format (15:28:b0:03:95:...). Upload key via POST /key, get fingerprint back, pass fingerprint to POST /boot/{id}/rescue as authorized_key.

---

How It All Fits Together

                     Karpenter (Go sidecar)
                     ├─ GetInstanceTypes() → cached from CNAP API
                     ├─ Create(NodeClaim) → POST /api/v1/clusters/{id}/nodes
                     ├─ Delete(NodeClaim) → DELETE /api/v1/clusters/{id}/nodes/{nodeId}
                     ├─ IsDrifted() → check CNAPNodeClass version hash
                     └─ List() → GET /api/v1/clusters/{id}/nodes

                              ↓ CNAP API

                     Temporal Workflows
                     ├─ provisionNode workflow
                     │   ├─ createServer (hcloud-go)        ← async, WaitFor
                     │   ├─ attachToNetwork (hcloud-go)     ← async, WaitFor
                     │   ├─ generateK0sJoinToken            ← k0s API
                     │   └─ [server boots, cloud-init runs, node self-registers]
                     │
                     ├─ provisionBareMetalNode workflow
                     │   ├─ activateRescue (Robot API)      ← sync
                     │   ├─ rebootServer (Robot API)        ← fire-and-forget
                     │   ├─ waitForSSH                      ← poll
                     │   ├─ gatherHardware (SSH)            ← sync
                     │   ├─ runInstallImage (SSH)           ← background process
                     │   ├─ waitForInstallComplete (SSH)    ← poll
                     │   ├─ rebootToOS (SSH)                ← fire-and-forget
                     │   ├─ waitForCloudInit (SSH)          ← poll
                     │   └─ [node self-registers via k0s token in cloud-init]
                     │
                     ├─ terminateNode workflow
                     │   ├─ removeFromLB (hcloud-go)
                     │   ├─ shutdownServer (hcloud-go)
                     │   ├─ waitForOff (poll)
                     │   └─ deleteServer (hcloud-go)
                     │
                     └─ remediateNode workflow
                         ├─ sshReboot → sleep(5m) → verify
                         ├─ softwareReboot → sleep(10m) → verify
                         ├─ hardwareReboot → sleep(10m) → verify
                         └─ markPermanentFailure → notify

What CAPH Doesn't Do That We Should

1. Firewall management — CAPH explicitly skips this. We should auto-create firewalls per cluster with rules for: k0s API (443), konnectivity (8132), kubelet (10250), node ports, Cilium WireGuard (51871). Use client.Firewall.SetRules() + label-selector targeting.
2. DNS management — Neither CAPH nor k0smotron manages DNS. We need to create/update DNS records for the SNI proxy hostnames.
3. Monitoring integration — Server metrics via client.Server.GetMetrics() for cost/utilization dashboards.

[robinbraemer]

Migrated from Linear — Robin, 2026-03-16T11:54:31.811Z

Source: https://linear.app/akua/issue/AK-278

Deep Comparison: CNAP KaaS vs k0smotron vs CAPH (Hetzner CAPI)

Cloned and analyzed both repos against our current gitops-kaas + Temporal architecture. TL;DR: Our current approach is more powerful for multi-tenant managed K8s. Cherry-pick specific implementations from CAPH rather than adopting the frameworks.

---

Where CNAP is Stronger

Capability	CNAP	k0smotron / CAPH
Multi-tenant routing	SNI proxy + PP2 DaemonSet — multiplexes ALL clusters through one endpoint	1 Service/LB per cluster ($$$, management nightmare at 100+ clusters)
GitOps-native	Git commit → ArgoCD ApplicationSet → Helm. Audit trail, PR reviews, git revert rollback	Operator-driven, state in etcd. No git audit, no PR workflow
Orchestration	Temporal: compensation stacks, idempotency keys, signal coordination, retry policies per activity	K8s reconcilers: giant switch statements on status conditions, no rollback
Full-stack per-cluster	Cilium, CoreDNS, Gateway API, metrics-server deployed per cluster via ApplicationSets	BYO CNI, BYO DNS — nothing managed
App marketplace	Registry proxy, Helm chart generation, install workflows	N/A
Incluster API proxy	Transparent kubernetes:443 replacement via PP2. Zero config for child pods	FRP tunneling or direct Service access

Where k0smotron is Stronger

• Upgrade strategies: InPlace (autopilot binary swap), Recreate, RecreateDeleteFirst — we have manual git commits
• HA control planes: Separate etcd StatefulSet with member management, defrag CronJob — we run single-replica SQLite
• Certificate rotation: Built-in detection and renewal — we generate once
• JoinTokenRequest CRD: Declarative token management — our workerconfig is static

Where CAPH is Stronger (and worth cherry-picking)

See next section.

---

Cherry-Pick Analysis: What to Self-Build Using CAPH as Reference

1. Bare Metal Provisioning State Machine

CAPH implements a 7-state machine: None → Preparing → Registering → PreProvisioning → ImageInstalling → EnsureProvisioned → Provisioned

Robot API calls to replicate:

• GET /server/{id} — server info + IPs
• POST /boot/{id}/rescue — activate rescue mode (pass SSH key fingerprint)
• POST /reset/{id} — reboot (types: sw, hw, power, man)
• GET/POST /key — SSH key management
• POST /server/{id} — set server name

SSH commands in rescue system:

• Hardware discovery: lsblk, lscpu, /proc/meminfo, NIC enumeration via /sys/class/net
• Disk health: smartctl checks on target WWNs
• installimage: Write /autosetup config, run /root/hetzner-installimage/installimage -a -c /autosetup -x /root/post-install.sh
• Cloud-init injection: Post-install script writes to /var/lib/cloud/seed/nocloud-net/user-data

Why Temporal is BETTER here than CAPH's reconciler:
CAPH's reconciler must re-derive its position from persisted status every loop — hostname checks, error type tracking, timeout calculations. A Temporal workflow is just linear: activateRescue() → reboot() → waitForSSH() → gatherHardware() → installImage() → waitForCompletion() → reboot() → waitForOS() → waitForCloudInit(). All intermediate state is implicit in workflow position.

What to build as Temporal activities (direct mapping):

• ActivateRescueMode — Robot API: delete old rescue, set rescue with SSH key
• RebootServer — Robot API: POST /reset/{id} with type parameter
• WaitForSSHReachable — Poll SSH connection with retry policy
• GatherHardwareDetails — SSH: run lsblk/lscpu/meminfo, return structured data
• CheckDiskHealth — SSH: smartctl on target WWNs
• RunInstallImage — SSH: write autosetup + post-install script, start background process
• WaitForInstallImageComplete — SSH: poll ps aux | grep installimage, check logs
• WaitForCloudInitComplete — SSH: poll cloud-init status
• InjectCloudInitData — Generate post-install script with k0s join token in nocloud-net seed

What to skip from CAPH:

• installimage.tgz bundling (they embed the binary in the controller image) — Hetzner rescue already has it
• PooledRemoteMachine CRD — we don't need a machine pool abstraction, Temporal can manage server inventory
• The actionResult type system — Temporal has native error handling

2. Health Remediation / Reboot Escalation Ladder

CAPH has a 3-tier escalation:

Tier	Method	Timeout	Robot API
1	SSH reboot (sshClient.Reboot())	5 min	N/A
2	Software reboot (CTRL+ALT+DEL)	10 min	POST /reset/{id} type=sw
3	Hardware reboot (power cycle)	10 min	POST /reset/{id} type=hw
—	Permanent failure	—	Manual intervention required

Verification after each tier: SSH back in, check hostname matches expected.

What to build:
A Temporal workflow remediateNode with timer-based escalation:

sshReboot() → sleep(5m) → verifyHostname()
  ↓ (failed)
softwareReboot() → sleep(10m) → verifyHostname()
  ↓ (failed)
hardwareReboot() → sleep(10m) → verifyHostname()
  ↓ (failed)
markPermanentFailure() → notify()

This is actually simpler in Temporal than in CAPH's reconciler where they track ErrorTypeSSHRebootTriggered, ErrorTypeSoftwareRebootTriggered, ErrorTypeHardwareRebootTriggered and compute timeouts from LastUpdated timestamps.

Additional CAPH patterns worth copying:

• connectionRefusedTimeout (10 min) — separate from reboot timeouts
• rebootWaitTime (15s) — grace period after reboot before checking
• Exponential backoff for repeated failures: 1m → 2m → 4m → ... → 8.5h (cap at errorCount=9)
• Not-available reboot type detection: skip software if HasSoftwareReboot() returns false

3. HCloud Resource Management

CAPH wraps the hcloud-go SDK with idempotency patterns we should copy:

Server lifecycle:

• Create: hcloud.ServerCreateOpts with name, labels, image (by name OR label — snapshots have no name), location, type, userdata, SSH keys, network, placement group
• Find: Two-tier lookup — first by provider ID, fallback to label selector caph-cluster-<name>=owned + machine.caph-name=<name>
• Delete: Remove from LB → graceful shutdown → wait → delete (handles stuck "off" state with power-on + timeout)
• Image resolution: Architecture-aware — filters by server type arch (ARM vs x86)

Load balancer:

• Auto-create for control plane: type (lb11/lb21/lb31), algorithm (round_robin/least_connections)
• Service diff: Compare desired vs actual by listen port, add/remove declaratively
• Protected LBs: Only remove ownership label, never delete
• Pre-existing LBs: Find by name, validate no other ownership labels, adopt

Network:

• Single network + subnet per cluster, find-or-create by label selector
• Auto-attach servers post-creation

Placement groups:

• Diff-based reconciliation: desired names vs existing, create missing, delete orphaned
• Spread type only (HA)

Universal patterns to replicate:

• Label-based ownership: cnap-cluster-<id>=owned on all resources for idempotent find-or-create
• Rate limit handling: Check ErrorCodeRateLimitExceeded on every API call, set condition, requeue with backoff
• Idempotent operations: ErrorCodeServerAlreadyAttached = success, ErrorCodeTargetAlreadyDefined = success, ErrorCodeNotFound on delete = success
• Graceful server deletion: Always shutdown before delete, track shutdown state, timeout after 10 min

What to build as Temporal activities:

• CreateServer — straightforward (build opts, call API, return ID)
• DeleteServer — workflow: removeFromLB() → shutdown() → waitForOff() → delete()
• CreateNetwork / DeleteNetwork — single API calls
• CreateLoadBalancer / DeleteLoadBalancer — single API calls
• AttachServerToNetwork — idempotent (already-attached = success)
• ReconcileLBTargets — diff-based add/remove

What to skip from CAPH:

• Bare metal LB target support (IP-based instead of server-based) — only needed if mixing bare metal + cloud
• CSR auto-approval controller — k0s handles this differently
• Firewall management — CAPH doesn't do it either, we'll need our own

---

Architecture Decision: Don't Adopt CAPI/k0smotron

Why not:

1. CAPH + k0smotron together = 3 operators + CRD sprawl, never battle-tested as a stack
2. CAPI is an abstraction for cloud-agnostic provisioning — CNAP is Hetzner-native initially, abstraction buys nothing
3. Replacing Temporal with reconcilers loses compensation, idempotency, and signal coordination
4. Replacing GitOps-driven cluster creation with CRD-driven loses audit trail and PR workflow
5. SNI proxy architecture (our key differentiator) has no equivalent in either project

What to cherry-pick from k0smotron (later):

• autopilot Plan CRD for InPlace k0s binary upgrades (can use autopilot directly without k0smotron)
• HA etcd StatefulSet pattern when we need HA control planes
• Certificate rotation timing logic

Trigger for re-evaluation: When managing 3+ clusters and doing k0s upgrades more than once a quarter.

[robinbraemer]

Migrated from Linear — Robin, 2026-03-16T11:09:01.390Z

Source: https://linear.app/akua/issue/AK-278

Worker config maintenance across K8s version upgrades

Current state (gitops-kaas)

The workerconfigs component maintains a static copy of k0s's auto-generated worker-config-default-{version} ConfigMap. This exists because the k0s controller's built-in worker-config reconciler is disabled (--disable-components=worker-config) — we manage it via gitops so apiServerAddresses points to the external SNI endpoint instead of the internal pod IP.

Problem: The ConfigMap has 1.33 hardcoded in 7+ places (names, labels, RBAC) with no templating.

What changes across k0s versions

Checked the k0s source (pkg/component/controller/workerconfig/reconciler.go):

1. ConfigMap name: worker-config-default-{major}.{minor} — always changes
2. Kubelet config content: k0s may change feature gate defaults, TLS cipher suites, add/deprecate kubelet fields, or bump the pause image between minor versions
3. The content is NOT just a version bump — new k0s versions may emit different kubelet configuration

Upgrade procedure

Simply find-replacing 1.33 → 1.34 is not safe. The correct process:

1. Run a temporary k0s v1.34 controller with worker-config enabled
2. Extract the generated ConfigMap: kubectl get cm worker-config-default-1.34 -n kube-system -o yaml
3. Replace our static copy with the extracted one, updating apiServerAddresses to use the template value
4. Update RBAC names/labels to match

Recommended improvement

Template the version number via a Helm value (k0sMinorVersion) so names/labels are automatic. But the kubelet config JSON still needs manual extraction per version — consider a CI job that diffs the generated ConfigMap against our static copy on k0s upgrades.

Karpenter interaction

With Karpenter, drift detection (IsDrifted()) can trigger rolling node replacement after the workerconfig is updated. But workers joining with an old k0s binary + old workerconfig would conflict. The upgrade order must be:

1. Update workerconfig ConfigMap (new version)
2. Update k0s controller (new version)
3. Update CNAPNodeClass k0sVersion (triggers Karpenter drift → new workers join with new binary + read new ConfigMap)

[robinbraemer]

Migrated from Linear — Robin, 2026-03-16T11:02:55.681Z

Source: https://linear.app/akua/issue/AK-278

Bonus: K8s version upgrades via drift detection

The IsDrifted() method in the CloudProvider interface enables automatic K8s version upgrades. When the desired K0s version changes in CNAPNodeClass, Karpenter detects drift on all nodes and does a rolling replacement — provision new, drain old, delete old. This makes CNAP-52 (allow upgrading K8s version) nearly free once Karpenter is in place.

See comment on CNAP-52 for the full upgrade flow.