GKE node pools don't scale up from zero nodes

[pluna]

What happened?

A GPU node pool with minNodeCount: 0 cannot cold-start. The model claims its GPU through a DRA ResourceClaim, which binds only against a ResourceSlice that the NVIDIA driver publishes per running GPU node. At zero nodes there are no slices, so the cluster autoscaler cannot tell that a new node would meet the claim, and refuses to create the first one. The deployment stays pending:

cannot allocate all claims

kubectl get resourceslices returns nothing while the pool sits at zero, and the L4 slices appear only once a node is running.

The limitation is known and worked around in the getting started examples, which pin minNodeCount: 1. Nothing enforces it: the schema accepts minNodeCount: 0 on a GPU pool with no validation (apis/inferenceclusters/definition.yaml), so anyone who sets it hits this.

How can we reproduce it?

1. Apply a GPU InferenceCluster whose pool has minNodeCount: 0 and maxNodeCount > 0, on GKE with an L4 pool.

2. Wait for the pool to idle to zero nodes.

3. Apply the ModelDeployment and ModelService.

4. The deployment stays pending. Check it:

   kubectl get resourceslices            # empty at zero nodes
   kubectl describe pod <model-pod>      # cannot allocate all claims

Workaround. Set minNodeCount: 1 on every GPU pool so a node, and its ResourceSlice, always exists.

What environment did it happen in?

• Cloud and cluster: GKE
• GPU: NVIDIA L4 (g2-standard)
• Scheduling uses Kubernetes Dynamic Resource Allocation with the NVIDIA DRA driver
• Scaling uses the GKE cluster autoscaler

Modelplane version: v0.1.0-rc.2

[negz]

To some extent this is a limitation of Kubernetes and the cluster autoscaler, and mostly out of our control.

That said it does seem like it should work for the NVIDIA DRA driver on GKE specifically. The problem is the autoscaler uses a label to know that a node pool with zero nodes does in fact support DRA, and we don't add that label today:
https://docs.cloud.google.com/kubernetes-engine/docs/how-to/set-up-dra

The GKE docs indicate that the autoscaler is currently handling this specially for NVIDIA DRA though - e.g. that the autoscaler can't do this for any old DRA driver.

[negz]

AFAICT the autoscaler doesn't support scale-up from zero on AWS yet though. I think kubernetes/autoscaler#7799 tracks that (in the sense that it tracks supporting it in general).