Skip to content

Label GKE GPU node pools so the autoscaler can cold-start from zero#332

Draft
negz wants to merge 1 commit into
modelplaneai:mainfrom
negz:dra-matic-entrance
Draft

Label GKE GPU node pools so the autoscaler can cold-start from zero#332
negz wants to merge 1 commit into
modelplaneai:mainfrom
negz:dra-matic-entrance

Conversation

@negz

@negz negz commented Jul 8, 2026

Copy link
Copy Markdown
Collaborator

Fixes #306.

A GKE GPU node pool with minNodeCount: 0 never scaled up. A model's pod claims its GPU through a DRA ResourceClaim, which binds only against a ResourceSlice the NVIDIA DRA driver publishes per running GPU node. At zero nodes there are no slices, and GKE's cluster autoscaler couldn't tell that a new node would satisfy the claim, so it refused to create the first one. The pod stayed pending with cannot allocate all claims.

This isn't a DRA architectural limitation. GKE's autoscaler recognises a DRA-capable node pool by the cloud.google.com/gke-nvidia-gpu-dra-driver=true node label and uses it to model the ResourceSlices such a node would publish when simulating a scale-up. compose-gke-cluster set only modelplane.ai/gpu and modelplane.ai/pool, so the pools were never recognised. This sets the label on every GPU node pool.

The getting-started GKE manifests pinned minNodeCount: 1 to work around the stall; they now allow minNodeCount: 0.

I kept the scope to the autoscaler label. GKE's DRA docs list three more node-pool requirements (gpu-driver-version=disabled, gke-no-default-nvidia-gpu-device-plugin=true, nvidia.com/gpu.present=true) for the manual-driver DRA path, but this repo runs the DEFAULT-driver path (GKE installs the driver at /home/kubernetes/bin/nvidia, layering the DRA driver on top) and #306 confirms DRA scheduling works once a node is up. The only gap for cold-start was the label.

Draft: not yet validated on a real GKE cluster. The scale-from-zero behaviour needs an end-to-end test against a GKE L4/A100 pool at minNodeCount: 0 before this merges.

docs/content/getting-started/what-youll-build.cast still shows the old minNodeCount: 1 line in its recording and needs regenerating separately.

A GPU node pool with minNodeCount 0 never scaled up. A model's pod
claims its GPU through a DRA ResourceClaim, which binds only against a
ResourceSlice the NVIDIA DRA driver publishes per running GPU node. At
zero nodes there are no slices, and GKE's cluster autoscaler couldn't
tell that a new node would satisfy the claim, so it refused to create
the first one. The pod stayed pending with "cannot allocate all claims".

GKE's autoscaler recognises a DRA-capable node pool by the
cloud.google.com/gke-nvidia-gpu-dra-driver=true node label, which lets
it model the ResourceSlices such a node would publish when simulating a
scale-up. compose-gke-cluster set only modelplane.ai/gpu and
modelplane.ai/pool, so the pools were never recognised.

This sets the label on every GPU node pool. The getting-started GKE
manifests pinned minNodeCount 1 to work around the stall; they now allow
minNodeCount 0.

Fixes modelplaneai#306.

Signed-off-by: Nic Cope <nicc@rk0n.org>

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR enables GKE GPU node pools to cold-start (scale up from minNodeCount: 0) for workloads that request GPUs via Kubernetes DRA ResourceClaims, by applying the node label that GKE’s cluster autoscaler uses to model DRA ResourceSlices during scale-up simulation.

Changes:

  • Add the cloud.google.com/gke-nvidia-gpu-dra-driver=true node label to GPU node pools composed by compose-gke-cluster.
  • Update compose-gke-cluster unit test expectations to include the new label.
  • Update getting-started GKE manifests to allow minNodeCount: 0 for GPU pools (removing the prior workaround that pinned it to 1).

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated no comments.

File Description
functions/compose-gke-cluster/function/fn.py Adds a constant for the autoscaler-recognized label and applies it to GPU node pool labels.
functions/compose-gke-cluster/tests/test_fn.py Updates golden expected node pool labels to include the new autoscaler label.
docs/manifests/getting-started/gke/platform.yaml Changes GPU pool minNodeCount from 1 to 0 in the starter manifest.
docs/manifests/getting-started/gke/platform-scale.yaml Changes GPU pool minNodeCount from 1 to 0 in the scale manifests.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

GKE node pools don't scale up from zero nodes

2 participants