Label GKE GPU node pools so the autoscaler can cold-start from zero#332
Draft
negz wants to merge 1 commit into
Draft
Label GKE GPU node pools so the autoscaler can cold-start from zero#332negz wants to merge 1 commit into
negz wants to merge 1 commit into
Conversation
A GPU node pool with minNodeCount 0 never scaled up. A model's pod claims its GPU through a DRA ResourceClaim, which binds only against a ResourceSlice the NVIDIA DRA driver publishes per running GPU node. At zero nodes there are no slices, and GKE's cluster autoscaler couldn't tell that a new node would satisfy the claim, so it refused to create the first one. The pod stayed pending with "cannot allocate all claims". GKE's autoscaler recognises a DRA-capable node pool by the cloud.google.com/gke-nvidia-gpu-dra-driver=true node label, which lets it model the ResourceSlices such a node would publish when simulating a scale-up. compose-gke-cluster set only modelplane.ai/gpu and modelplane.ai/pool, so the pools were never recognised. This sets the label on every GPU node pool. The getting-started GKE manifests pinned minNodeCount 1 to work around the stall; they now allow minNodeCount 0. Fixes modelplaneai#306. Signed-off-by: Nic Cope <nicc@rk0n.org>
There was a problem hiding this comment.
Pull request overview
This PR enables GKE GPU node pools to cold-start (scale up from minNodeCount: 0) for workloads that request GPUs via Kubernetes DRA ResourceClaims, by applying the node label that GKE’s cluster autoscaler uses to model DRA ResourceSlices during scale-up simulation.
Changes:
- Add the
cloud.google.com/gke-nvidia-gpu-dra-driver=truenode label to GPU node pools composed bycompose-gke-cluster. - Update
compose-gke-clusterunit test expectations to include the new label. - Update getting-started GKE manifests to allow
minNodeCount: 0for GPU pools (removing the prior workaround that pinned it to1).
Reviewed changes
Copilot reviewed 4 out of 4 changed files in this pull request and generated no comments.
| File | Description |
|---|---|
| functions/compose-gke-cluster/function/fn.py | Adds a constant for the autoscaler-recognized label and applies it to GPU node pool labels. |
| functions/compose-gke-cluster/tests/test_fn.py | Updates golden expected node pool labels to include the new autoscaler label. |
| docs/manifests/getting-started/gke/platform.yaml | Changes GPU pool minNodeCount from 1 to 0 in the starter manifest. |
| docs/manifests/getting-started/gke/platform-scale.yaml | Changes GPU pool minNodeCount from 1 to 0 in the scale manifests. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #306.
A GKE GPU node pool with
minNodeCount: 0never scaled up. A model's pod claims its GPU through a DRAResourceClaim, which binds only against aResourceSlicethe NVIDIA DRA driver publishes per running GPU node. At zero nodes there are no slices, and GKE's cluster autoscaler couldn't tell that a new node would satisfy the claim, so it refused to create the first one. The pod stayed pending withcannot allocate all claims.This isn't a DRA architectural limitation. GKE's autoscaler recognises a DRA-capable node pool by the
cloud.google.com/gke-nvidia-gpu-dra-driver=truenode label and uses it to model theResourceSlicessuch a node would publish when simulating a scale-up.compose-gke-clusterset onlymodelplane.ai/gpuandmodelplane.ai/pool, so the pools were never recognised. This sets the label on every GPU node pool.The getting-started GKE manifests pinned
minNodeCount: 1to work around the stall; they now allowminNodeCount: 0.I kept the scope to the autoscaler label. GKE's DRA docs list three more node-pool requirements (
gpu-driver-version=disabled,gke-no-default-nvidia-gpu-device-plugin=true,nvidia.com/gpu.present=true) for the manual-driver DRA path, but this repo runs the DEFAULT-driver path (GKE installs the driver at/home/kubernetes/bin/nvidia, layering the DRA driver on top) and #306 confirms DRA scheduling works once a node is up. The only gap for cold-start was the label.Draft: not yet validated on a real GKE cluster. The scale-from-zero behaviour needs an end-to-end test against a GKE L4/A100 pool at
minNodeCount: 0before this merges.docs/content/getting-started/what-youll-build.caststill shows the oldminNodeCount: 1line in its recording and needs regenerating separately.