fix(gpu): tear down cuda-lts VHD prebake on GRID nodes [backport #8919 to v20260709] - #8924
Merged
ganeshkumarashok merged 2 commits intoJul 13, 2026
Conversation
ganeshkumarashok
requested review from
AbelHu,
Devinwong,
SriHarsha001,
awesomenix,
calvin197,
cameronmeissner,
djsly,
karenychen,
lilypan26,
mxj220,
pdamianov-dev,
phealy,
r2k1,
runzhen,
sulixu,
surajssd,
timmy-wright,
titilambert,
xuexu6666 and
zachary-bailey
as code owners
July 13, 2026 22:42
Contributor
There was a problem hiding this comment.
Pull request overview
Backports the GPU provisioning fix from #8919 onto official/v20260709 to prevent NVML driver/library mismatches on GRID (A10 / NVv5) nodes booting from CUDA-prebaked shared Ubuntu VHDs by tearing down a mismatched CUDA prebake before the GRID managed driver install.
Changes:
- Add
cleanUpGridNodeCudaPrebaketo remove CUDA-prebaked driver artifacts when the node’s requested managed driver kind is GRID and the VHD marker kind is non-GRID/legacy. - Invoke the new cleanup early in
ensureGPUDrivers(before the config/validate dispatch) so it applies to both managed driver install paths. - Add ShellSpec coverage for the new decision logic (teardown vs no-op, including legacy marker and non-Ubuntu no-op).
Reviewed changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated no comments.
| File | Description |
|---|---|
| parts/linux/cloud-init/artifacts/cse_config.sh | Introduces cleanUpGridNodeCudaPrebake and runs it early in ensureGPUDrivers on Ubuntu to avoid GRID vs CUDA prebake collisions. |
| spec/parts/linux/cloud-init/artifacts/cse_config_spec.sh | Adds ShellSpec tests validating teardown/no-op behavior for GRID vs CUDA marker-kind combinations and non-Ubuntu behavior. |
Contributor
Changes cached containers or packages on windows VHDsPlease get a Windows SIG member to approve. The following dif file shows any additions or deletions from what will be cached on windows VHDs organised by VHD type.
diff --git a/vhd_files/2025-gen2-tl.txt b/vhd_files/2025-gen2-tl.txt
index a901175..5fc7b9e 100644
--- a/vhd_files/2025-gen2-tl.txt
+++ b/vhd_files/2025-gen2-tl.txt
@@ -34 +33,0 @@ HKLM:\SYSTEM\CurrentControlSet\Policies\Microsoft\FeatureManagement\Overrides\18
-HKLM:\SYSTEM\CurrentControlSet\Policies\Microsoft\FeatureManagement\Overrides\908168846=1
diff --git a/vhd_files/2025-gen2.txt b/vhd_files/2025-gen2.txt
index 4d7aa19..5c054c3 100644
--- a/vhd_files/2025-gen2.txt
+++ b/vhd_files/2025-gen2.txt
@@ -34 +33,0 @@ HKLM:\SYSTEM\CurrentControlSet\Policies\Microsoft\FeatureManagement\Overrides\18
-HKLM:\SYSTEM\CurrentControlSet\Policies\Microsoft\FeatureManagement\Overrides\908168846=1
diff --git a/vhd_files/2025.txt b/vhd_files/2025.txt
index 1441011..1fe4b78 100644
--- a/vhd_files/2025.txt
+++ b/vhd_files/2025.txt
@@ -34 +33,0 @@ HKLM:\SYSTEM\CurrentControlSet\Policies\Microsoft\FeatureManagement\Overrides\18
-HKLM:\SYSTEM\CurrentControlSet\Policies\Microsoft\FeatureManagement\Overrides\908168846=1 |
Comment on lines
+1345
to
+1352
| # Keep only when the prebake is explicitly grid (matches this grid node). An empty marker kind is | ||
| # a legacy cuda prebake; a "cuda" marker is a cuda prebake -- both mismatch a grid node, tear down. | ||
| if [ "${m_kind}" = "grid" ]; then | ||
| return 0 | ||
| fi | ||
| echo "AKS_GPU_PREBAKE event=grid_cuda_prebake_teardown driver_type=${NVIDIA_GPU_DRIVER_TYPE:-} marker_kind=${m_kind:-none} node_kind=${node_kind} action=teardown" | ||
| cleanUpPrebakedGPUDriver | ||
| } |
sulixu
approved these changes
Jul 13, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Backport of #8919 to the
official/v20260709release branch (prebake-enabled VHD).Fixes
nvidia-smifailing with "Failed to initialize NVML: Driver/library version mismatch" on agentpool GRID / converged A10 (NVv5) nodes and NAP GRID nodes booting the CUDA-prebaked shared Ubuntu VHD.Root cause: the shared Ubuntu VHD prebakes only the
cuda(-lts)driver + a DKMS marker. The existingcleanUpPrebakedGPUDriveronly runs on nodes that do not install a managed driver. A GRID node installs thegriddriver on top of the stale prebakedcudamodule +/usr/bin/lib64libs → NVML mismatch.Fix:
cleanUpGridNodeCudaPrebake, invoked fromensureGPUDriversbefore theconfigGPUDrivers/validateGPUDriversdispatch, tears down the prebake when the node installsgridand the marker is not grid (cuda / legacy kind-less). Pure driver-kind mismatch; CUDA nodes untouched.Cherry-pick of the squashed main commit
947d2def64— verified the introduced patch body is identical to what merged onmain.Related
main: fix(gpu): tear down cuda-lts VHD prebake on GRID nodes before managed install #8919 (merged)official/v20260702backport: fix(gpu): tear down cuda-lts VHD prebake on GRID nodes [backport #8919 to v20260702] #8921Test plan
cleanUpGridNodeCudaPrebake(incl. non-Ubuntu no-op)🤖 Generated with Claude Code