fix(gpu): tear down cuda-lts VHD prebake on GRID nodes [backport #8919 to v20260702] by ganeshkumarashok · Pull Request #8921 · Azure/AgentBaker

ganeshkumarashok · 2026-07-13T21:56:47Z

Summary

Backport of #8919 to the official/v20260702 release branch (the live prebake-enabled VHD).

Fixes nvidia-smi failing with "Failed to initialize NVML: Driver/library version mismatch" on agentpool GRID / converged A10 (NVv5) nodes and NAP GRID nodes booting the CUDA-prebaked shared Ubuntu VHD.

Root cause: the shared Ubuntu VHD prebakes only the cuda(-lts) driver + a DKMS marker. The existing cleanUpPrebakedGPUDriver only runs on nodes that do not install a managed driver. A GRID node installs the grid driver on top of the stale prebaked cuda module + /usr/bin/lib64 libs → NVML mismatch.

Fix: cleanUpGridNodeCudaPrebake, invoked from ensureGPUDrivers before the configGPUDrivers/validateGPUDrivers dispatch, tears down the prebake when the node installs grid and the marker is not grid (cuda / legacy kind-less). Pure driver-kind mismatch; CUDA nodes untouched.

Cherry-pick of the squashed main commit 947d2def64 — the introduced patch is identical (only git blob-index headers differ, due to surrounding file drift between main and this branch).

Relationship to #8918

This supersedes the earlier hand-built 0702 PR #8918 with an exact cherry-pick of what merged to main, so the release branch carries byte-identical GRID-fix content. #8918 will be closed.

Test plan

shellcheck clean; ShellSpec coverage for cleanUpGridNodeCudaPrebake (incl. non-Ubuntu no-op)
e2e on an agentpool GRID/A10 node + a NAP GRID node booting the prebaked VHD

🤖 Generated with Claude Code

… install (#8919)

Copilot

Pull request overview

Backports the GPU provisioning fix from #8919 to official/v20260702 to prevent CUDA-prebaked Ubuntu VHD remnants from colliding with managed GRID driver installs (avoids NVML driver/library mismatch on GRID/A10/NAP GRID nodes).

Changes:

Add cleanUpGridNodeCudaPrebake to tear down mismatched CUDA prebake state on Ubuntu GRID nodes before the GPU driver install/validate dispatch.
Invoke this cleanup early in ensureGPUDrivers so both configGPUDrivers and validateGPUDrivers flows are covered.
Add ShellSpec coverage validating teardown/no-op behavior across marker kinds, driver types, and OS.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated no comments.

File	Description
parts/linux/cloud-init/artifacts/cse_config.sh	Adds and wires `cleanUpGridNodeCudaPrebake` into `ensureGPUDrivers` to remove mismatched CUDA prebake before GRID driver paths.
spec/parts/linux/cloud-init/artifacts/cse_config_spec.sh	Adds ShellSpec tests covering teardown decision logic and expected no-op conditions.

github-actions · 2026-07-13T22:03:45Z

Changes cached containers or packages on windows VHDs

Please get a Windows SIG member to approve.

The following dif file shows any additions or deletions from what will be cached on windows VHDs organised by VHD type.

Additions are new things cached.
Deletions are things no longer cached.

diff --git a/vhd_files/2022-containerd-gen2.txt b/vhd_files/2022-containerd-gen2.txt
index e187816..db10c9e 100644
--- a/vhd_files/2022-containerd-gen2.txt
+++ b/vhd_files/2022-containerd-gen2.txt
@@ -129,0 +130 @@ mcr.microsoft.com/oss/v2/kubernetes-csi/secrets-store/driver:v1.5.4
+mcr.microsoft.com/oss/v2/kubernetes/azure-cloud-node-manager:v1.33.13-windows-hpc-1
@@ -131 +132 @@ mcr.microsoft.com/oss/v2/kubernetes/azure-cloud-node-manager:v1.33.14-windows-hp
-mcr.microsoft.com/oss/v2/kubernetes/azure-cloud-node-manager:v1.33.15-windows-hpc-1
+mcr.microsoft.com/oss/v2/kubernetes/azure-cloud-node-manager:v1.34.10-windows-hpc-1
@@ -133 +134 @@ mcr.microsoft.com/oss/v2/kubernetes/azure-cloud-node-manager:v1.34.11-windows-hp
-mcr.microsoft.com/oss/v2/kubernetes/azure-cloud-node-manager:v1.34.12-windows-hpc-1
+mcr.microsoft.com/oss/v2/kubernetes/azure-cloud-node-manager:v1.35.5-windows-hpc-1
@@ -135 +136 @@ mcr.microsoft.com/oss/v2/kubernetes/azure-cloud-node-manager:v1.35.6-windows-hpc
-mcr.microsoft.com/oss/v2/kubernetes/azure-cloud-node-manager:v1.35.7-windows-hpc-1
+mcr.microsoft.com/oss/v2/kubernetes/azure-cloud-node-manager:v1.36.1-windows-hpc-1
@@ -137 +137,0 @@ mcr.microsoft.com/oss/v2/kubernetes/azure-cloud-node-manager:v1.36.2-windows-hpc
-mcr.microsoft.com/oss/v2/kubernetes/azure-cloud-node-manager:v1.36.3-windows-hpc-1
diff --git a/vhd_files/2022-containerd.txt b/vhd_files/2022-containerd.txt
index 760db3c..94de353 100644
--- a/vhd_files/2022-containerd.txt
+++ b/vhd_files/2022-containerd.txt
@@ -129,0 +130 @@ mcr.microsoft.com/oss/v2/kubernetes-csi/secrets-store/driver:v1.5.4
+mcr.microsoft.com/oss/v2/kubernetes/azure-cloud-node-manager:v1.33.13-windows-hpc-1
@@ -131 +132 @@ mcr.microsoft.com/oss/v2/kubernetes/azure-cloud-node-manager:v1.33.14-windows-hp
-mcr.microsoft.com/oss/v2/kubernetes/azure-cloud-node-manager:v1.33.15-windows-hpc-1
+mcr.microsoft.com/oss/v2/kubernetes/azure-cloud-node-manager:v1.34.10-windows-hpc-1
@@ -133 +134 @@ mcr.microsoft.com/oss/v2/kubernetes/azure-cloud-node-manager:v1.34.11-windows-hp
-mcr.microsoft.com/oss/v2/kubernetes/azure-cloud-node-manager:v1.34.12-windows-hpc-1
+mcr.microsoft.com/oss/v2/kubernetes/azure-cloud-node-manager:v1.35.5-windows-hpc-1
@@ -135 +136 @@ mcr.microsoft.com/oss/v2/kubernetes/azure-cloud-node-manager:v1.35.6-windows-hpc
-mcr.microsoft.com/oss/v2/kubernetes/azure-cloud-node-manager:v1.35.7-windows-hpc-1
+mcr.microsoft.com/oss/v2/kubernetes/azure-cloud-node-manager:v1.36.1-windows-hpc-1
@@ -137 +137,0 @@ mcr.microsoft.com/oss/v2/kubernetes/azure-cloud-node-manager:v1.36.2-windows-hpc
-mcr.microsoft.com/oss/v2/kubernetes/azure-cloud-node-manager:v1.36.3-windows-hpc-1
diff --git a/vhd_files/2025-gen2-tl.txt b/vhd_files/2025-gen2-tl.txt
index a901175..4f3c531 100644
--- a/vhd_files/2025-gen2-tl.txt
+++ b/vhd_files/2025-gen2-tl.txt
@@ -34 +33,0 @@ HKLM:\SYSTEM\CurrentControlSet\Policies\Microsoft\FeatureManagement\Overrides\18
-HKLM:\SYSTEM\CurrentControlSet\Policies\Microsoft\FeatureManagement\Overrides\908168846=1
@@ -60,0 +60 @@ mcr.microsoft.com/oss/v2/kubernetes-csi/secrets-store/driver:v1.5.4
+mcr.microsoft.com/oss/v2/kubernetes/azure-cloud-node-manager:v1.33.13-windows-hpc-1
@@ -62 +62 @@ mcr.microsoft.com/oss/v2/kubernetes/azure-cloud-node-manager:v1.33.14-windows-hp
-mcr.microsoft.com/oss/v2/kubernetes/azure-cloud-node-manager:v1.33.15-windows-hpc-1
+mcr.microsoft.com/oss/v2/kubernetes/azure-cloud-node-manager:v1.34.10-windows-hpc-1
@@ -64 +64 @@ mcr.microsoft.com/oss/v2/kubernetes/azure-cloud-node-manager:v1.34.11-windows-hp
-mcr.microsoft.com/oss/v2/kubernetes/azure-cloud-node-manager:v1.34.12-windows-hpc-1
+mcr.microsoft.com/oss/v2/kubernetes/azure-cloud-node-manager:v1.35.5-windows-hpc-1
@@ -66 +66 @@ mcr.microsoft.com/oss/v2/kubernetes/azure-cloud-node-manager:v1.35.6-windows-hpc
-mcr.microsoft.com/oss/v2/kubernetes/azure-cloud-node-manager:v1.35.7-windows-hpc-1
+mcr.microsoft.com/oss/v2/kubernetes/azure-cloud-node-manager:v1.36.1-windows-hpc-1
@@ -68 +67,0 @@ mcr.microsoft.com/oss/v2/kubernetes/azure-cloud-node-manager:v1.36.2-windows-hpc
-mcr.microsoft.com/oss/v2/kubernetes/azure-cloud-node-manager:v1.36.3-windows-hpc-1
diff --git a/vhd_files/2025-gen2.txt b/vhd_files/2025-gen2.txt
index 4d7aa19..d0ea692 100644
--- a/vhd_files/2025-gen2.txt
+++ b/vhd_files/2025-gen2.txt
@@ -34 +33,0 @@ HKLM:\SYSTEM\CurrentControlSet\Policies\Microsoft\FeatureManagement\Overrides\18
-HKLM:\SYSTEM\CurrentControlSet\Policies\Microsoft\FeatureManagement\Overrides\908168846=1
@@ -60,0 +60 @@ mcr.microsoft.com/oss/v2/kubernetes-csi/secrets-store/driver:v1.5.4
+mcr.microsoft.com/oss/v2/kubernetes/azure-cloud-node-manager:v1.33.13-windows-hpc-1
@@ -62 +62 @@ mcr.microsoft.com/oss/v2/kubernetes/azure-cloud-node-manager:v1.33.14-windows-hp
-mcr.microsoft.com/oss/v2/kubernetes/azure-cloud-node-manager:v1.33.15-windows-hpc-1
+mcr.microsoft.com/oss/v2/kubernetes/azure-cloud-node-manager:v1.34.10-windows-hpc-1
@@ -64 +64 @@ mcr.microsoft.com/oss/v2/kubernetes/azure-cloud-node-manager:v1.34.11-windows-hp
-mcr.microsoft.com/oss/v2/kubernetes/azure-cloud-node-manager:v1.34.12-windows-hpc-1
+mcr.microsoft.com/oss/v2/kubernetes/azure-cloud-node-manager:v1.35.5-windows-hpc-1
@@ -66 +66 @@ mcr.microsoft.com/oss/v2/kubernetes/azure-cloud-node-manager:v1.35.6-windows-hpc
-mcr.microsoft.com/oss/v2/kubernetes/azure-cloud-node-manager:v1.35.7-windows-hpc-1
+mcr.microsoft.com/oss/v2/kubernetes/azure-cloud-node-manager:v1.36.1-windows-hpc-1
@@ -68 +67,0 @@ mcr.microsoft.com/oss/v2/kubernetes/azure-cloud-node-manager:v1.36.2-windows-hpc
-mcr.microsoft.com/oss/v2/kubernetes/azure-cloud-node-manager:v1.36.3-windows-hpc-1
diff --git a/vhd_files/2025.txt b/vhd_files/2025.txt
index 1441011..ab44d8b 100644
--- a/vhd_files/2025.txt
+++ b/vhd_files/2025.txt
@@ -34 +33,0 @@ HKLM:\SYSTEM\CurrentControlSet\Policies\Microsoft\FeatureManagement\Overrides\18
-HKLM:\SYSTEM\CurrentControlSet\Policies\Microsoft\FeatureManagement\Overrides\908168846=1
@@ -60,0 +60 @@ mcr.microsoft.com/oss/v2/kubernetes-csi/secrets-store/driver:v1.5.4
+mcr.microsoft.com/oss/v2/kubernetes/azure-cloud-node-manager:v1.33.13-windows-hpc-1
@@ -62 +62 @@ mcr.microsoft.com/oss/v2/kubernetes/azure-cloud-node-manager:v1.33.14-windows-hp
-mcr.microsoft.com/oss/v2/kubernetes/azure-cloud-node-manager:v1.33.15-windows-hpc-1
+mcr.microsoft.com/oss/v2/kubernetes/azure-cloud-node-manager:v1.34.10-windows-hpc-1
@@ -64 +64 @@ mcr.microsoft.com/oss/v2/kubernetes/azure-cloud-node-manager:v1.34.11-windows-hp
-mcr.microsoft.com/oss/v2/kubernetes/azure-cloud-node-manager:v1.34.12-windows-hpc-1
+mcr.microsoft.com/oss/v2/kubernetes/azure-cloud-node-manager:v1.35.5-windows-hpc-1
@@ -66 +66 @@ mcr.microsoft.com/oss/v2/kubernetes/azure-cloud-node-manager:v1.35.6-windows-hpc
-mcr.microsoft.com/oss/v2/kubernetes/azure-cloud-node-manager:v1.35.7-windows-hpc-1
+mcr.microsoft.com/oss/v2/kubernetes/azure-cloud-node-manager:v1.36.1-windows-hpc-1
@@ -68 +67,0 @@ mcr.microsoft.com/oss/v2/kubernetes/azure-cloud-node-manager:v1.36.2-windows-hpc
-mcr.microsoft.com/oss/v2/kubernetes/azure-cloud-node-manager:v1.36.3-windows-hpc-1

Copilot

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.

+# ---- hotfix: auto-generated by hotfix-generate GH Action ----
+- path: {{GetCSEConfigScriptFilepath}}
+  permissions: "0744"
+  encoding: gzip
+  owner: root
+  content: !!binary |
+    {{GetVariableProperty "cloudInitData" "provisionConfigs"}}
+


fix(gpu): tear down cuda-lts VHD prebake on GRID nodes before managed…

5a56ee2

… install (#8919)

Copilot AI review requested due to automatic review settings July 13, 2026 21:56

ganeshkumarashok requested review from AbelHu, Devinwong, SriHarsha001, awesomenix, calvin197, cameronmeissner, djsly, karenychen, lilypan26, mxj220, pdamianov-dev, phealy, r2k1, runzhen, sulixu, surajssd, timmy-wright, xuexu6666 and zachary-bailey as code owners July 13, 2026 21:56

ganeshkumarashok temporarily deployed to test July 13, 2026 21:56 — with GitHub Actions Inactive

ganeshkumarashok mentioned this pull request Jul 13, 2026

fix(gpu): tear down cuda-lts VHD prebake on GRID nodes before managed install #8918

Closed

4 tasks

Copilot started reviewing on behalf of ganeshkumarashok July 13, 2026 21:58 View session

chore: auto-generate hotfix template entries

138426c

aks-node-assistant Bot had a problem deploying to test July 13, 2026 21:58 Failure

sulixu approved these changes Jul 13, 2026

View reviewed changes

ganeshkumarashok merged commit bfd5a47 into official/v20260702 Jul 13, 2026
16 of 17 checks passed

ganeshkumarashok deleted the ganesh/grid-prebake-teardown-0702-backport branch July 13, 2026 21:59

Copilot AI reviewed Jul 13, 2026

View reviewed changes

Copilot AI review requested due to automatic review settings July 13, 2026 22:00

Copilot AI reviewed Jul 13, 2026

View reviewed changes

ganeshkumarashok mentioned this pull request Jul 13, 2026

fix(gpu): tear down cuda-lts VHD prebake on GRID nodes [backport #8919 to v20260709] #8924

Merged

2 tasks

Navigation Menu

Uh oh!

fix(gpu): tear down cuda-lts VHD prebake on GRID nodes [backport #8919 to v20260702] - #8921

fix(gpu): tear down cuda-lts VHD prebake on GRID nodes [backport #8919 to v20260702]#8921
ganeshkumarashok merged 2 commits into
official/v20260702from
ganesh/grid-prebake-teardown-0702-backport

ganeshkumarashok commented Jul 13, 2026

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

github-actions Bot commented Jul 13, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

ganeshkumarashok commented Jul 13, 2026

Summary

Relationship to #8918

Test plan

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

github-actions Bot commented Jul 13, 2026

Changes cached containers or packages on windows VHDs

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants