fix: clean up mismatched prebaked GPU drivers - #8915
Conversation
Ensure Ubuntu managed GPU driver installation removes a prebaked GPU driver when its marker indicates a different driver family than the requested node driver. This prevents GRID scenarios from keeping the prebaked CUDA driver alongside a dynamically installed GRID driver. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 4110a607-c4e2-49f0-b023-5b5f73cfb153
There was a problem hiding this comment.
Pull request overview
This PR improves Ubuntu GPU node provisioning when a shared VHD has a prebaked NVIDIA driver that doesn’t match the managed driver family requested by CSE (notably CUDA prebake + managed GRID install). It adds marker parsing + driver-kind normalization, and triggers the existing prebaked-driver teardown path when a mismatch is detected.
Changes:
- Add helper functions to normalize
NVIDIA_GPU_DRIVER_TYPE→ driver kind and to readdriver_kindfrom the prebake marker. - Add
cleanUpMismatchedPrebakedGPUDriverand invoke it early inensureGPUDriverson Ubuntu before managed driver installation. - Add ShellSpec coverage for absent markers, CUDA→GRID cleanup, legacy markers without
driver_kind, and matching no-op behavior.
Reviewed changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.
| File | Description |
|---|---|
| spec/parts/linux/cloud-init/artifacts/cse_config_spec.sh | Adds focused ShellSpec tests validating mismatch cleanup behavior and edge cases. |
| parts/linux/cloud-init/artifacts/ubuntu/cse_install_ubuntu.sh | Updates the cleanUpPrebakedGPUDriver header comment to reflect its expanded use cases. |
| parts/linux/cloud-init/artifacts/cse_config.sh | Introduces driver-kind/marker helpers and runs mismatch cleanup before managed GPU driver installation on Ubuntu. |
| # cleanUpPrebakedGPUDriver removes a driver pre-baked into the shared VHD on nodes that should not | ||
| # keep it: non-GPU VMs, GPU VMs opted out via --gpu-driver None or skip toggles, and managed GPU | ||
| # nodes whose requested driver kind does not match the prebaked driver marker. There the driver is | ||
| # either dead weight (wasted disk; nvidia.ko rebuilt on every kernel patch), unused attack surface, or | ||
| # the wrong driver family for the node SKU. No-op unless the marker exists. |
AgentBaker Linux gate detectiveRun: https://msazure.visualstudio.com/CloudNativeCompute/_build/results?buildId=172045051 Detective summary: Ubuntu 22.04 Gen2 containerd completed CIS assessment and failed baseline diff on Likely cause / signatures: Confidence: High for the two CIS rule regressions; low/medium for the Ubuntu 20.04 FIPS unknown exit-2 surface. Recommended owner/action: VHD/CIS owner should inspect the 22.04 Strongest alternative: PR #8915 GPU driver cleanup caused image content drift; possible but not proven for the CIS diffs and not supported for the 20.04 FIPS unknown surface from sampled logs. Evidence: ADO build summary, Test/Scan logs 538/545, source commit Wiki signatures: linux-vhd-prgate-cis-ubuntu2204-gen2-containerd-6131-pass-fail, linux-vhd-prgate-ubuntu2004-fips-gen2-containerd-exit2-unknown-surface, linux-vhd-prgate-cis-ubuntu2404-gen2-containerd-6141-pass-fail |
|
fixed in #8919 |
What this PR does / why we need it:
Fixes Ubuntu GRID GPU nodes that can otherwise keep a CUDA driver prebaked into the VHD when CSE dynamically installs a GRID driver. The CSE now reads the prebake marker before managed GPU driver installation, normalizes the prebaked and requested driver families, and invokes the existing prebaked driver cleanup path when they do not match.
Legacy markers without
driver_kindare treated as CUDA prebake only for GRID requests. This is intentionally driver-family aware only; CUDA-vs-CUDA version drift still needs separate marker/version comparison work.Validation:
GENERATE_TEST_DATA=true go test ./pkg/agent...passed.make generateis blocked by internal Go proxy authorization while installingginkgo.Which issue(s) this PR fixes:
N/A
🤖 Generated by GitHub Copilot