build(gpu): make the CUDA runtime base a build ARG (T4 fleet)#496
Open
kriszyp wants to merge 1 commit into
Open
build(gpu): make the CUDA runtime base a build ARG (T4 fleet)#496kriszyp wants to merge 1 commit into
kriszyp wants to merge 1 commit into
Conversation
CUDA 13.x supports Turing (SM 7.5 / T4) — Turing is the minimum architecture in the 13.x line, so the current 13.3.0 base runs on the GCP T4 fleet as-is. The 13.x runtime does, however, require an R580+ host driver. To avoid forking the Dockerfile on hosts pinned to an older driver branch, expose the runtime base as CUDA_RUNTIME_IMAGE (default unchanged at nvidia/cuda:13.3.0-cudnn-runtime-ubuntu22.04), overridable to a CUDA 12.x cudnn-runtime base at build time. No default behavior change. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Contributor
|
Reviewed; no blockers found. |
kriszyp
requested review from
Devin-Holland and
sleekmountaincat
and removed request for
a team and
kylebernhardy
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Small hardening for the upcoming GCP T4 GPU fleet, alongside host-manager#125.
TL;DR on the CUDA-13 question
Dockerfile-gpubuilds onnvidia/cuda:13.3.0-cudnn-runtime-ubuntu22.04, and I'd flagged a worry that CUDA 13 might have dropped Turing. It hasn't — CUDA 13.x makes Turing (SM 7.5 / T4) the minimum supported architecture; it's Maxwell/Pascal/Volta that were removed. So the current base runs on T4 as-is, no arch change needed.The one real caveat: the CUDA 13.x runtime requires an R580+ host driver. On a new cloud fleet, GPU host images often lag on driver branch (535/550 LTS are common), so a host could be too old for the 13.x runtime.
Change
Expose the runtime base as a build ARG so ops can pin a CUDA 12.x base on driver-constrained hosts without forking the Dockerfile:
Override example:
docker build -f Dockerfile-gpu \ --build-arg CUDA_RUNTIME_IMAGE=nvidia/cuda:12.6.3-cudnn-runtime-ubuntu22.04 .Default is unchanged (13.3.0), so existing builds are byte-identical.
Context / scope
In the shared-vLLM design (host-manager#125) tenant containers reach embeddings/generation by HTTP to the host vLLM, so this image's CUDA runtime is only exercised by in-instance GPU code (e.g. a native addon). For the embedding-first T4 rollout it's largely latent — this ARG is forward-looking flexibility, not a fix for a current break.
Sources for the CUDA-13/Turing + R580 facts:
🤖 Generated with Claude Code