[upstream_ut] test/dynamo/test_higher_order_ops.py test_dropout failed with RuntimeError: CUDA not available - #9
Draft
Stonepia wants to merge 1 commit into
Draft
[upstream_ut] test/dynamo/test_higher_order_ops.py test_dropout failed with RuntimeError: CUDA not available#9Stonepia wants to merge 1 commit into
Stonepia wants to merge 1 commit into
Conversation
[upstream_ut] test/dynamo/test_higher_order_ops.py test_dropout failed with RuntimeError: CUDA not available
7 tasks
|
This does not fix the issue as test_dropout:
The issue is rooted a bit deeper. I am currently working on a proper solution. |
Collaborator
Author
|
Hi @kdrozd-dev , Thanks for the comment! This is just an agent workflow test, so I didn't quite read the code yet. Thanks for the fix! |
chuanqi129
pushed a commit
that referenced
this pull request
Jun 24, 2026
…#188024) Some gfx950 (MI350) 2-GPU runner pods come up with a container that cannot read part of the KFD/HSA topology. RCCL then fails every collective init with "ncclUnhandledCudaError: Call to CUDA function failed / Could not read node #N" (N is a fixed topology-node index for that pod, e.g. #2 or #9). When a distributed shard lands on such a pod, the first collective crashes and a later test hangs the whole shard until the 270-minute job timeout. Host-side rocminfo still enumerates the GPUs on these pods, so the existing "Runner check GPU count" gate does not catch it -- the failure is the container's deeper topology read, not agent enumeration. Diagnosis: the same RCCL error appears across many different distributed tests and both worker-crash and downstream-hang forms; in the logs it is constant per pod and present from the very first collective, and world_size is 2 so "node #N" cannot be a rank -- it is a system topology node. So it is a per-pod container health problem, not a PyTorch or per-test bug. Fix: add a fast in-container RCCL pre-flight (.ci/pytorch/rocm_preflight.py) that spawns a min(2, ngpu)-rank process group and does one all_reduce. For distributed shards _rocm-test.yml runs it (wrapped in `timeout 180`) before the suite; on failure or hang the job fails in seconds with a clear message instead of hanging for 270 minutes, and the bad pod is identifiable for draining. Non-distributed shards are unchanged. Test Plan: ``` python -m py_compile .ci/pytorch/rocm_preflight.py lintrunner -a .ci/pytorch/rocm_preflight.py .github/workflows/_rocm-test.yml ``` Validated the pre-flight directly. Locally on 8x A100 (the nccl backend exercises the same init/collective path RCCL uses) the script passes: ROCm/RCCL pre-flight passed (2-rank all_reduce) and the failure path is exercised too: when the process group cannot initialize, mp.spawn raises, the script prints the "::error::" message and exits 1 -- the job fails fast instead of hanging. In CI, dispatched the gate on the gfx950.2 distributed pool: the pre-flight ran inside the container and passed on healthy pods (exit 0, suite proceeded). The broken-pod case is the same exit-1 fast-fail path; it only reproduces when a pod with the broken KFD topology is in rotation, and the pool was healthy during testing, so that specific case was not caught live. Authored with Claude. Pull Request resolved: pytorch#188024 Approved by: https://github.com/frgossen
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
[upstream_ut] test/dynamo/test_higher_order_ops.py test_dropout failed with RuntimeError: CUDA not available
Fixes intel/torch-xpu-ops#3361
Root Cause: The test
test_dropoutuses@requires_cuda_and_tritonwhich skips when CUDA is unavailable, but the XPU CI environment has CUDA libraries present (satisfying the decorator at module-load time) yet no working CUDA GPU. When AOTAutograd'screate_functionalized_rng_ops_wrapperis invoked withfunctionalize_rng_ops=True, it unconditionally callsCUDARngStateHelper.get_torch_state_as_tuple(graph_capture_wrappers.py:580) which hard-raisesRuntimeError('CUDA not available')with no CUDA guard. The functionget_torch_state_as_tupleis CUDA-only butfunctionalize_rng_opsis a generic config flag that can be activated on non-CUDA backends.Failed Tests:
test/dynamo/test_higher_order_ops.py::ActivationCheckpointingTests::test_dropoutDiff stat: