SCALE on AMD is missing CUDA Graphs conditional nodes

[diskdog]

SCALE 1.7.1 and the 2026.07.02 nightly, AMD target (gfx1100), compile-only. NVIDIA target and nvcc 13.1 are fine.

CUDA 12.4 conditional graph nodes are not declared on the AMD target, while the target advertises CUDA 12.9, so version-guarded users enable them.

Missing: cudaGraphConditionalHandle, cudaGraphConditionalHandleCreate, cudaGraphCondAssignDefault, cudaGraphCondTypeIf, and the conditional member of cudaGraphNodeParams. (The cudaGraphNodeTypeConditional enumerator itself is present in hostapi.h; the handle type, its creator, the cond-type enum and the params member are what is absent.)

repro.cu:

#include <cuda_runtime.h>
int main() {
  cudaGraph_t g;
  cudaGraphCreate(&g, 0);
  cudaGraphConditionalHandle h;
  cudaGraphConditionalHandleCreate(&h, g, 1, cudaGraphCondAssignDefault);
  cudaGraphNodeParams p{};
  p.type = cudaGraphNodeTypeConditional;
  p.conditional.handle = h;
  p.conditional.type = cudaGraphCondTypeIf;
  p.conditional.size = 1;
  return 0;
}

nvcc -std=c++17 -c repro.cu                          ok
targets/gfx1100/bin/nvcc -std=c++17 -c repro.cu      error: unknown type name 'cudaGraphConditionalHandle'
llvm/bin/nvcc -arch=sm_75 -std=c++17 -c repro.cu     ok (NVIDIA target has it)

Both SCALE versions fail identically (longstanding, not a regression).

Found building pytorch v2.12.1: fails CUDAGraph.cpp and CUDAGraph.cu (error: 'struct cudaGraphNodeParams' has no member named 'conditional').

[ChrisKitching]

This one is tricky to do pre-PM4, since the intended behaviour of conditional nodes is that they evaluate fully on-GPU. That basically requires a self-modifying argument buffer, which is impossible using the documented HSA mechanism for interacting with the GPU. We have an inflight project to switch to the undocumented-but-more-powerful PM4 queue, but it's a nontrivial job.

The fundamental issue is that the command buffer consumption hardware reads ahead an unspecified amount and/or performs caching, so simply mutating the HSA queue following the current kernel is not guaranteed to be picked up. This is the same issue we face trying to implement CUDA Dynamic Parallelism. PM4 offers a mechanism to jump to a different command buffer (which is necessarily uncacheed), allowing reliable self-modifying command buffers.

We may or may not do a short-term implementation of this which bounces control back to the host to evaluate each conditional node there, but that somewhat defeats the point of CUDA gaphs.