[Bug]: cuMemcpyBatchAsync segfaults or hangs on large/repeated compact scatter submissions

[coltonottley]

Your current environment

• Hardware: one TP=2 endpoint on 2x NVIDIA RTX PRO 6000 Blackwell Workstation Edition GPUs for the original large-submission failure; one TP=4 endpoint on 4x of the same GPUs for the repeated-submission hang
• GPU VBIOS: 98.02.81.00.07
• NVIDIA driver: 580.159.04
• CUDA driver API: 13.0
• CUDA toolkit: 13.0 (nvcc V13.0.88)
• CUDA runtime package: 13.0.88-1
• PyTorch: 2.11.0+cu130
• Host: Ubuntu 25.10, kernel 6.17.0-29-generic
• Container base: Ubuntu 24.04
• Affected vLLM integration: swap_blocks_batch in csrc/libtorch_stable/cache_kernels.cu, resolving the CUDA Driver API function cuMemcpyBatchAsync through cuGetProcAddress
• Public affected source before the workaround: 420f10899a
• Public workaround commit: 2e75dbf24e

🐛 Describe the bug

cuMemcpyBatchAsync is unsafe for a valid heterogeneous scatter-copy workload generated by compact CPU KV-cache transfers on RTX PRO 6000 Blackwell GPUs with CUDA 13.0 / driver 580.159.04.

We observed two failure shapes:

1. A TP=2 replay submitted 51,828 copy descriptors in one batch. Both TP workers segfaulted inside cuMemcpyBatchAsync.
2. After an intermediate mitigation reduced individual submissions, a TP=4 process completed two CPU-hop replays, then the third replay left one rank stuck below execute_model. The request contained approximately 19,634 logical descriptors per rank before physical-page span splitting. The affected rank remained runnable while its GPU did no useful work; the other ranks slept. Five minutes later the executor timed out the execute_model RPC and force-killed the survivor.

This is not a claim that there is a universal 32K-descriptor threshold. The behavior is non-monotonic: a single larger submission segfaulted, while repeated smaller submissions later produced a one-rank hang. Descriptor count, descriptor sizes, total byte volume, repetition, and driver-internal state have not been isolated independently.

The TP4 crash backtrace reaches the CUDA driver call directly:

!!!!!!! Segfault encountered !!!!!!!
  File "<unknown>", line 0, in cuMemcpyBatchAsync
  File "/workspace/vllm-src/csrc/libtorch_stable/cache_kernels.cu", line 145, in swap_blocks_batch(...)

The repeated-hang timeline was:

- two exact CPU-hop probes completed successfully
- third replay created compact load job 136 for 48,128 externally restored tokens
- one TP rank stopped returning from execute_model
- shared-memory response-starvation warning repeated once per minute
- client abort did not unwind the worker call
- after five minutes: TimeoutError: RPC call to execute_model timed out
- scheduler dump had zero running/waiting requests but load job 136 remained
- shutdown force-killed one surviving worker

There was no OOM, NVIDIA Xid, or ECC error in the accepted failure evidence.

Matched bypass/control

We kept the same descriptor planner, stream/event ownership, transfer-job lifecycle, model, and TP4 semantic workload, but prevented the extension from resolving cuMemcpyBatchAsync. That forced its existing ordered per-descriptor cudaMemcpyAsync fallback.

A CUDA-13-aware cuGetProcAddress_v2 interception confirmed exactly one blocked lookup per TP rank. With the unchanged old image and this batch-API bypass:

• all six strict cold-plus-CPU-hop semantic probes passed;
• the formerly fatal probe passed twice;
• each accepted hop restored 48,128 external tokens with zero local-cache hits and exact output;
• the same PID survived for more than one hour;
• there were zero restarts, RPC timeouts, starvation warnings, fatals, or Xids.

The final source workaround adds use_batch_api=False only for compact scatter transfers while preserving the canonical batch path for ordinary block swaps. On the rebuilt native extension:

• the normal small batch path passed on a dedicated stream;
• per-copy fallback passed on default and dedicated streams;
• a 20,000-descriptor × 128-byte per-copy integrity case passed;
• the full TP4 semantic harness passed 6/6 with the real workload and no interception shim.

Public workaround/reference: jasl#32

Relevant repair: jasl@2e75dbf

This differs from #39491. That issue was traced to invalid/null or out-of-bounds geometry and returned CUDA_ERROR_INVALID_VALUE. Here, the same planned transfers complete with exact data when submitted as ordered cudaMemcpyAsync operations; the batch path instead causes native process failure or rank-local non-return.

Expected behavior

For valid source pointers, destination pointers, sizes, and a dedicated non-default stream, cuMemcpyBatchAsync should either:

1. enqueue the copies successfully; or
2. return a documented CUDA error with a useful failIdx.

It should not segfault the process or leave one TP worker permanently stuck inside the call.

Current workaround

Avoid cuMemcpyBatchAsync for compact heterogeneous scatter transfers and submit ordered per-descriptor cudaMemcpyAsync operations on the same stream. We are not proposing a descriptor-count threshold because the observed failures are not monotonic.

Open questions

• Is there a documented descriptor-count, aggregate-byte, attribute, or repetition limit for cuMemcpyBatchAsync?
• Does CUDA 13.0 have a known Blackwell-specific failure in large or repeated host↔device batches?
• Would NVIDIA prefer a standalone Driver API reproducer extracted from this workload? We can provide one and can run bounded tests on 2x/4x RTX PRO 6000 Blackwell hardware.

Before submitting a new issue...

• I searched open and closed vLLM issues and GitHub for cuMemcpyBatchAsync, descriptor-count crashes, and swap_blocks_batch. The closest result, [Bug]: OffloadingConnector GPU->CPU KV offload crashes with cuMemcpyBatchAsync failed at index 1 (error 1 / CUDA_ERROR_INVALID_VALUE) #39491, has a different proven cause and failure mode.