[Bug]: Warp Illegal Address / MMU Fault** (Xid 13) during prefill when running GLM-5.2-NVFP4 on NVIDIA B200 GPUs

[bleedingfight]

System Info

Python: 3.12.3 (main, Mar 23 2026, 19:04:32) [GCC 13.3.0]
CUDA available: True
GPU 0,1,2,3,4,5,6,7: NVIDIA B200
GPU 0,1,2,3,4,5,6,7 Compute Capability: 10.0
CUDA_HOME: /usr/local/cuda
NVCC: Cuda compilation tools, release 13.0, V13.0.88
CUDA Driver Version: 580.82.07
PyTorch: 2.11.0+cu130
sglang: 0.5.13.post1
sglang-kernel: 0.4.3
flashinfer_python: 0.6.12
flashinfer_cubin: 0.6.12
flashinfer_jit_cache: 0.6.12+cu130
triton: 3.6.0
transformers: 5.8.1
torchao: 0.17.0+cu130
numpy: 2.3.5
aiohttp: 3.14.1
fastapi: 0.137.0
huggingface_hub: 1.19.0
interegular: 0.3.3
modelscope: 1.37.1
orjson: 3.11.9
outlines: 0.1.11
packaging: 26.2
psutil: 7.2.2
pydantic: 2.13.4
python-multipart: 0.0.32
pyzmq: 27.1.0
uvicorn: 0.49.0
uvloop: 0.22.1
vllm: Module Not Found
xgrammar: 0.2.1
openai: 2.6.1
tiktoken: 0.13.0
anthropic: 0.109.1
litellm: Module Not Found
torchcodec: 0.11.1+cu130
NVIDIA Topology:
GPU0 GPU1 GPU2 GPU3 GPU4 GPU5 GPU6 GPU7 NIC0 NIC1 NIC2 NIC3 NIC4 NIC5 NIC6 NIC7 CPU Affinity NUMA Affinity GPU NUMA ID
GPU0 X NV18 NV18 NV18 NV18 NV18 NV18 NV18 PIX NODE NODE NODE SYSSYS SYS SYS 0-71,144-215 0 N/A
GPU1 NV18 X NV18 NV18 NV18 NV18 NV18 NV18 NODE PIX NODE NODE SYSSYS SYS SYS 0-71,144-215 0 N/A
GPU2 NV18 NV18 X NV18 NV18 NV18 NV18 NV18 NODE NODE PIX NODE SYSSYS SYS SYS 0-71,144-215 0 N/A
GPU3 NV18 NV18 NV18 X NV18 NV18 NV18 NV18 NODE NODE NODE PIX SYSSYS SYS SYS 0-71,144-215 0 N/A
GPU4 NV18 NV18 NV18 NV18 X NV18 NV18 NV18 SYS SYS SYS SYS PIXNODE NODE NODE 72-143,216-287 1 N/A
GPU5 NV18 NV18 NV18 NV18 NV18 X NV18 NV18 SYS SYS SYS SYS NODE PIX NODE NODE 72-143,216-287 1 N/A
GPU6 NV18 NV18 NV18 NV18 NV18 NV18 X NV18 SYS SYS SYS SYS NODE NODE PIX NODE 72-143,216-287 1 N/A
GPU7 NV18 NV18 NV18 NV18 NV18 NV18 NV18 X SYS SYS SYS SYS NODE NODE NODE PIX 72-143,216-287 1 N/A
NIC0 PIX NODE NODE NODE SYS SYS SYS SYS X NODE NODE NODE SYSSYS SYS SYS
NIC1 NODE PIX NODE NODE SYS SYS SYS SYS NODE X NODE NODE SYSSYS SYS SYS
NIC2 NODE NODE PIX NODE SYS SYS SYS SYS NODE NODE X NODE SYSSYS SYS SYS
NIC3 NODE NODE NODE PIX SYS SYS SYS SYS NODE NODE NODE X SYSSYS SYS SYS
NIC4 SYS SYS SYS SYS PIX NODE NODE NODE SYS SYS SYS SYS X NODE NODE NODE
NIC5 SYS SYS SYS SYS NODE PIX NODE NODE SYS SYS SYS SYS NODE X NODE NODE
NIC6 SYS SYS SYS SYS NODE NODE PIX NODE SYS SYS SYS SYS NODE NODE X NODE
NIC7 SYS SYS SYS SYS NODE NODE NODE PIX SYS SYS SYS SYS NODE NODE NODE X

Legend:

X = Self
SYS = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)
NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node
PHB = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)
PXB = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge)
PIX = Connection traversing at most a single PCIe bridge
NV# = Connection traversing a bonded set of # NVLinks

NIC Legend:

NIC0: mlx5_0
NIC1: mlx5_1
NIC2: mlx5_2
NIC3: mlx5_3
NIC4: mlx5_4
NIC5: mlx5_5
NIC6: mlx5_6
NIC7: mlx5_7

ulimit soft: 1048576

Who can help?

The source of this bug comes from running GLM-5.2 nvfp4 on sglang，A Very Difficult to Reproduce Operator Anomaly. sgl-project/sglang#29160

Information

• The official example scripts
• My own modified scripts

Tasks

• An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
• My own task or dataset (give details below)

Reproduction

NVFP4 GEMM+SwiGLU Fused Kernel Causes Warp Illegal Address on B200 (SM100)

Summary

The CUTLASS DSL JIT-compiled bmm_E2m1_...swiGlu_dynB_sm100f kernel triggers a Warp Illegal Address / MMU Fault (Xid 13) during prefill when running GLM-5.2-NVFP4 on NVIDIA B200 GPUs. The crash is reproducible across multiple occurrences (observed on 2026-06-15 and 2026-06-24) and always hits the same kernel at the same call site (_forward_shared_experts SwiGLU path).

Environment

• GPU: NVIDIA B200 (SM100, 148 SMs)
• Driver: CUDA 13.0
• Framework: sglang 0.5.13.post1, flashinfer 0.6.12, PyTorch 2.11.0+cu130
• Model: GLM-5.2-NVFP4 (modelopt_fp4 quantization, NVFP4 block-scaled format)
• Serving config: TP=8, --quantization modelopt_fp4, --kv-cache-dtype fp8_e4m3, --speculative-algorithm EAGLE, --disable-piecewise-cuda-graph

Crash Details

Call Stack (from watchdog py-spy dump)

scheduler.run_scheduler_process
  → scheduler.run_batch
    → model_runner.forward_extend
      → deepseek_v2.forward (MoE layer)
        → deepseek_v2._forward_shared_experts
          → FusedLinearMethod.forward
            → nvfp4_gemm_swiglu_nvfp4_quant()  ← CUTLASS DSL JIT kernel
              → bmm_E2m1_...swiGlu_dynB_sm100f  ← CRASH: Warp Illegal Address

dmesg (Xid 13)

[Wed Jun 24 15:44:20 2026] NVRM: Xid (PCI:0000:70:00): 13, Graphics SM Warp Exception on (GPC 1, TPC 1, SM 1): MMU Fault
[Wed Jun 24 15:44:20 2026] NVRM: Xid (PCI:0000:70:00): 13, Graphics Exception: ESR 0x5167b0=0x80f0017 0x5167b4=0x20 0x5167a8=0x1f81fb60 0x5167ac=0x1174
[Wed Jun 24 15:44:20 2026] NVRM: Xid (PCI:0000:70:00): 13, Graphics SM Warp Exception on (GPC 2, TPC 7, SM 1): MMU Fault
[Wed Jun 24 15:44:20 2026] NVRM: Xid (PCI:0000:70:00): 13, Graphics Exception: ESR 0x52c7b0=0x80f0017 0x52c7b4=0x20 0x52c7a8=0x1f81fb60 0x52c7ac=0x1174
[Wed Jun 24 15:44:20 2026] NVRM: Xid (PCI:0000:70:00): 13, Graphics SM Warp Exception on (GPC 6, TPC 8, SM 0): MMU Fault
[Wed Jun 24 15:44:20 2026] NVRM: Xid (PCI:0000:70:00): 13, Graphics Exception: ESR 0x56d730=0x80f0017 0x56d734=0x20 0x56d728=0x1f81fb60 0x56d72c=0x1174
[Wed Jun 24 15:44:20 2026] NVRM: Xid (PCI:0000:70:00): 13, Graphics SM Warp Exception on (GPC 7, TPC 8, SM 0): MMU Fault
[Wed Jun 24 15:44:20 2026] NVRM: Xid (PCI:0000:70:00): 13, Graphics Exception: ESR 0x57d730=0x80f0017 0x57d734=0x20 0x57d728=0x1f81fb60 0x57d72c=0x1174

Key observation: 4 SMs across different GPCs report identical fault addresses (0x1f81fb60 / 0x1174), indicating the same kernel tile computed the same out-of-bounds address.

CUDA Coredump Analysis

A CUDA coredump was captured (cuda_coredump_36-0-4-174.149.1782275213, 562MB):

CUDA Exception: Warp Illegal Address
The exception was triggered at PC 0x78605d165d30

Kernel: bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x512u2_s5_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f<<<(4,16,1),(512,1,1)>>>

Active threads:
  (0,15,0)   (0,0,0)    (1,15,0) (511,0,0)  1024 threads at PC 0x78605d165d50
  (2,15,0)   (0,0,0)    (2,15,0) (319,0,0)   320 threads at PC 0x78605d165d80
* (2,15,0) (320,0,0)    (3,15,0) (511,0,0)   704 threads at PC 0x78605d165d50  ← faulting warp

The coredump contains:

• 99.7% global memory image (534.7 MB) — raw VRAM dump with no annotations
• 0.1% GPU register state (962 register files for active warps)
• 0% CPU-side state (no host stack, no Python variables)

Note: CUDA_COREDUMP_GDB=1 was not set, so the coredump lacks host CPU state. This makes it impossible to retrieve the M/N/K values and scale tensor values passed to the kernel from the coredump alone.

Process Impact Timeline (2026-06-24)

Time	Event
15:44:20	GPU Xid 13 MMU Fault triggered
15:44:23	Last normal decode batch logged
15:44:25	Watchdog detects segfault, begins py-spy dump of surviving workers
15:44:58	PID 151 (scheduler_3) crashes with exit code -11 (SIGSEGV)
15:45:04–10	7 surviving TP workers successfully py-spy dumped
15:45:10	CUDA coredump triggered, 60s wait begins
15:46:08	SM state collection at 50/148 (~34% complete)
15:46:10	kill_process_tree terminates all processes before coredump finishes
15:46:10	All in-flight HTTP requests timeout

Note: The 60-second coredump wait is insufficient for B200's 148 SMs — full coredump requires ~3–5 minutes. The coredump file was not finalized because the process was killed mid-collection.

Prior Occurrence

Same kernel crashed on 2026-06-15 21:38:49 (different input, same call site). This confirms the bug is reproducible across different request patterns.

Reproduction Attempts

A minimal reproduction script using random NVFP4 weights with matching model dimensions (K=6144, N=3072) and various M values (1 to 65536) did not reproduce the crash. Concurrent large prefill requests also did not trigger it. This suggests the bug is input-data dependent — specific weight scale values combined with specific input patterns may cause TMA descriptor address calculation to go out of bounds.

Root Cause Hypothesis

The kernel uses TMA (Tensor Memory Accelerator) + LDGSTS for data loading on Blackwell. The identical fault address across 4 SMs strongly suggests:

1. A tile coordinate calculation in the persistent GEMM scheduler computes an out-of-bounds TMA address, OR
2. A TMA descriptor field (base address, stride, or coordinate transform) is incorrectly populated for certain input shapes/scales

The dynB (dynamic batch) suffix indicates the kernel handles variable M dimensions per batch. The TMA tile mapping may have an edge case where certain M/N combinations cause the tile coordinate to exceed the allocated buffer bounds.

Recommended Next Steps

1. Enable full coredump capture for next occurrence:

   export CUDA_COREDUMP=1
   export CUDA_COREDUMP_DIR=/tmp/crashdump    # persistent volume, not /tmp
   export CUDA_COREDUMP_GDB=1                 # include host CPU state

   And increase the coredump wait timeout in sglang's crash handler (currently hardcoded to 60s, needs ~300s for B200).

2. Enable kernel API logging to capture the exact GEMM parameters at crash time:

   export SGLANG_KERNEL_API_LOGLEVEL=3
   export SGLANG_KERNEL_API_LOGDEST=/tmp/crashdump/kernel_api_%i.log

3. NVIDIA investigation: Since the kernel logic is CUTLASS/CuTe DSL compiled code from TensorRT-LLM, the TMA address calculation bug likely resides in the Sm100BlockScaledPersistentDenseGemmKernel implementation. NVIDIA needs to:

   • Provide debug symbols (DWARF info) for the bmm_E2m1_...swiGlu_dynB_sm100f cubin
   • Map PC 0x78605d165d30 back to the CuTe/CUTLASS source line
   • Verify TMA descriptor field validation for dynB (dynamic batch) configurations

Kernel Provenance

Layer	Responsibility
sglang	Call site, parameter construction, fp4_quantize (via flashinfer), TMA descriptor setup
flashinfer	Input quantization (fp4_quantize) only; does NOT launch the GEMM+SwiGLU kernel
TensorRT-LLM (CUTLASS DSL)	Kernel implementation: Sm100BlockScaledPersistentDenseGemmKernel with SwiGLU fusion and TMA+LDGSTS loading

Relevant Files

• sglang call site: python/sglang/srt/models/deepseek_v2.py — _forward_shared_experts() → self.shared_experts()
• sglang kernel wrapper: python/sglang/srt/layers/quantization/nvfp4_gemm_swiglu_nvfp4_quant.py
• flashinfer quantization: flashinfer/fp4_quantization.py — fp4_quantize()
• TensorRT-LLM kernel source: cpp/tensorrt_llm/kernels/trtllmGenKernels/gemmGatedAct/

Expected behavior

no crash

actual behavior

crash

additional notes

The bug originated from the call to flashinfer

Before submitting a new issue...

• Make sure you already searched for relevant issues, and checked the documentation and examples for answers to frequently asked questions.