I am encountering the following error while trying to run the ( https://huggingface.co/nvidia/MiniMax-M2.7-NVFP4 ) model on an NVIDIA DGX Spark system:
Using 'FLASHINFER_B12X' NvFp4 MoE backend out of potential backends: ['FLASHINFER_TRTLLM', 'FLASHINFER_CUTEDSL', 'FLASHINFER_CUTEDSL_BATCHED', 'FLASHINFER_CUTLASS', 'VLLM_CUTLASS', 'MARLIN', 'EMULATION'].
Loading safetensors using Fastsafetensor loader: 0% Completed | 0/8 [00:00<?, ?it/s]
Loading safetensors using Fastsafetensor loader: 12% Completed | 1/8 [00:11<01:22, 11.82s/it]
Loading safetensors using Fastsafetensor loader: 25% Completed | 2/8 [00:36<01:57, 19.53s/it]
Loading safetensors using Fastsafetensor loader: 38% Completed | 3/8 [00:53<01:31, 18.31s/it]
Loading safetensors using Fastsafetensor loader: 50% Completed | 4/8 [01:08<01:08, 17.04s/it]
Loading safetensors using Fastsafetensor loader: 62% Completed | 5/8 [01:20<00:45, 15.14s/it]
Loading safetensors using Fastsafetensor loader: 75% Completed | 6/8 [01:25<00:23, 11.80s/it]
Loading safetensors using Fastsafetensor loader: 88% Completed | 7/8 [01:31<00:09, 9.89s/it]
Loading safetensors using Fastsafetensor loader: 100% Completed | 8/8 [01:32<00:00, 7.05s/it]
Loading safetensors using Fastsafetensor loader: 100% Completed | 8/8 [01:32<00:00, 11.59s/it]
(Worker_TP0 pid=183)
(Worker_TP0 pid=183) INFO 05-17 08:08:42 [default_loader.py:397] Loading weights took 92.74 seconds
(Worker_TP0 pid=183) WARNING 05-17 08:08:42 [kv_cache.py:109] Checkpoint does not provide a q scaling factor. Setting it to k_scale. This only matters for FP8 Attention backends (flash-attn or flashinfer).
(Worker_TP0 pid=183) WARNING 05-17 08:08:42 [kv_cache.py:123] Using KV cache scaling factor 1.0 for fp8_e4m3. If this is unintended, verify that k/v_scale scaling factors are properly set in the checkpoint.
(Worker_TP0 pid=183) WARNING 05-17 08:08:42 [kv_cache.py:162] Using uncalibrated q_scale 1.0 and/or prob_scale 1.0 with fp8 attention. This may cause accuracy issues. Please make sure q/prob scaling factors are available in the fp8 checkpoint.
(Worker_TP0 pid=183) INFO 05-17 08:08:43 [nvfp4.py:500] Using MoEPrepareAndFinalizeNoDPEPModular
(Worker_TP0 pid=183) INFO 05-17 08:08:45 [gpu_model_runner.py:5091] Model loading took 62.75 GiB memory and 96.962267 seconds
(Worker_TP0 pid=183) INFO 05-17 08:08:49 [backends.py:1089] Using cache directory: /root/.cache/vllm/torch_compile_cache/c1f8e3def4/rank_0_0/backbone for vLLM's torch.compile
(Worker_TP0 pid=183) INFO 05-17 08:08:49 [backends.py:1148] Dynamo bytecode transform time: 3.27 s
(Worker_TP0 pid=183) INFO 05-17 08:08:51 [backends.py:292] Directly load the compiled graph(s) for compile range (1, 2048) from the cache, took 1.175 s
(Worker_TP0 pid=183) INFO 05-17 08:08:51 [decorators.py:311] Directly load AOT compilation from path /root/.cache/vllm/torch_compile_cache/torch_aot_compile/e4b2e56fe0274a20590224bfd211f740d7f3c8261561ed633ca1e58e8beb4725/rank_0_0/model
(Worker_TP0 pid=183) INFO 05-17 08:08:51 [monitor.py:53] torch.compile took 4.91 s in total
Error: kernel '@Kernels::@kernel_cutlass_kernel_flashinferfused_moecute_dslblackwell_sm12xmoe_dynamic_kernelMoEDynamicKernel_object_at__tensorptrbf16gmemalign16o307230721_tensorptri32gmemo1_tensorptrf32gmemo1_tens_0' launch shared memory exceeds current GPU arch sm_121a allowed. Allocated: 118784 bytes. Max: 101376 bytes.
(Worker_TP0 pid=183) ERROR 05-17 08:09:19 [multiproc_executor.py:962] WorkerProc hit an exception.
(Worker_TP0 pid=183) ERROR 05-17 08:09:19 [multiproc_executor.py:962] Traceback (most recent call last):
(Worker_TP0 pid=183) ERROR 05-17 08:09:19 [multiproc_executor.py:962] File "/usr/
launch command
VLLM_SPARK_EXTRA_DOCKER_ARGS="-v $HOME/DATA/hf/models/:/models" ./launch-cluster.sh --no-ray --apply-mod mods/drop-caches --apply-mod mods/exp-b12x -t vllm-node-210-1-40082:latest -e VLLM_USE_FLASHINFER_MOE_FP4=1 -e OMP_NUM_THREADS=8 -e VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS=1 -e VLLM_FLASHINFER_MOE_BACKEND=throughput -e FLASHINFER_DISABLE_VERSION_CHECK=1 -e VLLM_USE_FLASHINFER_MOE_FP16=1 -e VLLM_NVFP4_GEMM_BACKEND=flashinfer-b12x -e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 -e VLLM_FLASHINFER_ALLREDUCE_BACKEND=trtllm exec vllm serve --model /models/nvidia/MiniMax-M2.7-NVFP4 --max-model-len auto --gpu-memory-utilization 0.9 --port 8888 --host 0.0.0.0 --trust-remote-code --load-format fastsafetensors --moe-backend flashinfer_b12x --enable-prefix-caching --enable-auto-tool-choice --tool-call-parser minimax_m2 --reasoning-parser minimax_m2 --served-model-name my-minimax -tp 2 --distributed-executor-backend ray --max-num-seqs 4 --attention-backend flashinfer --quantization modelopt_fp4 --dtype bfloat16 --kv-cache-dtype fp8
I am encountering the following error while trying to run the ( https://huggingface.co/nvidia/MiniMax-M2.7-NVFP4 ) model on an NVIDIA DGX Spark system:
Using 'FLASHINFER_B12X' NvFp4 MoE backend out of potential backends: ['FLASHINFER_TRTLLM', 'FLASHINFER_CUTEDSL', 'FLASHINFER_CUTEDSL_BATCHED', 'FLASHINFER_CUTLASS', 'VLLM_CUTLASS', 'MARLIN', 'EMULATION'].
Loading safetensors using Fastsafetensor loader: 0% Completed | 0/8 [00:00<?, ?it/s]
Loading safetensors using Fastsafetensor loader: 12% Completed | 1/8 [00:11<01:22, 11.82s/it]
Loading safetensors using Fastsafetensor loader: 25% Completed | 2/8 [00:36<01:57, 19.53s/it]
Loading safetensors using Fastsafetensor loader: 38% Completed | 3/8 [00:53<01:31, 18.31s/it]
Loading safetensors using Fastsafetensor loader: 50% Completed | 4/8 [01:08<01:08, 17.04s/it]
Loading safetensors using Fastsafetensor loader: 62% Completed | 5/8 [01:20<00:45, 15.14s/it]
Loading safetensors using Fastsafetensor loader: 75% Completed | 6/8 [01:25<00:23, 11.80s/it]
Loading safetensors using Fastsafetensor loader: 88% Completed | 7/8 [01:31<00:09, 9.89s/it]
Loading safetensors using Fastsafetensor loader: 100% Completed | 8/8 [01:32<00:00, 7.05s/it]
Loading safetensors using Fastsafetensor loader: 100% Completed | 8/8 [01:32<00:00, 11.59s/it]
(Worker_TP0 pid=183)
(Worker_TP0 pid=183) INFO 05-17 08:08:42 [default_loader.py:397] Loading weights took 92.74 seconds
(Worker_TP0 pid=183) WARNING 05-17 08:08:42 [kv_cache.py:109] Checkpoint does not provide a q scaling factor. Setting it to k_scale. This only matters for FP8 Attention backends (flash-attn or flashinfer).
(Worker_TP0 pid=183) WARNING 05-17 08:08:42 [kv_cache.py:123] Using KV cache scaling factor 1.0 for fp8_e4m3. If this is unintended, verify that k/v_scale scaling factors are properly set in the checkpoint.
(Worker_TP0 pid=183) WARNING 05-17 08:08:42 [kv_cache.py:162] Using uncalibrated q_scale 1.0 and/or prob_scale 1.0 with fp8 attention. This may cause accuracy issues. Please make sure q/prob scaling factors are available in the fp8 checkpoint.
(Worker_TP0 pid=183) INFO 05-17 08:08:43 [nvfp4.py:500] Using MoEPrepareAndFinalizeNoDPEPModular
(Worker_TP0 pid=183) INFO 05-17 08:08:45 [gpu_model_runner.py:5091] Model loading took 62.75 GiB memory and 96.962267 seconds
(Worker_TP0 pid=183) INFO 05-17 08:08:49 [backends.py:1089] Using cache directory: /root/.cache/vllm/torch_compile_cache/c1f8e3def4/rank_0_0/backbone for vLLM's torch.compile
(Worker_TP0 pid=183) INFO 05-17 08:08:49 [backends.py:1148] Dynamo bytecode transform time: 3.27 s
(Worker_TP0 pid=183) INFO 05-17 08:08:51 [backends.py:292] Directly load the compiled graph(s) for compile range (1, 2048) from the cache, took 1.175 s
(Worker_TP0 pid=183) INFO 05-17 08:08:51 [decorators.py:311] Directly load AOT compilation from path /root/.cache/vllm/torch_compile_cache/torch_aot_compile/e4b2e56fe0274a20590224bfd211f740d7f3c8261561ed633ca1e58e8beb4725/rank_0_0/model
(Worker_TP0 pid=183) INFO 05-17 08:08:51 [monitor.py:53] torch.compile took 4.91 s in total
Error: kernel '@Kernels::@kernel_cutlass_kernel_flashinferfused_moecute_dslblackwell_sm12xmoe_dynamic_kernelMoEDynamicKernel_object_at__tensorptrbf16gmemalign16o307230721_tensorptri32gmemo1_tensorptrf32gmemo1_tens_0' launch shared memory exceeds current GPU arch sm_121a allowed. Allocated: 118784 bytes. Max: 101376 bytes.
(Worker_TP0 pid=183) ERROR 05-17 08:09:19 [multiproc_executor.py:962] WorkerProc hit an exception.
(Worker_TP0 pid=183) ERROR 05-17 08:09:19 [multiproc_executor.py:962] Traceback (most recent call last):
(Worker_TP0 pid=183) ERROR 05-17 08:09:19 [multiproc_executor.py:962] File "/usr/
launch command
VLLM_SPARK_EXTRA_DOCKER_ARGS="-v $HOME/DATA/hf/models/:/models" ./launch-cluster.sh --no-ray --apply-mod mods/drop-caches --apply-mod mods/exp-b12x -t vllm-node-210-1-40082:latest -e VLLM_USE_FLASHINFER_MOE_FP4=1 -e OMP_NUM_THREADS=8 -e VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS=1 -e VLLM_FLASHINFER_MOE_BACKEND=throughput -e FLASHINFER_DISABLE_VERSION_CHECK=1 -e VLLM_USE_FLASHINFER_MOE_FP16=1 -e VLLM_NVFP4_GEMM_BACKEND=flashinfer-b12x -e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 -e VLLM_FLASHINFER_ALLREDUCE_BACKEND=trtllm exec vllm serve --model /models/nvidia/MiniMax-M2.7-NVFP4 --max-model-len auto --gpu-memory-utilization 0.9 --port 8888 --host 0.0.0.0 --trust-remote-code --load-format fastsafetensors --moe-backend flashinfer_b12x --enable-prefix-caching --enable-auto-tool-choice --tool-call-parser minimax_m2 --reasoning-parser minimax_m2 --served-model-name my-minimax -tp 2 --distributed-executor-backend ray --max-num-seqs 4 --attention-backend flashinfer --quantization modelopt_fp4 --dtype bfloat16 --kv-cache-dtype fp8