[CI][cutlass-dsl-4.6][Blackwell] grouped_gemm_dglu FP8 NVVM compilation failure

[liujane-dev]

Summary

oss:rel:cutlass_dsl_4.6: [Blackwell, shard1] is failing in the scheduled OSS test lane pinned to nvidia-cutlass-dsl==4.6.1 on Blackwell. The failure is an NVVM backend compilation failure while compiling grouped GEMM DGLU FP8 kernels for sm_100a.

This is separate from the oss:rel:cutlass_dsl_internal failures in jobs 369173159 / 369173161 / 369173162 because this job uses the cutlass_dsl_4.6 lane, not the internal 4.7.0 lane.

CI job

• Job: oss:rel:cutlass_dsl_4.6: [Blackwell, shard1] (#369173158, lane=parent, ref=develop)
• Job URL: 369173158
• Parent FE pipeline: 58916903

Version / introduction window

• cuDNN frontend at failure: 1e7b36ba
• Related lane-introducing commit: 5cae4f044 (CI: test OSS suite across cutlass-dsl 4.5.0 / 4.6.1 / internal (4.7.0) versions)
• Recent checked history only has this exact oss:rel:cutlass_dsl_4.6: [Blackwell, shard1] lane in pipeline 58916903, so this is best treated as a new lane compatibility failure, not a proven cuDNN backend regression.
• cuDNN backend branch: dev
• cuDNN: 9.25.0.17
• GPU / SM: 17:41.959868Z 01O PyTorch GPU Name: NVIDIA Graphics Device / SM 10.0

Failing tests

• First failing test: test/python/fe_api/grouped_gemm/test_grouped_gemm_dglu.py::test_grouped_gemm_dglu_dense_compile_execute_fp8[False-32-sf_dtype0-True-cluster_shape_mn0-mma_tiler_mn0-acc_dtype0-n-n-d_dtype0-c_dtype1-ab_dtype0-static_sched]
• Exit code: 1

Error signature

TEST_FAILURE
2026-07-21T12:21:14.038662Z 01O FAILED test/python/fe_api/grouped_gemm/test_grouped_gemm_dglu.py::test_grouped_gemm_dglu_dense_compile_execute_fp8[False-32-sf_dtype0-True-cluster_shape_mn0-mma_tiler_mn0-acc_dtype0-n-n-d_dtype0-c_dtype1-ab_dtype0-static_sched] - cutlass.base_dsl.compiler.CompilerDiagnosticError: error: NVVM backend compilation failed
2026-07-21T12:21:14.038664Z 01O   error: libNVVM failed while compiling generated device IR.
2026-07-21T12:21:14.038664Z 01O   note: target architecture: sm_100a
2026-07-21T12:21:14.038665Z 01O   note: backend log:
2026-07-21T12:21:14.038673Z 01O         NVVM backend compilation failed
2026-07-21T12:21:14.038674Z 01O   suggestion: check that CUDA_TOOLKIT_PATH is set correctly and that the CUDA toolkit supports
2026-07-21T12:21:14.038674Z 01O               target architecture sm_100a; rerun with IR dump flags if backend context is needed
2026-07-21T12:21:14.038675Z 01O FAILED test/python/fe_api/grouped_gemm/test_grouped_gemm_dglu.py::test_grouped_gemm_dglu_dense_compile_execute_fp8[False-32-sf_dtype0-True-cluster_shape_mn0-mma_tiler_mn0-acc_dtype0-n-n-d_dtype0-c_dtype1-ab_dtype0-dynamic_sched] - cutlass.base_dsl.compiler.CompilerDiagnosticError: error: NVVM backend compilation failed
2026-07-21T12:21:14.038676Z 01O   error: libNVVM failed while compiling generated device IR.
2026-07-21T12:21:14.038676Z 01O   note: target architecture: sm_100a
2026-07-21T12:21:14.038676Z 01O   note: backend log:
2026-07-21T12:21:14.038677Z 01O         NVVM backend compilation failed

Existing tracker check

• Local NvBug matcher returned a broad overlap with NvBug #6479077 on Blackwell / oss:rel; that appears to be an overmatch because #6479077 tracks DSA sparse-attention backward mismatch, not grouped_gemm / NVVM compilation failure.
• This draft is intentionally separate from the cutlass_dsl_internal multi-arch issue draft.

Reproduction

pytest test/python/fe_api/grouped_gemm/test_grouped_gemm_dglu.py::test_grouped_gemm_dglu_dense_compile_execute_fp8[False-32-sf_dtype0-True-cluster_shape_mn0-mma_tiler_mn0-acc_dtype0-n-n-d_dtype0-c_dtype1-ab_dtype0-static_sched]