[Bug]: cutedsl kernel fails on upgrade to 4.6.0

[Anerudhan]

Before submitting

• I searched existing issues and did not find a duplicate.

Affected area

FE OSS kernels or CuTeDSL

Bug report

Description

All fp8 (MXF8 input / e4m3 output) variants of the grouped_gemm_dglu CuTe DSL kernel fail to compile with nvidia-cutlass-dsl==4.6.0 on sm_100a:

cutlass.base_dsl.compiler.CompilerDiagnosticError: error: NVVM backend compilation failed
  error: libNVVM failed while compiling generated device IR.
  note: target architecture: sm_100a
  note: backend log:
        NVVM backend compilation failed

NVVM_ERROR_COMPILATION is raised with an empty backend log, i.e. an internal libNVVM failure rather than a diagnosable IR error. The same kernel compiles and passes with nvidia-cutlass-dsl==4.5.0. This was previously masked in CI on the 4.6.0 upgrade branch (PR #368) by ReduxKind import errors that failed these tests at collection time before any kernel compiled.

Impact: 72/72 fp8 dglu tests fail (dense + discrete, dgeglu + dswiglu, static + dynamic sched, with and without dbias). All fp4 dglu variants (48/48) pass, as do all other kernel families (glu, glu_hadamard, dsrelu, srelu, quant, swiglu, dswiglu, wgrad, gemm_amax) on 4.6.0.

Steps to reproduce

Dependencies:

• nvidia-cutlass-dsl[cu13]==4.6.0 (with cuda-python, torch, apache-tvm-ffi — i.e. the [cutedsl] extra of nvidia-cudnn-frontend)
• Blackwell GPU (sm_100); reproduced on CUDA driver 590.xx, Python 3.12
• cudnn-frontend @ test/update_cutedsl (PR Update nvidia-cutlass-dsl version to 4.6.0 #368 head, d08c051) or any tree with the ReduxKind fix so the module imports

python -m pip install "nvidia-cutlass-dsl[cu13]==4.6.0" cuda-python torch apache-tvm-ffi pytest
python -m pytest -q \
  "test/python/fe_api/grouped_gemm/test_grouped_gemm_dglu.py::test_grouped_gemm_dglu_discrete_wrapper_fp8[n-dgeglu-static_sched]"

Fails in ~4 s at cute.compile() (grouped_gemm_dglu/api.py _compile_discrete). To reproduce without a test harness, dump the traced module and replay the pipeline:

CUTE_DSL_KEEP=all CUTE_DSL_DUMP_DIR=/tmp/dump CUTE_DSL_NO_CACHE=1 CUTE_DSL_DISABLE_FILE_CACHING=1 \
  python -m pytest -p no:cacheprovider -q "<test id above>"
# then parse /tmp/dump/cute_dsl_cutlass___call___*BlockScaledMoEGroupedGemmDgluDbiasKernel*.mlir (raw, non-_clean)
# in a cutlass MLIR context and run:
#   builtin.module(cute-to-nvvm{cubin-format=bin opt-level=3 rdc=false enable-assertions=false
#                  preserve-line-info=false enable-cuda-dialect=true cuda-dialect-external-module=true})
# -> NVVM_ERROR_COMPILATION reproduces in seconds

Expected behavior

Kernel compiles for sm_100a, as it does with nvidia-cutlass-dsl==4.5.0.

Actual behavior

NVVM_ERROR_COMPILATION / "libNVVM failed while compiling generated device IR", empty backend log. Also fails identically with opt-level=0.

Root-cause isolation (done via delta-debugging)

The failure is not attributable to any single construct — it appears to be a kernel-complexity/capacity failure in the libNVVM bundled with 4.6.0:

• Removing any one of these individually still fails identically: SFD row/col quantization (quant_sfd_row/col), the dgeglu/dswiglu activation, amax reduction, dbias atomics, dprob reduction, C-tensor loads, register→smem D copies, D TMA stores, or any single warp-specialized role.
• Removing the activation and the SFD quantization together makes the kernel compile and run — removing either alone does not.
• Every fp8-specific inline-asm helper (cvt.rn.satfinite.e4m3x2.f32, cvt.rp.satfinite.ue8m0x2.f32, cvt.rn.bf16x2.ue8m0x2, redux.sync.max.NaN.f32) compiles fine in small standalone sm_100a kernels.
• Reducing loop unrolling (unroll_full=True → unroll=1) throughout the kernel does not help.
• This explains the fp8/fp4 split: fp8 output enables generate_sfd (row+col quant epilogue), making dglu-fp8 the largest kernel variant (dual D outputs + derivative activation + SFD quant + dprob + dbias + amax); fp4 keeps generate_sfd=False and stays under the threshold.

Raw traced MLIR of a failing kernel and a standalone pipeline-replay script are available on request (~650 KB module; the replay reproduces the failure without a GPU test run).

Environment (where applicable)

cuDNN Frontend Version: 1.27.0
cuDNN Backend Version: 9.24.0
CUDA Toolkit/Runtime Version: 13.2 (torch), cutlass-dsl cu13 bundled toolchain
GPU: B200-class (sm_100)
CUDA driver: 590.xx
OS: Ubuntu 22.04 (Linux 5.15)
Python: 3.12
nvidia-cutlass-dsl: 4.6.0 (fails) / 4.5.0 (passes)