[CI][DSA][Blackwell] oss:rel shard1 sparse attention backward Tensor-likes mismatch

[liujane-dev]

Summary

oss:rel: [Blackwell, shard1] is failing on Blackwell with TEST_FAILURE in DSA sparse attention backward.

CI job

Jobs:

• Job: oss:rel: [Blackwell, shard1] (#368076777, lane=parent, ref=develop)
• Job URL: 368076777
• Parent FE pipeline: 58766219
• Anchor pipeline:
  • Last successful pipeline 58113302
  • First failing pipeline 58282151

Version window

• cuDNN frontend: 82b97bac → b641fef0
• cuDNN backend: 9.25.0.11-cb2f1a05-58087465 → 9.25.0.11-cb2f1a05-5808746 (branch: dev)
• Container: (cudnn/cudnn_frontend:cudnn_13.3.0)
• CUDA: 13.3
• cuDNN: 9.25.0.15-1c5ca8e8-58515561
• GPU / SM: NVIDIA Graphics Device / SM 10.0

Failing test

• First failing test: test/python/fe_api/dsa/test_DSA_sparse_attention_backward.py::test_DSA_sparse_attention_backward_wrapper[False-512-576-512-32-acc_dtype0-dtype0]
• Exit code: 1

Error snippet

FAILED test/python/fe_api/dsa/test_DSA_sparse_attention_backward.py::test_DSA_sparse_attention_backward_wrapper[False-512-576-512-32-acc_dtype0-dtype0] - AssertionError: Tensor-likes are not close!
Mismatched elements: 15 / 32 (46.9%)
Greatest absolute difference: 0.8858855962753296 at index (5,) (up to 0.05 allowed)
Greatest relative difference: 1.0 at index (0,) (up to 0.05 allowed)
FAILED test/python/fe_api/dsa/test_DSA_sparse_attention_backward.py::test_DSA_sparse_attention_backward_wrapper[True-512-576-512-32-acc_dtype0-dtype0] - AssertionError: Tensor-likes are not close!
Mismatched elements: 97741 / 18874368 (0.5%)

Bisect result:

• Target repository: cudnn_frontend
• Target commit: dc953fa
• Target title: Support SM90 DSA qh16 indexer forward and fix qh32 sparse backward (Support SM90 DSA qh16 indexer forward and fix qh32 sparse backward #388)

Root cause analysis:

• Commit dc953fa introduced/enabled the DSA sparse attention backward qh32 MLA shape (head_dim=576, head_dim_v=512, num_heads=32) in the L0 parameterization. This exact shape is the first failing test in the trace.
• The same commit rewrote the SM90 sparse_attention_backward scheduling/LSE+dPsum loading path in python/cudnn/deepseek_sparse_attention/sparse_attention_backward/dsa_bwd_sm90.py. The kernel now maps work tiles as one CTA per query with qhead tile indexing, then loads per-head LSE/dPsum via _load_f32_head_to_smem. For the qh32 MLA shape, the produced dq/dkv/d_sink do not match the PyTorch reference.
• Since both the passing anchor and first failing job use cuDNN 9.25.0.11-cb2f1a05, this is attributed to the cudnn_frontend FE change rather than a cuDNN backend version change.

Reproduction

pytest -q test/python/fe_api/dsa/test_DSA_sparse_attention_backward.py -k 'test_DSA_sparse_attention_backward_wrapper and 576 and 512 and 32'