[CI][cutlass-dsl-internal] OSS suite fails across Hopper/Blackwell lanes

[liujane-dev]

Summary

The scheduled OSS suite is failing across multiple oss:rel:cutlass_dsl_internal lanes after the new internal cutlass-dsl nightly jobs were added. These jobs all run FE commit 1e7b36ba in pipeline 58916903 with cuDNN 9.25.0.17 and install nvidia-cutlass-dsl-internal (unreleased 4.7.0 line).

This should be tracked separately from job 368076777 / issue 418: that older tracker is for oss:rel: [Blackwell, shard1] DSA sparse-attention backward mismatch on the normal OSS lane, not the new cutlass-dsl-internal lanes.

CI jobs

• oss:rel:cutlass_dsl_internal: [Hopper] — https://nv/cudnn-frontend/-/jobs/369173159 — FAILED test/python/fe_api/bsa/test_BSA_attention_forward.py::test_bsa_attention_forward_fixed_blocks - worker 'gw2' crashed while running 'fe_api/bsa/test_BSA_attention_forward.py::test_bsa_attention_forward_fixed_blocks'
• oss:rel:cutlass_dsl_internal: [Blackwell, shard0] — https://nv/cudnn-frontend/-/jobs/369173161 — timeout after 1h with crashed xdist worker / many failing tests before pytest summary
• oss:rel:cutlass_dsl_internal: [Blackwell, shard1] — https://nv/cudnn-frontend/-/jobs/369173162 — timeout after 1h with crashed xdist worker / many failing tests before pytest summary
• oss:rel:cutlass_dsl_internal: [Blackwell, shard1] — https://nv/cudnn-frontend/-/jobs/369173162 — timeout after 1h with crashed xdist worker / many failing tests before pytest summary

Introduction window

• Failure pipeline: #58916903 (https://nv/cudnn-frontend/-/pipelines/58916903)
• FE commit at failure: 1e7b36ba
• Lane-introducing commit: 5cae4f044 (CI: test OSS suite across cutlass-dsl 4.5.0 / 4.6.1 / internal (4.7.0) versions)
• The oss:rel:cutlass_dsl_internal lanes did not exist in the checked prior pipelines, so this appears introduced/exposed by the new CI lane rather than by the older 2a823092 Blackwell DSA fix.

Representative failures

Hopper

FAILED test/python/fe_api/bsa/test_BSA_attention_forward.py::test_bsa_attention_forward_fixed_blocks - worker 'gw2' crashed while running 'fe_api/bsa/test_BSA_attention_forward.py::test_bsa_attention_forward_fixed_blocks'
FAILED test/python/fe_api/dsa/test_DSA_indexer_top_k.py::test_DSA_indexer_top_k_compile_execute[True-1-705-acc_dtype0-dtype0] - AssertionError: Tensor-likes are not equal!
Greatest absolute difference: inf at index (0,)

Blackwell shard0 / shard1

xdist workers report `node down: Not properly terminated` / `Fatal Python error: Aborted`.
Jobs then time out after 1h before pytest can emit a complete failure summary.

Existing tracker check

• Local NvBug matcher only weakly overlaps with NvBug #6479077 on broad oss:rel; treat that as an overmatch for these internal cutlass-dsl lane failures.
• GitHub issue search found no existing issue for cutlass_dsl_internal, test_bsa_attention_forward_fixed_blocks, BSA_attention_forward, or DSA_indexer_top_k.

Reproduction

pytest test/python/fe_api/dsa/test_DSA_indexer_top_k.py::test_DSA_indexer_top_k_compile_execute[True-1-705-acc_dtype0-dtype0]