Skip to content

cuDNN frontend grouped GEMM fusion API is restricted to PyTorch #260

Description

@Anerudhan

The cuDNN frontend has added grouped GEMM fusions, such as grouped GEMM + swiglu + MXFP8 quantize. However, these are only accessible through the public API here, which depends on torch and torch Tensors.
https://github.com/NVIDIA/cudnn-frontend/blob/97f6cb3b88cacff507cca1280db5650a457d92b3/python/cudnn/grouped_gemm/grouped_gemm_swiglu/api.py

Since this depends on torch Tensors, we cannot call this from JAX/XLA. So we’d like one of two options:

  1. CuTeDSL kernel is made public directly and a stable direct CuTeDSL kernel interface is committed to from cuDNN frontend team. JAX can call CuTeDSL kernels directly, it’s the public API torch Tensor wrapper that is problematic for us.
  2. Expose this functionality through the C++ cuDNN frontend API. We can call the C++ cuDNN frontend API from JAX/XLA, but not the cuDNN frontend Python API as it depends on torch Tensors

Metadata

Metadata

Assignees

Labels

cat-featureRequests for new functionality, APIs, examples, or behavior improvements.mod-frontendcuDNN frontend APIs, operation graph construction, plans, and user-facing wrappers.orig-nv-engReported or requested by NVIDIA engineering.

Type

No type

Projects

No projects

Relationships

None yet

Development

No branches or pull requests

Issue actions