cudaPackages_13_0: tracking issue

[bryanhonof]

CC: @NixOS/cuda-maintainers

I've tried bumping the CUDA version in nixpkgs to CUDA 13.0 that was released earlier today. Most of it seemed to have gone smooth, except when I got to cuTENSOR. They seem to have changed the manifest format, making @ConnorBaker's cuda-redist-find-features script` fail when trying to generate the feature manifest JSON.

$ nix run github:connorbaker/cuda-redist-find-features -- \
   process-manifests \
   --log-level DEBUG \
   --version 2.3.0 \
   https://developer.download.nvidia.com/compute/cutensor/redist \
   ./pkgs/development/cuda-modules/cutensor/manifests
/nix/store/xf850yly6iknkw00x2ssa6p4mw3kjnxv-python3.12-cuda-redist-find-features-0.1.1/lib/python3.12/site-packages/cuda_redist_find_features/types/_version.py:12: UserWarning: `frozen` is set via both the `dataclass` decorator and `config` for dataclass 'Version'.This is not recommended. The `frozen` specification on `dataclass` will take priority.
  @dataclasses.dataclass(frozen=True, order=True, slots=True, config=model_config)
Set logging level to DEBUG.
Version set to 2.3.0.
Using URL https://developer.download.nvidia.com/compute/cutensor/redist.
Using dir pkgs/development/cuda-modules/cutensor/manifests
2025-08-26T13:25:52 DEBUG    Fetching manifests from pkgs/development/cuda-modules/cutensor/manifests...                                                                   _manifest_ref.py:125
                    DEBUG    Globbing for redistrib_2.3.0.json in pkgs/development/cuda-modules/cutensor/manifests.                                                        _manifest_ref.py:120
                    DEBUG    Found 1 manifests in 0 seconds.                                                                                                               _manifest_ref.py:138
                    INFO     Reading manifest from pkgs/development/cuda-modules/cutensor/manifests/redistrib_2.3.0.json...                                                 _manifest_ref.py:44
Traceback (most recent call last):
  File "/nix/store/xf850yly6iknkw00x2ssa6p4mw3kjnxv-python3.12-cuda-redist-find-features-0.1.1/bin/..cuda-redist-find-features-wrapped-wrapped", line 9, in <module>
    sys.exit(main())
             ^^^^^^
  File "/nix/store/9l7cdymnw9ck7l7hk1d7n6yc2ql3a0j0-python3.12-click-8.1.7/lib/python3.12/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/nix/store/9l7cdymnw9ck7l7hk1d7n6yc2ql3a0j0-python3.12-click-8.1.7/lib/python3.12/site-packages/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
         ^^^^^^^^^^^^^^^^
  File "/nix/store/9l7cdymnw9ck7l7hk1d7n6yc2ql3a0j0-python3.12-click-8.1.7/lib/python3.12/site-packages/click/core.py", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/nix/store/9l7cdymnw9ck7l7hk1d7n6yc2ql3a0j0-python3.12-click-8.1.7/lib/python3.12/site-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/nix/store/9l7cdymnw9ck7l7hk1d7n6yc2ql3a0j0-python3.12-click-8.1.7/lib/python3.12/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/nix/store/xf850yly6iknkw00x2ssa6p4mw3kjnxv-python3.12-cuda-redist-find-features-0.1.1/lib/python3.12/site-packages/cuda_redist_find_features/cmd/__init__.py", line 88, in process_manifests
    process_manifests_impl(
  File "/nix/store/xf850yly6iknkw00x2ssa6p4mw3kjnxv-python3.12-cuda-redist-find-features-0.1.1/lib/python3.12/site-packages/cuda_redist_find_features/cmd/process_manifests_impl.py", line 70, in process_manifests_impl
    ref.ref: (ref.version, ref.parse()) for ref in refs
                           ^^^^^^^^^^^
  File "/nix/store/xf850yly6iknkw00x2ssa6p4mw3kjnxv-python3.12-cuda-redist-find-features-0.1.1/lib/python3.12/site-packages/cuda_redist_find_features/manifest/nvidia/_manifest_ref.py", line 56, in parse
    manifest = NvidiaManifest.model_validate_json(content_bytes)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/nix/store/xq8y47ncc2bmb287wswaxv2vxzzmg3r8-python3.12-pydantic-2.9.2/lib/python3.12/site-packages/pydantic/main.py", line 625, in model_validate_json
    return cls.__pydantic_validator__.validate_json(json_data, strict=strict, context=context)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
pydantic_core._pydantic_core.ValidationError: 13 validation errors for NvidiaManifest
libcutensor.linux-x86_64.relative_path
  Field required [type=missing, input_value={'cuda12': {'relative_pat...', 'size': '386289552'}}, input_type=dict]
    For further information visit https://errors.pydantic.dev/2.9/v/missing
libcutensor.linux-x86_64.sha256
  Field required [type=missing, input_value={'cuda12': {'relative_pat...', 'size': '386289552'}}, input_type=dict]
    For further information visit https://errors.pydantic.dev/2.9/v/missing
libcutensor.linux-x86_64.md5
  Field required [type=missing, input_value={'cuda12': {'relative_pat...', 'size': '386289552'}}, input_type=dict]
    For further information visit https://errors.pydantic.dev/2.9/v/missing
libcutensor.linux-x86_64.size
  Field required [type=missing, input_value={'cuda12': {'relative_pat...', 'size': '386289552'}}, input_type=dict]
    For further information visit https://errors.pydantic.dev/2.9/v/missing
libcutensor.cuda_variant
  Input should be an object [type=model_type, input_value=['12', '13'], input_type=list]
    For further information visit https://errors.pydantic.dev/2.9/v/model_type
libcutensor.linux-sbsa.relative_path
  Field required [type=missing, input_value={'cuda12': {'relative_pat...', 'size': '385042432'}}, input_type=dict]
    For further information visit https://errors.pydantic.dev/2.9/v/missing
libcutensor.linux-sbsa.sha256
  Field required [type=missing, input_value={'cuda12': {'relative_pat...', 'size': '385042432'}}, input_type=dict]
    For further information visit https://errors.pydantic.dev/2.9/v/missing
libcutensor.linux-sbsa.md5
  Field required [type=missing, input_value={'cuda12': {'relative_pat...', 'size': '385042432'}}, input_type=dict]
    For further information visit https://errors.pydantic.dev/2.9/v/missing
libcutensor.linux-sbsa.size
  Field required [type=missing, input_value={'cuda12': {'relative_pat...', 'size': '385042432'}}, input_type=dict]
    For further information visit https://errors.pydantic.dev/2.9/v/missing
libcutensor.windows-x86_64.relative_path
  Field required [type=missing, input_value={'cuda12': {'relative_pat...', 'size': '150006949'}}, input_type=dict]
    For further information visit https://errors.pydantic.dev/2.9/v/missing
libcutensor.windows-x86_64.sha256
  Field required [type=missing, input_value={'cuda12': {'relative_pat...', 'size': '150006949'}}, input_type=dict]
    For further information visit https://errors.pydantic.dev/2.9/v/missing
libcutensor.windows-x86_64.md5
  Field required [type=missing, input_value={'cuda12': {'relative_pat...', 'size': '150006949'}}, input_type=dict]
    For further information visit https://errors.pydantic.dev/2.9/v/missing
libcutensor.windows-x86_64.size
  Field required [type=missing, input_value={'cuda12': {'relative_pat...', 'size': '150006949'}}, input_type=dict]
    For further information visit https://errors.pydantic.dev/2.9/v/missing

pkgs/development/cuda-modules/cutensor/manifests/redistrib_2.1.0.json

{
    "release_date": "2025-01-27",
    "release_label": "2.1.0",
    "release_product": "cutensor",
    "libcutensor": {
        "name": "NVIDIA cuTENSOR",
        "license": "cuTensor",
        "license_path": "libcutensor/LICENSE.txt",
        "version": "2.1.0.9",
        "linux-x86_64": {
            "relative_path": "libcutensor/linux-x86_64/libcutensor-linux-x86_64-2.1.0.9-archive.tar.xz",
            "sha256": "ee59fcb4e8d59fc0d8cebf5f7f23bf2a196a76e6bcdcaa621aedbdcabd20a759",
            "md5": "ed15120c512dfb3e32b49103850bb9dd",
            "size": "814871140"
        },
        "linux-sbsa": {
            "relative_path": "libcutensor/linux-sbsa/libcutensor-linux-sbsa-2.1.0.9-archive.tar.xz",
            "sha256": "cef7819c4ecf3120d4f99b08463b8db1a8591be25147d1688371024885b1d2f0",
            "md5": "fec00a1a825a05c0166eda6625dc587d",
            "size": "782008004"
        },
        "windows-x86_64": {
            "relative_path": "libcutensor/windows-x86_64/libcutensor-windows-x86_64-2.1.0.9-archive.zip",
            "sha256": "ed835ba7fd617000f77e1dff87403d123edf540bd99339e3da2eaab9d32a4040",
            "md5": "9efcbc0c9c372b0e71e11d4487aa5ffa",
            "size": "1514752712"
        }
    }
}

pkgs/development/cuda-modules/cutensor/manifests/feature_2.3.0.json

{
    "release_date": "2025-08-11",
    "release_label": "2.3.0",
    "release_product": "cutensor",
    "libcutensor": {
        "name": "NVIDIA cuTENSOR",
        "license": "cuTensor",
        "license_path": "libcutensor/LICENSE.txt",
        "version": "2.3.0.6",
        "linux-x86_64": {
            "cuda12": {
                "relative_path": "libcutensor/linux-x86_64/libcutensor-linux-x86_64-2.3.0.6_cuda12-archive.tar.xz",
                "sha256": "8903ea04daa32e25dda1b4d12cbdb475aefb2bb3f2317413b16c96fcadd8fd8b",
                "md5": "2a793f1808ab95eac80414e6f5f95450",
                "size": "429247820"
            },
            "cuda13": {
                "relative_path": "libcutensor/linux-x86_64/libcutensor-linux-x86_64-2.3.0.6_cuda13-archive.tar.xz",
                "sha256": "2d2768520b2d7f76a7081ddbe64c3cf61094e3e668ea296ac943c4f8435a1441",
                "md5": "fe9b4d5fcceab428189688f6c1518748",
                "size": "386289552"
            }
        },
        "cuda_variant": [
            "12",
            "13"
        ],
        "linux-sbsa": {
            "cuda12": {
                "relative_path": "libcutensor/linux-sbsa/libcutensor-linux-sbsa-2.3.0.6_cuda12-archive.tar.xz",
                "sha256": "fdf5b10e6d09a30dae7e2956333cdb02c580a9b61d447154b7de8127ed7fa9d8",
                "md5": "34cce96f713430945bdbbb0ca379111b",
                "size": "427353736"
            },
            "cuda13": {
                "relative_path": "libcutensor/linux-sbsa/libcutensor-linux-sbsa-2.3.0.6_cuda13-archive.tar.xz",
                "sha256": "380d7b2bf8f3aebbe467225345008ccb27256d60fd1dad38665f4c7d23278a76",
                "md5": "0c42604f9773f8032ed5de61a2dcc5a1",
                "size": "385042432"
            }
        },
        "windows-x86_64": {
            "cuda12": {
                "relative_path": "libcutensor/windows-x86_64/libcutensor-windows-x86_64-2.3.0.6_cuda12-archive.zip",
                "sha256": "0df80ccbfc1c607290e4ee0ef6f5400deaac674f1937f4609bff9215d1c39aef",
                "md5": "1b839d28bc670f5d5ee8ac2b47169ff8",
                "size": "218110876"
            },
            "cuda13": {
                "relative_path": "libcutensor/windows-x86_64/libcutensor-windows-x86_64-2.3.0.6_cuda13-archive.zip",
                "sha256": "643998686714108f178d1cd395aada3dd53ac89c29bc91995c42380cc4f66cec",
                "md5": "9858a29ec264d5bad8fe78c694e5c40a",
                "size": "150006949"
            }
        }
    }
}

It looks like NVIDIA did start nesting the library for every supported CUDA version? Instead of having multiple sub-directories in lib, as mentioned here?

I'm assuming this has some implications on how cuTENSOR is packaged?

[bryanhonof]

Related: NVIDIA/build-system-archive-import-examples#6
Related: #436186

[ConnorBaker]

Should be closed by #437723, when that lands.

[SomeoneSerge]

The extra nesting thing was addressed in #406740 but I need to rebase it...

[bryanhonof]

🥳 #437723 got merged.