Skip to content

GPU CI fails on main with CUDNN_STATUS_EXECUTION_FAILED_CUDART #222

Description

@ChrisRackauckas-Claude

GPU CI is failing on main with a cuDNN execution error. I hit the same failure while checking PR #221, then verified it also appears on the unmodified base branch.

Main run: https://github.com/SciML/DeepEquilibriumNetworks.jl/actions/runs/28940793272
Main GPU job: https://github.com/SciML/DeepEquilibriumNetworks.jl/actions/runs/28940793272/job/85862242539

PR run with same failure: https://github.com/SciML/DeepEquilibriumNetworks.jl/actions/runs/28940670571/job/85861840439

CI reproduction context from the job:

import Pkg
Pkg.test(;
    coverage = true,
    julia_args = ["--check-bounds=yes", "--compiled-modules=yes", "--depwarn=yes"],
    force_latest_compatible_version = false,
    allow_reresolve = true,
)

with GROUP=GPU on Julia 1.12.6.

Representative failure output:

Warning: No valid algorithm found, probably bad params for convolution.
x_size: (3, 3, 1, 3): Error During Test at test/layers_tests.jl:41
CUDNNError: CUDNN_STATUS_EXECUTION_FAILED_CUDART (code 5003)
...
Layers Tests | 734 pass, 36 error, 770 total
DEQ/cuda | 540 pass, 36 error, 576 total

The same CI log also shows a CUDA runtime artifact download data error earlier in the job:

Downloading artifact: CUDA_Runtime
ERROR: Data Error : jl_dgluVCHUZS-download
Failure artifact: CUDA_Runtime

I did not bisect locally because this checkout is shallow/grafted and I do not have a local GPU reproduction environment available here. This appears independent of PR #221's documentation-only changes because it reproduces on main.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions