Skip to content

Cuda backend silently returns standard (+, ×) results for tropical algebras — dispatch ignores the algebra parameter (follow-up to #6) #48

Description

@isPANN

Summary

Einsum::execute::<MaxPlus<f64>, f64, Cuda>(...) compiles, runs, and
returns a value. The value is the standard sum-product contraction,
not max-plus. The Cuda::contract dispatch table examines only
TypeId::of::<A::Scalar>() (i.e. f64) and routes every f64 call
through contract_cutensor, which is cuTENSOR's standard (+, ×)
kernel. The algebra parameter A is effectively discarded.

The same scenario on Cpu (execute::<MaxPlus<f64>, f64, Cpu>) is
correct — Cpu's contract path is generic over A and threads the
algebra through. Only the Cuda backend has this silent
misdispatch.

This appears to be unfinished work from #6, which delivered cuTENSOR +
storage + tests but never wired up the tropical path. The cuda
feature on current main is cuda = ["dep:cudarc"] and pulls
nothing tropical-specific; the tropical-gemm-cuda crate that #6
originally listed in the cuda feature is still declared as an
optional dependency in Cargo.toml but is no longer gated by any
feature, making it permanently dead in the dependency graph. RFC #17
lists "CUDA backend with cuTENSOR integration" and "Multiple algebras
(Standard, MaxPlus, MinPlus, MaxMul)" side by side under "What We
Have"; the cross-product of the two is the silent-bug surface.

Reproducer

Tested on current main (06bd29a, fix: skip ignored lockfile in
release target
), with cudarc 0.12.1 + CUDA toolkit 12.6 on an
HKUST-GZ HPC2 login node.

// repro/Cargo.toml
// omeinsum = { path = "../omeinsum-rs", features = ["tropical", "cuda"] }

// repro/src/lib.rs
use omeinsum::{Cuda, Einsum, MaxPlus, Tensor};

pub fn this_compiles_and_silently_misdispatches(
    ein: &Einsum,
    a: &Tensor<f64, Cuda>,
    b: &Tensor<f64, Cuda>,
) -> Tensor<f64, Cuda> {
    ein.execute::<MaxPlus<f64>, f64, Cuda>(&[a, b])
}

cargo check --features tropical,cuda finishes cleanly:

...
   Checking tropical-gemm v0.2.0
   ...
   Checking omeinsum v0.1.0 (.../omeinsum-rs)
warning: trait `CudaScalar` is never used
   --> src/backend/cuda/mod.rs:419:11
    |
419 | pub trait CudaScalar:
    |           ^^^^^^^^^^

warning: `omeinsum` (lib) generated 1 warning
    Checking repro v0.1.0 (.../repro)
    Finished `dev` profile [unoptimized + debuginfo] target(s) in 31.19s

Two things to notice in this build:

  • The call compiles. The trait bound T: BackendScalar<B> with
    T = f64, B = Cuda is satisfied by impl BackendScalar<Cuda> for f64 (src/backend/traits.rs:207). There is no compile-time guard
    catching the tropical-vs-Cuda mismatch.
  • tropical-gemm-cuda is never compiled or checked. With the
    cuda feature enabled, only tropical-gemm (CPU) and cudarc
    appear in the build graph. The dependency declared at
    Cargo.toml:33 is gated by no feature, so it never enters the
    graph.

Why the dispatch is wrong

src/backend/cuda/mod.rs, in the Backend for Cuda impl block:

fn contract<A: Algebra>(
    &self,
    a: &CudaStorage<A::Scalar>,
    /* shapes, strides, modes ... */
) -> CudaStorage<A::Scalar>
where
    A::Scalar: BackendScalar<Self>,
{
    let strides_c = Self::compute_strides(shape_c);
    use std::any::TypeId;

    if TypeId::of::<A::Scalar>() == TypeId::of::<f32>() {
        /* transmute &CudaStorage<A::Scalar> → &CudaStorage<f32>,
           call self.contract_cutensor(...), transmute back */
    } else if TypeId::of::<A::Scalar>() == TypeId::of::<f64>() {
        /* same with f64 */
    } else if TypeId::of::<A::Scalar>() == TypeId::of::<CudaComplex<f32>>() {
        /* ... */
    } else if TypeId::of::<A::Scalar>() == TypeId::of::<CudaComplex<f64>>() {
        /* ... */
    } else {
        panic!(
            "CUDA backend only supports f32, f64, CudaComplex<f32>, and \
             CudaComplex<f64> for contractions. Got type: {:?}",
            std::any::type_name::<A::Scalar>()
        );
    }
}

contract_cutensor is the standard cuTENSOR contract — (+, ×)
arithmetic, no semiring parameter. The A generic is bound but
never inspected. For any algebra whose Scalar is one of
{f32, f64, Complex32, Complex64} — which is every tropical algebra
over those scalars — the dispatch silently returns a standard-algebra
result. No panic, no error, no warning.

For comparison, src/backend/cpu/mod.rs's contract<A: Algebra> is
algebra-generic all the way down (calls A::add/A::mul through the
Semiring trait), so Cpu is unaffected by this issue.

Other related artifacts in the same file

  • Cuda::contract_with_argmax is an unconditional panic!(), with
    the comment "A custom kernel would be needed for tropical
    backpropagation on GPU.
    ". Tropical backprop on GPU is therefore
    also blocked, but at least loudly.
  • pub trait CudaScalar at line 419 has impls for f32 and f64
    only and is never referenced anywhere else in the codebase — rustc
    flags it dead_code. From the name and shape it looks like an
    earlier attempt at the scalar-bound that should have driven tropical
    dispatch, and was orphaned.

What tropical-gemm-cuda already provides (currently unused)

The crate declared at Cargo.toml:33 (TensorBFS/tropical-gemm v0.2,
tropical-gemm-cuda) already ships:

  • tropical_gemm_gpu<T: CudaKernel>(ctx, &a, &b, &mut c) and
    tropical_matmul_gpu for TropicalMaxPlus<f32/f64>,
    TropicalMinPlus<f32/f64>, TropicalMaxMul<f32/f64>, plus i32/i64
    variants.
  • A CudaKernelWithArgmax trait and launch_gemm_*_with_argmax_*
    entry points that return (C, argmax_indices) — the primitive
    needed for configuration recovery / tropical backprop.
  • A persistent CudaContext with an NVRTC plan cache that amortizes
    the ~7 s kernel-compile cost.

The non-trivial parts (column-major layout matching cuTENSOR,
argmax-tracking kernels, plan cache) are upstream of omeinsum-rs.
What's missing is dispatch glue and a feature wiring that pulls the
crate into the build.

Why this matters

A backend that returns a different mathematical operation than its
type signature claims is the worst kind of bug for a numerical
library — it goes undetected by typical sanity checks, contaminates
downstream results, and only surfaces when someone happens to compare
against a CPU baseline. Tropical TN contraction is the most common
use case for omeinsum-rs outside of standard linear algebra (MIS,
QUBO, factor-graph MAP, viterbi), so anyone moving such a workload
onto the Cuda backend hits this trap.

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions