Decode MXFP4 (fmt=7) on CUDA, so Kimi K3 has a CUDA expert tier

[ZacharyZcR]

The gap

Kimi K3's routed experts are QAT in MXFP4 and streamed un-re-encoded, so fmt=7 is the format its expert tier runs on. Only the Vulkan shader could decode it — the CUDA backend understood fmt 0/1/2/3/4/6 and nothing else. c/Makefile said so outright:

@echo "*** kimi_k3 has no CUDA backend (#783): building without it."
@echo "*** On NVIDIA, Kimi K3 runs on the Vulkan path"

…while forcing NOCUDA_CFLAGS to strip -DCOLI_CUDA back out. An NVIDIA host either went through Vulkan or ran the experts on CPU.

What this adds

fmt=7 in quant_matmul plus a stateless entry point (coli_cuda_matmul_mxfp4), and kimi_k3.c's expert_apply tries it first under K3_CUDA=1.

Decode only — at S>1 the CPU kernels amortise over the batch and a per-call upload would not pay. It returns 0 with the output untouched on any failure, so the caller falls through to disk+CPU exactly as it does when Vulkan declines. That is the same try-then-fall-back contract vLLM's MXFP4 backends use (FlashInfer/AITER when available, an emulation path when not).

Two decisions the CPU reference dictated

Exponent as a bit pattern, not exp2f. (uint32)s << 23 read as float is 2^(s-127) for s ∈ [1,254], and it reproduces the CPU path's documented edges exactly — s=0 → +0, s=255 → +inf. exp2f would agree across the normal range and diverge at precisely the two values where a silent mismatch would hide.

e2m1 computed arithmetically, not from a __constant__ table. A file-scope __constant__ array with static linkage is initialised per translation unit, and this file is also compiled into the HIP build and the Windows DLL.

Verified on hardware, not just compiled

tests/test_mxfp4_cuda.cu diffs the kernel against quant.h's matmul_mxfp4 — the CPU path the engine already trusts. On an RTX 4070 (sm_89):

ok   all 16 e2m1 codes decode exactly (cpu == gpu == spec)
ok   decode + matmul              S=1 I=64   O=32  worst rel 2.43e-06
ok   multi-row batch              S=4 I=128  O=64  worst rel 1.59e-05
ok   wide rows (many groups)      S=1 I=2048 O=16  worst rel 8.41e-07
ok   non-multiple-of-32 columns   S=1 I=96   O=8   worst rel 1.23e-07
ok   tail group (I%32 != 0)       S=1 I=80   O=8   worst rel 6.40e-06
ok   exponent 0 -> +0 / 255 -> inf / 127 -> unit scale
test_mxfp4_cuda: ok

The test earned its place on the first run. Results came back off by 1e38. One look showed why — quant_matmul ends with:

y[...] = (fmt && fmt != 4 && fmt != 6) ? partial[0] * scales[o] : partial[0];

fmt=7 was not on that exemption list, so the per-group result got multiplied again by scales[o] — and for MXFP4 that pointer is ue8m0 bytes, so it did not merely double-scale, it read garbage as float. A compile check passes this. Only a differential run against the CPU catches it.

Tolerance is 1e-4 relative, not bit-exactness: the two accumulate in a different order (CPU serial over columns, CUDA strided across threads then reduced). Decode errors are not subtle at that scale — the bug above moved results by 30+ orders of magnitude.

Also in scope

make test-c passes, all four engines build, kimi_k3.c compiles both with and without -DCOLI_CUDA, and the stale #783 message plus the NOCUDA_CFLAGS override are gone from the kimi_k3 rule.

Not covered

End-to-end Kimi K3 on CUDA. That needs the 1.6 TB checkpoint, which a 12 GB card cannot hold. What is proven here is that the kernel decodes MXFP4 correctly on real hardware; throughput on a real model still wants a machine that can load one.

Also unmeasured: whether the per-call upload is worth it at decode on a real expert tier. The knob is opt-in (K3_CUDA=1, default off) precisely because that trade has not been measured yet.