Gemma-4 text generation throws on every CUDA build: shared forward calls the ROCm-only vt::GeluMulSeparate (SACRED gate RED on main since 0c2827c1)

[localai-bot]

Summary

Gemma-4 text generation throws on any CUDA build at the first layer with per-layer embeddings. The SACRED gate test_gemma4_paged_engine is RED on current main:

test case THREW exception: vt::GeluMulSeparate: ROCm-only fast path in this build

This is not a test-harness problem. The same call site is on the production forward, so examples/vllm-server with a Gemma-4 checkpoint on CUDA hits it too.

Mechanism

src/vllm/model_executor/models/gemma4.cpp:652 — the shared Gemma-4 forward, not a ROCm branch — calls vt::GeluMulSeparate unconditionally in the ple > 0 (per-layer-embedding) path:

vt::GeluMulSeparate(d.q, gated.ptr(), gate_lin.ptr(), ple_l.ptr(), T * ple, DType::kBF16);

vt::GeluMulSeparate (src/vt/fused_ops.cpp:56-71) has exactly one implementation:

#if defined(VLLM_CPP_HIP)
  if (q.device.type == DeviceType::kROCM) { rocm::GeluMulSeparateRocm(...); return; }
#endif
  throw std::runtime_error("vt::GeluMulSeparate: ROCm-only fast path in this build");

There is no CUDA, CPU, Metal or Vulkan path, and no build flag can supply one — the fallback is the throw. Gemma-4 E4B has per-layer embeddings, so ple > 0 and the very first layer aborts.

When it broke

0c2827c1 — perf(gemma4): dual-GPU FP8 resident without host OOM + peer mix (#154), 2026-08-08, on main.

That commit replaced a portable op with the ROCm-only one in the same change that introduced the throw:

-      vt::GeluAndMul(d.q, gated.t(), gate_in.t());  // gelu_tanh(gate_lin)*ple_l
+      vt::GeluMulSeparate(d.q, gated.ptr(), gate_lin.ptr(), ple_l.ptr(), T * ple, DType::kBF16);

git log -S"vt::GeluMulSeparate" -- src/vllm/model_executor/models/gemma4.cpp and git log -S"ROCm-only fast path in this build" -- src/vt/fused_ops.cpp both return that single commit.

Evidence

Two CUDA builds on dgx (GB10, nvcc 13.0, Release, -DVLLM_CPP_CUTLASS_DIR=$HOME/cutlass-4.5.0), run back to back under one flock $HOME/gpu.lock:

arm	tree	result
baseline	main @ 87308dea	THREW ... ROCm-only fast path in this build
with PR #317	main + #317 @ a5bd7ad0	identical throw

Identical failure on both arms, so this is not attributable to #317 — it was found while hardware-checking that PR. gemma4.cpp.o compiles byte-identical between the two arms (cmp), which is what makes the attribution airtight.

Why it went unnoticed

test_gemma4_paged_engine is checkpoint-gated and dgx-only: on CI the snapshot is absent, so it emits a SKIP and returns green. The gate can only fail where nobody runs it routinely.

Suggested fix

Give vt::GeluMulSeparate a portable fallback instead of a throw — the semantics are out[i] = gelu_tanh(gate[i]) * up[i], which composes from the existing vt::GeluTanh plus an elementwise multiply, at the cost of one temporary. The ROCm fused kernel stays the fast path where it exists. Reverting the call site to vt::GeluAndMul over a [T, 2*ple] buffer would also work but gives back the ROCm win.

Worth a check on the rest of src/vt/fused_ops.cpp in the same pass: vt::DualRmsNormPlusRes throws "CPU compose not yet wired" on the same principle, so this may not be the only shared call site reachable on a non-ROCm build.

[localai-bot]

Fixed on main as 59b426ae0d85.

The SACRED Gemma-4 gate passes on CUDA for the first time since 0c2827c1 (2026-08-08):

gemma4-E4B STRICT: 32/32 token-exact vs vLLM 0.25.0 golden.

Proof

Three arms built from three trees and run back to back inside one flock $HOME/gpu.lock on dgx (GB10, nvcc 13.0, Release, cutlass 4.5.0), so the only variable between them is the tree:

arm	tree	result
g4base	main	THREW vt::GeluMulSeparate: ROCm-only fast path in this build
g4pr317	main + #317	identical throw
g4fix	main + this fix	32/32 token-exact, SUCCESS

The two RED arms are what make the green one mean something — same box, same checkpoint, same minute. (#317 was the PR being hardware-checked when this was found; it is unrelated, and its gemma4.cpp.o is byte-identical to baseline.)

The fix

vt::GeluAndMul computes exactly this math — gelu_tanh(gate) * up — but wants the halves adjacent as one [rows, 2D] tensor. The portable fallback stages gate and up into a temporary [1, 2n] laid out as [gate | up] and runs the shipped kernel on it, so it matches GeluAndMul by construction on every backend that registers it rather than being a second copy of the gelu math that could drift. This is the same composed-reference shape RmsNormPlusAdd already uses one function above it in the same file. ROCm keeps its fused kernel untouched.

The temporary is read by work queued on q, so it is freed only after Synchronize (no-op on CPU; on CUDA it is what stops Free racing the kernel), and on the exception path too.

Evidence chain

check	result
RED first	3 cases throwing for the intended reason
GREEN after	test_ops_activation 21/21, 277,289 assertions
Mutation	swapping gate/up in the staging copies fails all 3 new cases
Full CPU suite	380/381 — sole failure is the #294 test_async_llm flake (10/10 serial on this build; diff touches nothing in the engine)
Clean build	0 warnings under -Werror, CPU and CUDA
CUDA gate	32/32 token-exact

42 lines of src/vt/fused_ops.cpp, 64 lines of tests.

Left open deliberately

1. Speed, not correctness. The compose costs one allocation, two copies and a synchronize per call against ROCm's single fused kernel. A real portable kernel — or an elementwise multiply op, which vt does not have (only MulScalar) — would remove all three. Worth its own row if the CUDA Gemma-4 decode path is ever measured.
2. Three siblings still throw by the identical pattern in the same file: vt::DualRmsNormPlusRes ("non-ROCm GPU path not registered" / "CPU compose not yet wired"), vt::MatmulBTAlphaBeta and vt::MatmulBTFp8Channel. None is reached by the Gemma-4 gate, so they were left alone rather than fixed blind. Whether any is reachable from a shared forward is the open question — that is exactly how this one got shipped.

Why it survived three days

test_gemma4_paged_engine is checkpoint-gated and dgx-only: absent the snapshot it emits a SKIP and reads green on CI. A gate that can only fail where nobody runs it routinely.