mudler · localai-bot · Aug 11, 2026 · Aug 11, 2026 · Aug 11, 2026 · Aug 11, 2026
diff --git a/.agents/NOW.md b/.agents/NOW.md
@@ -29,7 +29,7 @@ Work: exact-chunks on main `1ce0d662b`; sm_120 measured at `3d2581551`.
 | `SERVE-METRICS` async (#277) | **`/metrics` was DEAD on the shipped server**: AsyncLLM folded nothing. Now live, ctest 366/366 | Config-gated families |
 | `ENG-LOAD-DIRECT-UPLOAD` (#150) | **default ON:** weights VIEW the mmap; 27B load **1.54x warm / 1.61x cold** | merged qkv/gate_up + lm_head |
 | Vulkan 27B | decode **MET 4.36 vs 4.35**. **LOADMEM: load held the model TWICE, 100.759 -> 53.413 GiB** | Load-phase host build is the new peak |
-| `BACKEND-ROCM` | **(b) fix in; #140 gfx1201 hipBLAS + Gemma-4 MoE landed; W0 green** | compile + M2 ([spec](specs/rocm-unified-memory-b.md)) |
+| `BACKEND-ROCM` | (b) in; #140 landed; W0 green; gfx1200 M0-M4 (#269) | APU compile+M2; GDN families claimed — `CLAIM-ROCM-GDN-KERNELS` ([spec](specs/rocm-gdn-kernels.md)) |
 | TP spike #287 (PR #143) | **TP-W1 LANDED**: rank-group table + TP handle (6/6); DSR leak FIXED (unblocks #127/#154/#155) | TP-W2 (linears + loader) |
 | Release | **ACTIVE; required W1-W11/W13 implemented in #196** | Finish hosted ten-SM proof; rebase/push; run full eight-tuple dry run |
 | `SAMPLE-PROMPT-LOGPROBS` (#223) | **LANDED** 21/21 | W2 `echo`; CUDA PENDING |

diff --git a/.agents/coordination.md b/.agents/coordination.md
@@ -1913,6 +1913,7 @@ this claim will meet. The tiled row is speed-gatable on dgx.
 | `CLAIM-PARAKEET-MODEL-P4` | `MODEL-AUDIO-PARAKEET-ENCODER`, `MODEL-AUDIO-PARAKEET-TRANSDUCER` | Claude Opus 5 (1M context) | isolated worktree `/home/mudler/_git/vllm.cpp-parakeet` (CPU-only) | branch `row/MODEL-AUDIO-PARAKEET-ENCODER`, base `main` | The encoder, the CTC head with greedy collapse, the log-mel front end, an HF-safetensors loader and the RNN-T/TDT transducer. Owns `src/vllm/model_executor/models/parakeet_*`, `src/vllm/multimodal/parakeet_audio_processor.*`, `examples/parakeet_transcribe/`, the Parakeet tests and the two model-matrix rows. | `ACTIVE` | 2026-08-07 CPU-LANDED + gated against a HF `ParakeetForCTC` oracle with EXACT token ids; transcribes ctc-0.6b/1.1b, rnnt-0.6b, tdt-0.6b-v3. No CUDA, no aarch64, no speed number claimed. |
 | `CLAIM-BACKEND-TENSTORRENT-SPIKE` | `BACKEND-TENSTORRENT` (`INVENTORIED`→`ACTIVE`; W0 skeleton, one op) | Claude Code (claude-sonnet-5) | isolated worktree `/home/lu_zero/Sources/vllmcpp-tenstorrent`; real Blackhole (P150) hardware available all session | branch `row/BACKEND-TENSTORRENT`, base `origin/main` `7534da65`, draft PR [#197](https://github.com/mudler/vllm.cpp/pull/197) | New extension platform (no upstream vLLM equivalent, same class as Metal/Vulkan): `DeviceType::kTENSTORRENT`, `vt::tenstorrent::Backend` (host-staged, discrete PCIe), `TenstorrentPlatform`, ONE op (`kMatmul`, F32/rank-2) via `ttnn::operations::matmul::matmul`. Owns `include/vt/device.h`, `src/vllm/platforms/{platform,tenstorrent}.cpp`, `src/vt/tenstorrent/`, `tests/vt/test_tenstorrent_backend.cpp`, the `VLLM_CPP_TENSTORRENT` CMake wiring, the backend-matrix/porting-inventory/spec, and this claim. | `ACTIVE` | 2026-08-09 W0 LANDED: 3/3 test cases, 8/8 assertions PASS on real hardware (registration, Platform/Backend mirror, kMatmul vs host F32 reference within bf16 tolerance, max_abs_diff 0.03375). Two bugs found+fixed in the same pass (nlohmann-json ABI collision between vllm.cpp's vendored copy and tt-metal's, isolated via a separate OBJECT library; a process-exit segfault from static MeshDevice teardown ordering, fixed by a deliberate leak) — both documented in the spec's Risks/decisions. Not reviewed. No model runs; one op only; host round-trip per call, no perf claim. |
 | `CLAIM-ENG-LOAD-DIRECT-UPLOAD` | `ENG-LOAD-DIRECT-UPLOAD` (engine-matrix, NEW row -> `ACTIVE`) | Claude Code (opus-5) | isolated worktree `.claude/worktrees/agent-a122a5ddc3546bf31`; GB10 `dgx.casa` tree `~/work/load150/src` (Vulkan `build-vk` + CUDA `build-cuda`), every GPU stage under `flock $HOME/gpu.lock`, `local-ai-worker` parked and restored | branch `row/ENG-LOAD-DIRECT-UPLOAD`, base `origin/main` `375a471e` | Issue #150. Owns ONLY: the refcounted safetensors mapping + byte counters (`include/vllm/model_executor/model_loader/safetensors_reader.h`, `src/vllm/model_executor/model_loader/safetensors_reader.cpp`), `BorrowStTensorBytes` + the `AdoptDeviceBytesAsHost` mmap branch + `OwnedTensor::mmap_src` (`include/vllm/model_executor/models/qwen3_5_weights.h`, `src/vllm/model_executor/models/qwen3_5_weights.cpp`), the qualifying call sites in `include/vllm/model_executor/models/dense_weight_loaders.h` and `src/vllm/model_executor/models/qwen3_5_dense_weights.cpp`, the upload counter in `include/vllm/model_executor/models/dense_attn_block.h`, the `VT_LOAD_STATS` phase timing in `src/vllm/entrypoints/model_loader.cpp`, NEW `tests/vllm/test_load_direct_upload.cpp` + its `tests/CMakeLists.txt` line, NEW `.agents/specs/load-direct-upload.md`, the NEW engine-matrix row + its Loading/Total rollup, the `#150` roadmap intake cell, two `docs/ENVIRONMENT.md` rows, this claim, `.agents/NOW.md`, and the `docs/STATUS.md`/`docs/BENCHMARKS.md` one-liners. **NON-COLLISION:** no new kernel, no model registry change, no CMake target change beyond one test line; every non-verbatim loader helper is untouched and the lever is same-binary A/B-able (`VT_LOAD_DIRECT_UPLOAD=0`). MUST NOT touch README, Metal/SACRED/apex/darwin, or any model/kernel source | `ACTIVE` | 2026-08-09 — mechanism test 6/6 (77) and RED under two mutations; GB10 Vulkan gates on the changed tree green (`test_vulkan_backend` 35/35·2650, `test_backend_cross_device` 11/11·132, `test_opt_paged_engine` 6/6 prompts token-exact 96/96, 0 declines, device type 3); GB10 CUDA full `ctest` 383/393 with BOTH SACRED gates PASS and all 10 failures reproduced on a clean `origin/main` build; 27B bf16 load 1.54x warm / 1.61x cold, bytes moved 100.196 -> 81.260 GiB; `local-ai-worker` restored (`--restart=always`, Up). Fresh scoped review OWED |
+| `CLAIM-ROCM-GDN-KERNELS` | `BACKEND-ROCM` (stays `ACTIVE` — M3 GDN kernel slice, issue #41) | pi (kimi-k3) | isolated worktree `/home/vikash/vllm.cpp-rocm-gdn-kernels`, base `origin/main` `5812b8b6`; 4x RX 7900 XTX gfx1100 discrete, ROCm 7.14, GPU jobs hold `~/gpu.lock` | `row/ROCM-GDN-KERNELS` | The M3 GDN kernel family unblocking Qwen3.5-0.8B M2 on discrete ROCm. Owns ONLY: NEW `.agents/specs/rocm-gdn-kernels.md`; NEW `src/vt/rocm/rocm_gdn.hip` (10 ops: kGdnStateGather/Scatter 77/78, kCausalConv1dFwd/Update 5/6, kRmsNormGated 9, kSigmoidGateBf16 58, kGdnPrefill/Decode 10/11, kGdnPostConv 65, kAttnQkNormRopeGate 67 — hand-translated from `src/vt/cuda/cuda_gdn.cu`/`cuda_ops.cu` donors); ADDITIVE registration entries in `src/vt/rocm/rocm_ops.hip`; ADDITIVE GDN cases + buffer helpers in `tests/vt/test_backend_cross_device.cpp`; this claim + `.agents/NOW.md`. **NON-COLLISION:** every new file is ROCm-only; the two shared-file edits are additive registration/case blocks that change no existing behavior (unregistered-op backends skip; CUDA/CPU/Vulkan paths byte-identical). Disjoint from #234 (sampling), #317 (gemma4 FP8/WMMA), #273 (near-tie docs). | `ACTIVE` | 2026-08-11 — **SPEC COMMITTED.** Pre-claim scratch validation on gfx1100 (recorded in the spec): all 10 kernels 68/68 standalone checks (state/conversions bit-exact); drop-in TU compiles with exact production flags; 5 red-first cross-device cases run-verified 16/16 against the real library. NEXT: family PRs in throw-order (GDN-STATE-IO first), each red-first cross-device case + focused gate + full gate + M2 rerun. |
 **CLOSED same-session claim — `CLAIM-TP-SPIKE-287` (task #287, 2026-08-08,
 records-only, helper `row/SPIKE-TENSOR-PARALLELISM` draft PR #143, base
 `b38f78a7`).** Not an active-claims row because every referenced row keeps its

diff --git a/.agents/specs/rocm-gdn-kernels.md b/.agents/specs/rocm-gdn-kernels.md
@@ -0,0 +1,120 @@
+# ROCm GDN kernel family (gfx1100) — BACKEND-ROCM M3 slice
+
+**Issue:** [#41](https://github.com/mudler/vllm.cpp/issues/41) (ROCm backend umbrella; roadmap
+issue-table row for `BACKEND-ROCM`).
+**Claim:** `CLAIM-ROCM-GDN-KERNELS` (coordination.md).
+**Base:** pinned `origin/main` `5812b8b6`.
+**Hardware:** 4× RX 7900 XTX (`gfx1100`, discrete), ROCm 7.14, Linux Mint 22.3 (reporter's box).
+
+## The gap, verified
+
+Qwen3.5-0.8B (`Qwen3_5ForConditionalGeneration`; hybrid GDN — `layer_types` 3:1
+linear:full, `full_attention_interval=4`, `linear_conv_kernel_dim=4`) on discrete gfx1100
+throws `vt: no kernel for op 77 on device type 5` (`src/vt/op_provider.cpp`). The CPU
+reference tier cannot install on a dGPU by design (`UnifiedMemory()=false`), so M2 for
+this model needs native ROCm kernels for the ten ops the model calls that have none
+(evidence: [#41 M2 attempt](https://github.com/mudler/vllm.cpp/issues/41#issuecomment-5230043080);
+staged-state note: [#41 follow-up](https://github.com/mudler/vllm.cpp/issues/41#issuecomment-5237179696)).
+Classic-dense M2 already runs all-native on this lane (Qwen3-0.6B, zero fallbacks —
+that model needs none of these ops).
+
+## Op inventory (the ten)
+
+| OpId | Name | Semantics (gdn-semantics.md) | CPU oracle | CUDA donor |
+|---|---|---|---|---|
+| 77 | `kGdnStateGather` | indexed state rows → f32 working | cpu_ops.cpp:1666 | cuda_gdn.cu:248-335 |
+| 78 | `kGdnStateScatter` | f32 working → indexed rows | cpu_ops.cpp:1708 | cuda_gdn.cu:278-335 |
+| 5 | `kCausalConv1dFwd` | causal depthwise conv, prefill (§2) | cpu_ops.cpp:1021 | cuda_gdn.cu:479-520 (scalar) |
+| 6 | `kCausalConv1dUpdate` | single-step decode conv (§3) | cpu_ops.cpp:1081 | cuda_gdn.cu:884-913 |
+| 9 | `kRmsNormGated` | gated RMSNorm, norm_before_gate (§5) | cpu_ops.cpp:1210 | cuda_gdn.cu RmsNormGatedRowKernel |
+| 58 | `kSigmoidGateBf16` | sigmoid·mul; **out bf16, attn f32/bf16, gate f32** (ops.cpp contract) | cpu_ops.cpp:2273 | no CUDA reg — CPU composite is the donor |
+| 10 | `kGdnPrefill` | gated-delta recurrence (§7/§8) | cpu_ops.cpp:1331 | cuda_gdn.cu:1856 GdnScanKernel |
+| 11 | `kGdnDecode` | single-step recurrence | cpu_ops.cpp:1368 | same scan, decode mode |
+| 65 | `kGdnPostConv` | fused conv-split+l2norm+g/beta | cpu_ops.cpp:2337 | cuda_gdn.cu:1155 |
+| 67 | `kAttnQkNormRopeGate` | fused full-attn preamble (gemma/plain) | cpu_ops.cpp:956 | cuda_ops.cu:1429 area |
+
+## Execution-path finding (decides the design)
+
+`qwen3_5.cpp` `IndexedGdnOpsNative()` gates the indexed state-I/O arm on
+`kCausalConv1dUpdate` + `kGdnDecode` + `kGdnStateGather` + `kGdnStateScatter` all being
+**natively** registered (`OpRegistered` excludes the reference tier). With the full set
+registered, the model takes the CUDA lane's device-resident path with **zero model or
+runner edits**. The row-copy arm is not a viable discrete fallback (host round trips per
+GDN layer per token; the Vulkan record measured the cost). `needs_weight_staging()` for
+discrete ROCm stays false in this row — the platform comment defers it to a measured
+residency row after M2 runs.
+
+## Routing decision
+
+The portable `GdnScanKernel` serves both `kGdnPrefill` (qsl != null) and `kGdnDecode`
+(qsl == null, optional state_idx + NULL-block zero-out). The CUDA lane's perf machinery —
+WMMA chunked prefill (`#if __CUDA_ARCH__ >= 800`), fused/packed decode, register/tiled
+conv variants, Triton-AOT cubins, spec-decode and fp8-quant variants — is NVIDIA-only or
+not on this model's path and stays unported (docs/ROCM.md §6). Perf variants are M5
+levers, and upstream's ROCm answer (Triton/CK) is a separate later decision.
+
+## PR slicing (one family per PR, throw-order; each ends with an M2 rerun)
+
+1. **GDN-STATE-IO**: 77/78 (the current throw) — this PR's first slice.
+2. **GDN-CONV**: 5/6. 3. **GDN-NORMGATE**: 9/58. 4. **GDN-CORE**: 10/11.
+5. **GDN-FUSED**: 65/67. After 4 the model completes prefill+decode; 5 restores the fused
+fast path (VT_GLUE_FUSE structure parity with the CUDA lane).
+
+## Pre-claim validation state (scratch, on gfx1100; landed through the lifecycle here)
+
+All ten kernels were hand-translated from the donors and validated BEFORE any tree
+change, in standalone harnesses against independent host references (68/68 checks:
+state paths and dtype conversions **bit-exact**; compute outputs at expf/FMA ulp level,
+max rel ≤ 2e-5). A complete drop-in TU compiles clean with the build's exact production
+flags (`-O3 -ffp-contract=off -std=c++20 --offload-arch=gfx1100`). Five red-first
+cross-device cases were assembled into a scratch copy of `test_backend_cross_device.cpp`
+and run against the real library: 16/16 green with GDN cases correctly skipping
+unregistered ops. Writing them surfaced four `ops.cpp` contract requirements the kernels
+satisfy (compact-arm per-token state rows; RmsNormGated rank-3 shape matching; rank-2
+preamble inputs; SigmoidGateBf16's f32 gate). Scratch root: `~/gdn-spike/` on the
+reporter's box (harnesses + byte-exact logs).
+
+## Test plan (red-first per family)
+
+1. New `test_backend_cross_device` cases per family, CPU-oracle compared — byte paths
+   (gather/scatter, conv-state write-back/roll) **bit-exact**; arithmetic at NMSE ≤ 5e-4.
+   Written RED before each registration lands (skip-today, run-once-registered).
+2. Model level: the §5.2 M2 rerun per family — Qwen3.5-0.8B vs `--device cpu`, greedy,
+   tokens + `VT_OP_PROVIDER_STATS` posted on #41; the final family carries the e2e claim
+   attempt.
+3. Full `ctest` suite per family PR; gates reported exactly as observed.
+
+## M2 disposition (ratified context)
+
+The near-tie regime is on record: three boards (gfx1100, gfx1103, gfx1200) show
+GPU-vs-CPU greedy near-tie flips on Qwen3-0.6B near-tied prompts, deterministic per
+backend; joral's #269 analysis (uniform 0.01–0.6% per-layer drift; two real vLLM-ROCm
+oracles disagreeing with *each other* at K=5) and the maintainer's #273 merge comment
+ratified the **distributional gate** for this regime ("a token-exact bar cannot close
+it"), and zero-fallbacks-under-`VT_OP_PROVIDER_STATS` as the discrete-M2 mechanism
+evidence. This row's M2 claim for Qwen3.5-0.8B follows that disposition: e2e completion,
+all-native op resolution, determinism, and the distributional comparison vs the CPU
+backend — not strict token-exactness.
+
+## Risks
+
+- **Scan perf at M5** (portable scan vs chunked/fused): accepted — correctness first;
+  the perf lever is named, not hidden.
+- **bf16 state arms at model level**: covered at op level by the bit-exact scratch
+  checks; the cross-device cases compare f32-state arms (matching the file's tier
+  structure) — model-level bf16 state behavior is observed in the M2 rerun.
+- **Wave64 (gfx9/CDNA)**: none of these kernels use warp-width-sensitive primitives;
+  the scan's `__syncthreads` structure is width-agnostic. gfx9 boards remain untested
+  (unverbraucht's MI50 offer on #41) — a later row.
+
+## Stop conditions
+
+- A family fails its cross-device gate and the defect is not resolvable from the CPU
+  oracle + donor re-read → stop, post the failing evidence on #41.
+- `IndexedGdnOpsNative(kROCM)` does not flip after family 4 (model path assumption
+  broken) → stop, NEEDS_DECISION on #41.
+- Any maintainer redirection of scope/slicing → this spec is amended in the same change.
+
+## Outcome
+
+(pending — filled at DONE)
diff --git a/CMakeLists.txt b/CMakeLists.txt
@@ -1312,6 +1312,8 @@ if(VLLM_CPP_HIP)
     src/vt/rocm/rocm_gemma4_expert_geglu.hip
     src/vt/rocm/rocm_fp8_channel_gemv.hip
     src/vt/rocm/rocm_moe_router.hip
+    src/vt/rocm/rocm_gdn_state.hip
+    src/vt/rocm/rocm_gdn_conv.hip
     src/vt/rocm/rocm_ops.hip)
   if(VLLM_CPP_HIP_ARCHITECTURES)
     set_source_files_properties(
@@ -1326,6 +1328,8 @@ if(VLLM_CPP_HIP)
       src/vt/rocm/rocm_gemma4_expert_geglu.hip
       src/vt/rocm/rocm_fp8_channel_gemv.hip
       src/vt/rocm/rocm_moe_router.hip
+      src/vt/rocm/rocm_gdn_state.hip
+      src/vt/rocm/rocm_gdn_conv.hip
       src/vt/rocm/rocm_ops.hip
       PROPERTIES HIP_ARCHITECTURES "${VLLM_CPP_HIP_ARCHITECTURES}")
   endif()

diff --git a/docs/FEATURES.md b/docs/FEATURES.md
@@ -297,7 +297,7 @@ CPU elementwise GEMM (f32/f16/bf16) runs AVX2 and AVX-512 tiers on x86 where the
 | LoRA end to end | CPU brick landed | Unwired standalone; not usable through the server |
 | Multimodal over HTTP | Image request path wired; forward + codec pending | `ROAD-V1-MM` W1-W3 landed (`server_main.cpp:826`). Open: no mm-forward consuming `Request.mm_features`; no image codec vendored (raw RGB only); video/audio/multi-image not started |
 | Reranking / classify models | Engine side only | Embeddings are LIVE (`LlamaModel`, `vllm_embed`, `/v1/embeddings`); the classify/score heads are landed ops with no registered arch |
-| ROCm | W0 verified by community, model e2e pending | Backend + platform + 1 op, ctest-green on gfx1151/1103/1100/1201 ([#41](https://github.com/mudler/vllm.cpp/issues/41)). APU UnifiedMemory fix in (managed allocs, unverified); M2 unblocks with it. [ROCM.md](ROCM.md) |
+| ROCm | W0-W1 community-verified on 5 gfx archs; classic-dense e2e runs all-native (near-tie regime, #269); GDN-hybrid blocked on remaining GDN kernels | Backend + platform + #140 op set + GDN state-I/O and causal-conv1d ops, ctest-green on gfx1151/1103/1100/1201/1200 ([#41](https://github.com/mudler/vllm.cpp/issues/41)). APU UnifiedMemory fix verified. [ROCM.md](ROCM.md) |
 | XPU, TPU | Not started | CUDA, CPU, Metal and Vulkan are the built backends |
 | Custom logits processors on CUDA | Open, not root-caused | Segfaults in a CUDA build, 232/232 green on CPU |
 | Memory budgeting (`ROAD-V1-MEM`, #83) | M1+M2 landed (absolute bytes) | `--kv-cache-memory` sizes the KV pool from an absolute byte budget (ABI v16, group-aware divisor); `--num-blocks` overrides; `--gpu-memory-utilization` needs the M3 profile run (dgx-gated). See `specs/kv-sizing.md` |

diff --git a/docs/USAGE.md b/docs/USAGE.md
@@ -32,6 +32,18 @@ never use, and its teardown can deadlock at process exit — every test passes,
 ([#132](https://github.com/mudler/vllm.cpp/issues/132)). Setting a build type,
 or putting your own `-O` in `CMAKE_HIP_FLAGS`, overrides it.
 
+### ROCm op coverage is incremental (and throws are by design)
+
+The ROCm backend registers native ops family by family
+([#41](https://github.com/mudler/vllm.cpp/issues/41)); landed GDN slices so far:
+the indexed state I/O pair (`kGdnStateGather`/`kGdnStateScatter`) and the causal
+conv1d pair (`kCausalConv1dFwd`/`kCausalConv1dUpdate`, incl. the exact-chunks
+descriptor form Qwen3.5 prefill passes). On a
+discrete card there is no CPU fallback tier, so a model whose layers call an op
+that is not registered yet fails loudly with `vt: no kernel for op N on device
+type 5` — that is the memory-safety design working, not a crash. Run with
+`VT_OP_PROVIDER_STATS=1` to see which ops resolve native.
+
 ### CUTLASS is fetched as headers only
 
 `-DVLLM_CPP_CUTLASS_FETCH=ON` downloads CUTLASS v4.5.0 and stops there: the