feat(tenstorrent): host-free decode forward — capture investigation + architecture

.agents/backend-matrix.md

@@ -246,7 +246,8 @@ this repository. State remains `ACTIVE`; no lifecycle transition is claimed.
 | `BACKEND-TENSTORRENT` | Tenstorrent Blackhole (Tensix multicore, discrete PCIe, no unified memory) — thin `vt::` adapter over ttnn's existing C++ op library rather than hand-written kernels, mirroring the Metal/MLX decision (E1); vLLM has no Tenstorrent platform anywhere | vllm.cpp extension through upstream seam `platforms/interface.py:134-229` (same pattern as Metal/Vulkan) | **ACTIVE 2026-08-10.** `vt::tenstorrent::Backend` + registrar [tenstorrent_backend.cpp](../src/vt/tenstorrent/tenstorrent_backend.cpp); shared mesh-device lifecycle [tenstorrent_device.cpp](../src/vt/tenstorrent/tenstorrent_device.cpp); 17 registered ops cover OPT-125m and the Qwen3-0.6B forward (`kMatmul`, `kMatmulBT`, `kAdd`, `kRelu`, `kEmbedding`, `kLayerNorm`, `kRmsNorm`, `kSiluAndMul`, bf16/f32 casts, three RoPE forms, `kQkvSplit`, `kReshapeAndCache`, host-oracle `kPagedAttention`, `kGreedyArgmax`) [tenstorrent_ops.cpp](../src/vt/tenstorrent/tenstorrent_ops.cpp); platform allow-list selects OPT and Qwen3 [platforms/tenstorrent.cpp](../src/vllm/platforms/tenstorrent.cpp). `DeviceType::kTENSTORRENT` [device.h](../include/vt/device.h) | [test_tenstorrent_backend.cpp](../tests/vt/test_tenstorrent_backend.cpp) carries real-Blackhole op gates; [test_qwen3_paged_engine.cpp](../tests/parity/test_qwen3_paged_engine.cpp) selects Tenstorrent device-specific anchor and teacher-forced near-tie goldens. OPT-125m STRICT 6/6 passed. Qwen3 short warm smoke ran 4 tokens at about 0.28 tok/s; full 16x16 gate remains pending behind host paged attention | [tenstorrent-backend.md](specs/tenstorrent-backend.md) | `ACTIVE` | `CLAIM-BACKEND-TENSTORRENT-SPIKE` |
 | `BACKEND-TENSTORRENT-RESIDUAL-GOLDEN` | Child of `BACKEND-TENSTORRENT` — the owed op-level numerics evidence at the residual-RMS device boundary (`kDeviceResidualMinRows == 32`): device path does `ttnn::add`+`ttnn::rms_norm` in bf16; host/CPU path accumulates in f32. Bot-flagged on #289; never measured at the boundary. | vllm.cpp CPU oracle `RmsNormKernel` mirrors vLLM `fused_add_rms_norm` (add in model dtype, variance in f32); `src/vt/cpu/cpu_ops.cpp:371-398` | `src/vt/tenstorrent/tenstorrent_ops.cpp:1067-1117` (host/device split, `kDeviceResidualMinRows=32`) | [test_tenstorrent_backend.cpp](../tests/vt/test_tenstorrent_backend.cpp) `kRmsNorm residual: device vs CPU f32 oracle across the rows=32 boundary`: 22/22 cases on real Blackhole P150. **Measured 2026-08-11:** host path `rows<32` bit-identical to CPU (`max_abs=0`); device bf16 path `rows>=32` diverges by constant **0.0459 abs** (1.9–2.6× rel on near-zero outputs) — bf16 rounding signature, not accumulation. Decision pending the e2e golden tie-break | [tenstorrent-residual-golden.md](specs/tenstorrent-residual-golden.md) | `SPIKE` | `CLAIM-BACKEND-TENSTORRENT-RESIDUAL-GOLDEN` |
 | `BACKEND-TENSTORRENT-MISTRAL` | Child of `BACKEND-TENSTORRENT` — allowlist `MistralForCausalLM` (Mistral-7B-v0.3: GQA 32/8, head_dim 128, plain rope theta 1e6, untied lm_head, full attention) on the TT platform + device-aware SACRED gate. Mistral reuses the Qwen3-dense forward verbatim (qk-norm skipped); every op already registered. No new kernel. | vLLM `mistral.py::MistralForCausalLM(LlamaForCausalLM)` (already ported to the shared dense machinery); gate pattern mirrored from `test_qwen3_paged_engine.cpp:221-296` | `src/vllm/platforms/tenstorrent.cpp:52-54` (allowlist) + `tests/parity/test_mistral_paged_engine.cpp` (device-aware wiring + Backend Proof) | **Gate PASSED on Blackhole P150 (2026-08-12):** [test_mistral_paged_engine.cpp](../tests/parity/test_mistral_paged_engine.cpp) 16/16 prompts PASS (12/16 strict-exact, 4/16 near-tie, 0 forward-divergent), max gap **0.062 nats**, BACKEND PROOF 0 declines (kMatmul selections=256 = untied lm_head on device, kPagedAttention=8192). Goldens `our_ids_tenstorrent.npy` + `neartie_gap_mnats_tenstorrent.npy` (transformers alternative-oracle; POL-ORACLE deviation recorded, same as Qwen3-0.6B TT precedent). Exit SIGSEGV 139 is the known MeshDevice teardown crash, not a gate failure | [tenstorrent-mistral.md](specs/tenstorrent-mistral.md) | `ACTIVE` | `CLAIM-BACKEND-TENSTORRENT-MISTRAL` |
-
+| `BACKEND-TENSTORRENT-TRACE-RUNNER` | Child of `BACKEND-TENSTORRENT` — wire the landed graph-capture foundation (#354 / `59568772`) into a capturable forward region. Handoff §8/§9 "not done". **Decision (2026-08-13): NO-GO for pure T=1 decode capture — MEASURED, not assumed.** | CUDA is the ONLY backend with `SupportsGraphCapture()==true` (`cuda_backend.cu:184-240`); Metal/Vulkan `false`. Shared decode-graph framework `Qwen3DenseDecodeGraph` (qwen3.cpp:489, used by Qwen3/Mistral/Llama/InternLM2) gated on `support_static_graph_mode()` which TT does NOT override (base `false`) | `src/vt/tenstorrent/tenstorrent_backend.cpp:70-76` (capture surface, landed) + `tenstorrent_ops.cpp` `Trace*` (landed); NOT wired into a dense forward | **Measured on Blackhole P150 (2026-08-13):** Q1 no host-free region at T=1; Q2 all-device-at-T=1 costs 12.5→10.7 tok/s; Q2b capture attempt with both overrides flipped → ttnn **`TT_FATAL: Reads are not supported during trace capture`** (backtrace through `to_vector<float>`), `0 replays`. The T=1 forward does device→host readbacks that ttnn trace prohibits; flipping the two thresholds is insufficient. Requires a host-free `ForwardLayers` redesign, not a threshold tweak. Next: prefill capture (separate row) must first audit its `to_vector` readbacks | [tenstorrent-trace-runner.md](specs/tenstorrent-trace-runner.md) | `SPIKE` (decision record complete) | `CLAIM-BACKEND-TENSTORRENT-TRACE-RUNNER` |
+| `BACKEND-TENSTORRENT-HOST-FREE-FORWARD` | Child of `BACKEND-TENSTORRENT` — make the per-decode-layer forward region host-free (zero `to_vector` readbacks) so mesh-trace capture can run. Decomposes into R1 RmsNorm+RoPE all-device, R2 QkvSplit+ReshapeAndCache device, R3 PA decode device-resident metadata, R4 capture wire+measure. Prerequisite revealed by the trace-runner spike (capture aborts on `to_vector`). | CUDA decode-graph capture contract (`cuda_backend.cu:184-197`): captured region is async, no host sync, no malloc, fixed ptrs. TT must match: no `to_vector` between Begin/EndCapture | `src/vt/tenstorrent/tenstorrent_ops.cpp` (RmsNorm:1067, PreferDeviceRope:1344, QkvSplit:1460, ReshapeAndCache:1527, PagedAttention:2009) | **R1-R3b MEASURED on P150** (env-gated `VT_TT_HOST_FREE_DECODE`, inert by default; 23/23 TT tests incl. a dedicated default-path inertness guard, M1-mutation-proved): R1 threshold flip landed; R2 device->device copy (`CopyDeviceDeviceIfCapture`, ttnn::copy+empty) landed; R3 program-cache warm (`enable_program_cache` + eager-warm) landed; R3b device zero-fill (`MemsetDeviceIfCapture`) landed. Capture enters the forward and reaches layer ops (CastBf16/RmsNorm fire). Remaining item-5 blocker = per-op enqueue_write; answer = persistent device tensors + before-replay populate (tt-metal vLLM plugin design). Full blocker map + architecture in [tenstorrent-host-free-r1.md](specs/tenstorrent-host-free-r1.md) | [tenstorrent-host-free-forward.md](specs/tenstorrent-host-free-forward.md) | `ACTIVE` (R1-R3b landed; item-5 port open) | `CLAIM-BACKEND-TENSTORRENT-HOST-FREE-FORWARD` |
 ## Native competitor and performance gates
 
 Every run records the competitor commit/release, model artifact hash, build

.agents/coordination.md

@@ -1997,6 +1997,8 @@ this claim will meet. The tiled row is speed-gatable on dgx.
 | `CLAIM-ENG-RELEASE-CONTAINERS-W1-W7` | `ENG-RELEASE-CONTAINERS` (`INVENTORIED`->`READY`->`ACTIVE`; spike spec + W1-W5/W7) | Claude Code (opus-5), helper role | isolated worktree `/home/mudler/_git/vllm.cpp-containers`; local Docker only -- NO GPU, NO registry push, NO publication | `row/ENG-RELEASE-CONTAINERS`, base `origin/main` `24306364`, PR [#307](https://github.com/mudler/vllm.cpp/pull/307) | Container spike spec, `docker/Dockerfile` (cpu/vulkan/cuda calling the release scripts), container matrix + checker, image validator, workflow + least-privilege guard, and the SIGTERM fix the boot gate surfaced (#312). Excludes W6 matching-hardware runtime evidence and any GHCR publication | `ACTIVE` | 2026-08-10 -- cpu lane built and gated e2e (config+layout+boot, clean SIGTERM 0.25 s, RED 137 before the fix); mutation suites 31/31 and 22/22; cuda/vulkan gated statically but never built here; nothing published |
 | `CLAIM-ROCM-GDN-KERNELS` | `BACKEND-ROCM` (stays `ACTIVE` — M3 GDN kernel slice, issue #41) | pi (kimi-k3) | isolated worktree `/home/vikash/vllm.cpp-rocm-gdn-kernels`, base `origin/main` `5812b8b6`; 4x RX 7900 XTX gfx1100 discrete, ROCm 7.14, GPU jobs hold `~/gpu.lock` | `row/ROCM-GDN-KERNELS` | The M3 GDN kernel family unblocking Qwen3.5-0.8B M2 on discrete ROCm. Owns ONLY: NEW `.agents/specs/rocm-gdn-kernels.md`; NEW `src/vt/rocm/rocm_gdn.hip` (10 ops: kGdnStateGather/Scatter 77/78, kCausalConv1dFwd/Update 5/6, kRmsNormGated 9, kSigmoidGateBf16 58, kGdnPrefill/Decode 10/11, kGdnPostConv 65, kAttnQkNormRopeGate 67 — hand-translated from `src/vt/cuda/cuda_gdn.cu`/`cuda_ops.cu` donors); ADDITIVE registration entries in `src/vt/rocm/rocm_ops.hip`; ADDITIVE GDN cases + buffer helpers in `tests/vt/test_backend_cross_device.cpp`; this claim + `.agents/NOW.md`. **NON-COLLISION:** every new file is ROCm-only; the two shared-file edits are additive registration/case blocks that change no existing behavior (unregistered-op backends skip; CUDA/CPU/Vulkan paths byte-identical). Disjoint from #234 (sampling), #317 (gemma4 FP8/WMMA), #273 (near-tie docs). | `ACTIVE` | 2026-08-11 — **ALL 5 FAMILIES LANDED AS STACKED DRAFT PRs + M2 E2E COMPLETE.** #334 (spec + state I/O), #336 (conv), #341 (postconv), #343 (recurrence), #345 (norm-gate/preamble + the SupportsCompressedGdnState capability seam). Qwen3.5-0.8B runs e2e all-native on discrete gfx1100, deterministic; 'The sky is' 8/8 CPU parity; open-prompt divergence characterized as not-state-rounding, GDN-path drift analysis named open. Gates per family: cross-device green (162/162 total), focused 3/3, full-suite zero-delta vs base. NEXT: fresh review + operator gate per PR; then the MoeRouterTopK bf16-logits arm (the named MoE-path blocker) and the GDN-path divergence characterization. |
 | `CLAIM-WEIGHT-OFFLOAD-W0A` | `ENG-WEIGHT-OFFLOAD` (engine-matrix, `READY`->`ACTIVE`; issue #797, the dense half of #149). | Claude Code (claude-opus-5), helper role, isolated worktree | `/home/mudler/.cache/sdd/mudler-vllm.cpp/weight-offload-w0`; CPU-only build (`-DVLLM_CPP_CUDA=OFF`) on the dev box -- W0a is a device-neutral config port, NO GPU/flock needed | branch `row/ENG-WEIGHT-OFFLOAD-W0`, base `origin/main` `b1cd4d8f6` | **W0a -- the weight-offload CONFIG SURFACE only.** Owns EXACTLY the NEW `include/vllm/config/offload.h` + `src/vllm/config/offload.cpp`, the NEW `tests/vllm/config/test_offload_config.cpp`, 1 `CMakeLists.txt` source line + 1 `tests/CMakeLists.txt` test line, and the records (this row, the engine-matrix row, `specs/weight-offload-uva.md` `## Now`, `docs/STATUS.md`). **NON-COLLISION:** purely additive and UNREACHABLE -- nothing constructs an `OffloadConfig` yet, so every existing gate is byte-identical BY CONSTRUCTION (`git diff` touches no existing source file beyond the two CMake registration lines). Does NOT own the offloader itself (W2/W5), the loader re-offload interaction (W3), the ABI/CLI field (the REMAINDER of W0), or anything in `v1/kv_offload/` (a different subject). | `ACTIVE` | 2026-08-14 -- **W0a LANDED: config surface + validator, RED-first then mutation-proven.** Transcribed from `vllm/config/offload.py` @ `555967922`: the three-value backend enum (:12), both sub-configs with their bounds and defaults (:23,34,54,62,66,70), the TWO hard validator errors (:100-112) and the THREE mismatch warnings (:114-135) -- warnings collected for the caller rather than raised, since C++ has no `warnings.warn`. Also ported: the dot-anchored segment match (`offloader/uva.py:91-93`), the `int(gb*1024**3)` truncation (`offloader/base.py:155`), the auto-selection ORDER (`base.py:139-149`), and the layer-grouping worked example (`offload.py:57`). **RED-first CAPTURED** on a compiling stub (build rc=0, 0 compile errors): 11/11 cases and 51/122 assertions failed for the intended reason. GREEN 11/11, 126/126. **MUTATION-PROVEN 6/6**, each reported with its compile status so a non-building mutation could not read as a pass: substring-instead-of-dot-anchored (4 asserts RED), grouping off-by-one (3), validator guard OR->AND (3), warnings-not-cleared (1), auto-order uva-before-prefetch (2), 1000^3-instead-of-1024^3 (3); tree restored byte-for-byte. One design correction the RED exposed: `ResolvedBackend()` now mirrors `create_offloader` exactly (an EXPLICIT backend is selected even at a zero budget) and the separate `is_offloading_enabled()` answers 'would anything actually move' -- conflating the two would let a zero-budget explicit backend read as offloading-on. OWED (the rest of W0): the `include/vllm.h` JSON field + server CLI flag. |
+| `CLAIM-BACKEND-TENSTORRENT-HOST-FREE-FORWARD` | `BACKEND-TENSTORRENT-HOST-FREE-FORWARD` (NEW child of `BACKEND-TENSTORRENT`; `INVENTORIED`→`SPIKE`) | Maki (zai/glm-5.2), helper role | isolated worktree `/home/lu_zero/Sources/vllmcpp-tenstorrent`; real Blackhole (P150) hardware | branch `row/BACKEND-TENSTORRENT-HOST-FREE-FORWARD` (planning; reuses the trace-runner worktree) | The prerequisite for decode mesh-trace capture, revealed by `BACKEND-TENSTORRENT-TRACE-RUNNER` (capture aborts on `to_vector`). Decomposes into R1 RmsNorm+RoPE all-device, R2 QkvSplit+ReshapeAndCache device, R3 PA decode device-resident metadata, R4 capture wire+measure. Owns ONLY: NEW `.agents/specs/tenstorrent-host-free-forward.md`, the NEW `BACKEND-TENSTORRENT-HOST-FREE-FORWARD` backend-matrix row, and this claim. No code, no model change, no perf claim — decision record only. | `SPIKE` | 2026-08-13 plan committed; no implementation yet. R1-R3 parallel-claimable once a maintainer accepts the decomposition. |
+| `CLAIM-BACKEND-TENSTORRENT-TRACE-RUNNER` | `BACKEND-TENSTORRENT-TRACE-RUNNER` (NEW child of `BACKEND-TENSTORRENT`; `INVENTORIED`→`SPIKE`) | Maki (zai/glm-5.2), helper role | isolated worktree `/home/lu_zero/Sources/vllmcpp-tenstorrent`; real Blackhole (P150) hardware | branch `row/BACKEND-TENSTORRENT-TRACE-RUNNER`, base `origin/main` `a89b3c45` | Feasibility spike (read-only analysis + on-card measurement) for wiring the landed #354 graph-capture foundation into a capturable Qwen3-dense/Mistral forward region on TT. Three questions: (1) is there a host-free decode region at T=1 (current hybrid thresholds route RoPE+residual to host)? (2) what does forcing all-device-at-T=1 cost in tok/s, and does capture recover it? (3) does ttnn program-cache warm-up work before begin_trace_capture? Surveys how CUDA/Metal/Vulkan structure capture (CUDA is the only `SupportsGraphCapture()==true` backend; Metal/Vulkan `false`). Owns ONLY: NEW `.agents/specs/tenstorrent-trace-runner.md`, the NEW `BACKEND-TENSTORRENT-TRACE-RUNNER` row, and this claim. No code, no model change, no perf claim — decision record only. | `SPIKE` | MEASURED 2026-08-12/13 (all on real P150): no host-free region at T=1 (residual rows<32 + RoPE T*H<64 host, every layer); all-device-at-T=1 costs 12.5→10.7 tok/s (reproduces handoff S6); capture attempt aborts (to_vector + enqueue_write fatals). Decision superseded for implementation by the host-free investigation (#694): capture IS achievable; the remaining work (persistent device buffers + before-replay input populate, ported from the tt-metal vLLM plugin design) is tracked there. PR #541 closed in favor of #694. |
 **CLOSED same-session claim — `CLAIM-TP-SPIKE-287` (task #287, 2026-08-08,
 records-only, helper `row/SPIKE-TENSOR-PARALLELISM` draft PR #143, base
 `b38f78a7`).** Not an active-claims row because every referenced row keeps its