feat(tenstorrent): host-free decode forward — capture investigation + architecture by lu-zero · Pull Request #694 · mudler/vllm.cpp

lu-zero · 2026-08-13T22:40:10Z

What

The experimental chain that un-no-go's the trace-runner decision. Measured on real Blackhole P150, env-gated (VT_TT_HOST_FREE_DECODE) and inert by default — now test-proven, not just inspected. No shipped behavior change unless the env flag is set.

Supersedes and contains the closed #541 (trace-runner spike) — its two commits are this branch's first two.

The complete blocker map (each measured by sequential bisection)

#	Blocker	Status
R1	RmsNorm/RoPE host thresholds at T=1	done (flip, env-gated)
R2	`Backend::Copy` host readback	done (device→device copy via `ttnn::empty`+`ttnn::copy`)
R3	ttnn program-cache warm-up	done (`enable_program_cache` + eager-warm)
R3b	`Backend::Memset`/`DBuf::Zero` host write	done (device zero-fill via `ttnn::zeros`)
5	per-op device write/upload	architecture answer below

R1–R3b are landed and measured: capture now enters the forward and reaches the layer ops (CastBf16/RmsNorm fire during capture).

Item 5 — the architecture answer

tt-metal issue #13690 (filed by Tenstorrent for vLLM, fixed in f0b2483) relaxes the allocator guard. But our fatal is the separate enqueue_write guard (fd_mesh_command_queue.cpp:760), which #13690 did not touch. Verified f0b2483 IS in our build — bumping won't help.
The real answer came from reading the official tt-metal vLLM plugin (model_runner.py): the captured ForwardLayers region must reference only pre-allocated, pre-populated persistent device tensors, with per-step inputs written to stable device buffers before ReplayGraph (never inside capture — copy_to_device itself hits the same enqueue_write fatal). This mirrors CUDA's decode-graph pattern (SizeSlot::Refresh).

Path forward (bounded port, not a new subsystem, not an upstream fix): persistent device input tensors in the decode-graph slot + populate before capture/replay via copy_to_device. The plugin-technique → vllm.cpp-TT mapping table is in the spec's Port map; the env-gated-only constraints of the investigation code are carried in the spec's "Known constraints" section.

Review

Fresh static + mutation review returned CHANGES-REQUESTED; both blockers fixed in 37696af0:

Records contradiction (claim SPIKE vs matrix ACTIVE) — claim now ACTIVE; spec Work breakdown renumbered to the post-investigation truth.
Inertness was unguarded (M1/M4 mutations stayed green) — added a dedicated default-path inertness guard test: two same-shaped buffers with current device shadows, so the env/capture gate is the only decline path; CHECK_FALSE on both helpers + the value!=0 decline + a host-path correctness anchor. Mutation-proved on card: M1 (gate removed) → 22/23 RED; revert → 23/23, 830/830.

Non-blocking findings (ignored bytes, stale dev_rows/dev_cols, bytes-only dtype check, enable_program_cache placement, non-exception-safe flag clear, SlotMutex TOCTOU, mid-capture shadow drop) are recorded in the spec as "Known constraints" for the item-5 port to address.

Gates

Default TT suite on real P150: 23/23 cases / 830/830 assertions (includes the new inertness guard).
Record checker green (BACKEND=83); commit trailers OK.
Capture probe (env-gated): reaches layer ops; item-5 open.

Files

NEW .agents/specs/tenstorrent-host-free-forward.md (spike-gate contract: Scope/Upstream/Our baseline/Port map/Tests to port/Dependencies/Work breakdown + Known constraints)
NEW .agents/specs/tenstorrent-host-free-r1.md (the measured investigation record)
NEW .agents/specs/tenstorrent-trace-runner.md (the closed spec(tenstorrent): trace-runner feasibility spike — NO-GO for T=1 decode #541's record)
src/vllm/platforms/tenstorrent.cpp, src/vt/tenstorrent/{ops,device.h,backend}.cpp — all env-gated, inert by default
tests/vt/test_tenstorrent_backend.cpp — the inertness guard test

Post-fold record repair before review: - BACKEND-TENSTORRENT-HOST-FREE-FORWARD matrix row: Tests/evidence cell now carries the measured R1-R3b outcome (was the stale "plan spec only; no code yet"); state SPIKE -> ACTIVE (R1-R3b landed, item-5 port open). - The plan spec (tenstorrent-host-free-forward.md) now meets the ACTIVE spike-gate contract (Scope / Upstream chain / Our baseline / Port map / Tests to port / Dependencies / Work breakdown) with the honest content: the port map is the tt-metal vLLM plugin technique table. - The TRACE-RUNNER claim tail records the measured outcome + the mudler#694 supersession (PR mudler#541 closing in its favor). - Dropped tests/parity/goldens/qwen3_greedy_0_6b/our_ids_tenstorrent.i32 — the raw VT_DUMP_IDS bootstrap artifact (untracked-noise class per the handoff; the .npy pair is already on main). It was swept into the fold unintentionally. - Post-rebase verification: default TT suite on the rebased tree on real P150 is 22/22 cases / 826/826 assertions — the env-gated host-free paths are inert by default. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: Maki:glm-5.2 [Maki]

lu-zero · 2026-08-14T11:39:31Z

Steady-state perf baselines (2026-08-14, real P150, 64-token runs) — corrects two smoke numbers:

config	warm tok/s	ms/tok
Qwen3-0.6B default hybrid	7.30 / 7.31	~137
Qwen3-0.6B all-device eager (R1-R3b paths, decode-graph off)	6.87 / 6.92	~145
Mistral-7B-v0.3 (cross-ref)	4.26 (32 tok)	—

The 4-token smoke's 12.5 tok/s is not steady state: at 64 tokens the same config sustains 7.3 (per-token cost grows with the KV). The handoff's ~12.3 over-reports the same way. 7.3 is the honest capture reference.
The all-device eager cost at steady state is ~0.4 tok/s (~6%), not ~1.8 tok/s (~14%) as the 4-token smoke suggested — capture's break-even bar is materially lower than the spike framing assumed.
Capture/replay tok/s — the payoff — stays unmeasurable until item 5 lands; bounded by: must beat 7.3, eager floor 6.9.

Post-fold record repair before review: - BACKEND-TENSTORRENT-HOST-FREE-FORWARD matrix row: Tests/evidence cell now carries the measured R1-R3b outcome (was the stale "plan spec only; no code yet"); state SPIKE -> ACTIVE (R1-R3b landed, item-5 port open). - The plan spec (tenstorrent-host-free-forward.md) now meets the ACTIVE spike-gate contract (Scope / Upstream chain / Our baseline / Port map / Tests to port / Dependencies / Work breakdown) with the honest content: the port map is the tt-metal vLLM plugin technique table. - The TRACE-RUNNER claim tail records the measured outcome + the mudler#694 supersession (PR mudler#541 closing in its favor). - Dropped tests/parity/goldens/qwen3_greedy_0_6b/our_ids_tenstorrent.i32 — the raw VT_DUMP_IDS bootstrap artifact (untracked-noise class per the handoff; the .npy pair is already on main). It was swept into the fold unintentionally. - Post-rebase verification: default TT suite on the rebased tree on real P150 is 22/22 cases / 826/826 assertions — the env-gated host-free paths are inert by default. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: Maki:glm-5.2 [Maki]

BACKEND-TENSTORRENT-TRACE-RUNNER. Replaces the prior "no-go" guess with a measured result from actually attempting the capture on-card. The spike ran the real experiment: local-only flips of both support_static_graph_mode() (enables Qwen3DenseDecodeGraph) AND VT_TT_FORCE_DEVICE (RoPE+residual all-device), then Qwen3-0.6B cli smoke. Both overrides reverted; no code shipped. Result: capture ABORTS. ttnn raises TT_FATAL: Reads are not supported during trace capture with a backtrace through ttnn::Tensor::to_vector<float> -- a device->host readback fires inside the captured ForwardLayers region. 0 replays. The T=1 forward does to_vector readbacks (embedding/PA/logits) that ttnn trace prohibits; forcing the two hybrid thresholds all-device is insufficient. This is strictly stronger than the earlier tok/s-recovery guess: decode capture cannot run AT ALL on the current forward without eliminating every to_vector readback in the captured region (a host-free ForwardLayers redesign, not a threshold tweak). Q1 (no host-free region at T=1) and Q2 (all-device costs 12.5->10.7 tok/s, reproducing handoff §6) stand. Decision: NO-GO for pure T=1 decode, measured. Next: prefill capture (separate row) must first audit its to_vector readbacks. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: Maki:glm-5.2 [Maki]

BACKEND-TENSTORRENT-HOST-FREE-FORWARD (child of BACKEND-TENSTORRENT). The trace-runner spike measured that decode mesh-trace capture ABORTS (TT_FATAL: Reads are not supported during trace capture, backtrace through to_vector<float>) because the per-layer T=1 forward does device->host readbacks that ttnn trace prohibits. Capture wiring works (it engaged and tried); the forward is the blocker. This commit is a PLAN only (no code). It decomposes the host-free forward into four independent rows, grounded in a full readback map of the current TT ops: R1 RmsNorm + RoPE all-device at T=1 (threshold flip; numerics already proven by BACKEND-TENSTORRENT-RESIDUAL-GOLDEN; eager cost 12.5->10.7 tok/s measured, recovered by capture). R2 QkvSplit + ReshapeAndCache device-resident (bit-exact memcpy/stride ops; reuse the landed paged-KV-shadow path). R3 PagedAttention decode with device-resident metadata (the scope risk -- device SDPA decode exists, but block_table/seq_lens/query_start_loc are read host-side around it). R4 flip support_static_graph_mode + wire capture + measure replay tok/s (integration; the payoff that justifies R1-R3: replay warm tok/s >= 12.5). R1-R3 parallel-claimable; R4 integrates. The R1 eager regression (12.5-> 10.7) is acceptable ONLY because R4 recovers it; R1 must gate on support_static_graph_mode() so non-capture runs keep the 12.5 baseline. The trace-runner row's "next" now points here instead of to a vague prefill-capture note: the host-free forward is the real unblock. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: Maki:glm-5.2 [Maki]

…ture answer The experimental chain that un-no-go's the trace-runner decision. Measured on real Blackhole P150, env-gated (VT_TT_HOST_FREE_DECODE) and inert by default (21/21 TT tests, 814/814 assertions unchanged). Sequential bisection of every blocker between "capture aborts" and "capture runs into the layer ops." Complete blocker map: R1 RmsNorm/RoPE host thresholds at T=1 -> flip (done) R2 Backend::Copy host readback -> device->device copy (done) R3 ttnn program-cache warm-up -> enable + eager-warm (done) R3b Backend::Memset/DBuf::Zero host write-> device zero-fill (done) 5 per-op device write/upload -> ARCHITECTURE ANSWER below R1-R3b are landed and measured: capture now enters the forward and reaches the layer ops (CastBf16/RmsNorm fire during capture). The bisection instrumentation (TT_OP_TRACE macro + EnsureHostBytes/DownloadToHost prints + a tt_capture_active() file-scope flag) is kept, all gated on VT_TT_TRACE_DEBUG, inert by default. Item 5 (the remaining "Writes are not supported during trace capture" fatal) was investigated upstream. tt-metal issue #13690 (filed by Tenstorrent for vLLM, fixed in f0b2483) relaxes the ALLOCATOR guard during a live trace -- but our fatal is the separate ENQUEUE_WRITE guard (fd_mesh_command_queue.cpp:760), which #13690 did NOT touch. So bumping won't help. The real answer is architectural, from reading the official tt-metal vLLM plugin (tt/vllm/plugins/vllm-tt-plugin/.../model_runner.py): the captured ForwardLayers region must reference only pre-allocated, pre-populated persistent device tensors, with per-step inputs written to stable device buffers BEFORE ReplayGraph (never inside capture). The plugin uses ttnn.copy_host_to_device_tensor (= C++ copy_to_device -> enqueue_write_tensor) only at warmup-populate and before each replay, never during capture -- because copy_to_device itself hits the same enqueue_write fatal. This is the same pattern CUDA's decode graph uses (SizeSlot::Refresh writes host buffers that a captured async-copy re-reads). Path forward (bounded port, not a new subsystem, not an upstream fix): make the TT decode-graph slot hold persistent device tensors for inputs, populate them before capture/replay via copy_to_device, and ensure captured ops read those device tensors without any internal from_vector/to_vector. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: Maki:glm-5.2 [Maki]

Post-fold record repair before review: - BACKEND-TENSTORRENT-HOST-FREE-FORWARD matrix row: Tests/evidence cell now carries the measured R1-R3b outcome (was the stale "plan spec only; no code yet"); state SPIKE -> ACTIVE (R1-R3b landed, item-5 port open). - The plan spec (tenstorrent-host-free-forward.md) now meets the ACTIVE spike-gate contract (Scope / Upstream chain / Our baseline / Port map / Tests to port / Dependencies / Work breakdown) with the honest content: the port map is the tt-metal vLLM plugin technique table. - The TRACE-RUNNER claim tail records the measured outcome + the mudler#694 supersession (PR mudler#541 closing in its favor). - Dropped tests/parity/goldens/qwen3_greedy_0_6b/our_ids_tenstorrent.i32 — the raw VT_DUMP_IDS bootstrap artifact (untracked-noise class per the handoff; the .npy pair is already on main). It was swept into the fold unintentionally. - Post-rebase verification: default TT suite on the rebased tree on real P150 is 22/22 cases / 826/826 assertions — the env-gated host-free paths are inert by default. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: Maki:glm-5.2 [Maki]

Fresh static + mutation review (POL-REVIEW-FRESH) returned CHANGES-REQUESTED with two merge-blockers and a set of non-blocking findings. All addressed: BLOCKER 1 (records contradiction): the live coordination claim said SPIKE/"no implementation yet" while the same branch promoted the matrix row to ACTIVE. Claim row now ACTIVE with the landed R1-R3b state + the open item-5 port. The spec's Work breakdown also contradicted the landed R-numbering (it still defined R2=QkvSplit/RAC, R3=PA metadata from the original plan); restructured to the post-investigation truth: landed R1-R3b, open item 5 (the payoff port), queued-behind-item-5 for the original items, plus a "known constraints" section carrying the env-gated-only issues (ignored bytes, stale dev_rows/dev_cols, bytes-only dtype check, enable_program_cache placement, non-exception-safe capture-flag clear, SlotMutex TOCTOU, mid-capture shadow drop) into the item-5 spec. BLOCKER 2 (M1/M4 mutations stayed green — inertness unguarded): added a dedicated default-path inertness guard test. Two same-shaped outputs each carry a CURRENT device shadow (via device Matmul), so the env/capture gate is the only thing that can make CopyDeviceDeviceIfCapture / MemsetDeviceIfCapture decline; CHECK_FALSE on both (plus the value!=0 decline), and a host-path correctness anchor. MUTATION-PROVED on card: removing the gate (M1) turns the suite 22/23 RED; reverting restores 23/23, 830/830. NITs: device.h comment now matches the code (ttnn::zeros, not ttnn::full); SharedMeshDevice() moved after the no-shadow bail; matrix tests cell 23/23 with the guard noted. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: Maki:glm-5.2 [Maki]

…ll-device 64-token runs on real P150 correct two earlier smoke numbers: - The 4-token smoke (12.5 tok/s) is not steady state: at 64 tokens the same default config sustains 7.3 tok/s. Per-token cost grows with context, so the handoff's ~12.3 over-reports too. 7.3 is the honest capture reference. - The all-device eager cost at steady state is ~0.4 tok/s (~6%), not the ~1.8 tok/s (~14%) the 4-token smoke suggested — capture's break-even bar is lower than the spike framing assumed. (All-device runs need VLLM_CPP_QWEN3_DENSE_DECODE_GRAPH=0, else the flag's support_static_graph_mode flip makes the framework attempt capture and abort on item 5.) Capture/replay tok/s — the actual payoff — remains unmeasurable until item 5 lands; the numbers above bound it (must beat 7.3, eager floor 6.9). Mistral-7B cross-ref: 4.26 tok/s warm (tenstorrent-mistral.md). FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: Maki:glm-5.2 [Maki]

…isection Two item-5 fixes landed, both env-gated and inert by default (23/23 TT tests, 830/830): 1. ttnn::zeros/full is NOT capture-safe (creation.cpp full_impl host-fills and to_device()s = enqueue_write). Replaced the R3b zero-fill with a persistent zero-tensor cache: entries created OUTSIDE capture, primed during the eager warmup by EnsureDevice2D (flag-gated), applied in-region via ttnn::copy(zero, shadow) — a capturable device->device program. Cache-miss during capture is a hard VT_CHECK. 2. Verified QkvSplit's existing device path (ttnn::slice) fires under MatmulBT's shadow — earlier suspicion wrong. Instrumented all 16 from_vector sites with capture-gated [TT-UP] prints (UploadRows incl. ptr+shape). Measured frontier: capture now runs deep into layer 0 (CastBf16 -> RmsNorm -> MatmulBT -> QkvSplit -> qk-norms) and dies at a precisely-identified site — the RoPE cos/sin table upload ([16,64], BuildCosSinFromPositions host-build inside RopeNeoxKernel). That is the plugin per-step-input case: persistent device cos/sin buffer populated before capture/replay. Remaining sites after rope: RAC KV writes, PA metadata, logits path — same pattern, rope is the template. Item 5 is now a scoped multi-site port, not an unknown; replay tok/s stays blocked behind the remaining sites. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: Maki:glm-5.2 [Maki]

…n card) Three fixes working together: 1. Persistent cos/sin device cache keyed by (T*H, rot/2); entries created or content-refreshed OUTSIDE capture; replayed in-region. Stale content during capture is a hard VT_CHECK (never silent corruption). 2. Driver warm hook WarmRopeCosSin called from the decode-graph Refresh slot (qwen3.cpp), warming the UNPADDED T-row positions (what rope sees), not ppos — the first attempt used ppos and always missed. 3. Byte-exact content: the default rope route (RopeFromCache) reads the per-step bf16 CACHE table, so warm content must round-trip f32->bf16 (0.540302 f32 vs 0.539062 bf16 — f32 warm never matched). Measured on P150: rope cache HIT during capture for both q (16x64) and k (8x64); capture proceeds PAST rope. Also documented: the dense decode path routes rope through RopeFromCacheKernel by default (not RopeNeox), which the mis-placed debug print exposed. New frontier: ReshapeAndCache — next fatal is a to_vector readback inside RAC right after rope. Then PA metadata, then logits. RAC carries a replay-semantics question (replayed KV appends against fixed slots — the CUDA-graph solution) alongside the mechanical fix. Default-path safety: 23/23 cases, 830/830 assertions after all changes. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: Maki:glm-5.2 [Maki]

The agent-record CI job flags production env vars read from src/ that are undocumented. Both Tenstorrent host-free investigation vars are now in docs/ENVIRONMENT.md: the capture-mode gate under Deployment knobs (with the incomplete/byte-identical caveat), the bisection traces under Diagnostic. check-env-doc: OK, 339 vars. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: Maki:glm-5.2 [Maki]

MUSIC3 W6 (aa3643b) placed the anon-namespace SpeechRegistry() helper — returning vllm::multimodal::SpeechRegistry& — INSIDE the extern "C" block (vllm_c.cpp:488-1955). Clang rejects a C-linkage function returning a C++ reference (-Werror,-Wreturn-type-c-linkage); every clang build, including the Tenstorrent one, is broken on that commit (MSVC/Windows compiled, which is why CI's windows jobs went green). Minimal fix: hoist the helper (with its anon namespace) to just before the extern "C" block opens. No behavior change — the function is C++ and always was; the linkage was accidental. Also on this commit: the rebase record repairs (BACKEND=84 with the landed MISTRAL row + HOST-FREE-FORWARD; dup-row cleanups) and the post-rebase benchmark confirmation in the host-free spec (7.13/7.23 default, 6.68/6.80 all-device — consistent with pre-rebase). FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: Maki:glm-5.2 [Maki]

…apped FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: Maki:glm-5.2 [Maki]

…ed cache TryReshapeAndCacheDeviceDecode + WarmRacIdx driver warm hook implemented. Shape-keyed idx cache (same content-refresh pattern as rope). Warm fires correctly; content matches (slot0=32 warmup, slot0=33 capture, idx matches). The device branch bails because the paged-KV device shadow was never created (k=0 v=0) — EnsurePagedKvTtnn runs inside PA's device path, which doesn't fire during eager warmup. Fix: eagerly prime the shadow from WarmRacIdx. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: Maki:glm-5.2 [Maki]

…xecutes WarmPagedKvShadow primes both k and v shadows (fixed KvSlice offset bug). TryReshapeAndCacheDeviceDecode EXECUTES during capture (verified: "RAC device->device update (capture-safe)" printed on card). The paged_update_cache call hits "Writes not supported" because it's not program-cache-warm — the eager forward's RAC bailed to host (shadow didn't exist at eager time). Warm attempt from WarmRacIdx hangs (paged_update_cache deadlocks from the warm-hook context, likely a device-state issue). NEXT: move shadow priming BEFORE the eager forward so the eager RAC takes the device branch and warms paged_update_cache naturally (same CQ context). The framework's flow: eager step -> Refresh -> capture step; priming must happen before the eager step, not at Refresh. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: Maki:glm-5.2 [Maki]

…not capture-safe RAC device branch now executes on cold + capture steps. Fixed en route: - shadow priming for ALL layers (was only layer 0; each attn_kv[l] has its own .data pointer) - used=block+1 (off-by-one: shadow must cover block+1 blocks) - idx tensor dtype INT32 (ttnn requires INT32, not UINT32) - input sharding (paged_update_cache requires height-sharded input) - embedding ids reverted to UINT32 (global replace collateral damage) But paged_update_cache itself triggers "Writes are not supported during trace capture" — the op allocates a new output tensor internally, which is an enqueue_write forbidden during capture. This is NOT a program-cache issue (the cold step compiled the program). It is a ttnn API limitation: paged_update_cache returns a new tensor rather than updating in-place. Options: (a) upstream ttnn change for in-place update; (b) pre-allocate result + different scatter op; (c) skip RAC from capture (but PA reads the KV cache inside the captured region, so it can't move out). FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: Maki:glm-5.2 [Maki]

…padded Key discovery: paged_update_cache::create_output_tensors returns cache_tensor (in-place, no allocation). The Writes fatal was from build_padded's ttnn::to_memory_config (sharding allocates) + other helper ops, NOT from paged_update_cache itself. Attempted pre-built sharded zero in warm + ttnn::copy in build_padded. Crashed (segfault 139) on the cold step — shape/lifetime mismatch on the pre-built sharded tensor. Root cause clear; fix is to make build_padded use only capture-safe ops against pre-warmed persistent tensors, or take approach (b) — replace paged_update_cache with manual ttnn::copy into a pre-sliced cache region. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: Maki:glm-5.2 [Maki]

… idx) The segfault was caused by WarmRacIdx running BEFORE WarmPagedKvShadow in the driver Refresh slot, so the shadow loop found 0 entries and the sharded_zero was default-constructed (empty) — ttnn::copy then crashed on the empty tensor's memory_config(). Fixed by swapping the order: prime the shadows first, then build the RAC idx tensors (which use the shadows' geometry for the sharded input). The Writes fatal persists: paged_update_cache is in-place (no allocation), but build_padded's ttnn::copy between TILE and height-sharded memory configs triggers an implicit layout conversion (a write). NEXT: approach (b) — manual ttnn::copy into a pre-sliced cache region, no sharding. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: Maki:glm-5.2 [Maki]

Skip RAC during capture (mark shadow stale, return true). Capture proceeds PAST RAC to TryPagedAttentionDeviceDecode, which hits Reads-not-supported (the read guard at :807, not the write guard at :760). PA is doing a to_vector readback (likely EnsurePagedKvTtnn re-uploading the stale shadow or EnsureHost on query metadata). The real RAC fix: move it OUT of the captured ForwardLayers region — do the KV write at the driver Refresh slot (before BeginCapture), same as the plugin's per-step pattern. The captured region starts after RAC, not at ForwardLayers. Remaining sites: PA metadata, PA shadow re-upload, PA output, logits. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: Maki:glm-5.2 [Maki]

…uploads During capture, skip EnsurePagedKvTtnn and use the cached shadow (primed by WarmPagedKvShadow). Capture passes PA KV shadow check, reaches the page_table + cur_pos from_vector uploads (the same persistent-buffer pattern as rope cos/sin). FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: Maki:glm-5.2 [Maki]

… bail) PA metadata (page_table + cur_pos) persistent device tensors warmed at the Refresh slot. Capture uses cached meta: "PA using cached meta (pt+cp)". But sdpa_decode hits "Cannot load new binaries" — the cold step's PA device path bails before sdpa (q_from_device OK cap=0 never prints). Debugging the bail point next. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: Maki:glm-5.2 [Maki]

The cold step's PA device path was bailing because KvSlice returns a non-contiguous strided view that EnsurePagedKvTtnn's VT_CHECK rejects. Fixed by using the cached shadow (from WarmPagedKvShadow) on BOTH cold and capture steps. The cold step now runs sdpa_decode on all 28 layers (program compiled); the capture step hits the program cache. Measured on real Blackhole P150 (Qwen3-0.6B, 4-token smoke): run 1 (cold+capture): 18.5s, 0.22 tok/s run 2 (warm replay): 0.046s, 86.5 tok/s run 3 (warm replay): 0.051s, 77.9 tok/s ~12x speedup over the 7.3 tok/s eager baseline. EXIT=0. Default-path safety: 23/23 cases, 831/831 assertions. Caveats: RAC is skipped during capture (stale KV); the real fix is moving RAC out of the captured region. The 4-token smoke over-reports (64-token steady-state measurement pending). FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: Maki:glm-5.2 [Maki]

RAC now runs INSIDE the captured region. paged_update_cache is in-place (verified) and capture-safe. The sharded input uses the pre-built sharded_zero. Capture completes: EXIT=0, 83/77 tok/s replay. The k/v copy into the sharded buffer fails during capture (ttnn::zeros + ttnn::concat are writes, caught by try/catch, falls back to zeros = stale KV). Fix: pre-build the padded k/v at warm time using only capture-safe ops (ttnn::copy from device shadow into the pre-allocated sharded buffer). FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: Maki:glm-5.2 [Maki]

lu-zero mentioned this pull request Aug 13, 2026

spec(tenstorrent): trace-runner feasibility spike — NO-GO for T=1 decode #541

Closed

lu-zero force-pushed the row/BACKEND-TENSTORRENT-HOST-FREE-R1 branch from 18c7df4 to eef9dcd Compare August 14, 2026 10:07

lu-zero marked this pull request as ready for review August 14, 2026 10:52

lu-zero force-pushed the row/BACKEND-TENSTORRENT-HOST-FREE-R1 branch from 0e3db6a to 4f0a6f9 Compare August 14, 2026 18:17

lu-zero added 18 commits August 15, 2026 00:05

spec(tenstorrent): RAC capture-safe design complete — the next step m…

bee5b7f

…apped FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: Maki:glm-5.2 [Maki]

lu-zero force-pushed the row/BACKEND-TENSTORRENT-HOST-FREE-R1 branch from 413b274 to b90cb15 Compare August 14, 2026 22:07

lu-zero force-pushed the row/BACKEND-TENSTORRENT-HOST-FREE-R1 branch from b90cb15 to 812ac3e Compare August 14, 2026 22:07

lu-zero added 2 commits August 15, 2026 00:19

Navigation Menu

feat(tenstorrent): host-free decode forward — capture investigation + architecture - #694

feat(tenstorrent): host-free decode forward — capture investigation + architecture#694
lu-zero wants to merge 21 commits into
mudler:mainfrom
lu-zero:row/BACKEND-TENSTORRENT-HOST-FREE-R1

lu-zero commented Aug 13, 2026 •

edited

Loading

Uh oh!

lu-zero commented Aug 14, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

lu-zero commented Aug 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What

The complete blocker map (each measured by sequential bisection)

Item 5 — the architecture answer

Review

Gates

Files

Uh oh!

lu-zero commented Aug 14, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

lu-zero commented Aug 13, 2026 •

edited

Loading