spec(tenstorrent): trace-runner feasibility spike — NO-GO for T=1 decode - #541
Draft
lu-zero wants to merge 2 commits into
Draft
spec(tenstorrent): trace-runner feasibility spike — NO-GO for T=1 decode#541lu-zero wants to merge 2 commits into
lu-zero wants to merge 2 commits into
Conversation
BACKEND-TENSTORRENT-TRACE-RUNNER. Replaces the prior "no-go" guess with a measured result from actually attempting the capture on-card. The spike ran the real experiment: local-only flips of both support_static_graph_mode() (enables Qwen3DenseDecodeGraph) AND VT_TT_FORCE_DEVICE (RoPE+residual all-device), then Qwen3-0.6B cli smoke. Both overrides reverted; no code shipped. Result: capture ABORTS. ttnn raises TT_FATAL: Reads are not supported during trace capture with a backtrace through ttnn::Tensor::to_vector<float> -- a device->host readback fires inside the captured ForwardLayers region. 0 replays. The T=1 forward does to_vector readbacks (embedding/PA/logits) that ttnn trace prohibits; forcing the two hybrid thresholds all-device is insufficient. This is strictly stronger than the earlier tok/s-recovery guess: decode capture cannot run AT ALL on the current forward without eliminating every to_vector readback in the captured region (a host-free ForwardLayers redesign, not a threshold tweak). Q1 (no host-free region at T=1) and Q2 (all-device costs 12.5->10.7 tok/s, reproducing handoff §6) stand. Decision: NO-GO for pure T=1 decode, measured. Next: prefill capture (separate row) must first audit its to_vector readbacks. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: Maki:glm-5.2 [Maki]
lu-zero
force-pushed
the
row/BACKEND-TENSTORRENT-TRACE-RUNNER
branch
from
August 12, 2026 22:16
690b49e to
f0bdc2d
Compare
BACKEND-TENSTORRENT-HOST-FREE-FORWARD (child of BACKEND-TENSTORRENT). The
trace-runner spike measured that decode mesh-trace capture ABORTS
(TT_FATAL: Reads are not supported during trace capture, backtrace through
to_vector<float>) because the per-layer T=1 forward does device->host
readbacks that ttnn trace prohibits. Capture wiring works (it engaged and
tried); the forward is the blocker.
This commit is a PLAN only (no code). It decomposes the host-free forward
into four independent rows, grounded in a full readback map of the current
TT ops:
R1 RmsNorm + RoPE all-device at T=1 (threshold flip; numerics already
proven by BACKEND-TENSTORRENT-RESIDUAL-GOLDEN; eager cost 12.5->10.7
tok/s measured, recovered by capture).
R2 QkvSplit + ReshapeAndCache device-resident (bit-exact memcpy/stride
ops; reuse the landed paged-KV-shadow path).
R3 PagedAttention decode with device-resident metadata (the scope risk --
device SDPA decode exists, but block_table/seq_lens/query_start_loc
are read host-side around it).
R4 flip support_static_graph_mode + wire capture + measure replay tok/s
(integration; the payoff that justifies R1-R3: replay warm tok/s
>= 12.5).
R1-R3 parallel-claimable; R4 integrates. The R1 eager regression (12.5->
10.7) is acceptable ONLY because R4 recovers it; R1 must gate on
support_static_graph_mode() so non-capture runs keep the 12.5 baseline.
The trace-runner row's "next" now points here instead of to a vague
prefill-capture note: the host-free forward is the real unblock.
FOLLOWING_AGENTS_PROTOCOL
Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: Maki:glm-5.2 [Maki]
localai-bot
added a commit
that referenced
this pull request
Aug 12, 2026
…_topk -- remove the literal bounds (#505) Lands the #505 fix: DsaTopkKernel sized `bool chosen[512]` and `int64_t picked[64]` by literal while index_topk is 512 (V4-Flash) / 1024 (V4-Pro). MEASURED on dgx.casa (GB10, sm_121a): the pre-fix kernel at Flash's own width takes `cudaStreamDestroy: an illegal memory access` and SIGABRTs; after the two-pass threshold rewrite the suite is 23/23 with 83913/83913 assertions and 0 skipped, re-verified from the merged tree with CUTLASS + FlashAttention-2 hard-verified in that run's own configure log. The fix removes the bounds rather than asserting them, so there is no configurable limit left to outgrow, and it drops the O(topk^2) emit sort. FRESH REVIEW: PASS. An independent reviewer reproduced both the defect and the fix on real sm_121a hardware, fuzzed 3,000,081 shapes across three independent implementations (host reference, kernel transcription, and its own O(n^2) rank-count oracle) with ZERO divergence, and ran a 12-row device mutation table. It found no blocking issue and 6 non-blocking findings, addressed in a follow-up. Notably it confirmed the tie-heavy case is uniquely load-bearing: it is the only case that catches a tie-break inversion or a value-only threshold. CI: windows-msvc-cpu and windows-msvc-vulkan are red at the repo-wide BASELINE, not from this change -- the same two jobs fail on #539, #541 and on #511 which already merged, and this PR's own log shows every target building with no `error C####`, failing instead in the release-packaging PowerShell step. Merged on direct developer instruction after the requested review pass. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [Claude Code]
localai-bot
added a commit
that referenced
this pull request
Aug 13, 2026
…opk, and correct two guard-comment overclaims (#552) Lands the six findings from the fresh review of #505, which returned PASS after failing to find any input where the two-pass threshold selection diverges from `DsaTopkSelect` -- 3,000,081 fuzzed shapes across three independent implementations, zero divergence. CODE: `DsaTopkLaunch` now checks `cudaGetLastError()` after the launch and refuses non-positive `topk` to match the host reference, which previously asserted while the device arm silently returned an empty vector. TESTS: two new device cases. One drives the window clamps -- under-run, over-run, both at once, and an in-range control -- which nothing exercised before. One asserts BOTH arms refuse topk=0 and topk=-1. PROSE: the `w < topk` guard comment claimed to defend against a NaN row writing past its own row. That cannot occur: pass 2's predicate is satisfied by exactly rank(th) <= topk elements and a NaN never satisfies it either. Corrected to belt-and-braces, bounded by construction. "Strictly cheaper" is now scoped to a comparison with the old kernel, not fitness for the real geometry. REFUTED IN FLIGHT: finding 4 held that the launch check would make the next fault attributable to this launch. The arm built to demonstrate it disproved it -- the pre-fix kernel WITH the check still reports `cudaStreamDestroy: an illegal memory access`, because a stack-overflow fault is asynchronous while `cudaGetLastError()` covers launch-configuration errors. The check is kept and its comment now says what it actually does; claiming otherwise would have repeated finding 1's defect inside the change that exists to correct one. DEVICE ARMS on dgx.casa (GB10 sm_121a), each a fresh nvcc rebuild from a pristine kernel with the binary mtime verified to advance, scoped with `-tc=` never `-ts=`, under `flock $HOME/gpu.lock`: baseline 6/6 SUCCESS; no_topk_guard FAILURE; no_ws_clamp exit=134 CRASHED; no_we_clamp 1052 failed assertions; no_launch_check SUCCESS (unobservable by construction); prefix_with_check exit=134; restored full suite 25/25 with 90062 assertions, 0 skipped, SUCCESS. Both new cases have teeth against the mutation each was written for. CPU 25/25 and 13/13, preflight fully clean including the contention-sensitive floor harness. Windows MSVC red is the repo-wide baseline (same two jobs fail on #539, #541 and on #511 which already merged; no `error C####`, the failure is the release-packaging PowerShell step). Merged on direct developer instruction. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [Claude Code]
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
BACKEND-TENSTORRENT-TRACE-RUNNER(child ofBACKEND-TENSTORRENT). A read-only feasibility spike (POL-SPIKE-FIRST, POL-NO-CEILING) deciding whether to wire the landed graph-capture foundation (#354 /59568772) into a capturable decode region. No code shipped — decision record only. The spike actually attempted the capture on-card rather than guessing.Findings
Q1 — host-free region at T=1? No. Static trace:
RmsNormKernel(rows=1 < 32) andPreferDeviceRope(T·H = 16 < 64) both route to host every layer. Bonus: the shared decode-graph frameworkQwen3DenseDecodeGraph(qwen3.cpp:489, used by Qwen3/Mistral/Llama/InternLM2) is disabled for TT because the platform doesn't overridesupport_static_graph_mode()(basefalse; only CUDA returnstrue).Q2 — all-device-at-T=1 tok/s cost? Measured on real Blackhole P150, Qwen3-0.6B, local-only
VT_TT_FORCE_DEVICEoverride (reverted, unshipped):Reproduces handoff §6 precisely (~12.3→10.5; 12.5→10.7 here; same ~14% / ~1.8 tok/s regression).
Q2b — capture attempted on-card: ABORTS. With both
support_static_graph_mode()→true(enablesQwen3DenseDecodeGraph, which wiresBeginCapture/ForwardLayers/EndCaptureGraph) ANDVT_TT_FORCE_DEVICEset, capture still fails:A device→host readback fires inside the captured
ForwardLayersregion. ttnnbegin_trace_captureprohibits any host read during capture; forcing the two hybrid thresholds all-device is insufficient because other ops (embedding / paged-attention / logits) still doto_vector.Decision
NO-GO for pure T=1 decode capture — measured, not assumed. Decode capture cannot run at all on the current forward without first eliminating every
to_vectorreadback in the captured region, which is a redesign of the TT forward's host-staging model, not a threshold tweak. The earlier "implausible to recover 1.8 tok/s" reasoning is superseded: capture aborts before perf even enters the picture.Open follow-on (separate row): prefill / multi-token chunk capture. At T≥32 the thresholds already go all-device; whether the prefill region is also free of
to_vectorreadbacks is the open question the next row must answer first (same ttnn host-read prohibition applies).How the other backends do it
CUDA is the ONLY
SupportsGraphCapture()==truebackend (cuda_backend.cu:184-240: contract = async/no host sync/no malloc in region, fixed pointers). Metal and Vulkan arefalsewith the eventual mapping named but unimplemented. Model-side capture sites captureForwardLayersdevice-resident, one eager step warms the pool first.Files
.agents/specs/tenstorrent-trace-runner.md(spec + measured## Outcome+ decision)BACKEND-TENSTORRENT-TRACE-RUNNERrow in.agents/backend-matrix.md(SPIKE, decision record complete).agents/coordination.mdscripts/check-agent-record.pyBACKEND 81→82src/change. Both local overrides (support_static_graph_mode,VT_TT_FORCE_DEVICE) were reverted.