SPEC-DSPARK W8: capture the T=1+k speculative VERIFY shape (the measured 0.870x-0.981x gap)

[localai-bot]

Why

Paired measurement against the pinned graphed oracle (.agents/specs/dspark-spec-decode.md §6l):

35B cell	ours	pinned graphed vLLM	ratio
"capital", 128 tok both	75.82	77.28	0.981x
"fibonacci", 89 tok both	135.39	155.60	0.870x

Our fibonacci reps span 134.81-135.41 (0.4%), so 0.870x is a reading, not noise.

The deficit tracks acceptance. It is largest where the block is mostly
accepted, so the T=1+k verify runs every step (0.870x), and smallest where
acceptance is low (0.981x). That localises the gap to the VERIFY forward, not
the drafter: acceptance is at parity with upstream and our proposals are near
token-identical to theirs (#430).

Mechanism. Both model families gate the decode CUDA graph on
input.pure_decode (qwen3_5_dense.cpp:159, qwen3_5_moe.cpp:128), and
pure_decode is num_actual_tokens == num_reqs. A speculative verify submits
num_reqs x (1+k) tokens, so it fails that predicate and runs EAGER every step,
while upstream captures the uniform 1+k shape. §6g already established that the
graph launches we DO see are the D13 draft-step graph, not the verify.

This is the same headroom DFlash left open as D12 Part C.

Scope

• A verify-shaped capture keyed on (num_reqs, 1+k). k is fixed per config and
  num_reqs <= max_num_seqs, so the shape count is small and static.
• It must route the SPEC paths, not the pure-decode ones: block-diagonal causal
  attention over the 1+k query span, and the GDN/Mamba T>1 spec recurrence with
  per-timestep snapshots that SPEC-GDN-SEGMENTS already landed
  (vt::GdnSpecDecode, vt::CausalConv1dSpecUpdate).
• Qwen3_5DenseDecodeGraph as it stands is documented pure-decode ("all
  query_len==1"), so this is a sibling class or an explicit generalisation, NOT
  a loosened predicate. Do not simply widen the pure_decode gate: that would
  send a spec batch through a graph captured for query_len==1.

Gate

• Replayed == eager BIT-IDENTICAL on both gate models, spec-ON and spec-OFF.
• spec-OFF byte-identical (the SACRED corpora), i.e. inert when not speculating.
• Then the paired cross-engine re-measure on the same two 35B cells above; the
  target is >= 1.0x on both.

Capture-safety warning

No function-local upload temporaries inside the captured region. This repo has
already shipped a use-after-free from exactly that (a captured graph baked stack
addresses, replay read freed memory), and a sanitizer-clean run is NOT proof of
capture safety. Compute-sanitizer plus an explicit replay-vs-eager bit-compare.

Follows #436 (W7 device sampling, landed, byte-identical), which removed the
host-side sampling waste but not this.

[localai-bot]

Budget measured: the whole gap is ~4.8 ms/step, and the verify is 87% of the step

Decomposed our 35B "fibonacci" cell (the 0.870x one), VT_SPEC_TRACE, warm reps:

quantity	value
tokens / steps	89 / 18 = 4.89 tokens per step (accepted 3.89 of k=8, + bonus)
our step, total	36.6 ms (0.659 s / 18 steps, and 0.658 s on the next rep)
our draft phase	4.64 ms (backbone 4.16 + sample 0.48)
our verify + overhead	~32.0 ms = 87% of the step
pinned graphed oracle, whole step	~31.8 ms (155.60 tok/s, same 89 tokens)

The oracle's ENTIRE step, including its own draft, costs about what our verify
alone costs. The deficit is ~4.8 ms per step, which is exactly the 13% that
0.870x represents.

That sizes this issue: the target is a ~4-5 ms/step recovery on a 35B MoE
forward at T=9, which is the right order for eager-vs-captured launch overhead
across that many layers and kernels. It also confirms the drafter is NOT worth
optimising further here -- the whole draft phase is 4.64 ms, so even eliminating
it entirely would not reach parity, whereas the verify is where the step lives.

Two consequences for whoever picks this up:

1. Measure replay-vs-eager on the VERIFY in isolation before wiring it, so the
   4.8 ms is confirmed recovered rather than assumed.
2. sample=0.48 ms is already at the bandwidth bound (SPEC-DSPARK W7: the sequential Markov sample loop is HOST-side — 28% of the draft step, 14.9 MB downloaded per step #436 §6k) and is 1.3% of
   the step. Do not spend effort there.

Evidence: dgx:~/work/dspark-w6/verify_budget.log.

[localai-bot]

Ruled out: the verify cost is NOT host-side waste

Worth stating because this repo's largest 35B wins have historically been
host-side waste rather than kernel speed, so it was the cheaper hypothesis and
had to be checked before committing to a capture.

It does not apply. RejectionSampler::forward takes
logits [num_logits, vocab] f32 on q's device (rejection_sampler.h:116-119),
so the expanded verify logits stay device-resident and the greedy rejection runs
on device. There is no [1+k, vocab] download and no host argmax to remove --
which for the 35B would have been ~2.2M floats per step and could plausibly have
accounted for the whole 4.8 ms.

So the ~32.0 ms verify is the eager forward itself, and the ~4.8 ms/step deficit
is launch overhead across a 35B MoE at T=9. That leaves the static-shape capture
in this issue as the lever, with no cheaper alternative in front of it.

[localai-bot]

ROOT CAUSE: we mirrored the model but NOT vLLM's cudagraph dispatcher

The gap is not a kernel, a drafter, or an acceptance difference -- those are all
at parity. It is the graph DISPATCH PREDICATE.

Upstream @ 555967922:

# v1/worker/gpu/cudagraph_utils.py:95-105 — "uniform" = all requests share a
# query_len. It is NOT "query_len == 1".
def get_uniform_token_count(num_reqs, num_tokens, max_query_len):
    if (max_query_len == num_tokens // num_reqs) and (num_tokens == max_query_len * num_reqs):
        return max_query_len
    return None

# v1/cudagraph_dispatcher.py:37 — and the captured decode length IS the verify shape
self.uniform_decode_query_len = 1 + self.vllm_config.num_speculative_tokens

# v1/cudagraph_dispatcher.py:143-146 — so a uniform 1+k batch takes the FULL graph
if uniform_decode and self.cudagraph_mode.has_mode(CUDAGraphMode.FULL):
    num_reqs = min(num_tokens_padded // uniform_decode_query_len, max_num_seqs)
    assert num_tokens_padded % uniform_decode_query_len == 0

vLLM therefore graphs the speculative verify BY CONSTRUCTION: uniform_decode_query_len
is literally 1 + num_speculative_tokens.

Ours: pure_decode = num_actual_tokens == num_reqs, i.e. query_len == 1 only
(qwen3_5_dense.cpp:159, qwen3_5_moe.cpp:128). Our verify never matches, so it
runs EAGER every step. That is the whole ~4.8 ms/step, and it explains why the
deficit tracks acceptance: the more of the block is accepted, the more often the
ungraphed verify runs (0.870x at high acceptance vs 0.981x at low).

What this changes about the work

It is a MIRRORING job with an upstream anchor, not an invention:

1. Generalise the predicate from "query_len == 1" to upstream's uniform test, and
   set the uniform decode query length to 1 + num_speculative_tokens.
2. Key captures on a descriptor (num_tokens, num_reqs, uniform) rather than one
   bespoke pure-decode graph, and pad token counts to a captured set the way
   _bs_to_padded_graph_size does, so a handful of shapes covers every batch.
3. The captured region must route the SPEC attention and GDN paths for a 1+k
   query span (already landed under SPEC-GDN-SEGMENTS), which is what makes
   this more than flipping the predicate — flipping it alone would send a spec
   batch through a graph captured for query_len == 1.

Gate unchanged: replay == eager bit-identical both gate models, spec-OFF
byte-identical, then the paired cross-engine re-measure, target >= 1.0x.

[localai-bot]

The wiring is SMALL: our capture is already keyed on padded TOKEN count

Reading Qwen3_5DenseDecodeGraph::Impl (qwen3_5.cpp:8706+) changes the estimate
of this work. The machinery already matches vLLM's shape-set design:

• slots is std::map<int64_t /*padded S*/, SlotRing> — a captured shape SET,
  not one graph, which is upstream's _bs_to_padded_graph_size idea.
• Step() computes B = token_ids.size() — the token count, not the request
  count — then S = PadToCaptureSize(B, max_num_reqs). A verify batch of
  num_reqs * (1+k) tokens therefore already has a natural slot key.
• SizeSlot::Refresh already copies the SPEC fields (gdn_meta.num_spec_decodes,
  num_decode_tokens, max_query_len, attn_meta.causal), so the captured
  region reads spec metadata rather than needing a new struct.
• Padding rows are already made inert the way vLLM does it (token/pos 0,
  slot_mapping = -1 so ReshapeAndCache skips, gdn state index -1 so the
  conv/ssm update skips), and the forward is documented ROW-INDEPENDENT.

So the port is not a new capture class. It is:

1. Caller gate: replace input.pure_decode with the landed
   IsUniformDecodeBatch(num_reqs, num_actual_tokens, max_query_len, num_spec)
   (src/vllm/v1/worker/gpu/cudagraph_dispatch.h) at qwen3_5_dense.cpp:159 and
   qwen3_5_moe.cpp:128.
2. Pad cap: PadToCaptureSize's cap is max_num_seqs today, and the comment
   says why — it keeps padding inside the GDN state cache. Under speculation that
   cache is ALREADY widened to max_num_reqs * (num_spec + 1)
   (runner.cpp:449-451, landed with Speculative KV widening allocates unbounded GDN state and takes the machine down instead of refusing #371), so the cap becomes
   max_num_reqs * UniformDecodeQueryLen(num_spec). Verify that arithmetic
   against the actual allocation rather than trusting this note.
3. Capture-time sizing: SizeSlot buffers (hidden [S,H], logits [S,vocab])
   follow S, so they scale automatically once the cap allows a larger S.

What still has to be PROVEN, not assumed

• The captured region must select the SPEC attention path for a 1+k query span.
  attn_meta.causal and max_query_len are refreshed, but confirm the kernel
  chosen at CAPTURE time is the one a 1+k batch needs — a graph bakes in the
  kernel, so capturing under a wrong max_query_len would silently freeze the
  wrong one. Capture with the spec shape, not a warm pure-decode shape.
• replay == eager BIT-IDENTICAL on both gate models, spec-ON and spec-OFF.
• spec-OFF byte-identical (SACRED corpora): with num_spec == 0 the predicate
  reduces to today's pure-decode shape, so this should be provable as inert.
• No function-local upload temporaries inside the captured region.

Then the paired cross-engine re-measure on the two 35B cells; target >= 1.0x
against 0.870x / 0.981x today.

[localai-bot]

One prerequisite the plan above glosses over

ModelForwardInput (model_registry.h:229-241) carries pure_decode but NOT the
speculation width, so IsUniformDecodeBatch(..., num_spec) cannot be evaluated at
the gate as things stand. Add a default-inert field:

int64_t num_speculative_tokens = 0;   // 0 => predicate reduces to pure decode

set it where the runner builds the verify forward, and the gate becomes a local
decision. Defaulting to 0 is what keeps every non-spec caller byte-identical, so
the field can land before the gate flip.

An alternative that avoids the field is to derive the uniform length from the
batch itself (num_actual_tokens / num_reqs compared against
attn_meta.max_query_len). That is tempting but is NOT what upstream does:
vLLM restricts the FULL-graph branch to uniform_decode_query_len specifically
(cudagraph_dispatcher.py:143-146) so the captured shape set stays bounded.
Deriving it would let any uniform shape capture a graph, including chunked
prefill batches that happen to be uniform, and each new shape costs a capture and
its buffers. Thread the field; mirror the bound.

Order that keeps every step landable and inert:

1. add the field (default 0) + set it in the runner — byte-identical everywhere;
2. raise the pad cap to max_num_reqs * UniformDecodeQueryLen(num_spec) — still
   unreachable while the gate rejects spec batches;
3. flip the gate — the first step that changes behavior, and the one whose gate
   is replay == eager bit-identical plus spec-OFF SACRED byte-identical.

[localai-bot]

CORRECTION to my "three edits" estimate: the spec GDN path is not capture-safe yet

I previously wrote that this was three edits plus a field. That was wrong, and the
evidence is specific.

1. The graph path explicitly refuses spec batches, by assertion.
detail::ValidateGdnDecodeGraphState (qwen3_5.cpp:422-441) hard-asserts

metadata.num_prefills == 0 && metadata.num_prefill_tokens == 0 &&
metadata.num_spec_decodes == 0 && metadata.num_spec_decode_tokens == 0 &&
metadata.num_decodes == real_batch && metadata.num_decode_tokens == real_batch &&
metadata.num_actual_tokens == real_batch
    "metadata must describe exact pure non-spec decode"

and qwen3_5.cpp:8472 says so in as many words: "spec never captures --
ValidateGdnDecodeGraphState rejects a spec batch". So the assertion is not an
oversight to delete; it is guarding an invariant the region was built on.

2. The blocker underneath it: the spec GDN branch uploads from FUNCTION-LOCAL
host buffers. In the spec path:

DBuf dqsl(d, DType::kI32, {nreq + 1}, qsl_full.data());   // qwen3_5.cpp:4261
DBuf dhis(d, DType::kI32, {nreq},     his.data());        // qwen3_5.cpp:4262
DBuf dcs (d, DType::kF32, cs_shape);                      // qwen3_5.cpp:4245

Those are H2D copies sourced from host vectors that live on the stack of the
forward. That is EXACTLY the failure this repo has already shipped once -- a
captured graph bakes the source address, and replay reads freed stack memory.
A sanitizer-clean first run does not catch it, because the stack is usually still
mapped.

The pure-decode path does not have this problem because it was given the
persistent treatment: StepDevInputs (qwen3_5.cpp:3643+) holds positions /
slot_mapping / block_table / seq_lens / query_start_loc / gdn_state_idx as
persistent device buffers, refreshed via pinned staging OUTSIDE the captured
region. The spec branch never received that treatment because it never captured.

So the real shape of this work

Before any predicate flip:

1. Give the spec GDN metadata the same persistent-buffer treatment the non-spec
   path already has -- extend StepDevInputs (or a spec sibling) to cover the
   spec query_start_loc / state indices / conv-state staging, and move the H2D
   out of the captured region.
2. Only then generalise ValidateGdnDecodeGraphState to accept a uniform spec
   batch, with the assertion re-expressed rather than removed
   (num_decode_tokens == num_reqs * (1+k), state indices sized num_reqs*(1+k)).
3. Then the gate + capture-size work, and capture with the SPEC shape so the
   graph bakes the spec attention kernel, not a pure-decode one.

Step 1 is the substance and is where the capture-safety review belongs. The
predicate landed at a170c81 remains correct and inert; it is simply not the
long pole.

[localai-bot]

Implementation attempted. The real blocker: our decode-graph driver conflates TOKENS with REQUESTS

I built the capture and ran it. It engages -- captured Qwen3.5 dense decode graph for padded size S=16 (real B=16), which is exactly 1+k at k=15 -- so the
predicate and the aux plumbing work. But three defects surfaced, and the third
is structural.

1. Padding rewrote the spec metadata. BuildPaddedDecode unconditionally
sets num_decodes = num_decode_tokens = num_actual_tokens = S and drops the spec
fields, tripping "GDN decode+prefill+spec tokens must equal actual tokens". Fixed
by bypassing it for spec (S == B, nothing to pad).

2. Step's eager fallback ignored aux, so once aux batches were routed to the
graph, a disabled graph produced no taps at all ("missing target aux multi-tap").
Fixed by delegating to the eager ForwardDeviceMultiTap.

3. The captured graph read STALE spec metadata -> incoherent tokens and 5x
SLOWER (26 vs 136 tok/s). StageStepInputs refills only the pure-decode
fields, so a replay re-read the PREVIOUS step's num_accepted -- the value
GdnSpecDecode uses to select each request's initial state (column
num_accepted-1) and to advance the conv window. Zero acceptance then multiplies
the step count, which is where the slowdown came from. I added
StageSpecStepInputs, and forced the persistent StepDevInputs path on for spec
(the per-call one is a pool block freed on return, whose address a graph bakes).

And then the structural one. With staging added, the capture arm dies on
cudaMemcpyAsync: invalid argument, because the staging path sizes PER-REQUEST
arrays by the TOKEN count:

block_table = AllocPinned(sizeof(int32_t) * S * cols);   // rows are REQUESTS
seq_lens    = AllocPinned(sizeof(int32_t) * S);
qsl         = AllocPinned(sizeof(int32_t) * (S + 1));
...
std::memcpy(pin.block_table, s.attn_meta.block_table_tensor.data(),
            sizeof(int32_t) * S * cols);

That is correct ONLY because pure decode has S == num_reqs. Under speculation
S = num_reqs * (1+k), so this overruns the host block table (UB) and the device
buffer (the invalid argument). The conflation is not local to one function: the
slot, the pinned staging, and their consumers all use one S for both meanings.

This is exactly why upstream carries both. vLLM's key is
BatchDescriptor(num_tokens=..., num_reqs=..., uniform=...)
(cudagraph_dispatcher.py:150-156) -- two separate fields, because under
speculation they differ. We collapsed them into one when the only captured shape
was pure decode.

Revised scope

Before the capture can be correct, the decode-graph driver needs a request count
threaded alongside the token count: PinnedStepInputs sized R for per-request
arrays and S for per-token ones, StageStepInputs likewise, and every consumer
audited for which meaning it wanted. That is a refactor of the driver, not an
edit to a predicate, and it should be its own reviewed slice with the pure-decode
path proven byte-identical first (R == S there, so it is provable).

State: the work is on row/SPEC-DSPARK-LOADER, spec capture DEFAULT OFF behind
VT_SPEC_DECODE_GRAPH, so the tree is correct with it off (eager arm measured
128-136 tok/s, matching pre-change). Not landed on main.

[localai-bot]

LANDED: the verify is captured, MoE lane 0.870x -> 0.986x

Paired against the pinned graphed oracle in one lock session, matched token
counts, median of 3 warm reps:

35B cell	before	with capture	pinned vLLM	ratio
"capital", 128 tok both	72.23	78.37	78.76	0.995x (was 0.981x)
"fibonacci", 89 tok both	134.55	140.82	142.88	0.986x (was 0.870x)

Same-binary A/B (VT_SPEC_DECODE_GRAPH=0 = eager verify): +8.5% and +4.7%, text
byte-identical on both lanes. Gated with capture ON and OFF across the four e2e
spec-decode suites with identical assertion counts (9 / 27 / 9 / 5), default ON
re-verified with the env unset.

Still NOT >= 1.0x. The residual is ~1.4% against reps that spread 0.3%, so it
is a real gap and the row does not claim parity. Leaving this issue open for it.

The three defects worth knowing about are in the commit message and spec §6n; the
one with the widest blast radius is that our decode-graph driver collapsed TOKENS
and REQUESTS into a single count, which is correct only for pure decode. Upstream
keys graphs on BatchDescriptor(num_tokens, num_reqs, uniform) precisely because
speculation makes them differ.

Owed: the residual ~1.4%, the 27B dense lane re-measure (its cells were never
like-for-like), and padded / multi-request spec shapes -- capture takes the EXACT
shape today, bounded by max_num_seqs, which leaves batching on the table.