SPEC-DSPARK W7: the sequential Markov sample loop is HOST-side — 28% of the draft step, 14.9 MB downloaded per step

[localai-bot]

Measured

[spec-phase] on the 27B lane, steady state:

backbone=27.44ms  sample=10.50ms  logits=3724800
backbone=27.36ms  sample=10.49ms  logits=3724800

The sequential Markov sampling is 10.5 ms per draft step, 28% of the step,
and it is pure host-side work:

1. every block-forward variant returns std::vector<float>
   (qwen3_dflash.h:155,202,248,284), so the logits are ALWAYS downloaded —
   nqpr x draft_vocab = 15 x 248320 = 3,724,800 floats = 14.9 MB per draft
   step on the 27B;
2. SampleDsparkBlockDrafts then runs, per step i of k, a device->host Markov
   bias download plus a host argmax over the full 248320 vocab
   (dspark/speculator.cpp:44-80).

The 35B lane's draft is reduced-vocab (32000), so it moves 9 x 32000 = 1.15 MB
instead of 14.9 MB — which is part of why that lane already sits at 0.92-0.98x
while the 27B does not.

This is risk R5 in .agents/specs/dspark-spec-decode.md, recorded when the
lane landed and never closed: "An N-iteration host loop with a device round-trip
per step would be a decode-path regression at exactly the point the feature is
supposed to win." Upstream captures the whole draft step, backbone plus the
N-step Markov loop, in ONE CUDA graph (dspark/speculator.py:22-24).

The fix

Keep the loop on device. Every primitive already exists, so this is composition,
not a new kernel:

• a device-returning block-forward variant (today's four all download);
• per step: IndexSelect(markov_w1, prev) -> GEMV against markov_w2 -> add the
  step's base row -> vt::GreedyArgmax (already lowest-index tie-break, the same
  convention the host loop uses) -> d2t map (identity when the checkpoint ships
  no d2t, as the 27B does) -> prev stays on device;
• download only the final [num_reqs, k] ids.

prev must stay device-resident or the per-step sync defeats the point.

Gate

Device path must be token-identical to the host path on both lanes (same drafts,
same acceptance), then the speed delta measured on matched content. Correctness
first: a faster drafter that changes drafts is not a win.

Context: this is the real lever behind #430, whose premise (a draft-quality gap)
was refuted by direct measurement — our proposals are nearly token-identical to
upstream's and acceptance is at parity on matched content.

[localai-bot]

W7 LANDED (0f2b12ed9, spec §6k): the sequential Markov sample now runs on device — one base-logits upload and a [num_reqs] i64 readback per step replace k downloads of [num_reqs, draft_vocab] f32 and k host argmaxes. Text byte-IDENTICAL on both lanes; sampling −11.5% (27B) / −15% (35B), ~3% e2e on the 35B.

Staying OPEN for the half this issue also names and W7 explicitly deferred: the device-returning block-forward variant. All four variants still perform the [nqpr, draft_vocab] D→H (qwen3_dflash.h:155,202,248,284) inside backbone=27.4ms. §6k sizes it at 14.9 MB ≈ ~0.15 ms on GB10 — a small prize that should be SIZED before it is built.

The residual in the sampling phase itself is NOT overhead: markov_w2 is [draft_vocab, markov_rank] bf16 so the per-step bias GEMV re-reads the whole matrix, and both lanes land on the ~205 GB/s bound 13x apart in vocab. Upstream runs the same math. Do not chase it as if it were host cost.

Listed in .agents/roadmap_v1.md's intake table as of #536 / #539 — it had been tracking work while absent from the table the protocol requires it to agree with.