SPEC-DSPARK: dense 27B lane accepts 12.2% where the pinned oracle accepts 49.3% (cross-engine 0.35x)

[localai-bot]

What

The binding cross-engine A/B against the pinned, graphed oracle
(0.23.1rc1.dev1511+g555967922, FlashInfer 0.6.15.post1) is recorded in
.agents/specs/dspark-spec-decode.md §6h. It splits into two verdicts:

• 35B A3B MoE lane: near parity. 0.92–0.98x cross-engine, and our
  acceptance MATCHES upstream: 20.8% vs 20.4% (212/1040, 1.63 accepted per
  draft of k=8).
• 27B dense lane: the gap, and it is DRAFT QUALITY. Upstream accepts
  49.3% (281/570, 7.39 accepted per draft of k=15) where we accept
  12.2% (~1.8 per draft). Upstream turns that into a 5.23x self-speedup on
  the "capital" prompt; we get 1.76x. Cross-engine that cell is 0.350x
  (17.41 vs 49.71 tok/s).

Warm tok/s, completion_tokens / whole-request wall on both sides, same
target+draft+k, max_num_seqs=2, greedy, one flock, cold run discarded.

lane / prompt	ours	pinned vLLM	ours/vLLM
35B spec-off, "capital" (128 tok both)	71.47	73.91	0.967x
35B DSpark k=8, "capital" (128 tok both)	74.10	75.92	0.976x
35B DSpark k=8, "fibonacci" (89 tok both)	134.85	146.41	0.921x
27B spec-off, "fibonacci" (126 tok both)	9.80	9.51	1.031x
27B DSpark k=15, "fibonacci"	31.83	34.06	0.935x
27B DSpark k=15, "capital"	17.41	49.71	0.350x

Ruled out (checked, not assumed)

• Mis-resolved aux taps. The 27B draft declares five taps at absolute
  target indices [1, 16, 31, 46, 61], and the 27B target is a hybrid
  (linear-attention + full-attention), so a tap resolved against a filtered
  layer list would silently degrade drafts. Both MaybeCaptureAuxTap call
  sites are inside the full for (l = 0; l < num_hidden_layers; ++l) loop and
  index by absolute l (qwen3_5.cpp:6707, :7546).
• The Markov step loop, which matches upstream's shape: prev seeded from
  the anchor, re-biased per step, argmax over base + bias, then d2t
  (speculator.py:120-121,148).

Next traceable hypothesis

The 27B draft uses the flat/native config layout and carries no
draft_vocab_size; the 35B carries 32000 and reaches upstream-matching
acceptance through the Speculators translation path. So the d2t / reduced-vocab
handling on the flat path is the thing to instrument next — compare our
proposed draft token ids against upstream's for an identical prefix, rather
than comparing acceptance totals.

No ceiling is being claimed here: upstream demonstrably extracts 7.39 tokens
per draft from the same checkpoint on the same hardware.

Also fixed in passing

The first run of this A/B measured the wrong oracle: ~/venvs/vllm-oracle
is a symlink to the preserved v0.25.0-stage rollback (issue #375's failure
mode). The harness now asserts vLLM commit + FlashInfer version and aborts on
mismatch. The rollback misleads in both directions — its DSpark is far slower
than the pin's (35B 89.8 vs 146.4 tok/s).

[localai-bot]

REFUTED: there is no 4x draft-quality gap. Correcting my own root cause.

Direct instrumentation of both engines on the same prompt kills this issue's premise.

Our drafts are nearly token-identical to upstream's. Dumping upstream's
_sample_sequential (anchor, and the k proposed ids) against our VT_SPEC_TRACE
at the same anchors:

anchor	upstream proposes	we propose
11751	13 271 9764 6511 220 6511 314 279 369 279 198 13 198 ...	13 271 9764 6511 220 6511 314 279 369 279 198 13 198 ...
248068	198 248069 271 760 6511 314 9338 369 ...	198 248069 271 760 6511 314 9338 369 ...
2972	57590 332 369 279 6511 314 9338 9338 13 248046 ...	57590 332 369 279 6511 314 9338 9338 13 248046 ...

Acceptance is at parity on matched content. Same prompt, both engines
emitting the same output tokens: ours 32 tokens in 11 draft steps (12.1%
accepted of k=15), upstream 32 tokens in 12 steps (11.1%). We are marginally
ahead, not 4x behind.

Where the 49.3% came from. Upstream's graphed benchmark run on
"The capital of France is" degenerated into a repetition loop: 8 distinct
token ids across 128 tokens ("The capital of France is Paris." repeated ten
times), while upstream's OWN spec-off arm on the same prompt produced ordinary
reasoning text (<think>Here's a thinking process: ...). Trivially predictable
repeated text is trivially draftable, which is what produced 7.39 accepted per
draft and 49.7 tok/s. Our engine generated real reasoning text on that cell, so
the 0.350x ratio compared 84 tokens of reasoning against 128 tokens of a loop.

That also disposes of the "next hypothesis" recorded here: the d2t/reduced-vocab
path is irrelevant, because this checkpoint has no d2t at all (both Markov
tables are full target vocab [248320, 256]).

Also ruled out along the way, each checked in source rather than assumed: aux
tap indexing (both MaybeCaptureAuxTap sites index by absolute l), the Markov
step loop (matches speculator.py:120-121,148), logit_scale (upstream routes
base logits AND bias through the same logits_processor, so it cancels in the
argmax), head aliasing (we correctly keep the draft's own shipped
embed_tokens/lm_head), attn_output_gate (risk R3 RESOLVED: q_proj is
[4096, 5120], not doubled, so it is inert metadata), and the non-causal
resolution (ours mirrors upstream's inverted causal = is_sliding rule, so all
five draft layers are non-causal as upstream has them).

An A/B on our own engine, same weights with only sample_from_anchor flipped in
a copied config, confirmed our anchor layout is correct: forcing the 1+N layout
drives acceptance to ZERO on every step (15.6 -> 6.24 tok/s) because the
proposals land off-by-one.

The real gap is PER-STEP COST, and it is measured

Closing this and opening the actual lever. Our draft step splits as
backbone 27.4 ms + host Markov sampling 10.5 ms, so 28% of the draft step is
spent downloading nqpr x draft_vocab logits and running k host argmaxes over
the full vocab. For the 27B that is 15 x 248320 floats = 14.9 MB downloaded per
draft step; the 35B's reduced 32000-vocab draft moves only 1.15 MB, which is
part of why that lane sits closer to parity. Upstream keeps the whole draft step
on device under one captured graph (risk R5 in the spec, never closed).

[localai-bot]

Premise refuted by direct measurement (see above). The per-step cost lever is tracked separately.