A draft proposes tokens, the full model verifies them in the same step, and an accepted proposal skips a full model step. It is opt-in and off by default; with no speculative config the engine is byte-identical to the non-speculative one.
Selection mirrors vLLM's own CLI: --speculative-config '<json>', taking the
same JSON object vLLM takes, so a config written for vLLM works here.
The engine accepts five method strings. Anything else is refused at load with
the list of accepted ones (src/vllm/config/speculative.cpp).
method |
Draft | k | State |
|---|---|---|---|
mtp |
the target's own mtp.* head |
1 | Gated, and the default recommendation. Token-exact against vLLM at concurrency 1, ~1.04x its speculative-on decode |
dflash |
a separate z-lab block-diffusion checkpoint | block size, e.g. 16 | Gated. 2.9x over speculative-off at concurrency 1, at or above vLLM's own DFlash-on |
dspark |
a separate DSpark checkpoint | block size, at least the draft's own | Runs end to end, not gated. Token-identical to speculative-off on the 35B, but currently about 2% slower than plain decode; no speed win is claimed |
ngram |
none, drafts come from the prompt's own suffix | required, no default | Accepted and wired; no published measurement |
draft_model |
a separate full model | from the draft | Config only. The JSON parses, but the engine has no branch for it and refuses the load |
MTP and DFlash are the two with binding numbers behind them; the per-method detail below and in BENCHMARKS.md says which is which.
- Models: the Qwen3.5 / 3.6 gate checkpoints that ship an
mtp.*draft head in their safetensors (Qwen3.6-27B and Qwen3.6-35B-A3B). Both are GDN hybrids, and the speculative path is wired through the linear-attention (GDN) recurrence and short causal convolution as well as the attention layers. - k = 1 only.
num_speculative_tokensgreater than 1 is not accepted for this method. Depth (k>1, dynamic, adaptive) is unbuilt and tracked in #81. - Correctness: at concurrency 1 the speculative-on greedy output is token-for-token identical to both the speculative-off output and vLLM's own MTP speculative greedy output on the same prompt.
- Speed: measured about 1.04x faster than vLLM's own speculative-on decode at concurrency 1 (see Measured result).
DFlash drafts a whole block in one non-autoregressive pass from a separate
z-lab draft checkpoint over the same target. num_speculative_tokens is
required and is the draft's block size, e.g. 16.
server --model /models/Qwen3.6-27B \
--speculative-config '{"method":"dflash","model":"<draft-path>","num_speculative_tokens":16}'Gated on the 27B: end-to-end output matches the vLLM DFlash-on golden under the ratified near-tie rule, and at concurrency 1 it is 2.9x our own speculative-off throughput and at or above vLLM's own DFlash-on decode. Speculative-off stays byte-identical.
The draft-free proposer: candidates come from matching the sequence's own
suffix, so there is no draft checkpoint to load. num_speculative_tokens is
required and the prompt-lookup window defaults to 5/5.
server --model /models/Qwen3.6-27B \
--speculative-config '{"method":"ngram","num_speculative_tokens":4}'It is wired end to end and reuses the same verify machinery as MTP. No throughput number is published for it.
DSpark drafts a whole block in one parallel pass and then adds intra-block dependency with a small sequential head, so a block draft stops being k conditionally independent guesses. It is the DFlash drafter plus a low-rank Markov transition bias.
Current state (SPEC-DSPARK): the config, the Markov head and draft model, both
published checkpoint layouts (native deepseek-ai/dspark_qwen3_* and
Speculators RedHatAI/*.dspark), the sequential sampler and the runner wiring
have landed, and the path runs end to end on the Qwen3.6-35B-A3B gate model.
DSpark now genuinely speculates on the Qwen3.6-35B-A3B gate model: draft tokens
are proposed and accepted, and throughput went from 6.78 to 41.89 tok/s once an
engine-wide bug was fixed that had been silently discarding EVERY speculator's
drafts on the CLI and server path (check_for_draft_tokens was never threaded
into EngineCoreProc, so post_step returned before installing anything).
On that model its greedy output is token-identical to speculative-off across all 48 tokens and reproducible (both arms run on the synchronous path; the speculative path forces async scheduling off, so a fair comparison has to force it off on the other side too).
It is still not gated. Speculation is currently about 2% SLOWER than plain decode at concurrency 1 — the sequential stage is a host-side loop with a device round-trip per step — and the cross-engine comparison against vLLM's own DSpark, the acceptance-rate band, and the other target families are all still owed. No speed win is claimed. A GGUF target, and a target architecture with no aux multi-tap, are both refused by name.
main --model /models/Qwen3-4B \
--speculative-config '{"method":"dspark","model":"deepseek-ai/dspark_qwen3_4b_block7","num_speculative_tokens":7}'num_speculative_tokens is required for a DSpark draft (a native DSpark config
carries no n_predict), and it must be at least the checkpoint's block size —
a smaller value produces incorrect output rather than merely lower acceptance,
so it is rejected.
On the OpenAI server:
server --model /models/Qwen3.6-27B \
--speculative-config '{"method":"mtp","num_speculative_tokens":1}'On the example CLI:
vllm-cli --model /models/Qwen3.6-27B \
--prompt "The capital of France is" --max-tokens 64 \
--speculative-config '{"method":"mtp","num_speculative_tokens":1}'Over the C ABI (include/vllm.h, ABI v6):
vllm_model_params mp = vllm_model_params_default();
mp.model_path = "/models/Qwen3.6-27B";
mp.speculative_config = "{\"method\":\"mtp\",\"num_speculative_tokens\":1}";
vllm_engine *engine = NULL;
vllm_engine_load(&mp, &engine); /* NULL/"" speculative_config => no speculation */The JSON is parsed into the same vllm::SpeculativeConfig the C++ API takes
programmatically (EngineParams::speculative_config). The examples above use
mtp, where num_speculative_tokens must be 1; see Methods for the
other spellings and what each one requires. A malformed document, an unsupported
method, a missing num_speculative_tokens where the method needs one, or a
checkpoint with no mtp.* head fails the load loudly at startup rather than
running silently without speculation.
Concurrency 1, Qwen3.6-27B (NVFP4) on GB10, our speculative-on decode versus vLLM 0.25.0 running the same speculative config:
| Metric | Ours (spec on) | vLLM (spec on) |
|---|---|---|
| Time per output token, prose | 66.2 ms | 69.1 ms |
| Time per output token, code | 62.95 ms | 65.3 ms |
| Decode throughput, prose | 15.1 tok/s | 14.4 tok/s |
| Decode throughput, code | 15.7 tok/s | 15.1 tok/s |
| Drafter acceptance rate | 0.85 prose / 0.92 code | 0.84 |
Speculation helps both engines about 1.5 to 1.6x at this operating point, and our engine is already about 4% faster than vLLM with speculation off, so the lead is preserved with it on. The extra state speculation needs (a doubled recurrent-state slot at k=1, plus the draft cache and head) costs about 3.6 GB, well inside the box's unified memory. The full A/B, including the higher-concurrency numbers, is in BENCHMARKS.md.
The concurrent (multi-request) path, where a speculative request shares a scheduler step with an ordinary prefill, is implemented. The GDN layer splits the batch into its speculative and ordinary rows, runs each through its own recurrence over disjoint per-request state, and merges the results back; because the state is disjoint, the mixed batch's output equals the two requests run alone, bit-for-bit, at the real 27B and 35B dimensions. Speculation keeps helping at concurrency 2, 4 and 8 on the 27B (about 1.5x our own speculative-off throughput).
Each of these names the method it applies to.
- MTP is k=1 only.
num_speculative_tokensgreater than 1 is not accepted formtp. The block drafters take their k from the draft's block size instead, andngramrequires one. Depth for MTP (k>1, dynamic, adaptive) is unbuilt, #81. - EAGLE, EAGLE3, Medusa and the rest are not wired. Of vLLM's thirteen
SpeculativeMethodstrings the engine accepts five, anddraft_modelamong those is config-only. The remainder are inventoried, not implemented. - DSpark is not gated. It runs end to end and is token-identical to speculative-off on the 35B, but it is about 2% slower than plain decode at concurrency 1, and the comparison against vLLM's own DSpark, the acceptance-rate band and the other target families are all still owed.
- MTP needs Qwen3.5/3.6 with an
mtp.*head. The target may be a safetensors directory or a.ggufconverted WITH the head (llama.cpp's layer-indexednextnblock); a GGUF exported--no-mtpis refused and says so. - On an NVFP4 safetensors target the concurrency-1 identity above is not currently reliable. Measured on the 35B A3B NVFP4 safetensors: its logits land on a coarse grid that yields EXACT ties between distinct tokens, and at such a tie speculative-on and speculative-off, and even two speculative-off runs, can pick differently. The same weights loaded from GGUF, which expands to bf16, show no such ties and are token-identical. Open; see docs/STATUS.md.
- Concurrency above 1 is not token-stable for the 27B. Its greedy output is not bit-stable across batch shapes even with speculation off (changing the batch size flips a few near-tie tokens), so exact token-for-token agreement between speculative-on and speculative-off is a concurrency-1 property only. Correctness above concurrency 1 rests on the model-independent bit-exact GDN split-merge proof and on matching drafter acceptance, not on identical token streams.
The flag is a thin wrapper over the library surface:
vllm::entrypoints::EngineParams params;
params.speculative_config =
vllm::ParseSpeculativeConfigJson(R"({"method":"mtp","num_speculative_tokens":1})");
auto engine = vllm::entrypoints::LoadedEngine::FromModelDir(model_dir, params);Leaving speculative_config unset (the default) loads the ordinary engine, which
is byte-identical to a build with no speculative code compiled in.