diff --git a/.agents/issue-index.md b/.agents/issue-index.md index c30465bd0..9799397fc 100644 --- a/.agents/issue-index.md +++ b/.agents/issue-index.md @@ -209,3 +209,7 @@ rather than merged. `scripts/check-agent-record.py` gates both. | [#776](https://github.com/mudler/vllm.cpp/issues/776) | `GATE-OP-PARITY-MANIFEST` | `test_op_parity` THREW `json.exception.type_error.302` out of the CPU golden pass instead of failing an assertion, so the walker's `no runner for op` guard — the check that caught #559's missing runner arm — stopped running for every golden after the offender. The artifact was `tests/parity/goldens/minimax_music3_oracle/manifest.json`, and that half is #755, already fixed by `043e56862`. #755 closed the walker's INPUT set; it did not close its EXCEPTION surface, and reproducing #776 on the fixed tree shows the difference: nulling one tensor `dtype` in `rmsnorm_f32_8x128` still threw `type_error.302` at the TEST_CASE line and cut the pass from 142 assertions to 37, leaving 45 committed goldens unchecked. Both remaining throw sites — `json::parse` and any runner field read — now funnel through `GuardGoldenStage`, which turns a `std::exception` into a `FAIL_CHECK` naming `goldens//manifest.json` and continues. `doctest::detail::TestFailureException` is deliberately not a `std::exception` (`third_party/doctest/doctest.h:2563`), so the #559 `FAIL` still aborts loudly and the widening cannot mute it. FIXED IN FLOW | bug | | [#740](https://github.com/mudler/vllm.cpp/issues/740) | `MODEL-TEXT-qwen3-5-qwen3-5-moe-for-causal-lm` | MoE loader: bf16 / 3-D-stacked routed experts — the arm every published Qwen MoE repo ships, and the one Qwen3.8 needs | feature | | [#861](https://github.com/mudler/vllm.cpp/issues/861) | `GATE-SQUASH-SEPARATOR` | `PR_BODY` did not remove the `---------` separator: GitHub writes it above the `Co-authored-by:` block it appends, under both settings, so `617d6f452` landed failing the trailer gate with ONE block and one marker. #829 and #850 concluded otherwise from a simulation that omitted it. The fuse now steps over the separator when trailers sit on both sides (spec [`squash-separator.md`](specs/squash-separator.md)) | bug | +| [#436](https://github.com/mudler/vllm.cpp/issues/436) | `SPEC-DSPARK` | W7: the sequential Markov sample loop is HOST-side — 28% of the draft step, 14.9 MB downloaded per step | perf | +| [#442](https://github.com/mudler/vllm.cpp/issues/442) | `SPEC-DSPARK` | W8: capture the T=1+k speculative VERIFY shape (the measured 0.870x-0.981x gap) | perf | +| [#513](https://github.com/mudler/vllm.cpp/issues/513) | `SPEC-DSPARK` | DFlash/DSpark draft loading hardcodes the VL weight prefix, so a text-only Qwen3.5 target cannot share its embedding | bug | +| [#536](https://github.com/mudler/vllm.cpp/issues/536) | `ROAD-V1-C3` | C3's named tail records DSpark as unspiked three days after it landed, and files TLI under the wrong lane | bug | diff --git a/.agents/specs/dspark-spec-decode.md b/.agents/specs/dspark-spec-decode.md index e6f599a9d..df9a85099 100644 --- a/.agents/specs/dspark-spec-decode.md +++ b/.agents/specs/dspark-spec-decode.md @@ -17,7 +17,7 @@ | Dependencies | Landed: `SPEC-DFLASH` (`DONE`), `SPEC-REJECTION` verify half, `SPEC-GDN-SEGMENTS`. External, PENDING developer authority: checkpoint downloads (2.79-8.80 GB), dgx GPU time, push/draft-PR. Blocking unknown: R1, whether the pinned oracle runs DSpark at all. | | Work breakdown | §4 — W1 config, W2 Markov head + draft model, W3 loader (native + Speculators), W4 speculator (anchor layout + sequential sampling), W5 runner + one-surface, W6 gates. W1-W4 are CPU-gateable. | | Risks/decisions | §6 — R1 oracle runnability (V2 runner), R2 Speculators format is a new subsystem, R3 the community 27B checkpoint's `attn_output_gate`, R4 the `k >= dspark_block_size` garbling trap, R5 sequential sampling vs CUDA-graph capture, R6 greedy before probabilistic, R7 GB10 host-RAM pressure. | -| Status | W1-W5 LANDED and DSpark now genuinely speculates on the 35B gate model (real acceptance, 6.78 -> 41.89 tok/s) after fixing an engine-wide `check_for_draft_tokens` wiring bug that silently disabled EVERY speculator on the CLI/server path. W6 PARTIAL: spec-ON output is token-identical to spec-OFF and reproducible on the 35B gate model (§6b), but speed is ~2% BEHIND spec-off and the cross-engine + acceptance-band gates are still owed. R1 answered (§6a). | +| Status | **`ACTIVE`. W1-W8 LANDED and GPU-gated** (last refreshed 2026-08-12, [#536](https://github.com/mudler/vllm.cpp/issues/536); §8 records why this field had drifted). W1-W5 gave a working speculator on the 35B gate model after fixing an engine-wide `check_for_draft_tokens` wiring bug that silently disabled EVERY speculator on the CLI/server path (§6b); W7 ([#436](https://github.com/mudler/vllm.cpp/issues/436)) moved the sequential Markov sample on device, byte-identical (§6k); W8 ([#442](https://github.com/mudler/vllm.cpp/issues/442)) captured the T=1+k verify by mirroring vLLM's uniform-decode dispatch predicate, +12.2%/+3.5% same-binary (§§6m-6n). **Cross-engine under pinned clocks the 35B-A3B MoE lane is 0.975x (code cell, non-overlapping) / 1.012x (prose cell) — NOT parity**, with the residual localised to `marlin_moe_wna16::Marlin` and attributed to a 12.9% effective-DRAM-bandwidth gap on byte-equivalent machine code (§§6s-6aa). R1 answered (§6a). Still owed for a binding W6: the SACRED-corpus token gate under the ratified near-tie protocol, the 27B dense re-measure, the Gemma4 `1 + N` layout on real weights, padded/multi-request capture shapes, and the next bandwidth lever (§6aa). ~~W1-W5 LANDED … W6 PARTIAL: … speed is ~2% BEHIND spec-off …~~ (the 2026-08-10 text, superseded by §§6c-6aa; the "~2% behind" was a cold single-shot reading corrected in §6c). | | Goal (developer, 2026-08-09) | a FULL DSpark implementation in vllm.cpp, mirrored from vLLM | ## 0. Verdict @@ -2053,3 +2053,46 @@ as one. Sources: pin files cited inline at `555967922`; HF API queried 2026-08-09; our anchors cited against `bc6e3d72`. + +## 8. Record reconciliation (2026-08-12, [#536](https://github.com/mudler/vllm.cpp/issues/536)) + +`ROAD-V1-C3`'s named tail still read "DSpark (`SPEC-DSPARK`) + +heterogeneous-vocabulary TLI (`SPEC-TLI`) **unspiked** — overlaps `ROAD-V1-D3`" +three days after this row's W1-W8 landed and were measured cross-engine. Every +clause was wrong, and a punch-list reader would have dispatched a port that +already exists. + +| Surface | Said | Now | +|---|---|---| +| [roadmap-v1-completion.md](roadmap-v1-completion.md) §2 C3 row, §3 item 17 | DSpark + TLI "unspiked", "overlaps D3", Size M | superseded in place; DSpark is a perf tail plus owed gates (S-M), TLI is a separate row (M) | +| [roadmap_v1.md](../roadmap_v1.md) rows 3 and C3 | "DFlash Part C + DSpark/TLI remain"; "their dedicated spikes are not written" | `SPEC-DFLASH` is `DONE` (D13/D14, `489a7544`); this spike exists (`2b342620e`) | +| [roadmap_v1.md](../roadmap_v1.md) §DSpark grounding note | `SPEC-DSPARK` "(engine matrix, `INVENTORIED`)", spike "future" | `ACTIVE`, spike written | +| [spec-decode-inventory.md](spec-decode-inventory.md) `dspark` row + lifecycle summary | **INVENTORIED** | **ACTIVE** | +| [spec-decode-inventory.md](spec-decode-inventory.md) §HF speculators | "no `speculators`-format adapter" | W3 shipped the DSpark one (`qwen3_dspark.cpp:227-300`) | +| [docs/STATUS.md](../../docs/STATUS.md) method surface | `dspark` INVENTORIED, contradicting the same page's own DSpark paragraph | ships DSpark | +| This spec's `Status` field | 2026-08-10 "W1-W5 … W6 PARTIAL … ~2% BEHIND spec-off" | W1-W8; cross-engine 0.975x/1.012x on the pre-reimage box, SUPERSEDED by 0.834x matched-and-warm on the rebuilt stack | +| [roadmap_v1.md](../roadmap_v1.md) Open issues | #436, #442, #513 absent from the intake table | listed | + +**The TLI half is filed under the wrong row, and that is the more useful +finding.** Upstream TLI is `use_heterogeneous_vocab` +(`config/speculative.py:150`) plus `VocabMapping` +(`v1/spec_decode/vocab_mapping.py:68`), and both are consumed ONLY by +`v1/spec_decode/llm_base_proposer.py:432-495,688-691,831-837` +(`SpecDecodeBaseProposer`) and `v1/spec_decode/draft_model.py:19,34-58`. The +V2-runner speculators this row ports (`v1/worker/gpu/spec_decode/dspark/`, +`.../dflash/`) contain no heterogeneous-vocab path at all, so nothing DSpark +lands moves TLI, and TLI's host is `SPEC-DRAFT-MODEL` — locally a CPU propose +brick with no runner construction. In particular this row's `d2t` work is NOT +TLI: `d2t` offsets ids inside ONE tokenizer's vocabulary +(`draft_id + d2t[draft_id]`, §2 D), while `VocabMapping` builds a string-level +intersection ACROSS tokenizer families, probing the space prefix at init to +handle a BPE draft against a SentencePiece target. + +**Why the drift happened, since the row can be told from the outside.** The +punch-list is written from row summaries, and this row's summaries were the +thing not updated: the spike went straight from "planned" to twenty-odd +`measure(SPEC-DSPARK)` commits without either the inventory's lifecycle summary +or `STATUS.md`'s one-line method surface following. Both are derived statements +about a row that lives elsewhere, which is exactly the shape AGENTS.md's Records +section warns about — a fact stored away from the thing it describes drifts +silently, because nothing fails when it does. diff --git a/.agents/specs/roadmap-v1-completion.md b/.agents/specs/roadmap-v1-completion.md index c8184459f..481c0c4f2 100644 --- a/.agents/specs/roadmap-v1-completion.md +++ b/.agents/specs/roadmap-v1-completion.md @@ -37,7 +37,7 @@ carry more than one class list the dominant one first. | `ROAD-V1-C1` extensibility | **DONE** (cornerstone) | Drop-in kernel ABI W0, Platform seam, model self-registration, and the **portable op-fusion framework ORDER-1 milestone** (W0–W4 merged+gated, `KERNEL-FUSION-FRAMEWORK`); consistency-audit CI check landed. `BACKEND-ABI-VT`/`BACKEND-CUDA-ARCH-ADDITIVITY` seams gated on sm_121a. | Row stays SPIKE-open only for **non-blocking** tail: Tier-1 fusion perf interpreter (composite-only → single-launch), `FUSION-DENSE-MIGRATE` (route 5 drift models — CLOSED 2026-08-10, [#299](https://github.com/mudler/vllm.cpp/issues/299)), a real Metal/Vulkan catalog realization (M4-reachable / HW-blocked), and migrating a production kernel family onto the common adapter. Correctness cornerstone is closed. | | `ROAD-V1-C2` model families | **RI** (+HW/EXT sub) | First additive model (Qwen3 dense) + a broad **text sweep correctness-complete + SACRED-gated**: Qwen3/Qwen3Moe/Coder, Llama/Yi/InternLM3, Mistral, GLM-4-9B/GLM-4.7-Flash, Gemma-1/2/3, OPT, DeepSeek-V2-Lite (MLA), OLMo-2, Phi-3/4, Phi-1/2, Granite-3, StableLM, InternLM2, MiniCPM, MiniCPM3 (MLA). 20 ACTIVE model rows. | **SPEED close** on every one (all 20 are "correctness-complete, speed pending"). MoE/SSM breadth (Qwen3-Next, Falcon, Falcon-H1, GraniteMoe*, Cohere2Moe, PhiMoE, Mamba/Jamba/Zamba2/NemotronH) = RI (INVENTORIED/SPIKE). Frontier: Kimi-Linear-48B fits (RI, +KDA kernel); DeepSeek-V3/GLM-5/MiniMax-M2/M3/Kimi-K2 = HW (>119 GiB); Command-R = EXT (HF token). | | `ROAD-V1-C2-LOCAL-BF16` | **RI** (S) | Local Qwen3.5-4B plain-BF16 diagnostic rebased onto current additive seams; CPU/CUDA + direct ON/OFF token-equivalence green; H32 AOT / plain-BF16 graphs / ratio-4 FA2 landed + trace-proven. | Port device-resident sampled-token mapping to discrete CUDA (remove the measured main-stream wait) and rerun the exact 4B series. Small. | -| `ROAD-V1-C3` spec-decode | **DONE** (core) | **MTP k=1 DONE + gated on BOTH gate models** (`SPEC-MTP`, c1 token-exact + above vLLM, c2–c8 on-par-or-above); **DFlash DONE + speed gate MET** (`SPEC-DFLASH` D14, our-ON ≥ vLLM-ON). | Named tail only: DSpark (`SPEC-DSPARK`) + heterogeneous-vocabulary TLI (`SPEC-TLI`) unspiked — overlaps `ROAD-V1-D3`. Core spec-decode is gate-closed. | +| `ROAD-V1-C3` spec-decode | **DONE** (core) | **MTP k=1 DONE + gated on BOTH gate models** (`SPEC-MTP`, c1 token-exact + above vLLM, c2–c8 on-par-or-above); **DFlash DONE + speed gate MET** (`SPEC-DFLASH` D14, our-ON ≥ vLLM-ON). | ~~Named tail only: DSpark (`SPEC-DSPARK`) + heterogeneous-vocabulary TLI (`SPEC-TLI`) unspiked — overlaps `ROAD-V1-D3`.~~ **SUPERSEDED 2026-08-12** ([#536](https://github.com/mudler/vllm.cpp/issues/536), see §3 item 17): `SPEC-DSPARK` is `ACTIVE` — W1–W8 implemented and GPU-gated, 35B-A3B MoE **0.975x** code / **1.012x** prose vs the pinned graphed oracle (#442), remaining work a perf tail plus owed gates. `SPEC-TLI` is genuinely untouched and belongs under `SPEC-DRAFT-MODEL`, whose W3 blocks it. `ROAD-V1-D3` excludes both by its own spec, so it overlaps nothing here. Core spec-decode is gate-closed. | | `ROAD-V1-C4` quantization | **RI** | **3 schemes DONE**: NVFP4-MO-W4A16, NVFP4-CT-W4A4, FP8-MO-STATIC (all R/M/C/E/P). **GGUF CPU vs llama.cpp is CLOSED** (2026-07-22, aarch64 binding host): decode **at parity** (1.03× behind, inside llama.cpp's ±1.8% run spread — the elementwise f16/bf16 GEMM lever `KERNEL-GEMM-CPU-ELEM` E1-E4 `18094ee2` took it 3.38×→1.03×), prefill **1.18× ahead** (q8_0 repack-at-load G7), RSS **1.01×**, byte-identical greedy tokens. | NVFP4-CT-W4A16 perf gate. FP8-generic dispatch (static/dyn × tensor/channel/token/block). Breadth: AWQ/GPTQ/Marlin-wiring, i-quants, MXFP4/MX, bitsandbytes, KV-quant — all INVENTORIED. (GGUF-vs-llama.cpp speed is no longer an open C4 blocker.) | | `ROAD-V1-C5` sliding/YaRN | **RI** | Joint spike accepted; all W1–W8 leaves implemented and CPU/oracle/sanitizer green. **CUDA GPU CLOSURE 2026-07-27 (`CLAIM-ROADMAP-C5`, dgx GB10 sm_121a, clean build of `489f7771`, oracle vLLM 0.26.0.dev0):** shared scaled-RoPE + local-mask CUDA path compiles `-Werror`-clean + RUNS on GB10; feature-positive correctness gates PASS — SWA Gemma-2/Gemma-3 48/48, LongRoPE Phi-4-mini 16/16 (RED-first), llama3 Llama-3.2-1B 16/16, dynamic-NTK InternLM2 16/16; both RoPE 0.26-oracle recaptures BIT-IDENTICAL to goldens. `ATTN-SLIDING-WINDOW`/`ATTN-ROPE-{LLAMA3,LONGROPE,DYNAMIC-NTK}`/`ATTN-YARN` → `ACTIVE`; `ATTN-CHUNKED-LOCAL` + `KV-*-SPEC` honest. | **Honest residual (vehicle-blocked, not skipped):** YaRN model e2e (no cached Nomic/gpt-oss consumer) + chunked-local model e2e (no Llama4 row) REACHABLE-BLOCKED; long-context positive-mask (prompt > W) SWA e2e + KV-memory G8; every-axis **SPEED** tail (all leaves correctness-complete, speed-pending). | | `ROAD-V1-C6` async/priority serving | **RI** | `ENG-ASYNC-SCHED` **DONE** (`6ea7856`, default-ON, DGX token-neutral). W1/W2/W4 landed. | `SERVE-ASYNC-LLM` (GATING → prod-ON, blocks the SGLang floor + `ROAD-V1-A`), `ENG-PRIORITY-SCHED` + `ENG-CORE-BUSY-LOOP` GPU gates (GATING, held behind SERVE-GATE-ONLINE). | @@ -213,7 +213,56 @@ gate → size (S/M/L) → vehicle model. `[H]` = user-directed headline. leaving only entries that need the shared layer extended. The dgx paged-engine confirmation for those five is OWED. 16. **`ROAD-V1-C9` 0.26 denominators/goldens refresh** (recurring). **Size S, ongoing.** -17. **`ROAD-V1-C3` DSpark + TLI** (core spec-decode done; overlaps D3). **Size M.** +17. ~~**`ROAD-V1-C3` DSpark + TLI** (core spec-decode done; overlaps D3). **Size M.**~~ + **RECONCILED 2026-08-12** ([#536](https://github.com/mudler/vllm.cpp/issues/536)): + the item was written when both halves were untouched. Neither half is what + it says, and they are not one item. + **DSpark is not unspiked.** Its spike spec landed 2026-08-09 (`2b342620e`, + [dspark-spec-decode.md](dspark-spec-decode.md)), `SPEC-DSPARK` has been + `ACTIVE` since, and W1–W8 are implemented and GPU-gated: the Markov head, + the sequential sampler, native **and** Speculators-format loading, the `d2t` + reduced draft vocab, the runner/one-surface wiring, the device sequential + sample ([#436](https://github.com/mudler/vllm.cpp/issues/436)) and the T=1+k + verify capture ([#442](https://github.com/mudler/vllm.cpp/issues/442), + mirroring vLLM's `uniform_decode_query_len = 1 + num_speculative_tokens`). + En route it fixed an engine-wide defect: `EngineCoreProc` never threaded + `check_for_draft_tokens`, so EVERY speculator's drafts were dropped on the + CLI and server paths. Measured against the pinned graphed oracle under + pinned clocks the 35B-A3B MoE lane measured **0.975x** (code cell, + non-overlapping distributions) and **1.012x** (prose cell) ON THE + PRE-REIMAGE BOX. Both figures are SUPERSEDED: that machine no longer + exists, and on the rebuilt stack the matched-and-warm paired ratio is + **0.834** -- see the benchmark record entries of 2026-08-15, which also + record that every earlier ratio used a single COLD oracle invocation. With the + residual localised to one kernel and attributed to a **12.9% + effective-DRAM-bandwidth** gap on byte-equivalent machine code (94 registers + / 3664 SASS instructions on both sides, spec §§6t–6aa). **Remaining is a + perf tail plus owed gates, not a port: Size S–M**, and its next lever is + named (`cudaMemAdvise`/placement on the expert slab; upstream `ncu` counters + are BLOCKED in both replay modes, so a standalone `moe_wna16_marlin_gemm` + harness is the only remaining route). Owed for a binding W6: the SACRED-corpus + token gate under the ratified near-tie protocol, the 27B dense re-measure + (its earlier cells were never like-for-like), the Gemma4 `1 + N` layout on + real weights, and padded/multi-request spec capture shapes. + **TLI is untouched — and it is not a DSpark tail.** No commit, no code, no + spec, no issue; `SPEC-TLI` is `INVENTORIED`. Upstream TLI is + `use_heterogeneous_vocab` (`config/speculative.py:150`) + `VocabMapping` + (`v1/spec_decode/vocab_mapping.py:68`), consumed ONLY by + `v1/spec_decode/llm_base_proposer.py` (`SpecDecodeBaseProposer`) and + `v1/spec_decode/draft_model.py:19`. The V2-runner speculators our + DFlash/DSpark port mirrors (`v1/worker/gpu/spec_decode/{dflash,dspark}/`) + have no heterogeneous-vocab path at all, so TLI's host is `SPEC-DRAFT-MODEL` + — a CPU propose brick with no runner construction — and TLI is + prerequisite-blocked behind that row's W3, not merely unspiked. DSpark's + `d2t` does not cover it: `d2t` is an offset table inside ONE tokenizer's + vocabulary (`draft_id + d2t[draft_id]`), while TLI builds a string-level + intersection ACROSS tokenizer families (BPE `Ġ` vs SentencePiece `▁`, + probed at init). **Size M, and it should be re-filed under `SPEC-DRAFT-MODEL`.** + **The D3 overlap is backwards.** + [spec-decode-breadth-d3.md](spec-decode-breadth-d3.md) §Scope puts DSpark and + TLI explicitly *out of* `ROAD-V1-D3` and back under `ROAD-V1-C3`; D3's + landing covers no part of this tail. What DSpark reused is C3's own + MTP/DFlash verify/reject loop. ## 4. Bottom line diff --git a/.agents/specs/spec-decode-inventory.md b/.agents/specs/spec-decode-inventory.md index 49d3ff854..c916e4f0f 100644 --- a/.agents/specs/spec-decode-inventory.md +++ b/.agents/specs/spec-decode-inventory.md @@ -42,7 +42,7 @@ vLLM, it is not an addition vLLM lacks. | `mtp` (canonical) | `EagleProposer` via `use_eagle()`; special `Gemma4Proposer`/`Step3p5MTPProposer` (`gpu_model_runner.py:627-630`); MRV2 `MTPSpeculator` | head-on-base per family (nextn layer on target) | V1 default + MRV2 | **DONE** `SPEC-MTP` (Qwen3.5/3.6 k=1); family breadth `SPEC-MTP-FAMILY` | | `deepseek_mtp` ... `inkling_mtp` (20 family strings) | `MTPModelTypes` `speculative.py:37-59` -> deprecate-remap to `mtp` (`:686-690`); models `deepseek_mtp.py`, `glm4_moe_mtp.py`, `ernie_mtp.py`, `nemotron_h_mtp.py`, ... | head-on-base per family | V1 default (dispatched by `draft_model_config.hf_config.model_type`) | **ACTIVE** DeepSeek-V4 (`MODEL-SPEC-deepseek-v4-deep-seek-v4-mtp`, W1, weight-blocked); rest **INVENTORIED** (`SPEC-MTP-FAMILY`, model-matrix `MODEL-SPEC-*-mtp`) | | `dflash` | `vllm/v1/spec_decode/dflash.py` (DFlashProposer); models `laguna_dflash.py`, `qwen3_dflash.py`; speculators algo `algos.py:93` | separate block-diffusion draft + aux | V1 default + MRV2 | **DONE** `SPEC-DFLASH` (ported from vLLM; +GGUF `SPEC-DFLASH-GGUF`) | -| `dspark` | MRV2 `vllm/v1/worker/gpu/spec_decode/dspark/speculator.py:37`; models `gemma4_dspark.py`, `qwen3_dspark.py`; speculators algo `algos.py:133` | separate SAR block draft + aux | V1 default (via `use_eagle()`) + MRV2 | **INVENTORIED** `SPEC-DSPARK` | +| `dspark` | MRV2 `vllm/v1/worker/gpu/spec_decode/dspark/speculator.py:37`; models `gemma4_dspark.py`, `qwen3_dspark.py`; speculators algo `algos.py:133` | separate SAR block draft + aux | V1 default (via `use_eagle()`) + MRV2 | ~~**INVENTORIED**~~ **`ACTIVE`** `SPEC-DSPARK` (W1–W8 implemented + GPU-gated; 35B-A3B MoE 0.975x code / 1.012x prose vs the pinned graphed oracle, #442; superseded here 2026-08-12, #536) | ## User-visible config axes (apply on top of a method) @@ -51,25 +51,34 @@ vLLM, it is not an addition vLLM lacks. | Acceptance / rejection sampler | `speculative.py:77,216` (`rejection_sample_method`) | `standard` \| `synthetic` \| `block` | `standard` **ACTIVE** `SPEC-REJECTION`; `synthetic`/`block` **ABSENT** `SPEC-ACCEPT-VARIANTS` | | Draft sampling | `speculative.py:78,283` (`draft_sample_method`) | `greedy` \| `probabilistic` | `greedy` only; `probabilistic` **ABSENT** `SPEC-ACCEPT-VARIANTS` | | Dynamic k (per batch size) | `speculative.py:1336-1337`; `vllm/v1/spec_decode/dynamic/utils.py` (`num_speculative_tokens_per_batch_size`) | schedule list | **ABSENT** `SPEC-DYNAMIC` | -| Heterogeneous draft/target vocab | `vllm/v1/spec_decode/vocab_mapping.py` | shared-token / ID translation | **INVENTORIED** `SPEC-TLI` | +| Heterogeneous draft/target vocab (TLI) | `vllm/config/speculative.py:150` (`use_heterogeneous_vocab`); `vllm/v1/spec_decode/vocab_mapping.py:68` (`VocabMapping`); consumed ONLY by `llm_base_proposer.py:432-495,688-691,831-837` (`SpecDecodeBaseProposer`) and `draft_model.py:19,34-58` | cross-tokenizer-family string intersection, target↔draft ID translation, constrained draft logits | **INVENTORIED** `SPEC-TLI` — untouched, and it hangs off the V1 `SpecDecodeBaseProposer`, NOT the V2-runner DFlash/DSpark speculators we ported, so its host row is `SPEC-DRAFT-MODEL` and it is prerequisite-blocked behind that row's W3 (2026-08-12, #536). DSpark's `d2t` does NOT cover it: `d2t` offsets ids inside ONE vocabulary | | GDN spec metadata + slot-snapshot rollback | `vllm/v1/attention/backends/gdn_attn.py` | — | **ACTIVE** `SPEC-GDN-SEGMENTS` | ## HF `speculators` checkpoint-format integration `vllm/transformers_utils/configs/speculators/algos.py` registers `SUPPORTED_SPECULATORS_TYPES = {eagle3, peagle, dflash, dspark}` (`:15,55,93,133`) -and rewrites a HF `speculators`-format config into a vLLM draft arch. Our side +and rewrites a HF `speculators`-format config into a vLLM draft arch. ~~Our side loads native draft configs directly (no `speculators`-format adapter); tracked under the EAGLE/DFlash/DSpark rows as a loader-format residual, not a separate -method. +method.~~ **SUPERSEDED 2026-08-12 (#536): `SPEC-DSPARK` W3 landed the DSpark +adapter** — `speculators_model_type == "dspark"` detection, the +`speculators_config` proposal-method unwrap, and the `algos.py:133-165` field +translation (`aux_hidden_state_layer_ids` → `target_layer_ids = [i-1]`, +`sample_from_anchor`, `draft_vocab_size`, `mask_token_id`, `markov_rank`, +`block_size`), in `src/vllm/model_executor/models/qwen3_dspark.cpp:227-300`. +It is the path the `RedHatAI/*speculator.dspark` gate-model drafts load through. +The residual is now the EAGLE3/PEAGLE/DFlash algos, not the whole subsystem. ## Our lifecycle summary (engine matrix §Speculative decoding) - **DONE**: `SPEC-MTP`, `SPEC-MTP-GGUF`, `SPEC-DFLASH`, `SPEC-DFLASH-GGUF`. - **ACTIVE**: `SPEC-REJECTION`, `SPEC-GDN-SEGMENTS`, `SPEC-NGRAM`, - `SPEC-DRAFT-MODEL`. + `SPEC-DRAFT-MODEL`, **`SPEC-DSPARK`** (moved out of INVENTORIED here + 2026-08-12, #536: it has been `ACTIVE` in the engine matrix since the + 2026-08-09 spike, with W1–W8 implemented and GPU-gated). - **SPIKE**: `SPEC-MEDUSA`. **BLOCKED**: `SPEC-EAGLE3`. -- **INVENTORIED**: `SPEC-DSPARK`, `SPEC-TLI`, and the nine enumerated here: +- **INVENTORIED**: ~~`SPEC-DSPARK`,~~ `SPEC-TLI`, and the nine enumerated here: `SPEC-NGRAM-GPU`, `SPEC-SUFFIX`, `SPEC-EAGLE`, `SPEC-MTP-FAMILY`, `SPEC-ACCEPT-VARIANTS`, `SPEC-DYNAMIC`, `SPEC-CUSTOM-CLASS`, `SPEC-EXTRACT-HIDDEN`, `SPEC-MLP-SPECULATOR`. diff --git a/docs/STATUS.md b/docs/STATUS.md index e1aae20d9..d526699b3 100644 --- a/docs/STATUS.md +++ b/docs/STATUS.md @@ -194,7 +194,7 @@ concurrency-1 A/B our-on 29.32 tok/s vs vLLM-on 29.24, non-overlapping bands, vLLM 0.26.0.dev0 stack (which resolves vllm#40898), and it remains gated behind a spike while its user-facing serving surface is finalized. -**Method surface (enumerated from vLLM source 2026-08-06, `.agents/specs/spec-decode-inventory.md`).** Of the 13 vLLM `SpeculativeMethod` strings we ship MTP (k=1), DFlash and n-gram; draft_model is a CPU brick and Medusa a spike; EAGLE1/EAGLE3, ngram-gpu, suffix, dspark, custom_class, extract_hidden_states, dynamic-k and the synthetic/block acceptance variants are INVENTORIED; mlp_speculator is upstream-deprecated (no V1 proposer). Draft DEPTH (k>1, dynamic, adaptive) unbuilt (`ROAD-V1-D3-SPEC-K`, #81). +**Method surface (enumerated from vLLM source 2026-08-06, `.agents/specs/spec-decode-inventory.md`).** Of the 13 vLLM `SpeculativeMethod` strings we ship MTP (k=1), DFlash, DSpark and n-gram; draft_model is a CPU brick and Medusa a spike; EAGLE1/EAGLE3, ngram-gpu, suffix, custom_class, extract_hidden_states, dynamic-k and the synthetic/block acceptance variants are INVENTORIED; mlp_speculator is upstream-deprecated (no V1 proposer). Draft DEPTH (k>1, dynamic, adaptive) unbuilt (`ROAD-V1-D3-SPEC-K`, #81). **DeepSeek-V4 native MTP** (`DeepSeekV4MTPModel`, ACTIVE — W1 self-spec wiring, 2026-07-30) has its nextn draft head wired to the same lossless spec-decode path.