diff --git a/README.md b/README.md index 5bfd906df..c5d190940 100644 --- a/README.md +++ b/README.md @@ -41,11 +41,10 @@ Big updates have landed in Speculators! To get a more in-depth look, check out t Some of the exciting new features include: +- **DSpark Training Algorithm**: Added support for the DSpark training algorithm, which extends DFlash's anchored-block drafting with a Markov head that conditions each draft position on the previous token within the block, plus a confidence head that predicts per-position acceptance probability. DSpark checkpoints can warm-start from existing DFlash checkpoints. - **P-EAGLE Training Support**: Added support for the [P-EAGLE training algorithm](https://docs.vllm.ai/projects/speculators/en/latest/user_guide/algorithms/peagle), which extends EAGLE-3's architecture with parallel multi-token prediction via Conditional-On-Distribution (COD) sampling. Rather than generating draft tokens sequentially, P-EAGLE predicts multiple tokens in a single forward pass, reducing drafting latency. The Red Hat team published a [P-EAGLE speculator for Qwen3-8B](https://huggingface.co/RedHatAI/Qwen3-8B-speculator.peagle). - **MTP Finetuning Support**: Added support for finetuning the native Multi-Token Prediction (MTP) heads of models like Qwen3-Next on domain-specific data, following the [FastMTP](https://arxiv.org/abs/2509.18362) approach. Because the MTP head is small (~100M–400M params), it can be trained on pre-extracted hidden states without loading the full verifier - **Sliding Window Attention for DFlash and DSpark**: DFlash and DSpark speculators use sliding window attention on all draft layers by default. Use `--sliding-window` to set the window size and `--full-attention-indices` to opt specific layers into full attention. Sliding window attention reduces KV cache allocation for long-context sequences and can improve per-position acceptance rates compared to full attention. -- **Qwen3-8B DFlash Speculator**: The RedHat team published a [DFlash speculator for Qwen3-8B](https://huggingface.co/RedHatAI/Qwen3-8B-speculator.dflash), achieving average speculative token acceptance lengths of up to 3.74 on `math_reasoning`. -- **Gemma 4 Speculators**: The RedHat team published speculators for Gemma 4 31B-it, including both [DFlash](https://huggingface.co/RedHatAI/gemma-4-31B-it-speculator.dflash) and [EAGLE-3](https://huggingface.co/RedHatAI/gemma-4-31B-it-speculator.eagle3) checkpoints, enabling production-grade speculative decoding for Gemma 4 models. - **DFlash Training Algorithm**: Added support for the DFlash training algorithm with anchored-block drafting, using auxiliary hidden states from multiple verifier layers. Includes CLI options for block size and max anchors, plus DFlash metrics, utilities, and draft model. DFlash models trained through Speculators can now run seamlessly in vLLM as of [vLLM PR #38300](https://github.com/vllm-project/vllm/pull/38300). - **Online Training Support**: Added support for online training using the new [vLLM hidden extraction system](https://github.com/vllm-project/vllm/pull/33736), enabling real-time hidden state generation during training without requiring separate offline data generation steps. @@ -118,11 +117,16 @@ The following table summarizes the models that have been trained end-to-end by o ✅ - Qwen3 MoE + Qwen3 MoE 30B-Instruct EAGLE-3 - ✅ + ✅
DFlash ✅ + ✅ + + + 30B + DFlash ✅ ✅ @@ -149,10 +153,10 @@ The following table summarizes the models that have been trained end-to-end by o ✅ -Mistral 3 Large -675B-Instruct -EAGLE-3 ⏳ -⏳ +Mistral Small 4 +119B +DFlash ✅ +✅ Gemma 4 @@ -166,6 +170,18 @@ The following table summarizes the models that have been trained end-to-end by o EAGLE-3 ✅ ✅ + +NVIDIA Nemotron 3 Ultra +550B-A55B +DFlash ✅ +✅ + + +NVIDIA Nemotron 3 Super +120B-A12B +DFlash ✅ +✅ + diff --git a/docs/index.md b/docs/index.md index 112554304..cff0781c9 100644 --- a/docs/index.md +++ b/docs/index.md @@ -81,7 +81,7 @@ The following table summarizes the models that have been trained end-to-end by o Qwen3 8B -Eagle-3 ✅ +Eagle-3 ✅
DFlash ✅
P-EAGLE ✅ ✅ @@ -108,11 +108,16 @@ The following table summarizes the models that have been trained end-to-end by o ✅ - Qwen3 MoE + Qwen3 MoE 30B-Instruct Eagle-3 - ✅ + ✅
DFlash ✅ + ✅ + + + 30B + DFlash ✅ ✅ @@ -139,15 +144,9 @@ The following table summarizes the models that have been trained end-to-end by o ✅ -Mistral 3 Large -675B-Instruct -Eagle-3 ⏳ -⏳ - - -Qwen3.5 -9B -MTP ✅ +Mistral Small 4 +119B +DFlash ✅ ✅ @@ -162,6 +161,18 @@ The following table summarizes the models that have been trained end-to-end by o Eagle-3 ✅ ✅ + +NVIDIA Nemotron 3 Ultra +550B-A55B +DFlash ✅ +✅ + + +NVIDIA Nemotron 3 Super +120B-A12B +DFlash ✅ +✅ +