Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
349 changes: 348 additions & 1 deletion .agents/specs/nemotron-h-model.md

Large diffs are not rendered by default.

5 changes: 5 additions & 0 deletions CMakeLists.txt
Original file line number Diff line number Diff line change
Expand Up @@ -787,6 +787,11 @@ add_library(vllm STATIC
src/vllm/model_executor/models/kimi_linear_forward.cpp
src/vllm/model_executor/models/kimi_linear.cpp
src/vllm/model_executor/models/kimi_linear_device.cpp
# MODEL-NEMOTRON-H W3 (#517): the additive NemotronH structural bring-up —
# config descent, the on-disk name map, the registry entry and the het-KV
# topology. No forward: nemotron_h.cpp is W4.
src/vllm/model_executor/models/nemotron_h_registry.cpp
src/vllm/model_executor/models/nemotron_h_weights.cpp
src/vllm/model_executor/models/glm4_registry.cpp
src/vllm/model_executor/models/glm4_weights.cpp
src/vllm/model_executor/models/glm4.cpp
Expand Down
10 changes: 5 additions & 5 deletions docs/FEATURES.md
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,7 @@ are our reading of their documented behavior, not measurements.
| Embeddable behind a C ABI | ✅ | ☐ | ☐ | ✅ |
| Weight formats | Safetensors + GGUF | Safetensors | Safetensors | GGUF |
| Correctness gate | token-exact vs vLLM | reference | own | own |
| Architectures | 37 registered, 27 gated | 130+ | 100+ | 100+ |
| Architectures | 38 registered, 27 gated | 130+ | 100+ | 100+ |
| Downloadable server binaries | ✅ v0.0.2: eight indexed archives with checksums, provenance, manifests, and SBOMs. Windows ZIP downloads do not exist; native CPU/Vulkan lanes await hosted runtime, dry-run, prerelease, and authenticated audit gates | ✅ wheels/containers | ✅ wheels/containers | ✅ host-specific binaries |
| Native Windows builds | ◐ CPU/Vulkan: `/MT /W4 /WX`, central `NOMINMAX`, UTF-8, aligned allocation, runtime ISA dispatch. Local closure includes the float-domain DeepSeek probe; hosted compile/runtime/release pending | ✅ | ✅ | ✅ |

Expand Down Expand Up @@ -87,7 +87,7 @@ are our reading of their documented behavior, not measurements.
The supported set is exactly what the C++ registry registers: every
architecture self-registers via `REGISTER_VLLM_MODEL`, and
`scripts/check-supported-models.py` gates this list against the source so it
cannot drift. Today that is **37 registered architectures**. Each row names the
cannot drift. Today that is **38 registered architectures**. Each row names the
checkpoint it was gated against and the verdict; caveats are in
[STATUS.md](STATUS.md), agent detail in `.agents/model-matrix.md`. A mergeable
gate/up MLP routes through one shared merged-GEMM method, so a tuned arm added
Expand Down Expand Up @@ -136,6 +136,7 @@ speed-pending, which [BENCHMARKS.md](BENCHMARKS.md) tracks.
| `LagunaForCausalLM` | poolside/Laguna-S-2.1-NVFP4, GGUF-Q4_K, Laguna-XS | byte-exact near-tie (distributional vs vLLM) | vLLM parity+ 1.03x, default on, via the `laguna-gen` CLI; the registered engine forward VT_CHECKs non-bf16 (`ARCH-ONE-SURFACE` fold) |
| `KimiLinearForCausalLM` | Kimi-Linear-48B-A3B (KDA + NoPE-MLA + MoE) | **Folded onto the shared paged runner (ROW 7 §21, #122): engine==CLI 128/128 byte-identical; vs golden 122/128 (the intrinsic near-tie profile); FA2 paged MLA default-ON; SACRED post-fold green** | Served via `vllm_engine_load` + `vllm_complete_tokens` (ABI v13); server 19.0 tok/s wall vs vLLM ~21 (~0.90×), speed residual open |
| `KimiK3ForConditionalGeneration` | Kimi-K3 (2.8T MoE) | scaffold: registry+config+enumeration gated, forward refuses | HW-infeasible (~1.56 TB); no run |
| `NemotronHForCausalLM` | Nemotron-3.5-Lightning-30B-A3B-NVFP4 (`nvidia` @`29f2d174`) | scaffold: registry+config+enumeration+KV-shape gated, forward refuses. 18487/18487 tensors claimed; a bf16 config claims that set minus its scale companions. Nothing runs yet (spec #517, blocked on #496) | no run; GGUF k-quants refuse by name and are owed |
| `MuseGlimmerForCausalLM` | real tensors, **bf16 depth 4/52 only**: 5 prefill argmax positions match a torch transcription of vllm#51655 and HF. GGUF full depth generates coherently (#347, #359) but is **NOT token-exact** | text forward + loader vs an fp32 reference, per-mechanism property tests, scaffold 11/11, GGUF gate 17/17. An ABSENT config key now takes the architecture's constant (#412): GGUF post-norms ran at 1e-5, not 1e-8 | no vLLM denominator (pin cannot load it); SECONDARY llama.cpp, same GGUF, GB10 CPU: prefill tie **0.997x**, decode 0.232x, RSS 1.92x (#333) |
| `MuseGlimmerForConditionalGeneration` | vision: **no reference run of any kind**; enumeration gated vs the released 30B index (1436/1436). Image/video need bf16 safetensors: `mmproj-kquant.gguf` is refused by name | perception encoder loaded and wired, so an image or video prompt runs; `perception_emb_norm` now armed by default (#405). Reachability plus placeholder scatter only, no image or video correctness | not measurable; anchored to open vllm#51655 |
| `LlamaModel` | landed tiny synthetic embedding fixture (engine path == direct pooler path, identical vectors; f64 LAST+normalize reference); real checkpoint (e5-mistral class) is a NAMED residual | pooling/embed only, text paths refuse by task; `vllm_embed` + `/v1/embeddings` | n/a (CPU correctness-grade embeddings) |
Expand Down Expand Up @@ -166,11 +167,10 @@ Enumerated in `.agents/model-matrix.md`, not registered, no runnable GB10 gate:
| `DeepseekV3ForCausalLM`, `DeepseekV32ForCausalLM` | DeepSeek-V3 / V3.2 | 671B, ~642 GiB fp8 vs 119 GiB unified; V3.2 also DSA-indexer dep-blocked |
| `GlmMoeDsaForCausalLM` | GLM-5 (DSA) | ~1404 GiB bf16; dep-blocked (GLM-5.x is DeepSeek-V3.2 verbatim) |
| `MiniMaxM2ForCausalLM` | MiniMax-M2 | ~230B, ~428 GiB bf16, ~4x over the unified pool |
| `NemotronHForCausalLM` | Nemotron-H / Nemotron-3.5-Lightning-30B-A3B | capability-blocked, not HW-blocked (20.1 GiB fits the pool): Mamba2 SSD unported ([#496](https://github.com/mudler/vllm.cpp/issues/496)), MoE and loader owed; spec [#517](https://github.com/mudler/vllm.cpp/issues/517) |

27 of the 31 registered text-generation architectures carry a passing
27 of the 32 registered text-generation architectures carry a passing
correctness gate today; the rest are honestly marked scaffold or blocked above.
(The 37 registered total also covers 3 Parakeet ASR entry points and the
(The 38 registered total also covers 3 Parakeet ASR entry points and the
`LlamaModel` embedding arch, which are not text generation.)
vLLM registers 130+ text architectures, so this is a curated, gated subset, not
a breadth claim. The first EMBEDDING architecture is registered and live
Expand Down
1 change: 1 addition & 0 deletions docs/USAGE.md
Original file line number Diff line number Diff line change
Expand Up @@ -342,6 +342,7 @@ tokens quietly.
| Architecture | Why it refuses |
|---|---|
| `KimiK3ForConditionalGeneration` | Needs ~1.56 TB (MXFP4); no host here can run it |
| `NemotronHForCausalLM` | The Mamba2 forward is not ported yet (#517 W4, blocked on #496). Safetensors resolve and parse; a GGUF file is refused by name, since no GGUF arm exists for it |

This is a deliberate state, not a bug: registering the architecture is what lets
the config parse and weight-name mapping be tested before the forward exists.
Expand Down
Loading
Loading