From d8e69bce6ea7822651607825466a06a274547b96 Mon Sep 17 00:00:00 2001 From: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com> Date: Tue, 11 Aug 2026 03:22:07 +0000 Subject: [PATCH 1/3] spec(docs): ground public surface repair for issue 342 FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: Codex:gpt-5 [Codex] --- .agents/roadmap_v1.md | 1 + .../docs-public-surface-alignment-342.md | 78 +++++++++++++++++++ 2 files changed, 79 insertions(+) create mode 100644 .agents/specs/docs-public-surface-alignment-342.md diff --git a/.agents/roadmap_v1.md b/.agents/roadmap_v1.md index 3e02ef89b..e94786ed5 100644 --- a/.agents/roadmap_v1.md +++ b/.agents/roadmap_v1.md @@ -36,6 +36,7 @@ issue is not yet placed. Keyed record: update in place, never append. | Issue | Row | Title | Kind | |---:|---|---|---| +| [#342](https://github.com/mudler/vllm.cpp/issues/342) | `DOCS-PUBLIC-SURFACE-ALIGN-342` | Public docs drift from shipped CLI, ABI v17, endpoints, registry counts, and benchmark scope | bug | | [#223](https://github.com/mudler/vllm.cpp/issues/223) | `SAMPLE-PROMPT-LOGPROBS` | `prompt_logprobs` is a silent no-op: the runner never computes logits at prompt positions | bug | | [#298](https://github.com/mudler/vllm.cpp/issues/298) | `PERF-27B-LMHEAD-DSR` | `main` is RED on the `device-leakage` DSR ratchet: `PrepareLmHeadResident` added a 33rd `vt_ifdef` to the shared layer | bug | | [#287](https://github.com/mudler/vllm.cpp/issues/287) | `KV-MOONCAKE-STORE` | `MooncakeStoreConnector`: the KV store half is linkable native C++ and gateable over TCP on one box | feature | diff --git a/.agents/specs/docs-public-surface-alignment-342.md b/.agents/specs/docs-public-surface-alignment-342.md new file mode 100644 index 000000000..1793cf9f4 --- /dev/null +++ b/.agents/specs/docs-public-surface-alignment-342.md @@ -0,0 +1,78 @@ +# DOCS-PUBLIC-SURFACE-ALIGN-342 + +Issue: [#342](https://github.com/mudler/vllm.cpp/issues/342) + +## Scope + +Repair the public README and usage guide where they have drifted from the +shipped command, C ABI, conditional HTTP routes, registry-backed architecture +inventory, and benchmark scope. This is a documentation-only projection repair. +It changes no code, checker semantics, lifecycle state, or accepted measurement. + +Authority is limited to this spec, the issue table in `.agents/roadmap_v1.md`, +one timeless positioning sentence in `.agents/mission.md`, `README.md`, and +`docs/USAGE.md`. `docs/STATUS.md` and `docs/BENCHMARKS.md` remain untouched. + +## Source anchors and design + +- `examples/CMakeLists.txt:86` sets target `server`'s output name to + `vllm-server`. Public commands must invoke `build/examples/vllm-server`. +- `include/vllm.h:39-145` is the ABI evolution record and defines + `VLLM_ABI_VERSION 17`. Public prose will name ABI v17 and describe additions + through v17 without preserving an aging export count. +- `src/vllm/entrypoints/openai/api_server.cpp:1024-1068` conditionally + registers `POST /v1/embeddings` when an embedder is attached and + `POST /v1/audio/transcriptions` when a transcriber is attached. The endpoint + table will state those conditions rather than implying every server exposes + the routes. +- `docs/FEATURES.md:82-163`, checked against `REGISTER_VLLM_MODEL` by + `scripts/check-supported-models.py`, is the keyed public model projection. The + README will distinguish 35 registered architectures from the 27 that carry a + passing correctness gate. It will not claim that every registered + architecture is token-gated. +- `docs/BENCHMARKS.md:18-20,40-42` separates checkpoint-specific speed results. + The passing six-concurrency table is the Qwen3.6-27B NVFP4 `unsloth` + checkpoint at revision `890bdef7`. NVIDIA ModelOpt 27B at `0893e160` and the + 35B-A3B grid remain speed-pending. README performance prose will make that + boundary explicit and will not change any measured value. + +The writing pass keeps the existing LocalAI project structure, uses plain +language, and introduces no em dashes. + +## Risks and stop conditions + +- Counts can drift again if they are copied without their registry/gate meaning. + Every count in README must say whether it is registered or correctness-gated. +- A broad "Qwen3.6-27B" speed claim can accidentally include a different + checkpoint layout. Every passing grid claim must name `unsloth` and + `890bdef7`; current ModelOpt and 35B gaps must remain visible. +- Endpoint wording must preserve conditional registration. Do not describe the + embedding or transcription route as present on a text-only server. +- Stop if a requested public-doc gate requires checker changes or an edit beyond + the authority above. + +## Tests and evidence + +Capture a focused stale-string scan before the implementation and require it to +find the old server path, ABI v10/export-count wording, conflicting architecture +counts, blanket correctness-gate language, and unscoped throughput language. +After the repair, require that scan to return no matches. Then run: + +```sh +python3 scripts/check-readme-structure.py +python3 scripts/check-public-doc-tables.py +python3 scripts/check-supported-models.py +python3 scripts/check-surface-coverage.py +python3 scripts/check-doc-checkpoint.py --base 5812b8b6 --head HEAD +cmake -S . -B build-docs-cpu -DVLLM_CPP_CUDA=OFF -DVLLM_CPP_METAL=OFF -DVLLM_CPP_VULKAN=OFF -DVLLM_CPP_ROCM=OFF +cmake --build build-docs-cpu --target server vllm-cli -j2 +build-docs-cpu/examples/vllm-server --help +``` + +Run `scripts/agent-preflight.sh --staged` before each commit. The known baseline +`check-test-registration` configure failure and `audit-live-rows` debt may be +reported only if unchanged; they do not expand this row. + +## Outcome + +Pending implementation and verification. From 599e607e5418286c9da8e7f572358882de8e466c Mon Sep 17 00:00:00 2001 From: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com> Date: Tue, 11 Aug 2026 03:24:33 +0000 Subject: [PATCH 2/3] docs: align public surface with shipped interfaces FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: Codex:gpt-5 [Codex] --- .agents/mission.md | 4 ++ .../docs-public-surface-alignment-342.md | 6 +- README.md | 69 +++++++++---------- docs/USAGE.md | 15 +++- 4 files changed, 54 insertions(+), 40 deletions(-) diff --git a/.agents/mission.md b/.agents/mission.md index 90962ec41..4dee4d79f 100644 --- a/.agents/mission.md +++ b/.agents/mission.md @@ -26,6 +26,10 @@ persistent-batch, paged-KV design. Packaging is llama.cpp-style: usable as a library (`libvllm` + stable C API), with example CLI / OpenAI-server binaries shipped from this repo. +Public positioning distinguishes registered architectures from those with a +passing correctness gate, and scopes every performance claim to the exact +checkpoint and workload that produced it. + `MANIFESTO.md` is the public statement of this mission: the "why" the landing page links from its header. It deliberately carries no benchmark numbers, so it stays quotable without ageing — `docs/BENCHMARKS.md` and `docs/STATUS.md` hold diff --git a/.agents/specs/docs-public-surface-alignment-342.md b/.agents/specs/docs-public-surface-alignment-342.md index 1793cf9f4..7e3d11150 100644 --- a/.agents/specs/docs-public-surface-alignment-342.md +++ b/.agents/specs/docs-public-surface-alignment-342.md @@ -75,4 +75,8 @@ reported only if unchanged; they do not expand this row. ## Outcome -Pending implementation and verification. +README and `docs/USAGE.md` now use the shipped `vllm-server` path, ABI v17, +conditional embedding and transcription routes, registry/gate-qualified model +counts, and checkpoint-scoped performance wording. No benchmark value or +lifecycle state changed. Verification evidence is carried by the implementation +commit and its handoff report. diff --git a/README.md b/README.md index d2c9f0050..e7a57a425 100644 --- a/README.md +++ b/README.md @@ -7,8 +7,8 @@

- Same tokens as vLLM. Same throughput. 140x less to install.
- Continuous batching, paged KV, 25+ architectures, CUDA / CPU / Metal / Vulkan. No Python anywhere. + Same tokens as vLLM. Checkpoint-scoped speed. 140x less to install.
+ Continuous batching, paged KV, 35 registered architectures, CUDA / CPU / Metal / Vulkan. No Python anywhere.

@@ -57,8 +57,8 @@ scheduling ideas (RadixAttention, LPM cache-aware admission, jump-forward decodi ABI, GGUF straight off the shelf, and compute directly on the quantized blocks). MLX's GEMM where it wins on Apple Silicon. Safetensors and GGUF, CUDA and CPU and Metal and Vulkan, from one source tree. -Every architecture is gated **token-for-token against vLLM** on the same workload. Speed claims use -the reference engine's production configuration. +The registry has **35 architectures**; **27 have a passing correctness gate**. Scaffolds and +oracle-blocked rows stay labelled, and speed claims name their checkpoint and workload. ![vllm.cpp vs vLLM on Qwen3.6-27B: identical output at every concurrency](benchmarks/media/concurrency_race.gif) @@ -71,23 +71,21 @@ Where that stands today: - **Small.** **66 MiB** of binary against a **9.1 GiB** vLLM install, both measured on the same GB10: about **140x less to deploy**, serving the same model in **24.88 GiB of peak host memory against vLLM's 28.18**. No interpreter in the process, and 0 bytes of bundled CUDA userspace. -- **Fast.** On Qwen3.6-27B we **match vLLM's throughput** against its graphed production config. - We are ahead at all six concurrencies we measured, but only **c1 (4.5%)** is outside our noise - band; the other five, 0.7% to 1.7%, are ties. Also **1.18x llama.cpp's prefill** on the same - GGUF file, and **ahead of MLX-LM on prefill** on Apple Silicon. Most other architectures are - correct but speed-pending, and each one says so. -- **Everything.** 25+ architectures, tool calling (36 parser families), structured output including +- **Fast.** The Qwen3.6-27B NVFP4 `unsloth` checkpoint at `890bdef7` matches graphed vLLM throughput. + Only the **c1 4.5% lead** clears noise; c2 to c32 are ties. NVIDIA ModelOpt 27B and current 35B-A3B + grids remain speed-pending. We also measure **1.18x llama.cpp prefill** on the same GGUF and lead + MLX-LM on Apple prefill. +- **Everything.** 35 registered architectures, tool calling (36 parser families), structured output including GBNF, three speculative decoders, image and video and audio input, external KV offload, Prometheus metrics, and the SGLang knobs, all in a library you can `dlopen`. -- **Grounded.** Every architecture is **gated token-for-token against a pinned vLLM oracle**, and - where vLLM's own greedy decode is non-deterministic at bf16 near-ties, the gate says so instead of - quietly loosening. +- **Grounded.** 27 registered architectures pass a pinned-vLLM correctness gate. Ratified near-tie, + scaffold, and oracle-blocked rows say so instead of becoming broader support claims. ## Performance -Qwen3.6-27B (NVFP4) on NVIDIA GB10, greedy, closed loop, against the vLLM oracle in its -**production graphed config** (not `--enforce-eager`). Output is token-for-token identical at every -point on this curve: +Qwen3.6-27B NVFP4, `unsloth` checkpoint revision `890bdef7`, on NVIDIA GB10, greedy and closed loop, +against the vLLM oracle in its **production graphed config** (not `--enforce-eager`). Output is +token-for-token identical at every point on this curve: | Concurrency | 1 | 2 | 4 | 8 | 16 | 32 | |---|---|---|---|---|---|---| @@ -99,6 +97,9 @@ We are ahead at all six, but only c1 at 4.5% is clearly outside our 0.5% run-to- treat c2 through c32 as ties. The tokens are identical either way, and the install is 66 MiB against 9.1 GiB. +This grid does not cover every Qwen3.6 checkpoint. NVIDIA ModelOpt 27B at `0893e160` and the current +35B-A3B grid are correctness-gated but still speed-pending. See [BENCHMARKS](docs/BENCHMARKS.md). + Cold start to first `/health`: **36.5 s vs vLLM's 221.5 s (6.1x)**, provisional ([detail](.agents/benchmark-record.md)). @@ -151,9 +152,8 @@ Full per-axis grids, memory tables, the nine residual axes, and exact reproducti numbers by [`benchmarks/demo/`](benchmarks/demo/), which reads its values from a committed spec, so every figure traces back to the run that produced it. -> **Pre-release, under heavy development.** Correctness is gated token-for-token against a pinned -> vLLM oracle across 25+ architectures. Speed is proven on one GPU (NVIDIA GB10 / DGX Spark, -> sm_121a) plus a CPU path that matches or beats llama.cpp on GGUF. Every capability is labelled +> **Pre-release, under heavy development.** 35 architectures are registered and 27 correctness-gated. +> Speed applies only to named checkpoints and workloads. Every capability is labelled > honestly in [docs/STATUS.md](docs/STATUS.md): *correctness-complete*, *speed-pending*, > *build-only*, or *hardware-blocked*. @@ -166,7 +166,7 @@ cmake -S . -B build && cmake --build build -j ```sh # Serve an OpenAI-compatible endpoint -build/examples/server --model /path/to/Qwen3.6-27B --port 8000 --max-num-seqs 32 +build/examples/vllm-server --model /path/to/Qwen3.6-27B --port 8000 --max-num-seqs 32 ``` ```sh @@ -187,7 +187,7 @@ configs, token-for-token the same output. Switching to it should be boring. Ever you get on top, most of it borrowed from whichever engine does it best: - **One 66 MiB binary instead of a 9.1 GiB install.** A flat, exception-free, llama.cpp-style C ABI - ([`include/vllm.h`](include/vllm.h), 19 symbols) you can `dlopen` from C, C++, Go, or Rust. No + ([`include/vllm.h`](include/vllm.h), ABI v17) you can `dlopen` from C, C++, Go, or Rust. No Python interpreter in the process, ever. - **GGUF as a first-class citizen.** Load the same quantized files llama.cpp uses, and on CPU **compute directly on the compressed blocks** (Q4_0/Q8_0/Q3_K/Q4_K/Q5_K/Q6_K) with no BF16 @@ -237,24 +237,19 @@ Per-capability lifecycle state, active gaps, and the next gate for each: ## Supported models -Every architecture below passes a token-for-token correctness gate against the pinned vLLM oracle on -GB10. Where vLLM's own greedy is deterministic the bar is strict token-exact; where vLLM is -self-inconsistent at bf16 near-ties, the bar is a near-tie-robust check. "Speed" is a separate bar -(match or beat vLLM on every axis). +The source registry has 35 architecture names, 27 with a passing correctness gate. The compact view +below keeps scaffolds and oracle-blocked rows visible. "Speed" is a separate, every-axis bar. -**Gate models:** Qwen3.6-27B and Qwen3.6-35B-A3B (hybrid GDN + MoE, NVFP4), both token-exact, the -27B at or above vLLM throughput on every axis. **Also running:** Llama-3.x, Mistral, Qwen3/Qwen2 -dense and MoE, DeepSeek-V2 and V4-Flash (MLA), GLM-4 and GLM-4.7-Flash, Laguna-S/XS-2.1, -Kimi-Linear-48B, Gemma-1 through Gemma-4, Phi-1 through Phi-4, OLMo-2, Granite-3, StableLM, -InternLM2/3, MiniCPM and MiniCPM3, Yi, OPT, plus Qwen3-VL and Qwen3.6-27B vision (image + video) -and Voxtral (audio). +**Gate models:** Qwen3.6-27B and Qwen3.6-35B-A3B (GDN + MoE, NVFP4), both correctness-gated. +`unsloth` 27B at `890bdef7` meets the speed floor; NVIDIA ModelOpt 27B and current 35B-A3B do not. +The registry-bound checkpoint list is in [docs/FEATURES.md](docs/FEATURES.md).

-The full architecture matrix (28 rows, with per-model correctness and speed state) +Compact architecture family view | Architecture | Example checkpoint | GGUF | Correctness | Speed | |---|---|:---:|---|---| -| Qwen3.5/3.6 hybrid (GDN + MoE) | Qwen3.6-27B, Qwen3.6-35B-A3B | 35B only | Token-exact | 27B at/above vLLM; 35B prefill-pending | +| Qwen3.5/3.6 hybrid (GDN + MoE) | Qwen3.6-27B, Qwen3.6-35B-A3B | 35B only | Token-exact | `unsloth` 27B @`890bdef7` at/above vLLM; ModelOpt 27B and current 35B grids speed-pending | | Qwen3 / Qwen2 dense | Qwen3-4B, Qwen3-32B | dense qwen35 | Token-exact (near-tie-robust) | Speed-pending | | Qwen3-MoE | Qwen3-Coder-30B-A3B | - | Token-exact (near-tie-robust) | Speed-pending | | Llama-3.x dense | Llama-3.2-1B | - | Token-exact (near-tie-robust) | Speed-pending | @@ -365,7 +360,7 @@ tokenizer). For SPEECH put the spoken line in the prompt. Recipe: [docs/USAGE.md ## OpenAI-compatible server ```sh -build/examples/server --model /path/to/Qwen3.6-27B --port 8000 --max-num-seqs 32 +build/examples/vllm-server --model /path/to/Qwen3.6-27B --port 8000 --max-num-seqs 32 ``` ```python @@ -386,7 +381,7 @@ behind a model gallery, multi-model serving, the full OpenAI API surface, auth, ## Use it as a library (C API) Link `libvllm` and include [`include/vllm.h`](include/vllm.h): a flat, exception-free, -llama.cpp-style C ABI (`VLLM_ABI_VERSION 10`, 19 exported symbols) suitable for `dlopen` / FFI. +llama.cpp-style C ABI (`VLLM_ABI_VERSION 17`) suitable for `dlopen` / FFI. ```c vllm_model_params mp = vllm_model_params_default(); @@ -458,8 +453,8 @@ from: - **[vLLM](https://github.com/vllm-project/vllm)** is the reference this project is measured against and the origin of the serving core: continuous batching, block-paged KV, the V1 scheduler, sampling - order, speculative decoding. It is also the oracle: every architecture here has to emit the same - tokens vLLM does before it counts as working. + order, speculative decoding. It is also the oracle: an architecture counts as correctness-gated + only after it passes its strict or ratified near-tie comparison. - **[SGLang](https://github.com/sgl-project/sglang)** contributed serving ideas nobody else ships, which we carry as documented toggles: RadixAttention, LPM cache-aware scheduling, and jump-forward decoding ([docs/SGLANG-COMPAT.md](docs/SGLANG-COMPAT.md)). diff --git a/docs/USAGE.md b/docs/USAGE.md index 89e172f76..9afce2cce 100644 --- a/docs/USAGE.md +++ b/docs/USAGE.md @@ -423,6 +423,8 @@ Registered in |---|---|---| | POST | `/v1/completions` | Text completion (JSON or `text/event-stream`) | | POST | `/v1/chat/completions` | Chat completion (JSON or streaming SSE) | +| POST | `/v1/embeddings` | Embeddings, registered only when the loaded engine has an embedding handler; otherwise absent (404) | +| POST | `/v1/audio/transcriptions` | Multipart audio transcription, registered only when the loaded engine has a transcription handler; otherwise absent (404) | | GET | `/v1/models` | List the served model | | GET | `/health` | Process liveness (200) | | GET, POST | `/ping` | Liveness probe (200, mirrors `/health`) | @@ -808,8 +810,10 @@ that invokes `ffmpeg`, path configurable with `--video-ffmpeg`). ## Consuming it as a library (C ABI) Link `libvllm` (static or shared) and include [`include/vllm.h`](../include/vllm.h). -It exposes a flat, exception-free, llama.cpp-style C ABI (`VLLM_ABI_VERSION 10`, -19 exported symbols) suitable for `dlopen` / FFI / LocalAI integration. +It exposes a flat, exception-free, llama.cpp-style C ABI (`VLLM_ABI_VERSION 17`) +suitable for `dlopen` / FFI / LocalAI integration. The header is the current +symbol and ownership contract; use `vllm_abi_version()` to check the loaded +library at runtime. ```c #include "vllm.h" @@ -848,6 +852,13 @@ concurrent requests, memory helpers, and diagnostics. Later ABI versions add: | v8 | Custom logits processors | | v9 | Engine sizing: chunked-prefill token budget, scheduling policy, external KV connector / LMCache | | v10 | Jump-forward decoding (tri-state, default off) | +| v11 | Audio transcription engines, parameters, results, and task-aware dispatch | +| v12 | Video and audio generation engine, results, and mux argument helpers | +| v13 | Completion from pre-tokenized prompts with generated token ids | +| v14 | Explicit text-engine device selection (`auto`, CPU, or CUDA) | +| v15 | Embedding entry points and task-aware pooling dispatch | +| v16 | Absolute KV-cache sizing and GPU-memory-utilization fields | +| v17 | `vllm_server_main`, the OpenAI server entry point on the public ABI | Chat templates render through the vendored google/minja engine, the same renderer llama.cpp ships. From e91fb3fa372bbf38c645252af7f57b83bd77b4da Mon Sep 17 00:00:00 2001 From: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com> Date: Tue, 11 Aug 2026 03:32:07 +0000 Subject: [PATCH 3/3] docs: scope README badges to recorded gates FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: Codex:gpt-5 [Codex] --- .agents/mission.md | 6 +++--- README.md | 4 ++-- 2 files changed, 5 insertions(+), 5 deletions(-) diff --git a/.agents/mission.md b/.agents/mission.md index 4dee4d79f..bb5495f9e 100644 --- a/.agents/mission.md +++ b/.agents/mission.md @@ -26,9 +26,9 @@ persistent-batch, paged-KV design. Packaging is llama.cpp-style: usable as a library (`libvllm` + stable C API), with example CLI / OpenAI-server binaries shipped from this repo. -Public positioning distinguishes registered architectures from those with a -passing correctness gate, and scopes every performance claim to the exact -checkpoint and workload that produced it. +Public positioning, including badges, distinguishes registered architectures +from those with a passing correctness gate, and scopes every performance claim +to the exact checkpoint and workload that produced it. `MANIFESTO.md` is the public statement of this mission: the "why" the landing page links from its header. It deliberately carries no benchmark numbers, so it diff --git a/README.md b/README.md index e7a57a425..23c09db95 100644 --- a/README.md +++ b/README.md @@ -13,8 +13,8 @@

License - vs vLLM - Architectures + Qwen3.6-27B unsloth 890bdef7 vs vLLM + Architectures: 35 registered, 27 correctness-gated Binary size LocalAI