mudler · localai-bot · Aug 12, 2026
diff --git a/.agents/backend-matrix.md b/.agents/backend-matrix.md
@@ -256,7 +256,7 @@ memory. Floating competitor versions do not count.
 |---|---|---|---|---|---|---|---|
 | `BACKEND-GATE-CUDA-VLLM` | CUDA correctness and every-axis parity vs vLLM v0.25.0 | current pin `e24d1b24`; oracle `702f481`; async default `vllm/config/vllm.py:952-1043`; depth-2 engine path `vllm/v1/engine/core.py:519-607`; async output `vllm/v1/worker/gpu/async_utils.py:12-70`; [benchmark protocol](verification.md) | Existing schema-v5 validator/summary remain. The production [packed component driver](../scripts/dgx-gdn-packed-component.sh#L1) and fail-closed [component finalizer](../tools/bench/gdn_packed_component.py#L1) enforce exact corpus/build/oracle provenance, exact and bounded raw-metric validation, duration consistency, exact commands, pairwise stability, one-lock correctness, thermal/memory health, symlink-safe sealing and marker-last provenance | [Component/summary tests](../tests/tools/test_gdn_packed_component.py#L1) pass and the [2026-07-15 new-binding ledger row](parity-ledger.md) records the sealed evidence. **NEW BINDING `9ecd9d0`: 114/124** (full production default set = async + Triton GDN cubin + bit-identical fast RMSNorm + bit-identical fast gated-RMSNorm; supersedes `246a23c`'s 49/124, `3f256ab`'s 55/124, all retained immutable in the ledger). Mem **4/4**, c1 **20/20**, c2 **20/20**, c16 19/20, c4 & c32 18/20, c8 15/20. The bit-identical (0-ulp) fast decode-kernel stack (`348d12d`+`9ecd9d0`) closed **+62 axes** — confirming the decode deficit was norm/quant/act kernel glue. Ratified as "effective parity" via two-grid totality (114+111→115/124; 110 pass-in-both + 5 coin-flip), NOT a clean 124/124. Remaining gate: 10 axes (35B every-axis closure c1/c2 low-conc residual). 35B performance blocked until 27B reaches a clean 124/124. (2026-07-31 audit `CLAIM-ROADMAP-V1-AUDIT`: this row previously lagged the roadmap by two bindings; now synced) | [CUDA inventory](specs/cuda-architecture-inventory.md); [online gate](specs/cuda-online-serving-gate.md); [packed decode](specs/gdn-packed-decode.md); [W3-I](specs/nvfp4-fused-silu-producer.md) | `PARTIAL` | - |
 | `BACKEND-BENCH-CUDA-SGLANG-PREFLIGHT` | SGLang corpus, harness, image and exact-checkpoint/token-ID preflights; P1 CPU harness is implemented/gated while P2 image/model/GPU classification remains | [SGLang v0.5.13](https://github.com/sgl-project/sglang/releases/tag/v0.5.13), commit `28b095c`; digest-pinned CUDA 13 image | [corpus](../tools/bench/make_serve_low_corpus.py#L143); [client/preflights](../tools/bench/run_serve_low.py#L124); [summary](../tools/bench/summarize_serve_low.py#L238); [memory](../tools/bench/sample_process_memory.py#L198); [driver](../scripts/dgx-sglang-low-concurrency.sh#L1) | [16 CPU contract cases](../tests/tools/test_serve_low_client.py#L103); [CTest registration](../tests/CMakeLists.txt#L12); dry-run manifest green; no image/model/GPU evidence or performance result | [SGLang low-concurrency spike](specs/cuda-sglang-low-concurrency.md) | `GATING` | - |
-| `BACKEND-GATE-CUDA-SGLANG` | Binding CUDA low-concurrency serving vs SGLang | same pinned SGLang/checkpoints as `BACKEND-BENCH-CUDA-SGLANG-PREFLIGHT` | [bench main:1](../examples/bench/main.cpp#L1), [server main:1](../examples/server/main.cpp#L1) | no binding run; HTTP TTFT/ITL cannot be measured honestly yet | [SGLang low-concurrency spike](specs/cuda-sglang-low-concurrency.md) | `BLOCKED` on `SERVE-ASYNC-LLM` and successful exact-equivalence preflight | - |
+| `BACKEND-GATE-CUDA-SGLANG` | Binding CUDA low-concurrency serving vs SGLang | same pinned SGLang/checkpoints as `BACKEND-BENCH-CUDA-SGLANG-PREFLIGHT` | [bench main:1](../examples/bench/main.cpp#L1), [server main:1](../examples/server/main.cpp#L1) | no binding run; ~~HTTP TTFT/ITL cannot be measured honestly yet~~ — **CORRECTED 2026-08-12 ([#534](https://github.com/mudler/vllm.cpp/issues/534)): that rationale describes the pre-W2 buffered-SSE server.** The exit criterion this row attached to `SERVE-ASYNC-LLM` (*"Real incremental SSE, in-flight batching, and streaming timing test pass"*, spike §Non-overlapping work breakdown) was met 2026-07-10; the arrival-time assertion is `tests/vllm/entrypoints/openai/test_api_server.cpp:622-634` (`dispatched < first_arrival < completed`), and `SERVE-GATE-ONLINE` has since measured HTTP TTFT/ITL on this exact path against vLLM across five recorded bindings (current `9ecd9d0` 114/124). The remaining dependency is this row's own exact-equivalence preflight | [SGLang low-concurrency spike](specs/cuda-sglang-low-concurrency.md) | `BLOCKED` on ~~`SERVE-ASYNC-LLM` and~~ successful exact-equivalence preflight (the `SERVE-ASYNC-LLM` half is met — see the evidence cell) | - |
 | `BACKEND-GATE-CUDA-SGLANG-PREFIX` | Binding deterministic shared-prefix cache-on serving vs the faster equivalent vLLM/SGLang floor, separate from cache-neutral serving | SGLang v0.5.15 `f63458b`; digest `d0a667e`; [DGX recipe correction/results](https://github.com/Weschera/qwen-sglang-dgx-spark/tree/03253ef98c01de59a21c85b9a5cc6a27a871c383); vLLM v0.25.0 explicit Qwen hybrid `mamba_cache_mode=align` | current prefix hashes/managers [kv_cache_utils.cpp:259](../src/vllm/v1/core/kv_cache_utils.cpp#L259), [kv_cache_manager.cpp:124](../src/vllm/v1/core/kv_cache_manager.cpp#L124); no local GDN/Mamba aligned-state retention or binding long-prefix harness yet | Source/config audit rejects the original cache-asymmetric 10--40x claim and treats the reported residual 25--45% SGLang lead as unproven: cache-on vLLM is 0.23.1, KV dtypes/memory differ, MTP is enabled, only 35B is measured, and full axes/hits/memory/traces/repetitions are absent. PX1/PX2 implementation, exact 27B equivalence/hit proof and all performance evidence remain pending | [SGLang shared-prefix extension](specs/cuda-sglang-low-concurrency.md#shared-prefix-extension-2026-07-12) | `READY` for PX1 harness/counter work; PX2 begins with the `KV-MAMBA-ALIGN` leaf spike, and binding execution also needs exact v0.5.15 equivalence plus `SERVE-ASYNC-LLM` | - |
 | `BACKEND-GATE-ROCM-VLLM` | ROCm parity vs vLLM | pinned vLLM ROCm backend | - | - | [CUDA inventory](specs/cuda-architecture-inventory.md) | `INVENTORIED` | - |
 | `BACKEND-GATE-ROCM-SGLANG` | ROCm low-concurrency serving vs SGLang | pinned SGLang ROCm build | - | - | [competitive benchmark spike](specs/competitive-benchmarks.md) | `INVENTORIED` | - |

diff --git a/.agents/coordination.md b/.agents/coordination.md
@@ -2016,11 +2016,11 @@ full-vocab identically.
 | Priority | Row/block | Dependency | Next handoff | State |
 |---|---|---|---|---|
 | 1 | `KERNEL-GEMM-BF16` (qkvz `W2`) for `SERVE-GATE-ONLINE` | `KERNEL-GDN-PACKED-DECODE` closed `DONE` (W1D3 EQUIVALENCE, `e47b4d6`); qkvz W2A implemented test-first 2026-07-15, CPU gates green; `3f256ab` remains 55/124 | run the qkvz DGX gates (model 235/235 default+rollbacks, GDN suite, 35B/GGUF inertness, memcheck, 145→97 BF16 trace; commands in state.md), then the AUTHORIZED fresh-SHA exact-grid rerun (fresh vLLM denominators; explicit `--mamba-ssm-cache-dtype float32`; cite `702f481`) before 35B | `GATING` |
-| 2 | `SERVE-ASYNC-LLM` block (`ENG-CORE-BUSY-LOOP`, `SERVE-ASYNC-LLM`, `ENG-ASYNC-SCHED`, `ENG-PRIORITY-SCHED`) | joint spike accepted ([async-serving.md](specs/async-serving.md)); W1/W2/W4 implemented/`GATING`; **W3 host-side + runner device-input + sampler-OUTPUT half all LANDED + CPU-gated 2026-07-16 (`ACTIVE`, `CLAIM-ASYNC-SCHED-W3`)** — `AsyncScheduler` + depth-2 `step_with_batch_queue` + config resolution + `VT_ASYNC_SCHED` rollback + `combine_sampled_and_draft_tokens` (input) + `vt::Backend` event/pinned + `AsyncGPUModelRunnerOutput` non-blocking D2H + `sample_tokens_async`/`runner_supports_async` + the depth-2 async-output seam + the ENABLE-FLIP (`LoadedEngine` `AsyncScheduler`+mcb=2 construction + the `cuda_combine_tokens.cu` device combine/scatter kernel) — all default OFF, landed 2026-07-16 | the W3 discriminator COMPLETED 2026-07-17 (`CLAIM-W3-ASYNC-DISC`): token gates 6/6, vLLM self-A/B proves the TTFT premium is vLLM's own async trade (+26/+31/+28 % at −0.7 to −0.9 % tput), both binding ITL-tail anomalies flip under W3-on, axis arithmetic nets positive ⇒ NEXT: `CLAIM-ASYNC-SCHED-W3` lands the default-ON flip (mirror `vllm/config/vllm.py:1040`) + fresh token gates, then the next grid | `GATING` |
+| 2 | `SERVE-ASYNC-LLM` block (`ENG-CORE-BUSY-LOOP`, `SERVE-ASYNC-LLM`, `ENG-ASYNC-SCHED`, `ENG-PRIORITY-SCHED`) | joint spike accepted ([async-serving.md](specs/async-serving.md)); W1/W2/W4 implemented/`GATING`; **W3 host-side + runner device-input + sampler-OUTPUT half all LANDED + CPU-gated 2026-07-16 (`ACTIVE`, `CLAIM-ASYNC-SCHED-W3`)** — `AsyncScheduler` + depth-2 `step_with_batch_queue` + config resolution + `VT_ASYNC_SCHED` rollback + `combine_sampled_and_draft_tokens` (input) + `vt::Backend` event/pinned + `AsyncGPUModelRunnerOutput` non-blocking D2H + `sample_tokens_async`/`runner_supports_async` + the depth-2 async-output seam + the ENABLE-FLIP (`LoadedEngine` `AsyncScheduler`+mcb=2 construction + the `cuda_combine_tokens.cu` device combine/scatter kernel) — all default OFF, landed 2026-07-16 | the W3 discriminator COMPLETED 2026-07-17 (`CLAIM-W3-ASYNC-DISC`): token gates 6/6, vLLM self-A/B proves the TTFT premium is vLLM's own async trade (+26/+31/+28 % at −0.7 to −0.9 % tput), both binding ITL-tail anomalies flip under W3-on, axis arithmetic nets positive ⇒ ~~NEXT: `CLAIM-ASYNC-SCHED-W3` lands the default-ON flip (mirror `vllm/config/vllm.py:1040`) + fresh token gates, then the next grid~~ **— DONE 2026-07-17 (`a0013a2`); superseded 2026-08-12 ([#534](https://github.com/mudler/vllm.cpp/issues/534)). The remaining NEXT for this block is narrower than the row reads: `ENG-ASYNC-SCHED` is `DONE`, `SERVE-ASYNC-LLM` is already the production serving path and holds on every-axis parity (27B ratified, 35B under `ROAD-V1-A`), `ENG-CORE-BUSY-LOOP`'s G1/G4 are covered by `test_qwen36_async_serving` (`1718bf155`) + binding `9ecd9d0`, and the ONE open leaf is `ENG-PRIORITY-SCHED` — whose priority-vs-FCFS engine gate has to be WRITTEN before it can be run.** | `GATING` |
 | 3 | `SERVE-E2E-NIGHTLY` | `SERVE-GATE-ONLINE` evidence where benchmarks overlap | write spike and CI/nightly split | `INVENTORIED` |
 | 4 | C1 kernel drop-in alignment | accepted kernel-family inventory + [drop-in ABI spike](specs/dropin-kernel-abi.md) | `BACKEND-ABI-VT` W0 is implemented and CPU-green; run cross-build + flock-held GB10 runtime/capture/model/trace handoff, close scalar-forwarder/backend-shim debts, then migrate one family per independently gated checkpoint | `GATING` |
 | 5 | `BACKEND-BENCH-CUDA-SGLANG-PREFLIGHT` | accepted spike; no dependency on async serving | P1 deterministic corpus/client/summary/memory/dry-run harness is implemented and `GATING`; provision the digest-pinned image only in a non-benchmark window, resolve the pinned client's missing raw E2E/TPOT sample gap without changing timed semantics, then P2 classifies each exact checkpoint | `GATING` |
-| 6 | `BACKEND-GATE-CUDA-SGLANG` | `BACKEND-BENCH-CUDA-SGLANG-PREFLIGHT`, then `SERVE-ASYNC-LLM` | run the binding c1-c16 campaign only after both dependencies close | `BLOCKED` |
+| 6 | `BACKEND-GATE-CUDA-SGLANG` | `BACKEND-BENCH-CUDA-SGLANG-PREFLIGHT`, ~~then `SERVE-ASYNC-LLM`~~ | ~~run the binding c1-c16 campaign only after both dependencies close~~ **— corrected 2026-08-12 ([#534](https://github.com/mudler/vllm.cpp/issues/534)): the `SERVE-ASYNC-LLM` half is MET.** The exit criterion the SGLang spike attached to it (*"Real incremental SSE, in-flight batching, and streaming timing test pass"*) was delivered by W2 on 2026-07-10; `test_api_server.cpp:622-634` asserts `dispatched < first_arrival < completed`, and `SERVE-GATE-ONLINE` has measured HTTP TTFT/ITL on that exact path across five recorded bindings. The remaining dependency is the preflight alone | `BLOCKED` |
 | 7 | `BACKEND-GATE-CUDA-SGLANG-PREFIX` | accepted shared-prefix extension; PX1 can proceed, while binding execution depends on exact SGLang v0.5.15 equivalence, a dedicated `KV-MAMBA-ALIGN` spike/implementation and `SERVE-ASYNC-LLM` | after the priority 27B cache-off parity grid/trace, implement the deterministic 64k/256k corpus and counters, write then execute the Mamba-align retention leaf, prove cache hits/no eviction, and bind every axis to the faster equivalent vLLM/SGLang result before 35B | `READY` |
 | 8 | C2/C4 models + quantizations | accepted comprehensive inventories | `MODEL-FACTORY-registry` and `QUANT-GGUF-CPU-THREADPOOL` are implemented and `GATING`: the registry awaits its exact two-model GPU greedy/performance/memory handoff after `CLAIM-SERVE-GATE-1`; the threadpool has 1/3/20 determinism + TSAN evidence but still needs the exclusive-idle-host B4 ≥10x/RSS series. Both claims are released. Keep-quant loader + CIQ GEMM follow that CPU checkpoint, then Llama dense | `GATING` |
 | 9 | C3/C5/C7/C8 engine/API work | C3/C5/C6 specs accepted; C7/C8 leaf specs missing | C3 **M-mtp-0 is CLOSED (2026-07-24, `CLAIM-MTP-I1-HEAD-ORACLE`)** — the standalone MTP draft head is oracle-parity-proven on both gate checkpoints; `SPEC-MTP` stays `GATING` on M-mtp-1..4 (scheduler plumbing, greedy rejection, GDN spec slots, k>1, CUDA graphs). All C5 W1-W8 implementation leaves are now `GATING`; restored feature-positive model rows, CUDA/runtime/oracle/trace and every-axis campaigns remain. An idle GPU still gates runtime closure; then spike C7/C8 | `GATING` |