diff --git a/.agents/backend-matrix.md b/.agents/backend-matrix.md index 7a1fa0ba3..f00c1e243 100644 --- a/.agents/backend-matrix.md +++ b/.agents/backend-matrix.md @@ -256,7 +256,7 @@ memory. Floating competitor versions do not count. |---|---|---|---|---|---|---|---| | `BACKEND-GATE-CUDA-VLLM` | CUDA correctness and every-axis parity vs vLLM v0.25.0 | current pin `e24d1b24`; oracle `702f481`; async default `vllm/config/vllm.py:952-1043`; depth-2 engine path `vllm/v1/engine/core.py:519-607`; async output `vllm/v1/worker/gpu/async_utils.py:12-70`; [benchmark protocol](verification.md) | Existing schema-v5 validator/summary remain. The production [packed component driver](../scripts/dgx-gdn-packed-component.sh#L1) and fail-closed [component finalizer](../tools/bench/gdn_packed_component.py#L1) enforce exact corpus/build/oracle provenance, exact and bounded raw-metric validation, duration consistency, exact commands, pairwise stability, one-lock correctness, thermal/memory health, symlink-safe sealing and marker-last provenance | [Component/summary tests](../tests/tools/test_gdn_packed_component.py#L1) pass and the [2026-07-15 new-binding ledger row](parity-ledger.md) records the sealed evidence. **NEW BINDING `9ecd9d0`: 114/124** (full production default set = async + Triton GDN cubin + bit-identical fast RMSNorm + bit-identical fast gated-RMSNorm; supersedes `246a23c`'s 49/124, `3f256ab`'s 55/124, all retained immutable in the ledger). Mem **4/4**, c1 **20/20**, c2 **20/20**, c16 19/20, c4 & c32 18/20, c8 15/20. The bit-identical (0-ulp) fast decode-kernel stack (`348d12d`+`9ecd9d0`) closed **+62 axes** — confirming the decode deficit was norm/quant/act kernel glue. Ratified as "effective parity" via two-grid totality (114+111→115/124; 110 pass-in-both + 5 coin-flip), NOT a clean 124/124. Remaining gate: 10 axes (35B every-axis closure c1/c2 low-conc residual). 35B performance blocked until 27B reaches a clean 124/124. (2026-07-31 audit `CLAIM-ROADMAP-V1-AUDIT`: this row previously lagged the roadmap by two bindings; now synced) | [CUDA inventory](specs/cuda-architecture-inventory.md); [online gate](specs/cuda-online-serving-gate.md); [packed decode](specs/gdn-packed-decode.md); [W3-I](specs/nvfp4-fused-silu-producer.md) | `PARTIAL` | - | | `BACKEND-BENCH-CUDA-SGLANG-PREFLIGHT` | SGLang corpus, harness, image and exact-checkpoint/token-ID preflights; P1 CPU harness is implemented/gated while P2 image/model/GPU classification remains | [SGLang v0.5.13](https://github.com/sgl-project/sglang/releases/tag/v0.5.13), commit `28b095c`; digest-pinned CUDA 13 image | [corpus](../tools/bench/make_serve_low_corpus.py#L143); [client/preflights](../tools/bench/run_serve_low.py#L124); [summary](../tools/bench/summarize_serve_low.py#L238); [memory](../tools/bench/sample_process_memory.py#L198); [driver](../scripts/dgx-sglang-low-concurrency.sh#L1) | [16 CPU contract cases](../tests/tools/test_serve_low_client.py#L103); [CTest registration](../tests/CMakeLists.txt#L12); dry-run manifest green; no image/model/GPU evidence or performance result | [SGLang low-concurrency spike](specs/cuda-sglang-low-concurrency.md) | `GATING` | - | -| `BACKEND-GATE-CUDA-SGLANG` | Binding CUDA low-concurrency serving vs SGLang | same pinned SGLang/checkpoints as `BACKEND-BENCH-CUDA-SGLANG-PREFLIGHT` | [bench main:1](../examples/bench/main.cpp#L1), [server main:1](../examples/server/main.cpp#L1) | no binding run; HTTP TTFT/ITL cannot be measured honestly yet | [SGLang low-concurrency spike](specs/cuda-sglang-low-concurrency.md) | `BLOCKED` on `SERVE-ASYNC-LLM` and successful exact-equivalence preflight | - | +| `BACKEND-GATE-CUDA-SGLANG` | Binding CUDA low-concurrency serving vs SGLang | same pinned SGLang/checkpoints as `BACKEND-BENCH-CUDA-SGLANG-PREFLIGHT` | [bench main:1](../examples/bench/main.cpp#L1), [server main:1](../examples/server/main.cpp#L1) | no binding run; ~~HTTP TTFT/ITL cannot be measured honestly yet~~ — **CORRECTED 2026-08-12 ([#534](https://github.com/mudler/vllm.cpp/issues/534)): that rationale describes the pre-W2 buffered-SSE server.** The exit criterion this row attached to `SERVE-ASYNC-LLM` (*"Real incremental SSE, in-flight batching, and streaming timing test pass"*, spike §Non-overlapping work breakdown) was met 2026-07-10; the arrival-time assertion is `tests/vllm/entrypoints/openai/test_api_server.cpp:622-634` (`dispatched < first_arrival < completed`), and `SERVE-GATE-ONLINE` has since measured HTTP TTFT/ITL on this exact path against vLLM across five recorded bindings (current `9ecd9d0` 114/124). The remaining dependency is this row's own exact-equivalence preflight | [SGLang low-concurrency spike](specs/cuda-sglang-low-concurrency.md) | `BLOCKED` on ~~`SERVE-ASYNC-LLM` and~~ successful exact-equivalence preflight (the `SERVE-ASYNC-LLM` half is met — see the evidence cell) | - | | `BACKEND-GATE-CUDA-SGLANG-PREFIX` | Binding deterministic shared-prefix cache-on serving vs the faster equivalent vLLM/SGLang floor, separate from cache-neutral serving | SGLang v0.5.15 `f63458b`; digest `d0a667e`; [DGX recipe correction/results](https://github.com/Weschera/qwen-sglang-dgx-spark/tree/03253ef98c01de59a21c85b9a5cc6a27a871c383); vLLM v0.25.0 explicit Qwen hybrid `mamba_cache_mode=align` | current prefix hashes/managers [kv_cache_utils.cpp:259](../src/vllm/v1/core/kv_cache_utils.cpp#L259), [kv_cache_manager.cpp:124](../src/vllm/v1/core/kv_cache_manager.cpp#L124); no local GDN/Mamba aligned-state retention or binding long-prefix harness yet | Source/config audit rejects the original cache-asymmetric 10--40x claim and treats the reported residual 25--45% SGLang lead as unproven: cache-on vLLM is 0.23.1, KV dtypes/memory differ, MTP is enabled, only 35B is measured, and full axes/hits/memory/traces/repetitions are absent. PX1/PX2 implementation, exact 27B equivalence/hit proof and all performance evidence remain pending | [SGLang shared-prefix extension](specs/cuda-sglang-low-concurrency.md#shared-prefix-extension-2026-07-12) | `READY` for PX1 harness/counter work; PX2 begins with the `KV-MAMBA-ALIGN` leaf spike, and binding execution also needs exact v0.5.15 equivalence plus `SERVE-ASYNC-LLM` | - | | `BACKEND-GATE-ROCM-VLLM` | ROCm parity vs vLLM | pinned vLLM ROCm backend | - | - | [CUDA inventory](specs/cuda-architecture-inventory.md) | `INVENTORIED` | - | | `BACKEND-GATE-ROCM-SGLANG` | ROCm low-concurrency serving vs SGLang | pinned SGLang ROCm build | - | - | [competitive benchmark spike](specs/competitive-benchmarks.md) | `INVENTORIED` | - | diff --git a/.agents/coordination.md b/.agents/coordination.md index 3dc1288fa..9e61505db 100644 --- a/.agents/coordination.md +++ b/.agents/coordination.md @@ -2016,11 +2016,11 @@ full-vocab identically. | Priority | Row/block | Dependency | Next handoff | State | |---|---|---|---|---| | 1 | `KERNEL-GEMM-BF16` (qkvz `W2`) for `SERVE-GATE-ONLINE` | `KERNEL-GDN-PACKED-DECODE` closed `DONE` (W1D3 EQUIVALENCE, `e47b4d6`); qkvz W2A implemented test-first 2026-07-15, CPU gates green; `3f256ab` remains 55/124 | run the qkvz DGX gates (model 235/235 default+rollbacks, GDN suite, 35B/GGUF inertness, memcheck, 145→97 BF16 trace; commands in state.md), then the AUTHORIZED fresh-SHA exact-grid rerun (fresh vLLM denominators; explicit `--mamba-ssm-cache-dtype float32`; cite `702f481`) before 35B | `GATING` | -| 2 | `SERVE-ASYNC-LLM` block (`ENG-CORE-BUSY-LOOP`, `SERVE-ASYNC-LLM`, `ENG-ASYNC-SCHED`, `ENG-PRIORITY-SCHED`) | joint spike accepted ([async-serving.md](specs/async-serving.md)); W1/W2/W4 implemented/`GATING`; **W3 host-side + runner device-input + sampler-OUTPUT half all LANDED + CPU-gated 2026-07-16 (`ACTIVE`, `CLAIM-ASYNC-SCHED-W3`)** — `AsyncScheduler` + depth-2 `step_with_batch_queue` + config resolution + `VT_ASYNC_SCHED` rollback + `combine_sampled_and_draft_tokens` (input) + `vt::Backend` event/pinned + `AsyncGPUModelRunnerOutput` non-blocking D2H + `sample_tokens_async`/`runner_supports_async` + the depth-2 async-output seam + the ENABLE-FLIP (`LoadedEngine` `AsyncScheduler`+mcb=2 construction + the `cuda_combine_tokens.cu` device combine/scatter kernel) — all default OFF, landed 2026-07-16 | the W3 discriminator COMPLETED 2026-07-17 (`CLAIM-W3-ASYNC-DISC`): token gates 6/6, vLLM self-A/B proves the TTFT premium is vLLM's own async trade (+26/+31/+28 % at −0.7 to −0.9 % tput), both binding ITL-tail anomalies flip under W3-on, axis arithmetic nets positive ⇒ NEXT: `CLAIM-ASYNC-SCHED-W3` lands the default-ON flip (mirror `vllm/config/vllm.py:1040`) + fresh token gates, then the next grid | `GATING` | +| 2 | `SERVE-ASYNC-LLM` block (`ENG-CORE-BUSY-LOOP`, `SERVE-ASYNC-LLM`, `ENG-ASYNC-SCHED`, `ENG-PRIORITY-SCHED`) | joint spike accepted ([async-serving.md](specs/async-serving.md)); W1/W2/W4 implemented/`GATING`; **W3 host-side + runner device-input + sampler-OUTPUT half all LANDED + CPU-gated 2026-07-16 (`ACTIVE`, `CLAIM-ASYNC-SCHED-W3`)** — `AsyncScheduler` + depth-2 `step_with_batch_queue` + config resolution + `VT_ASYNC_SCHED` rollback + `combine_sampled_and_draft_tokens` (input) + `vt::Backend` event/pinned + `AsyncGPUModelRunnerOutput` non-blocking D2H + `sample_tokens_async`/`runner_supports_async` + the depth-2 async-output seam + the ENABLE-FLIP (`LoadedEngine` `AsyncScheduler`+mcb=2 construction + the `cuda_combine_tokens.cu` device combine/scatter kernel) — all default OFF, landed 2026-07-16 | the W3 discriminator COMPLETED 2026-07-17 (`CLAIM-W3-ASYNC-DISC`): token gates 6/6, vLLM self-A/B proves the TTFT premium is vLLM's own async trade (+26/+31/+28 % at −0.7 to −0.9 % tput), both binding ITL-tail anomalies flip under W3-on, axis arithmetic nets positive ⇒ ~~NEXT: `CLAIM-ASYNC-SCHED-W3` lands the default-ON flip (mirror `vllm/config/vllm.py:1040`) + fresh token gates, then the next grid~~ **— DONE 2026-07-17 (`a0013a2`); superseded 2026-08-12 ([#534](https://github.com/mudler/vllm.cpp/issues/534)). The remaining NEXT for this block is narrower than the row reads: `ENG-ASYNC-SCHED` is `DONE`, `SERVE-ASYNC-LLM` is already the production serving path and holds on every-axis parity (27B ratified, 35B under `ROAD-V1-A`), `ENG-CORE-BUSY-LOOP`'s G1/G4 are covered by `test_qwen36_async_serving` (`1718bf155`) + binding `9ecd9d0`, and the ONE open leaf is `ENG-PRIORITY-SCHED` — whose priority-vs-FCFS engine gate has to be WRITTEN before it can be run.** | `GATING` | | 3 | `SERVE-E2E-NIGHTLY` | `SERVE-GATE-ONLINE` evidence where benchmarks overlap | write spike and CI/nightly split | `INVENTORIED` | | 4 | C1 kernel drop-in alignment | accepted kernel-family inventory + [drop-in ABI spike](specs/dropin-kernel-abi.md) | `BACKEND-ABI-VT` W0 is implemented and CPU-green; run cross-build + flock-held GB10 runtime/capture/model/trace handoff, close scalar-forwarder/backend-shim debts, then migrate one family per independently gated checkpoint | `GATING` | | 5 | `BACKEND-BENCH-CUDA-SGLANG-PREFLIGHT` | accepted spike; no dependency on async serving | P1 deterministic corpus/client/summary/memory/dry-run harness is implemented and `GATING`; provision the digest-pinned image only in a non-benchmark window, resolve the pinned client's missing raw E2E/TPOT sample gap without changing timed semantics, then P2 classifies each exact checkpoint | `GATING` | -| 6 | `BACKEND-GATE-CUDA-SGLANG` | `BACKEND-BENCH-CUDA-SGLANG-PREFLIGHT`, then `SERVE-ASYNC-LLM` | run the binding c1-c16 campaign only after both dependencies close | `BLOCKED` | +| 6 | `BACKEND-GATE-CUDA-SGLANG` | `BACKEND-BENCH-CUDA-SGLANG-PREFLIGHT`, ~~then `SERVE-ASYNC-LLM`~~ | ~~run the binding c1-c16 campaign only after both dependencies close~~ **— corrected 2026-08-12 ([#534](https://github.com/mudler/vllm.cpp/issues/534)): the `SERVE-ASYNC-LLM` half is MET.** The exit criterion the SGLang spike attached to it (*"Real incremental SSE, in-flight batching, and streaming timing test pass"*) was delivered by W2 on 2026-07-10; `test_api_server.cpp:622-634` asserts `dispatched < first_arrival < completed`, and `SERVE-GATE-ONLINE` has measured HTTP TTFT/ITL on that exact path across five recorded bindings. The remaining dependency is the preflight alone | `BLOCKED` | | 7 | `BACKEND-GATE-CUDA-SGLANG-PREFIX` | accepted shared-prefix extension; PX1 can proceed, while binding execution depends on exact SGLang v0.5.15 equivalence, a dedicated `KV-MAMBA-ALIGN` spike/implementation and `SERVE-ASYNC-LLM` | after the priority 27B cache-off parity grid/trace, implement the deterministic 64k/256k corpus and counters, write then execute the Mamba-align retention leaf, prove cache hits/no eviction, and bind every axis to the faster equivalent vLLM/SGLang result before 35B | `READY` | | 8 | C2/C4 models + quantizations | accepted comprehensive inventories | `MODEL-FACTORY-registry` and `QUANT-GGUF-CPU-THREADPOOL` are implemented and `GATING`: the registry awaits its exact two-model GPU greedy/performance/memory handoff after `CLAIM-SERVE-GATE-1`; the threadpool has 1/3/20 determinism + TSAN evidence but still needs the exclusive-idle-host B4 ≥10x/RSS series. Both claims are released. Keep-quant loader + CIQ GEMM follow that CPU checkpoint, then Llama dense | `GATING` | | 9 | C3/C5/C7/C8 engine/API work | C3/C5/C6 specs accepted; C7/C8 leaf specs missing | C3 **M-mtp-0 is CLOSED (2026-07-24, `CLAIM-MTP-I1-HEAD-ORACLE`)** — the standalone MTP draft head is oracle-parity-proven on both gate checkpoints; `SPEC-MTP` stays `GATING` on M-mtp-1..4 (scheduler plumbing, greedy rejection, GDN spec slots, k>1, CUDA graphs). All C5 W1-W8 implementation leaves are now `GATING`; restored feature-positive model rows, CUDA/runtime/oracle/trace and every-axis campaigns remain. An idle GPU still gates runtime closure; then spike C7/C8 | `GATING` | diff --git a/.agents/engine-matrix.md b/.agents/engine-matrix.md index 531cda073..0c1a70154 100644 --- a/.agents/engine-matrix.md +++ b/.agents/engine-matrix.md @@ -62,10 +62,10 @@ forensics: roadmap_v1.md and the parity ledger. | `ENG-CUDAGRAPH` | Decode graph capture/replay modes (host-cluster cleanup: capture-size set derived from `max_num_seqs` mirroring vLLM `_set_cudagraph_sizes`; 2026-07-18 graph-baked-scratch use-after-free fix — the 35B c2+ online-serving IMA blocker) | T0 | `vllm/config/compilation.py:53,1319,683-684,1438-1444`; `vllm/config/vllm.py:1667-1770`; `vllm/v1/worker/gpu/cudagraph_utils.py:116`; `tests/compile/test_config.py:122,229` | `src/vt/cuda/cuda_backend.cu:76,97,105`; `include/vllm/model_executor/models/decode_graph_sizes.h`; `src/vllm/model_executor/models/qwen3_5.cpp:3754,3952`; `src/vllm/v1/worker/gpu/runner.cpp:577,597`; graph-safe scratch (retire-on-grow so graph-baked scratch pointers stay valid) `src/vt/cuda/graph_safe_scratch.h`, `src/vt/cuda/cuda_moe_marlin.cu:75`, `src/vt/cuda/cuda_matmul_nvfp4.cu:766`, `src/vt/cuda/cuda_matmul_nvfp4_cutlass.cu:105`, `src/vt/cuda/cuda_matmul_fp8_cutlass.cu:95` | `tests/vt/test_cuda_backend.cpp:98`; `tests/vllm/models/test_decode_graph_sizes.cpp`; `tests/vt/test_graph_safe_scratch.cpp`; explicit 35B gate `tests/parity/test_qwen36_paged_engine.cpp:140` | [blocktable-host-cluster-cleanup.md](specs/blocktable-host-cluster-cleanup.md); [decode-graph-scratch-uaf-2026-07-18.md](specs/decode-graph-scratch-uaf-2026-07-18.md) | `PARTIAL` | - | | `ENG-BATCH-INVARIANT` | Opt-in deterministic execution across scheduler batch sizes (`VLLM_BATCH_INVARIANT=1`): batch-invariant matmul/norm/attention/collectives plus persistent-scheduler NVFP4; production default remains off | T1 | default/env `vllm/envs.py:89,576-578`; initialization `vllm/v1/worker/gpu_worker.py:1262`; NVFP4 dispatch `csrc/libtorch_stable/quantization/fp4/nvfp4_scaled_mm_sm120_kernels.cu:212-220`; suite fixture `tests/v1/determinism/conftest.py:9-12`; operator/e2e `tests/v1/determinism/test_nvfp4_batch_invariant_scaled_mm.py`, `tests/v1/determinism/test_nvfp4_batch_invariant.py` @ `702f481` | - | [W3-C3R executed contract](specs/nvfp4-persistent-plan-cache.md#w3-c3r-batch-shape-localization-and-gate-correction-2026-07-13): production-default ours and vLLM both change outputs across batch shapes; no local opt-in implementation is claimed | `planned: specs/batch-invariant-execution.md` | `INVENTORIED` | - | | `ENG-ASYNC-SCHED` | Async/overlap scheduling (AsyncScheduler placeholders + depth-2 batch-queue step + async D2H on a copy stream); vLLM's DEFAULT at the pin — mirror obligation per B3. **Host-side machinery + runner device-input half + sampler-OUTPUT half LANDED + CPU-gated (2026-07-16):** `AsyncScheduler` placeholder accounting, `step_with_batch_queue` depth-2, `ResolveAsyncScheduling` default-ON-when-compatible + `MaxConcurrentBatches`, `VT_ASYNC_SCHED` rollback; the runner device-input path `combine_sampled_and_draft_tokens`; PLUS the sampler-OUTPUT half — `vt::Backend` event/pinned primitives (`AllocPinned`/events, CUDA cudaHostAlloc+cudaEvent, CPU sync-degeneration), `AsyncGPUModelRunnerOutput` (device sampled-id snapshot → non-blocking D2H on a copy queue + event; `get_output()` waits only that event; MAIN queue never blocked), `Sampler::forward(sampled_ids_out)` device-resident greedy, `GPUModelRunner::sample_tokens_async` + `runner_supports_async`, and the `Executor`+`step_with_batch_queue` seam resolving `get_output()` at CONSUME time. All behind `VT_ASYNC_RUNNER`/`set_async_input_combine`, default OFF. Sync path byte-identical (placeholder sites INERT while count 0; combine off; `sample_tokens_async` degenerates to sync when async off; `sampled_ids_out=nullptr`). **ENABLE-FLIP LANDED + CPU-gated (2026-07-16):** (1) `LoadedEngine` now reorders `runner_` before the scheduler and builds an `AsyncScheduler` + `max_concurrent_batches=2` when `ResolveAsyncScheduling(runner_.runner_supports_async())` resolves ON (else the byte-identical synchronous `Scheduler` + depth-1); the resolved mcb threads into `AsyncLLM`→`EngineCoreProc` (`step_with_batch_queue`) and the "Asynchronous scheduling is enabled/disabled" log mirrors vLLM for A/B audit; (2) the device combine/scatter kernel (`_combine_sampled_and_draft_tokens_kernel` + last_sampled scatter) is ported to CUDA (`src/vt/cuda/cuda_combine_tokens.cu`), main-stream-ordered on the CUDA async path so it DELETES `sample_tokens_async`'s pre-scatter `Synchronize`; the CPU backend keeps the host loop. `VT_ASYNC_RUNNER=1` engages full W3; `VT_ASYNC_SCHED=0` is the same-binary rollback. Production default (no env) stays synchronous byte-identical. **FULL W3 DGX proof RAN twice** — `f086b64` (5/5 gates PASS; c16 TPOT −5.4 ms WIN, tput neutral, TTFT +36 % = Little's-law repayment) and the 2026-07-16 re-proof on the THROUGHPUT-lever fix (persistent pooled sampled-id/pinned buffers + `Sampler` greedy scratch removing ALL per-step `cudaMalloc`/`cudaFree`/`cudaHostAlloc`/event-create from the sampled-id path, incl. the overlap-killing `cudaFree` inside `get_output`; mirrors `gpu_model_runner.py:873-878` + `async_utils.py:12-70`): token-exactness **6/6 PASS**, interleaved c16 **tput −0.32 % (gate ≥+1.5 % FAILS), TPOT −4.95 ms retained, TTFT +34.8 %** — the allocator lever is REFUTED as the tput unlock (≤0.1 % of a ~165 ms c16 step). **DEFAULT FLIPPED ON 2026-07-17** (`VT_ASYNC_RUNNER` default ON via the pure `AsyncRunnerFlagIsOn` predicate, mirroring `vllm/config/vllm.py:992-1044`): the discriminator (`6ea7856`) proved vLLM's own async pays the identical +26–31 % TTFT / −0.7 to −0.9 % tput / −2.6 to −4.3 ms TPOT envelope and W3-ON nets positive (both binding ITL-tail anomalies flip to PASS), so the "needs a throughput lever" ship-gate is RETIRED — W3 is a parity/mirror obligation with a tails+TPOT win. The flip is TOKEN-NEUTRAL (async-ON ≡ async-OFF bit-identical on DGX). `VT_ASYNC_RUNNER=0` = runner-level rollback, `VT_ASYNC_SCHED=0` = scheduler-level rollback. TTFT means rise into vLLM's async envelope BY DESIGN — the next binding grid runs async by default and its TTFT must NOT be misread as a regression. **ROBUSTNESS FIX 2026-07-20 (`discard_request_mask`):** the runner was missing vLLM's `discard_request_mask`, so `GPUModelRunner` emitted a sampled token for prefill-CHUNK requests too; under async this drained a `num_output_placeholders` never reserved (the `is_prefill_chunk` path adds none) → the `async_scheduler.cpp` `num_output_placeholders >= 0` assertion aborted on c8 + short-output (chunked prefill + preemption). FIX mirrors vLLM: `execute_model` computes `exec_state_.discard[i] = seq_len < num_tokens` (`gpu_model_runner.py:2048`); `sample_tokens` clears those rows to empty (`outputs.py:303`), the async path passes `invalid_req_indices` to `AsyncGPUModelRunnerOutput::get_output` (`gpu_model_runner.py:3625` + `outputs.py:303`). Scheduler UNCHANGED (assertion kept — it was correct once the runner honors `scheduler.py:1888-1890`). Sync/non-chunked decode byte-identical (mask all-zero); DGX 27B 235/235 + 35B 315/315, `vllm-bench` c8+short-output+chunked+kv-pressure no longer crashes, memcheck 0. Ledger [parity-ledger.md](parity-ledger.md) 2026-07-20 row | T1 | `vllm/v1/core/sched/async_scheduler.py:12`; `vllm/config/vllm.py:490,990,1038`; `vllm/v1/engine/core.py:519`; `vllm/v1/worker/gpu/input_batch.py:304-406`; `vllm/v1/worker/gpu/async_utils.py:12-70`; `vllm/v1/worker/gpu/gpu_model_runner.py:242-332`; `vllm/v1/outputs.py:298-307` | `src/vllm/v1/core/sched/async_scheduler.cpp:10,45`; placeholder plumbing `src/vllm/v1/core/sched/scheduler.cpp:148,164,605`; `src/vllm/v1/engine/core.cpp:91` (`step_with_batch_queue`, async-output seam); `src/vllm/v1/engine/core_proc.cpp:32,46`; config `include/vllm/config/scheduler.h:117,165,188`, `src/vllm/config/scheduler.cpp:12`; `include/vllm/v1/request.h:187`; runner input leaf `src/vllm/v1/worker/gpu/prepare_inputs.cpp`, `src/vllm/v1/worker/gpu/input_batch.cpp`; runner output leaf `include/vt/backend.h`+`src/vt/backend.cpp`+`src/vt/cuda/cuda_backend.cu` (event/pinned), `include/vllm/v1/worker/gpu/async_output.{h,cpp}` (`AsyncGPUModelRunnerOutput`), `src/vllm/v1/sample/sampler.cpp` (`sampled_ids_out`), `src/vllm/v1/worker/gpu/runner.cpp` (`sample_tokens_async`/`runner_supports_async`), `src/vllm/v1/executor/executor.cpp`+`include/vllm/v1/worker/gpu/model_runner_base.h` (async seam); enable-flip `include/vllm/entrypoints/model_loader.h`+`src/vllm/entrypoints/model_loader.cpp` (`runner_` before scheduler, `ResolveAsyncEnabled`/`MakeScheduler`, `AsyncScheduler`+mcb=2, log), `include/vllm/v1/engine/async_llm.h`+`src/vllm/v1/engine/async_llm.cpp` (mcb param → `EngineCoreProc`); device kernel `include/vt/cuda/combine_tokens.h`+`src/vt/cuda/cuda_combine_tokens.cu`, wired `src/vllm/v1/worker/gpu/runner.cpp` (CUDA combine/scatter branch removes the pre-sync) | `tests/vllm/v1/test_async_scheduler.cpp:1` (6 cases, 54 asserts; RED vs base Scheduler 2/6 fail); depth-2 engine cycle `tests/vllm/v1/test_engine_core_proc.cpp:479` (mcb=2, async-output seam); config resolution `tests/vllm/test_scheduler_config.cpp:75`; enable-flip construction matrix `tests/vllm/entrypoints/test_loaded_engine_dense.cpp` (runner×VT_ASYNC_SCHED → scheduler type + mcb; RED = un-flipped engine, 3/3 ON-arm asserts fail); runner input leaf `test_combine_tokens.cpp` (RED = stale → 5/7 fail), `test_input_batch.cpp`, `test_runner.cpp` (async-ON≡sync); output leaf `tests/vt/test_backend.cpp` (event/pinned contract), `tests/vllm/v1/worker/test_async_output.cpp` (materialize/flush/snapshot; RED = +1 splice), `test_runner.cpp` (`sample_tokens_async` decode ≡ sync); full CPU ctest 111/111, tools 164/164. Prior diagnostic `3812d8` six-leg control: total **1.002153×**, TTFT **0.862159×**, no GPU-time reduction (neutral for speed). **DEFAULT-FLIP (2026-07-17):** new pure CPU flag test [test_async_runner_flag.cpp](../tests/vllm/v1/worker/test_async_runner_flag.cpp) (11 asserts, default-ON/'0'-off); construction matrix [test_loaded_engine_dense.cpp](../tests/vllm/entrypoints/test_loaded_engine_dense.cpp) INVERTED (default → AsyncScheduler+mcb=2; RED verified 5 asserts fail vs un-flipped). CPU clean `-Werror` rebuild, full serial ctest **116/116**, tools **164/164**. **DGX re-confirmation** (evidence `dgx:~/work/vllm.cpp-async-flip`, CUTLASS+FA2 hard-verified, one flock): shipping default (async ON + RMSNorm-fast OFF) → **27B 235/235 + 35B 315/315** with the "Asynchronous scheduling is enabled (mcb=2)" log, and both rollback arms (`VT_ASYNC_RUNNER=0`, `VT_ASYNC_SCHED=0`) 235/235 + 315/315 log "disabled"; async arms BIT-IDENTICAL (token-neutral). Closing record [parity-ledger.md#L502](parity-ledger.md#L502) | [async-serving.md](specs/async-serving.md) | `DONE` | `6ea7856` | -| `ENG-PRIORITY-SCHED` | Priority request queue + policy + priority preemption + `priority` plumbing (Request/EngineCoreRequest/OpenAI field); W4 of the async-serving block. Default stays FCFS. GATING: full CPU tier green (93/93; 12 ported priority-scheduler cases + 14 priority-queue cases incl. the seeded random property test); GPU G1 (both greedy engine gates, priority-vs-fcfs token-exactness) deferred to the next GPU-idle window — GPU held by the `SERVE-GATE-ONLINE` campaign | T1 | `vllm/v1/core/sched/request_queue.py:131,201`; `vllm/v1/core/sched/scheduler.py:546`; `vllm/config/scheduler.py:109`; `tests/v1/core/test_scheduler.py:2382,2978`; `tests/v1/core/test_priority_scheduler_random.py:1` | `src/vllm/v1/core/sched/request_queue.cpp:101,186`; `src/vllm/v1/core/sched/scheduler.cpp:178`; `src/vllm/v1/request.cpp:92`; `src/vllm/config/scheduler.cpp:21` | `tests/vllm/v1/test_scheduler.cpp:674,916`; `tests/vllm/v1/test_request_queue.cpp:238,429` | [async-serving.md](specs/async-serving.md) | `GATING` | - | +| `ENG-PRIORITY-SCHED` | Priority request queue + policy + priority preemption + `priority` plumbing (Request/EngineCoreRequest/OpenAI field); W4 of the async-serving block. Default stays FCFS. GATING: full CPU tier green (93/93; 12 ported priority-scheduler cases + 14 priority-queue cases incl. the seeded random property test); ~~GPU G1 (both greedy engine gates, priority-vs-fcfs token-exactness) deferred to the next GPU-idle window — GPU held by the `SERVE-GATE-ONLINE` campaign~~. **BLOCKER CORRECTED 2026-08-12 ([#534](https://github.com/mudler/vllm.cpp/issues/534)): the GPU is not what is stopping this, and G1 cannot be "rerun" because it does not exist.** `--scheduling-policy priority` is plumbed to the production server (`src/vllm/entrypoints/openai/server_main.cpp:408-411,672-673` -> `SchedulerPolicyFromString` -> `SchedulerConfig::policy`), but NO priority-vs-fcfs gate exists at the ENGINE/MODEL level. What exists is the scheduler-unit tier (`test_scheduler.cpp:674,916` admission/preemption ordering, `test_request_queue.cpp` heap ordering) plus C-ABI wire-name validation (`tests/capi/test_capi.cpp:1159`); nothing anywhere drives a real engine with `policy=kPriority` and compares token streams against the fcfs arm. A next owner writes that gate RED-first, then runs it. The struck deferral is a 2026-07-10 scheduling note that five successive `SERVE-GATE-ONLINE` bindings (`3f256ab`, `246a23c`, `a875397`, `f0fb727`, `9ecd9d0`) have since expired. This is the ONLY genuinely open leaf of the `ROAD-V1-C6` async-serving block | T1 | `vllm/v1/core/sched/request_queue.py:131,201`; `vllm/v1/core/sched/scheduler.py:546`; `vllm/config/scheduler.py:109`; `tests/v1/core/test_scheduler.py:2382,2978`; `tests/v1/core/test_priority_scheduler_random.py:1` | `src/vllm/v1/core/sched/request_queue.cpp:101,186`; `src/vllm/v1/core/sched/scheduler.cpp:178`; `src/vllm/v1/request.cpp:92`; `src/vllm/config/scheduler.cpp:21` | `tests/vllm/v1/test_scheduler.cpp:674,916`; `tests/vllm/v1/test_request_queue.cpp:238,429` | [async-serving.md](specs/async-serving.md) | `GATING` | - | | `ENG-PARTIAL-PREFILL` | Concurrent partial-prefill and long-prompt limits | T1 | `vllm/config/scheduler.py:70-80` | - | - | `planned: specs/partial-prefill-concurrency.md` | `INVENTORIED` | - | | `ENG-BATCH-QUEUE` | Pipelined `step_with_batch_queue` | T1 | `vllm/v1/engine/core.py:519` | - | - | `planned: specs/batch-queue-step.md` | `INVENTORIED` | - | -| `ENG-CORE-BUSY-LOOP` | Busy loop with input/output queue split (in-proc analog of the ZMQ EngineCoreProc boundary); W1 of the async-serving block. Implemented: `EngineCoreProc` (queues, run_busy_loop, shutdown drain/abort, WAKEUP, ENGINE_CORE_DEAD) + `InprocClient` on a dedicated engine thread; sync `LLMEngine` path untouched; UTILITY/DP/aborts-queue/step_with_batch_queue deferred per spec. GATING: CPU suites green; GPU G1 (token-exact twins) + G4 (no-throughput-regression) deferred to the gating handoff — GPU held by the `SERVE-GATE-ONLINE` campaign | T0 | `vllm/v1/engine/core.py:915,1259`; `vllm/v1/engine/core_client.py:467` | `include/vllm/v1/engine/core_proc.h:150`; `src/vllm/v1/engine/core_proc.cpp:51`; `src/vllm/v1/engine/core_client.cpp:33` | `tests/vllm/v1/test_engine_core_proc.cpp:188,314,395` (9 cases, 82 asserts; CPU ctest 93/93) | [async-serving.md](specs/async-serving.md) | `GATING` | - | +| `ENG-CORE-BUSY-LOOP` | Busy loop with input/output queue split (in-proc analog of the ZMQ EngineCoreProc boundary); W1 of the async-serving block. Implemented: `EngineCoreProc` (queues, run_busy_loop, shutdown drain/abort, WAKEUP, ENGINE_CORE_DEAD) + `InprocClient` on a dedicated engine thread; sync `LLMEngine` path untouched; UTILITY/DP/aborts-queue/step_with_batch_queue deferred per spec. GATING: CPU suites green; ~~GPU G1 (token-exact twins) + G4 (no-throughput-regression) deferred to the gating handoff — GPU held by the `SERVE-GATE-ONLINE` campaign~~. **BLOCKER CORRECTED 2026-08-12 ([#534](https://github.com/mudler/vllm.cpp/issues/534)): both are covered in substance and were never re-read onto this row.** G1 — `tests/parity/test_qwen36_async_serving.cpp` (`1718bf155`, 2026-08-05) drives `LoadedEngine::async_engine()` -> `AsyncLLM` -> `EngineCoreProc::step_with_batch_queue` (depth-2) token-exact against the SAME pinned oracle continuation the SACRED sync gate uses, proven RED->GREEN on GB10 with `compute-sanitizer` 0 errors, joined by the classic-dense arm `tests/parity/test_qwen3_dense_async_serving.cpp` (`52d76f3a9`). G4 — the `SERVE-GATE-ONLINE` binding `9ecd9d0` 114/124 runs exactly this path with async default ON. Promotion is the operator's rerun, not a records edit; the state is left unchanged deliberately | T0 | `vllm/v1/engine/core.py:915,1259`; `vllm/v1/engine/core_client.py:467` | `include/vllm/v1/engine/core_proc.h:150`; `src/vllm/v1/engine/core_proc.cpp:51`; `src/vllm/v1/engine/core_client.cpp:33` | `tests/vllm/v1/test_engine_core_proc.cpp:188,314,395` (9 cases, 82 asserts; CPU ctest 93/93) | [async-serving.md](specs/async-serving.md) | `GATING` | - | | `ENG-SCHED-KNOBS` | Reserve-full-ISL, scheduler class seam, stream interval | T1 | `vllm/config/scheduler.py:26,127,140,163` | `include/vllm/config/scheduler.h:71,95`; `src/vllm/config/scheduler.cpp:43,52`; `src/vllm/v1/core/sched/scheduler.cpp:237`; `src/vllm/v1/engine/output_processor.cpp:68` | `tests/vllm/test_scheduler_config.cpp:10,20`; `tests/vllm/v1/test_kv_cache_manager.cpp:425` | `planned: specs/scheduler-knobs.md` | `PARTIAL` | - | | `ENG-CASCADE-ATTN` | Cascade attention for shared prefixes. **SPIKED 2026-07-22: VERIFIED NOT OWED on the path we mirror, for three independent reasons** — (1) `disable_cascade_attn` defaults `True` (opt-in per its own docstring), (2) the implementation exists ONLY on the legacy V1 runner `gpu_model_runner.py:504,2544`; a recursive grep for `cascade` over the MRV2 tree `vllm/v1/worker/gpu/` returns ZERO hits and we port MRV2, (3) FlashAttention is the sole implementing backend while FlashInfer/Triton/Flex/ROCm all hard-return `False`, and Blackwell resolves FlashInfer FIRST — so it is unreachable on our GB10 gate box. Also forces full CUDA graphs OFF, and the shared-block count is hard-0 for SWA/chunked-local/Mamba (our hybrid gate models). The scheduler-side input is already computed/plumbed, so if upstream moves it into MRV2 only the backend half remains | T2 | `vllm/config/model.py:238,239-244`; V1-only impl `vllm/v1/worker/gpu_model_runner.py:504,2544,3869`; backend support `vllm/v1/attention/backends/flash_attn.py:670-671,1307-1319` vs `flashinfer.py:1445-1452`; Blackwell priority `vllm/platforms/cuda.py:145-151`; `tests/v1/e2e/general/test_cascade_attention.py:20` | common-prefix input only (not consumed): `src/vllm/v1/core/sched/scheduler.cpp:330,381`; `include/vllm/v1/core/sched/output.h:151` | per-manager zero policies `tests/vllm/v1/test_single_type_kv_cache_manager.cpp:263` | [prefix-prompt-caching-parity.md](specs/prefix-prompt-caching-parity.md) | `SPIKE` | `CLAIM-PREFIX-PROMPT-CACHING` | | `ENG-DBO-UBATCH` | DBO and ubatch overlap | T2 | `vllm/config/parallel.py:208,524` | - | - | `planned: specs/dbo-ubatch.md` | `INVENTORIED` | - | @@ -203,7 +203,7 @@ claims it. | `SERVE-STREAM-USAGE` | Completion/chat `stream_options`: final and continuous native-ID usage frames, non-stream validation, and force-usage server mode. GATING: the host implementation is CPU/sanitizer-green; void `31d053f` 27B execution proved exact native counts on all 2,016 standard timed requests, but fresh passing 27B→35B online and serialization A/B gates remain mandatory | T1 | `vllm/entrypoints/openai/engine/protocol.py:241-243`; completion `protocol.py:66,471-478`, `serving.py:298-305,359-454`; chat `protocol.py:214,731-737`, `serving.py:459-512,570-760`; `entrypoints/serve/utils/api_utils.py:276-289`; `tests/entrypoints/openai/completion/test_completion.py:400-553`; `tests/entrypoints/openai/chat_completion/test_chat.py:348-445` | schema/parser `include/vllm/entrypoints/openai/protocol.h:62,203,318`, `src/vllm/entrypoints/openai/protocol.cpp:103,223,278`; selection `src/vllm/entrypoints/openai/serving_utils.cpp:8`; completion SSE `src/vllm/entrypoints/openai/serving_completion.cpp:22,160`; chat SSE `src/vllm/entrypoints/openai/serving_chat.cpp:232,450`; force CLI `examples/server/main.cpp:123` | protocol/selection `tests/vllm/entrypoints/openai/test_protocol.cpp:130,189`; sync completion/chat `tests/vllm/entrypoints/openai/test_serving.cpp:484,647`; production final/continuous/validation/force/disconnect `tests/vllm/entrypoints/openai/test_api_server.cpp:403,442,498,607,652,687,712`; help `examples/CMakeLists.txt:36`. CPU CTest 105/105; focused 63 cases/658 assertions; API repeat 100/100; ASan+UBSan 3/3; TSan 1/1. `31d053f` retained all 36 standard 27B raw points / 2,016 successful requests with exact native 128-token usage | [stream-options.md](specs/stream-options.md) | `GATING` | - | | `SERVE-UTILITY-ENDPOINTS` | Tokenize, detokenize, ready, ping, server info, prefix reset. **LANDED + CPU-GATED 2026-07-27 (`CLAIM-ROADMAP-C8`, NOT pushed):** `/tokenize` (prompt form → `{count,max_model_len,tokens,token_strs?}`) + `/detokenize` (`{tokens[]}`→`{prompt}`) over the existing tokenizer, `/ping` (liveness, mirrors `/health`), `/server_info` (`{vllm_config,vllm_env,system_env}`), `/reset_prefix_cache` (`{"success":bool}` via an injected callback). All ADDITIVE + opt-in (tokenize/detokenize/reset registered only when their backing is attached). **CHAT-FORM `/tokenize` LANDED + CPU-GATED 2026-07-28 (`CLAIM-C8-CHAT-TOKENIZE`, NOT pushed):** `/tokenize` now accepts BOTH arms of the vLLM `TokenizeRequest` union — the raw `prompt` form AND the `TokenizeChatRequest{messages, add_generation_prompt, continue_final_message, add_special_tokens, tools?}`; the chat form renders through `chat_.prompt_fn()` (the IDENTICAL model chat template `create_chat_completion` tokenizes through), applies the `check_generation_prompt` mutual-exclusion (→400), tokenizes with the chat-form `add_special_tokens` default False (vs completion-form True), returns the same `{count,max_model_len,tokens,token_strs?}`. **`/tokenizer_info` LANDED + CPU-GATED 2026-07-28 (`CLAIM-C8-SERVE-ENDPOINTS`, NOT pushed):** `GET /tokenizer_info` gated behind a `set_tokenizer_info_enabled` flag mirroring vLLM's `enable_tokenizer_info_endpoint` CLI arg (off by default → the route is not registered → 404; on + tokenizer attached → 200). Surfaces the `tokenizer_config.json`-equivalent fields our byte-level/SentencePiece BPE tokenizer can GENUINELY back — `tokenizer_class` (the BPE family name), `model_max_length`, `vocab_size`, `bos_token_id`/`eos_token_id` (omitted when -1), and `added_tokens_decoder` (id → `{content,special,lstrip,rstrip}`); NAMED gaps OMITTED (never fabricated): the raw `chat_template` string (lives in the ChatPromptFn render seam, not the tokenizer), the HF `init_kwargs` (`clean_up_tokenization_spaces`/`add_bos_token`/`model_input_names`/padding-truncation defaults — not parsed), and the added-token `normalized`/`single_word` flags. **PRODUCTION `main.cpp` WIRING LANDED + CPU-GATED 2026-07-28 (`CLAIM-C8-SERVE-PROD-WIRING`, NOT pushed):** the shipped `vllm-server` binary now lights `/tokenize`+`/detokenize` (on by default when a tokenizer exists) and `/tokenizer_info` (behind the new `--enable-tokenizer-info-endpoint` flag, mirroring vLLM's `enable_tokenizer_info_endpoint`) from the LIVE engine+tokenizer through the shared `ConfigureUtilityEndpoints` seam — the SAME seam the gate drives over a real socket. `/metrics` + `/reset_prefix_cache` stay UNWIRED (named residuals): the production `AsyncLLM` frontend exposes no live `PrometheusStatLogger` (async stats deferred; missing `LoadedEngine::stat_logger()` + a `Record()` site in `AsyncLLM::RunOutputHandler`) and no thread-safe prefix-cache reset RPC (`reset_prefix_cache()` is `KVCacheManager`-private, mutated only on the EngineCore thread; missing `AsyncLLM::reset_prefix_cache`), so attaching either would be a fabricated wiring that never reaches the live engine — library handlers+tests retained. RESIDUAL: `chat_template_kwargs`/`continue_final_message` full template-render passthrough (the ChatPromptFn seam renders only via the `add_generation_prompt` gate), `/ready`, full server_info config dump, live `/metrics` + `/reset_prefix_cache` backing on the AsyncLLM path | T1 | `vllm/entrypoints/serve/tokenize/api_router.py:37,63,95-108`; production gating `vllm/entrypoints/openai/api_server.py:222`, `vllm/entrypoints/serve/__init__.py:11-31`, `vllm/entrypoints/openai/cli_args.py:140`; `vllm/entrypoints/serve/tokenize/protocol.py:24,50,156,166,181,185`; `vllm/entrypoints/serve/tokenize/serving.py:57,70-124,154-195`; `vllm/entrypoints/serve/sagemaker/api_router.py:47`; `vllm/entrypoints/serve/dev/server_info/api_router.py:43`; `vllm/entrypoints/serve/dev/cache/api_router.py:20` | handlers `src/vllm/entrypoints/openai/api_server.cpp:262` (`handle_tokenize`, prompt+chat union),`:368,404,245,422` (`handle_detokenize`/`handle_reset_prefix_cache`/`handle_ping`/`handle_server_info`),`:438` (`handle_tokenizer_info`); chat render seam `include/vllm/entrypoints/openai/serving_chat.h:197` (`prompt_fn()`); opt-in setters + routes `include/vllm/entrypoints/openai/api_server.h:118` (`set_tokenizer_info_enabled`); production seam `include/vllm/entrypoints/openai/api_server.h` (`ConfigureUtilityEndpoints`) + `src/vllm/entrypoints/openai/api_server.cpp` (impl); production call + CLI flags `examples/server/main.cpp` (`--enable-tokenizer-info-endpoint`, `ConfigureUtilityEndpoints(...)`) | `tests/vllm/entrypoints/openai/test_api_server.cpp:879` (prompt round-trip+schema+raw-form exact ids),`:938` (chat-form renders template + exact tokens, RED-first),`:1061` (`/tokenizer_info` backed fields + named-gap omissions + no-tokenizer 500),`:1250` (opt-in route gate: 404 flag-off → 200 flag-on over a real socket, RED-first),`:1319` (**production `ConfigureUtilityEndpoints` seam over a real socket: no-seam→404 RED, defaults→tokenize/detokenize 200 + info/abort 404, flags-on→200, exact abort delta-count**) — 32/32 / 420-assertion suite | [utility-endpoints.md](specs/utility-endpoints.md) | `ANCHOR-BACKFILL` | `CLAIM-C8-SERVE-PROD-WIRING` | | `SERVE-CHAT-TEMPLATE` | Full-surface Jinja chat templates (vendored google/minja `021c229` + documented lstrip guard) | T0 | `vllm/renderers/hf.py:673,986`; `vllm/entrypoints/chat_utils.py:1248,1335` | `src/vllm/entrypoints/chat_template.cpp:101,168,181,220` | `tests/vllm/entrypoints/test_chat_template.cpp:65,75,84,96` | `planned: specs/chat-templating.md` | `ANCHOR-BACKFILL` | - | -| `SERVE-ASYNC-LLM` | AsyncLLM-equivalent streaming engine API: per-request collectors, concurrent submit/generate/abort, live completion/chat SSE with disconnect abort, additive nonblocking C requests, and enough HTTP delivery capacity for configured concurrent streams. GATING: deterministic c32 capacity is implemented and GPU-classified; broader every-axis parity remains open | T0 | `vllm/v1/engine/async_llm.py:70,280,524,637,709`; `vllm/v1/engine/output_processor.py:45-105`; asyncio server path `vllm/entrypoints/openai/api_server.py:1`; `tests/v1/engine/test_async_llm.py:109,157,228,306,340,598` | existing async path `include/vllm/v1/engine/async_llm.h:45`, `src/vllm/v1/engine/async_llm.cpp:32`; fixed/legacy pool API `include/vllm/entrypoints/openai/api_server.h:41-57,101-104`; capacity selection `src/vllm/entrypoints/openai/api_server.cpp:23-62`; production max-seqs wiring + `VLLM_CPP_HTTP_FIXED_POOL=0` A/B `examples/server/main.cpp:230-247`; cpp-httplib defect `third_party/httplib/httplib.h:161-169,10359-10377` | persistent 32-client + control reserve, validation and diagnostic-mode cases `tests/vllm/entrypoints/openai/test_api_server.cpp:937-1000`; focused Release/help pass, API **100/100**, ASan+UBSan **1/1**, TSan **1/1**; known unrelated serial C-API flake isolated. Exact fixed/legacy c32 AB/BA/AB is healthy and steady-state-neutral: **1097.031/1097.290 tok/s = 0.999764×**, 8/20 axes, 1,152/1,152 requests and six memory returns; neither legacy arm samples the rare old stall. Exact `4e1d8ca` fixed c32 is healthy 3/3 and 0.9910× vLLM | [async-serving.md](specs/async-serving.md) | `GATING` | - | +| `SERVE-ASYNC-LLM` | AsyncLLM-equivalent streaming engine API: per-request collectors, concurrent submit/generate/abort, live completion/chat SSE with disconnect abort, additive nonblocking C requests, and enough HTTP delivery capacity for configured concurrent streams. GATING: deterministic c32 capacity is implemented and GPU-classified; broader every-axis parity remains open. **CLARIFIED 2026-08-12 ([#534](https://github.com/mudler/vllm.cpp/issues/534)) — this row is NOT waiting on a "prod-ON" flip,** which is what punch-list item 9 and the `ROAD-V1-A` SGLang clause both read it as. It IS the production serving path (`src/vllm/entrypoints/openai/server_main.cpp:731-734`, *"the production server uses AsyncLLM over EngineCoreProc's dedicated engine thread"*), with the capacity-derived fixed HTTP pool as the default and `VLLM_CPP_HTTP_FIXED_POOL=0` retained only as a same-binary diagnostic; the separate runner-side `VT_ASYNC_RUNNER`/`runner_supports_async` default is `ENG-ASYNC-SCHED`'s and has been ON since `a0013a2`. What remains is exactly the every-axis parity named above: 27B ratified (two-grid 115/124 effective), 35B open under `ROAD-V1-A`, plus open bug [#294](https://github.com/mudler/vllm.cpp/issues/294). Its GPU token-exact gate is `tests/parity/test_qwen36_async_serving.cpp` (`1718bf155`) — NOT `qwen36_paged_engine`, which drives the sync depth-1 path and structurally cannot see this row's defects | T0 | `vllm/v1/engine/async_llm.py:70,280,524,637,709`; `vllm/v1/engine/output_processor.py:45-105`; asyncio server path `vllm/entrypoints/openai/api_server.py:1`; `tests/v1/engine/test_async_llm.py:109,157,228,306,340,598` | existing async path `include/vllm/v1/engine/async_llm.h:45`, `src/vllm/v1/engine/async_llm.cpp:32`; fixed/legacy pool API `include/vllm/entrypoints/openai/api_server.h:41-57,101-104`; capacity selection `src/vllm/entrypoints/openai/api_server.cpp:23-62`; production max-seqs wiring + `VLLM_CPP_HTTP_FIXED_POOL=0` A/B `src/vllm/entrypoints/openai/server_main.cpp:874-883` (moved verbatim out of `examples/server/main.cpp` by ARCH-ONE-SURFACE #189; the example is now a one-line `vllm_server_main` client); cpp-httplib defect `third_party/httplib/httplib.h:161-169,10359-10377` | persistent 32-client + control reserve, validation and diagnostic-mode cases `tests/vllm/entrypoints/openai/test_api_server.cpp:937-1000`; focused Release/help pass, API **100/100**, ASan+UBSan **1/1**, TSan **1/1**; known unrelated serial C-API flake isolated. Exact fixed/legacy c32 AB/BA/AB is healthy and steady-state-neutral: **1097.031/1097.290 tok/s = 0.999764×**, 8/20 axes, 1,152/1,152 requests and six memory returns; neither legacy arm samples the rare old stall. Exact `4e1d8ca` fixed c32 is healthy 3/3 and 0.9910× vLLM | [async-serving.md](specs/async-serving.md) | `GATING` | - | | `SERVE-HTTP-TRANSPORT` | Serving-socket transport parity: mirror vLLM's uvicorn/asyncio default `TCP_NODELAY` on every accepted SSE socket so per-token stream frames are not held by Nagle against the peer's delayed ACK. Implemented + CPU-tested; the non-binding localhost A/B sizing is COMPLETE and NEUTRAL within noise on c1/c2 ITL/TPOT/throughput (loopback ACKs are instant, so Nagle never coalesces ~100 ms-cadence token frames) — no gate-axis credit expected; the mirror stays for real-network parity. Future keep-alive / read-write-timeout / listening-socket option parity noted, not done | T0 | vLLM serves via uvicorn over asyncio `vllm/entrypoints/launcher.py:71,76`, `vllm/entrypoints/openai/api_server.py:591,630`; asyncio disables Nagle per accepted TCP stream socket `asyncio/base_events.py:192-197` (`_set_nodelay`) called from `asyncio/selector_events.py:950`; cpp-httplib default-off `third_party/httplib/httplib.h:142`, applied on accept only when set `third_party/httplib/httplib.h:12083` | `src/vllm/entrypoints/openai/api_server.cpp:69` (`set_tcp_nodelay(true)` in the ApiServer setup) | behavioral accepted-socket `getsockopt(TCP_NODELAY)` case `tests/vllm/entrypoints/openai/test_api_server.cpp:1076` (helper `:380`); RED accepted `TCP_NODELAY` 0 → GREEN 1, full `test_openai_api_server` **22/22 cases / 242 assertions**; non-binding sizing root `~/work/vllm.cpp-tcpnodelay-sizing/ff915e8…` (raw-set SHA `f5b52900…2128`) neutral within noise; closure [ledger](parity-ledger.md#L451) | [serve-tcp-nodelay.md](specs/serve-tcp-nodelay.md) | `DONE` | `ff915e8` | | `SERVE-C-ABI` | Stable LocalAI-style C FFI (**19** exported `VLLM_API` symbols at `VLLM_ABI_VERSION 10`; blocking and nonblocking request handles. Count corrected 2026-07-24 from a stale `17`, which predated ABI v4/v5 adding `tool_parser`/`reasoning_parser` and the chat entry points; `include/vllm.h` is the source of truth and README:231 already said 19). **ABI v9 2026-07-28 (`CLAIM-CAPI-ENGINE-CONFIG-V9`): the ABI carried strictly LESS engine config than `EngineParams` does** - `max_num_batched_tokens`, the scheduler `scheduling_policy` (`fcfs` / `priority` / `lpm`), and `kv_transfer_config` (the external KV connector / LMCache JSON) were reachable from the bundled server's flags and from NO embedder. All three added, inert at their defaults (zero-filled v8 growth == byte-identical pre-v9 engine); the connector NAME is validated against `KVConnectorFactory` at load, mirroring the server's startup check. `tokenizer_config_path` stopped being a declared-since-v1 no-op and now selects the chat template's source file. Malformed `speculative_config`/`kv_transfer_config` documents now report `VLLM_ERR_INVALID_ARGUMENT` (the contract vllm.h documented since v6) instead of `VLLM_ERR_MODEL_LOAD`, via a catch scoped to the parse block so a real `FromModelDir` failure still reports MODEL_LOAD. Driver: the LocalAI vllm-cpp backend could not expose LMCache or the prefill budget in a model config) | T0 | Original project ABI; pinned vLLM has no C ABI | `include/vllm.h:143,181,207`; `src/capi/vllm_c.cpp:229,264,327,391` | `tests/capi/test_capi.cpp:320,428,505,574,606,640`; `tests/capi/test_dlopen.cpp:77,86`; `tests/capi/c_header_compile.c:1` | [c-api-library.md](specs/c-api-library.md) | `ANCHOR-BACKFILL` | `CLAIM-SERVE-C-ABI-SPIKE` | | `SERVE-CPP-API` | Rich `LLM` and `AsyncLLM` C++ API | T1 | `vllm/entrypoints/llm.py:66,422`; `vllm/v1/engine/async_llm.py:70` | - | - | `planned: specs/cpp-api.md` | `INVENTORIED` | - | diff --git a/.agents/roadmap_v1.md b/.agents/roadmap_v1.md index de40bea64..1ac6037fe 100644 --- a/.agents/roadmap_v1.md +++ b/.agents/roadmap_v1.md @@ -131,6 +131,7 @@ issue is not yet placed. Keyed record: update in place, never append. | [#362](https://github.com/mudler/vllm.cpp/issues/362) | — | First VALID 27B per-kernel decode attribution: whole-run trace windowed by the profiler's own cudaProfilerStart/Stop | bug | | [#223](https://github.com/mudler/vllm.cpp/issues/223) | `SAMPLE-PROMPT-LOGPROBS` | `prompt_logprobs` is a silent no-op: the runner never computes logits at prompt positions | bug | | [#294](https://github.com/mudler/vllm.cpp/issues/294) | `SERVE-ASYNC-LLM` | `test_async_llm`: reusing an aborted request id races the core abort — 26% failure rate under contention, on main | bug | +| [#534](https://github.com/mudler/vllm.cpp/issues/534) | `SERVE-ASYNC-LLM` | Punch-list item 9 (`ROAD-V1-C6`) directs a depth-2 throughput lever the project refuted (`6ea7856`) and a `runner_supports_async` flip that landed (`a0013a2`) 10 days before the item was written; the one real residual is `ENG-PRIORITY-SCHED`'s priority-vs-FCFS gate, which does not exist | bug | | [#222](https://github.com/mudler/vllm.cpp/issues/222) | `SERVE-GATE-ONLINE` | The online-serving gate aborts on current main: harness looks for `examples/server`, the build produces `examples/vllm-server` — paths repaired by `2b262622`/`8fce04d3`; residue is the drift guard, the `27n` key and a model gate that recorded a skip as passed | bug | | [#277](https://github.com/mudler/vllm.cpp/issues/277) | `SERVE-METRICS` | `/metrics` is dead on the production server: AsyncLLM never folds IterationStats into the logger | bug | | [#321](https://github.com/mudler/vllm.cpp/issues/321) | `SERVE-METRICS` | AsyncLLM output thread outlives metrics logger and uses freed PromRegistry | bug | diff --git a/.agents/specs/async-serving.md b/.agents/specs/async-serving.md index f32730f35..64e403e8a 100644 --- a/.agents/specs/async-serving.md +++ b/.agents/specs/async-serving.md @@ -7,12 +7,65 @@ split), `ENG-ASYNC-SCHED` (async/overlap scheduling), `ENG-PRIORITY-SCHED` `ROAD-V1-A` (see priority justification). Upstream pin: `/home/mudler/_git/vllm` @ `e24d1b24`. +## Now (record reconciliation 2026-08-12, [#534](https://github.com/mudler/vllm.cpp/issues/534)) + +Punch-list item 9 of [roadmap-v1-completion.md](roadmap-v1-completion.md) §3 was +this block's plan of record. Re-verified against the tree, **two of its three +clauses were already stale when that spec was committed** (`489f7771`, +2026-07-27), so it is reconciled there and here. Nothing below is deleted; the +checkpoint text that follows is history and is kept as written. + +| Clause | Reality | +|---|---| +| land the depth-2 throughput lever | Landed `6ea7856` (2026-07-16) and **REFUTED in the same commit**; the objective itself was retired by the 2026-07-17 discriminator addendum below (*"There is NO depth-2 throughput unlock to find"*). Acting on it re-opens a closed negative result. | +| flip `runner_supports_async` prod-ON | Flipped `a0013a2` (2026-07-17), default-ON via `AsyncRunnerFlagIsOn`. The item's title conflates this with `SERVE-ASYNC-LLM`, a different row, which is **also** already the production path (`src/vllm/entrypoints/openai/server_main.cpp:731-734`). | +| close priority/busy-loop GPU gates | `ENG-CORE-BUSY-LOOP` covered in substance; `ENG-PRIORITY-SCHED` genuinely OPEN — see below. | + +**The one real residual in this block is W4's gate, and it does not exist yet.** +`--scheduling-policy priority|fcfs` is plumbed all the way to the production +server (`server_main.cpp:408-411,672-673` → `SchedulerPolicyFromString` → +`SchedulerConfig::policy`), and the CPU tier is green (93/93, 12 ported +priority-scheduler cases + 14 priority-queue cases incl. the seeded random +property test) — but **no priority-vs-FCFS gate exists at the engine/model +level**. Everything that exists is scheduler-unit (`test_scheduler.cpp:674,916` +admission/preemption ordering, `test_request_queue.cpp` heap ordering) or C-ABI +wire-name validation (`tests/capi/test_capi.cpp:1159`); nothing drives a real +engine with `policy=kPriority` and compares its token stream against the fcfs +arm. G1 as written here cannot be "rerun"; a next owner writes it first, +RED-first, then runs it. The blocker both W1 and W4 still record — *"GPU +held by the `SERVE-GATE-ONLINE` campaign"* — is a scheduling note from +2026-07-10 that five successive bindings have since expired; it is not what is stopping +anyone. + +**W1 `ENG-CORE-BUSY-LOOP`'s GPU gates are covered in substance, unrecorded on the +row.** G1 (token-exact twins): `tests/parity/test_qwen36_async_serving.cpp` +(`1718bf155`, 2026-08-05) drives `LoadedEngine::async_engine()` → `AsyncLLM` → +`EngineCoreProc::step_with_batch_queue` (depth-2) token-exact against the same +pinned oracle continuation the SACRED sync gate uses, proven RED→GREEN on GB10 +under `compute-sanitizer`; `tests/parity/test_qwen3_dense_async_serving.cpp` +(`52d76f3a9`) adds the classic-dense arm. G4 (no-throughput-regression): the +`SERVE-GATE-ONLINE` binding `9ecd9d0` 114/124 runs exactly this path with async +default ON. Promoting the row is the operator's call, not this reconciliation's. + +**W2 `SERVE-ASYNC-LLM` is not waiting on a flip.** It holds at `GATING` on its +declared *broader every-axis parity*: 27B ratified (115/124 two-grid effective +parity), 35B open and owned by `ROAD-V1-A`. Its own open bug is +[#294](https://github.com/mudler/vllm.cpp/issues/294). The consumer dependency +recorded in [cuda-sglang-low-concurrency.md](cuda-sglang-low-concurrency.md) — +exit *"Real incremental SSE, in-flight batching, and streaming timing test +pass"* — was met by W2 on 2026-07-10; its "current endpoint buffers the whole +result" rationale describes the pre-W2 server, and the arrival-time assertion +that retired it is `tests/vllm/entrypoints/openai/test_api_server.cpp:622-634` +(`dispatched < first_arrival < completed`). + ## Current checkpoint and priority **W3 `ENG-ASYNC-SCHED` is DONE (2026-07-17): the async-scheduling default is FLIPPED ON** (`VT_ASYNC_RUNNER` default ON, mirror `vllm/config/vllm.py:992-1044`), DGX-re-confirmed token-neutral (235/235 + 315/315, log "enabled, mcb=2"), owner -`6ea7856`; rollbacks `VT_ASYNC_RUNNER=0` / `VT_ASYNC_SCHED=0`. See the 2026-07-17 +`6ea7856` (the SHA the discriminator ran on; the flip commit itself is +`a0013a2` — see the reconciliation above); rollbacks `VT_ASYNC_RUNNER=0` / +`VT_ASYNC_SCHED=0`. See the 2026-07-17 addendum below. The historical W3 landing narrative (machinery → runner halves → enable-flip → discriminator) is retained below for reference. diff --git a/.agents/specs/roadmap-v1-audit-2026-07-31.md b/.agents/specs/roadmap-v1-audit-2026-07-31.md index c3cc7114c..c16ec60fb 100644 --- a/.agents/specs/roadmap-v1-audit-2026-07-31.md +++ b/.agents/specs/roadmap-v1-audit-2026-07-31.md @@ -29,6 +29,7 @@ they rest on the ledger record of past GB10 runs. 2. **Gemma-4 image** — ledger implies STRICT 18/18; the dgx record is **16/18 near-tie** (2 bf16-near-tie punctuation diffs); **audio e2e is UNBUILT**. Row is RI/EXT, not DONE. 3. **ROAD-V1-A perf** — `ENG-ASYNC-SCHED` code is DONE-PROVEN, but the ROW is NOT: both SGLang-floor arms (`BACKEND-GATE-CUDA-SGLANG` BLOCKED on `SERVE-ASYNC-LLM` prod-ON; `-PREFIX` READY) have **never produced a binding run** (task #137 still pending); 27B is **114/124 "effective parity" via coin-flip totality**, not a clean 124/124; 35B c1/c2 low-conc still fail. 4. **C6 async/priority serving** — DONE marking not genuine: `SERVE-ASYNC-LLM` is **GATING** (engine-matrix.md:189), not prod-ON — the explicit blocker for the SGLang arm. + > **SUPERSEDED 2026-08-12 ([#534](https://github.com/mudler/vllm.cpp/issues/534)) — kept as the audit said it, do not act on it.** "Not prod-ON" reads a `GATING` lifecycle state as a production default. `SERVE-ASYNC-LLM` IS the production serving path (`src/vllm/entrypoints/openai/server_main.cpp:731-734`); it is `GATING` on broader every-axis parity, and the runner-side default that the punch-list conflates with it (`runner_supports_async`) has been ON since `a0013a2` (2026-07-17). The C6 row's one genuinely open leaf is `ENG-PRIORITY-SCHED`, whose priority-vs-FCFS gate does not exist yet. Item 3's SGLang clause carries the same misreading. Plus **D5 LoRA** (RI, not refuted-from-DONE but flagged): the headline LoRA is only a **standalone unwired CPU float-math brick** (`src/vllm/lora/punica_cpu.cpp` + `test_punica_cpu`) — no engine/runtime/endpoint integration. Most of D5 is unimplemented. ## Concrete debt / staleness found (fixable) diff --git a/.agents/specs/roadmap-v1-completion.md b/.agents/specs/roadmap-v1-completion.md index c8184459f..01c5fdc03 100644 --- a/.agents/specs/roadmap-v1-completion.md +++ b/.agents/specs/roadmap-v1-completion.md @@ -32,7 +32,7 @@ carry more than one class list the dominant one first. | Row | Class | Delivered + gated (anchor) | Remaining to close the row | |---|---|---|---| -| `ROAD-V1-A` perf/SGLang floor | **RI** | 27B **effective parity-or-better ratified** (two-grid totality 115/124: 110 pass-in-both + 5 coin-flip; residuals are the net-positive determinism tradeoff). `ENG-ASYNC-SCHED` DONE. | 35B every-axis closure (first binding 70/124, c4–c32 already win, c1/c2 residual — `BACKEND-GATE-CUDA-VLLM` PARTIAL); then the SGLang floor arms (`BACKEND-GATE-CUDA-SGLANG` BLOCKED on `SERVE-ASYNC-LLM` prod-ON; `-PREFIX` READY). | +| `ROAD-V1-A` perf/SGLang floor | **RI** | 27B **effective parity-or-better ratified** (two-grid totality 115/124: 110 pass-in-both + 5 coin-flip; residuals are the net-positive determinism tradeoff). `ENG-ASYNC-SCHED` DONE. | 35B every-axis closure (first binding 70/124, c4–c32 already win, c1/c2 residual — `BACKEND-GATE-CUDA-VLLM` PARTIAL); then the SGLang floor arms (`BACKEND-GATE-CUDA-SGLANG` BLOCKED on ~~`SERVE-ASYNC-LLM` prod-ON~~ — **corrected 2026-08-12 ([#534](https://github.com/mudler/vllm.cpp/issues/534)): no such flip exists; the exit criterion that row attached to `SERVE-ASYNC-LLM` was met by W2 on 2026-07-10 and the residual dependency is its own exact-equivalence preflight** — ; `-PREFIX` READY). | | `ROAD-V1-MM` multimodal | **RI** (+EXT sub) | **Image + video STRICT token-exact 32/32** on Qwen3.6-27B and Qwen3-VL-4B; **audio e2e** on Voxtral-Mini-3B (near-tie-robust; decoder 48/48). Correctness is the user's #1 priority and it LANDED. SPEED progress: tower lever #1 CLOSED (2114→148 ms, faster than vLLM eager encode); decode lever #2 CLOSED 2026-07-27 (on-GPU greedy argmax + no embed round-trip; bit-exact; 27B decode NEUTRAL at parity, audio ~0.4% win — multimodal-speed.md §8); lever #3 FIRST BRICK 2026-07-27 (the 27B image+video decode now routes through the production `Qwen3_5DenseDecodeGraph` captured decode = GRAPH-CAPTURABLE, token-exact 32/32 held, NEUTRAL at the 27B bandwidth floor — multimodal-speed.md §9); lever #3 **W1 LANDED 2026-07-27** (new `VoxtralDecodeGraph` graph-captures the Voxtral audio decode; bit-exact 14/14 held; A/B steady TPOT graphed 60.94 vs eager 61.71 ms/tok, non-overlapping — a small real win that NARROWS the audio gap 1.52×→1.49× but does NOT close it — multimodal-speed.md §10); decode-kernel efficiency ATTRIBUTED + VALIDATED ceiling 2026-07-27 (`CLAIM-MM-SPEED-DECODE-KERN`, multimodal-speed.md §11) — the whole ~20 ms/tok audio residual is the naive scalar `PagedAttentionKernel` decode attention (723 µs × 30 = 21.7 ms/step); the 1:1 vLLM lever (FA2 `flash_attn_varlen` decode) is already in-binary, gated off only because the driver's single KV block (444) isn't ÷16; `block_size÷16` → TPOT 59.4→38.2 ms/tok (−21.2, ~36%) = 0.94× vLLM 40.8 ms (BEATS parity), FA2 sequence teacher-force-VALID (0 divergences, gap 0.0), but it flips the committed near-tie golden → blocked byte-exact; RECORDS-ONLY, 14/14 held, win one `block_size÷16` + golden regen away. **ADOPTED 2026-07-27 (USER-APPROVED, `CLAIM-MM-SPEED-DECODE-KERN-ADOPT`, multimodal-speed.md §12): FA2 decode SHIPS as the Voxtral default — audio DECODE BEATS vLLM (0.97×), the LAST mm decode-speed gap CLOSED.** One-line `block_size÷16` (nsys: `flash_fwd_splitkv` 1410 @ 18.5 µs, zero `PagedAttentionKernel`); `test_voxtral_e2e` → ratified near-tie DISTRIBUTIONAL gate (binding = teacher-force PASS, kernel-independent; strict prefix exact to first bf16 tie, FA2 pos 18; determinism anchor to the FA2 seq); `voxtral_neartie.json` regenerated (md5 `937b9ad3…`), STRICT greedy golden UNCHANGED; gate PASS 16/16; teacher-force vLLM 0.25.0 = 0 divergent, gap 0.0, PASS; capture-safe (graph 46 replays + compute-sanitizer 0 errors + 3-run byte-identical) ⇒ DEFAULT graph path; A/B scalar 60.50 → FA2 **39.50 ms/tok** (−21.0, ~35%, NON-OVERLAPPING) = 0.97× vLLM 40.8. Audio DECODE now correctness+speed DONE; umbrella MM row stays PARTIAL (audio TTFT/encoder + c2+ batched serving). **ENCODER TTFT MEASURED + warp-attention brick 2026-07-27 (`CLAIM-MM-SPEED-AUDIO-ENC`, multimodal-speed.md §13):** routed the Whisper encoder self-attention (hd-64, non-causal) from the naive O(t²) `kAttention` to the warp-scoped `vt::AttentionDenseFast` (§7 tower fix; text byte-identical) — encoder forward **8870→1890 ms (4.7×, NON-OVERLAPPING)**, `test_voxtral_e2e` **16/16** with ZERO token flips, goldens md5 unchanged; **NOT at parity** (~1.89 s vs vLLM 43 ms, ~44× — the warp kernel is STILL 31.8 ms/layer, O(t²) memory-bound), closing needs a flash-TILED non-causal hd-64 attention (LARGE) + resident one-time encoder weights (MEDIUM, byte-exact). | Every-axis **SPEED** gate on all mm rows — none is DONE (`MODEL-MM-*` all PARTIAL/ACTIVE, speed-pending). Dominant residual = **lever #3: batched/graphed mm serving (c2+)**; the audio decode-KERNEL residual is now FULLY ATTRIBUTED (the scalar `PagedAttentionKernel`) and the fix is a VALIDATED bf16-near-tie ceiling (FA2 decode beats vLLM but changes the golden's near-tie branch — §11). **W1 REFINED the audio attribution:** graphing the Voxtral decode removed the per-step launch overhead but it was only ~1.25% of TPOT, so the ~20 ms/tok gap vs vLLM's 40.8 ms is per-step COMPUTE (the scalar decode-attention kernel), NOT launch overhead as §9.5 hypothesized. The audio decode win is now ADOPTED (FA2 SHIPS, §12 — audio decode BEATS vLLM 0.97×). Remaining lever-#3 W-plan (multimodal-speed.md §9.5/§13): audio TTFT NOW MEASURED our-side (32-layer Whisper encoder forward 1890 ms after the §13 warp-attention brick, still ~44× vs vLLM 43 ms — a flash-TILED non-causal hd-64 encoder attention + resident one-time encoder weights close it); W2 batched multi-seq (c2+); W3 `image_url`/`audio_url` serving ingestion. Gemma-4 mm/audio = EXT (below). Qwen3.6-35B mm needs a vision-inclusive checkpoint download + M2/M3 tower attach. | | `ROAD-V1-C1` extensibility | **DONE** (cornerstone) | Drop-in kernel ABI W0, Platform seam, model self-registration, and the **portable op-fusion framework ORDER-1 milestone** (W0–W4 merged+gated, `KERNEL-FUSION-FRAMEWORK`); consistency-audit CI check landed. `BACKEND-ABI-VT`/`BACKEND-CUDA-ARCH-ADDITIVITY` seams gated on sm_121a. | Row stays SPIKE-open only for **non-blocking** tail: Tier-1 fusion perf interpreter (composite-only → single-launch), `FUSION-DENSE-MIGRATE` (route 5 drift models — CLOSED 2026-08-10, [#299](https://github.com/mudler/vllm.cpp/issues/299)), a real Metal/Vulkan catalog realization (M4-reachable / HW-blocked), and migrating a production kernel family onto the common adapter. Correctness cornerstone is closed. | | `ROAD-V1-C2` model families | **RI** (+HW/EXT sub) | First additive model (Qwen3 dense) + a broad **text sweep correctness-complete + SACRED-gated**: Qwen3/Qwen3Moe/Coder, Llama/Yi/InternLM3, Mistral, GLM-4-9B/GLM-4.7-Flash, Gemma-1/2/3, OPT, DeepSeek-V2-Lite (MLA), OLMo-2, Phi-3/4, Phi-1/2, Granite-3, StableLM, InternLM2, MiniCPM, MiniCPM3 (MLA). 20 ACTIVE model rows. | **SPEED close** on every one (all 20 are "correctness-complete, speed pending"). MoE/SSM breadth (Qwen3-Next, Falcon, Falcon-H1, GraniteMoe*, Cohere2Moe, PhiMoE, Mamba/Jamba/Zamba2/NemotronH) = RI (INVENTORIED/SPIKE). Frontier: Kimi-Linear-48B fits (RI, +KDA kernel); DeepSeek-V3/GLM-5/MiniMax-M2/M3/Kimi-K2 = HW (>119 GiB); Command-R = EXT (HF token). | @@ -40,7 +40,7 @@ carry more than one class list the dominant one first. | `ROAD-V1-C3` spec-decode | **DONE** (core) | **MTP k=1 DONE + gated on BOTH gate models** (`SPEC-MTP`, c1 token-exact + above vLLM, c2–c8 on-par-or-above); **DFlash DONE + speed gate MET** (`SPEC-DFLASH` D14, our-ON ≥ vLLM-ON). | Named tail only: DSpark (`SPEC-DSPARK`) + heterogeneous-vocabulary TLI (`SPEC-TLI`) unspiked — overlaps `ROAD-V1-D3`. Core spec-decode is gate-closed. | | `ROAD-V1-C4` quantization | **RI** | **3 schemes DONE**: NVFP4-MO-W4A16, NVFP4-CT-W4A4, FP8-MO-STATIC (all R/M/C/E/P). **GGUF CPU vs llama.cpp is CLOSED** (2026-07-22, aarch64 binding host): decode **at parity** (1.03× behind, inside llama.cpp's ±1.8% run spread — the elementwise f16/bf16 GEMM lever `KERNEL-GEMM-CPU-ELEM` E1-E4 `18094ee2` took it 3.38×→1.03×), prefill **1.18× ahead** (q8_0 repack-at-load G7), RSS **1.01×**, byte-identical greedy tokens. | NVFP4-CT-W4A16 perf gate. FP8-generic dispatch (static/dyn × tensor/channel/token/block). Breadth: AWQ/GPTQ/Marlin-wiring, i-quants, MXFP4/MX, bitsandbytes, KV-quant — all INVENTORIED. (GGUF-vs-llama.cpp speed is no longer an open C4 blocker.) | | `ROAD-V1-C5` sliding/YaRN | **RI** | Joint spike accepted; all W1–W8 leaves implemented and CPU/oracle/sanitizer green. **CUDA GPU CLOSURE 2026-07-27 (`CLAIM-ROADMAP-C5`, dgx GB10 sm_121a, clean build of `489f7771`, oracle vLLM 0.26.0.dev0):** shared scaled-RoPE + local-mask CUDA path compiles `-Werror`-clean + RUNS on GB10; feature-positive correctness gates PASS — SWA Gemma-2/Gemma-3 48/48, LongRoPE Phi-4-mini 16/16 (RED-first), llama3 Llama-3.2-1B 16/16, dynamic-NTK InternLM2 16/16; both RoPE 0.26-oracle recaptures BIT-IDENTICAL to goldens. `ATTN-SLIDING-WINDOW`/`ATTN-ROPE-{LLAMA3,LONGROPE,DYNAMIC-NTK}`/`ATTN-YARN` → `ACTIVE`; `ATTN-CHUNKED-LOCAL` + `KV-*-SPEC` honest. | **Honest residual (vehicle-blocked, not skipped):** YaRN model e2e (no cached Nomic/gpt-oss consumer) + chunked-local model e2e (no Llama4 row) REACHABLE-BLOCKED; long-context positive-mask (prompt > W) SWA e2e + KV-memory G8; every-axis **SPEED** tail (all leaves correctness-complete, speed-pending). | -| `ROAD-V1-C6` async/priority serving | **RI** | `ENG-ASYNC-SCHED` **DONE** (`6ea7856`, default-ON, DGX token-neutral). W1/W2/W4 landed. | `SERVE-ASYNC-LLM` (GATING → prod-ON, blocks the SGLang floor + `ROAD-V1-A`), `ENG-PRIORITY-SCHED` + `ENG-CORE-BUSY-LOOP` GPU gates (GATING, held behind SERVE-GATE-ONLINE). | +| `ROAD-V1-C6` async/priority serving | **RI** | `ENG-ASYNC-SCHED` **DONE** (lever `6ea7856`; the default-ON flip is `a0013a2`, DGX token-neutral 27B 235/235 + 35B 315/315). W1/W2/W4 landed; the AsyncLLM depth-2 serving path is the production path and has carried its own GPU token-exact gate since `1718bf155`. | **CORRECTED 2026-08-12 ([#534](https://github.com/mudler/vllm.cpp/issues/534)); the superseded text read** ~~"`SERVE-ASYNC-LLM` (GATING → prod-ON, blocks the SGLang floor + `ROAD-V1-A`), `ENG-PRIORITY-SCHED` + `ENG-CORE-BUSY-LOOP` GPU gates (GATING, held behind SERVE-GATE-ONLINE)"~~. There is no prod-ON flip left: `SERVE-ASYNC-LLM` IS the production serving path (`server_main.cpp:731-734`) and `runner_supports_async` has been default-ON since 2026-07-17. `SERVE-ASYNC-LLM` holds at `GATING` on its declared broader every-axis parity — 27B ratified (115/124 two-grid effective), 35B open under `ROAD-V1-A` — plus open bug [#294](https://github.com/mudler/vllm.cpp/issues/294). `ENG-CORE-BUSY-LOOP`'s G1/G4 are covered in substance (`1718bf155`/`52d76f3a9`, binding `9ecd9d0`); the one genuinely open leaf is `ENG-PRIORITY-SCHED`, whose priority-vs-FCFS gate does not exist yet in `tests/`. See §3 item 9. | | `ROAD-V1-C7` sampling/logprobs | **RI** | **W1-W4 LANDED + CPU-GATED 2026-07-27 (`CLAIM-ROADMAP-C7`, NOT pushed):** `SAMPLE-CORE` + `SAMPLE-LOGIT-FILTERS` -> `ACTIVE` — the full sampling-control surface (temperature/top_p/top_k/min_p/penalties/seed/stop/min_tokens/logit_bias/allowed_token_ids/bad_words/logprobs-count) WIRED end-to-end params->protocol->InputProcessor->InputBatch->SamplingMetadata->Sampler + gated exactly on the CPU reference backend (RED-first; default/greedy byte-identical). | **RESIDUAL:** W5 `SAMPLE-LOGPROBS` payload end-to-end (LogprobsProcessor + OpenAI `CompletionLogProbs` serialization, PARTIAL — count wired, sampler produces tensors) + `SAMPLE-PROMPT-LOGPROBS` need the engine-output plumbing + a running-engine gate; then `n>1`, philox-RNG, logprobs_mode, beam-search, custom logits processors (INVENTORIED). | | `ROAD-V1-C8` tokenize/parse/metrics | **RI** | Tokenizer engines have code (`LOAD-HF-BPE` ANCHOR-BACKFILL, `LOAD-SENTENCEPIECE` ACTIVE 6/6). **`SERVE-METRICS` + `SERVE-UTILITY-ENDPOINTS` LANDED + CPU-GATED 2026-07-27 (`CLAIM-ROADMAP-C8`):** the **oldest open T0 debt** `/metrics` Prometheus is CLOSED — self-contained registry + text-0.0.4 exposition + always-on vLLM metric catalog (names/labels/buckets 1:1), gated by the vLLM scrape spec `EXPECTED_METRICS_V1` (RED-first, `test_prometheus_metrics` 4/4/81); plus `/tokenize`,`/detokenize`,`/ping`,`/server_info`,`/reset_prefix_cache` (`test_openai_api_server` 26/26/277). **`TOOLS-STREAMING-PARSER` engine core LANDED + CPU-GATED 2026-07-27 (`CLAIM-ROADMAP-C8-PARSER`):** vLLM 0.26 declarative `StreamingParserEngine` + qwen3/seed_oss/kimi_k2 configs + unified registry, EXACT event-for-event gate (`test_streaming_parser_engine` 586/586). **Parser ASSEMBLY layer LANDED + CPU-GATED 2026-07-27 (`CLAIM-ROADMAP-C8-ASSEMBLY`):** vLLM 0.26 `ParserEngine` (SemanticEvent -> streaming `DeltaMessage` + one-shot `ExtractedToolCallInformation`) + qwen3/seed_oss/kimi_k2 assembled parsers + `parser_manager` dispatch, EXACT field-for-field gate (`test_parser_engine_assembly` 9 scenarios, 1652/1652, RED-first 32 asserts). **Serving-SSE dispatch swap LANDED + CPU-GATED 2026-07-27 (`CLAIM-ROADMAP-C8-SERVING`):** the OpenAI chat streaming path routes engine-backed `--tool-call-parser` names through `parser_manager get_parser_engine` (drives `parse_delta`/`parse`), EXACT chunk-for-chunk gate vs vLLM 0.26 `chat_completion_stream_generator` (`test_openai_serving_chat_stream` 9 scenarios, 210/210, RED-first 6 CHECKs); OFF by default, legacy seam byte-identical. **5 more engine CONFIG families LANDED + CPU-GATED 2026-07-27 (`CLAIM-ROADMAP-C8-CONFIGS`):** minimax_m2/glm47_moe/deepseek_v4/deepseek_v32/nemotron_v3 as additive `ParserEngineConfig` builders + regex arg-converters + `Glm47MoeParser` name-`.strip()`, EXACT field-for-field (`test_parser_engine_assembly` 19 scenarios, 3510/3510, RED-first 2 asserts). **LAST 2 CONFIG families gemma4 + inkling LANDED + CPU-GATED 2026-07-27 (`CLAIM-ROADMAP-C8-CONFIGS-2`) — vLLM tool-parser family parity CLOSED:** 4 default-inert assembly-core virtual seams (`preprocess_feed`, virtual `events_to_delta`/`single_pass_parse`/`reset`/`extract_reasoning`, `args_wrapper_keys`) + `gemma4_config`/`inkling_config` + `Gemma4Parser` (channel-injection + `thought\n`-strip) / `InklingParser` (args-key unwrap + trailing-text flush), EXACT field-for-field (`test_parser_engine_assembly` 26 scenarios, 4526/4526, adds a non-streaming parse() gate, RED-first for all 4 seams; engine-core 586/586 + serving-SSE 210/210 byte-identical). **LIVE PER-STEP METRIC WIRING LANDED + CPU-GATED 2026-07-27 (`CLAIM-ROADMAP-C8-METRICS-WIRE`) — oldest T0 metrics debt RETIRED:** `/metrics` serves LIVE values — `Scheduler::make_stats()`→`EngineCoreOutputs.scheduler_stats` + `OutputProcessor`-built `IterationStats` fold into the logger at the `LLMEngine` step site (`llm_engine.py:308-329`); behavioural CPU gate `test_llm_engine.cpp` case 6 (44 asserts, RED-first 14 flip 0→correct) — gauges track the batch, token counters == exact counts, request_success + TTFT/ITL/e2e/TPOT/iteration histograms correct; catalog gate 4/4/81 + greedy stream unchanged. **PER-REQUEST TIMING via EngineCoreEvents LANDED + CPU-GATED 2026-07-27 (`CLAIM-ROADMAP-C8-RESPONSE-METRICS`) — `SERVE-RESPONSE-METRICS` INVENTORIED→ACTIVE:** the scheduler records QUEUED/SCHEDULED/PREEMPTED `EngineCoreEvent`s 1:1 vLLM (gated on `log_stats_`), drained onto `EngineCoreOutput.events`; `OutputProcessor.update_from_events` fills `FinishedRequestStats.{queued,prefill,inference}_time` + `IterationStats.num_preempted_reqs`, so `vllm:request_{queue,prefill,inference}_time_seconds` + `vllm:num_preemptions_total` carry real durations (the live-metrics wiring left them at 0); `test_scheduler` +1/15 (RED-first, real KV-exhaustion preemption) + `test_llm_engine` +1/26 (RED-first: 5 flip 0→positive), no-logger path byte-identical. **CHAT-FORM `/tokenize` LANDED + CPU-GATED 2026-07-28 (`CLAIM-C8-CHAT-TOKENIZE`):** `/tokenize` now accepts BOTH arms of the `TokenizeRequest` union — the chat form (`TokenizeChatRequest{messages, add_generation_prompt, continue_final_message, add_special_tokens, tools?}`) renders through the SAME chat template `create_chat_completion` tokenizes through (`chat_.prompt_fn()`, no reinvention), applies `check_generation_prompt` (→400), tokenizes with the chat-form `add_special_tokens` default False, returns `{count,max_model_len,tokens,token_strs?}` identically to vLLM 0.26 `serving_tokenization`; exact-gated tokens == render→Encode, RED-first (`test_openai_api_server` 27/27/337). **JSON-SCHEMA ARG-TYPE COERCION LANDED + CPU-GATED 2026-07-28 (`CLAIM-C8-ARG-COERCION`):** `_fix_arg_types`/`_streamable_string_keys`/`find_tool_properties` (parser_engine.py:227,269,365,348) ported over the already-ported `extract_types_from_schema`/`coerce_to_schema_type` helpers — `ParserTool` carries the function `parameters` schema (threaded from `serving_chat.cpp` `ToParserRequest`), so tools declaring typed params get their assembled `tool_calls[].function.arguments` coerced to int/number/bool/string/array/null in BOTH streaming and one-shot (priority null>int>number>bool>object>array>string, uncoercible left as-is); no-schema/absent-tools identity byte-identical. EXACT field-for-field (`test_parser_engine_assembly` 30 scenarios, 5038/5038, scenarios 27-30 qwen3 typed-schema whole+char + schema-mismatch + kimi JSON-native `"5"`->int, RED-first 38 asserts; engine-core 586/586 + serving-SSE 210/210 byte-identical). | **ASYNC SERVING-PATH METRIC WIRING LANDED + CPU-GATED 2026-08-10 (`CLAIM-SERVE-METRICS-ASYNC`, #277) — punch-list item 7 CLOSED:** `AsyncLLM::RunOutputHandler` folds each step's `SchedulerStats` + `IterationStats` into the attached logger, `EngineCore::step_with_batch_queue` stamps the `scheduler_stats`/`timestamp` upstream stamps in the shared `update_from_output` path, the logger takes a leaf mutex for scrape/record overlap, and `server_main.cpp` attaches one logger to both frontends; RED-first on three gates, CPU `ctest` 366/366. Residual: the chat/completion RESPONSE-BODY timing surface + CLI validation; config-gated families (spec-decode/kv-connector/mm/LoRA); `chat_template_kwargs`/`continue_final_message` full render passthrough. | | `ROAD-V1-C9` recurring sync | **RI** (recurring) | Pin **advanced to `555967922` / vLLM 0.26.0.dev0 + transformers 5.14.1** (`CLAIM-PIN-ADVANCE-W5`); re-gate 296/299 GREEN, zero golden drift. This unblocked OLMo-3 W5, DFlash, Gemma-4 module. | Refresh exact performance denominators + target goldens/tests on 0.26; then the ongoing mechanical cycle. Never terminally "done" (recurring). | @@ -106,8 +106,16 @@ gate → size (S/M/L) → vehicle model. `[H]` = user-directed headline. logprobs_mode. **Gate:** end-to-end parity vs vLLM payloads on the server. **Size M.** **Vehicle:** any gated dense model via the OpenAI server. 5. **`ROAD-V1-A` 35B closure + SGLang floor** `[H]` (27B already ratified). W: close the - 35B c1/c2 residual (merged-projection fp8 glue + aux-stream slices); flip - `SERVE-ASYNC-LLM` prod-ON to unblock the SGLang arm. **Gate:** 35B every-axis ≥ vLLM; + 35B c1/c2 residual (merged-projection fp8 glue + aux-stream slices); ~~flip + `SERVE-ASYNC-LLM` prod-ON to unblock the SGLang arm~~ — **CORRECTED 2026-08-12 + ([#534](https://github.com/mudler/vllm.cpp/issues/534)): there is no such flip.** + `SERVE-ASYNC-LLM` is already the production serving path, and the exit criterion the + SGLang spike actually attached to it — *"Real incremental SSE, in-flight batching, + and streaming timing test pass"* + ([cuda-sglang-low-concurrency.md](cuda-sglang-low-concurrency.md) §Non-overlapping + work breakdown) — was met by W2 on 2026-07-10. Its "current endpoint buffers the + whole result" rationale describes the pre-W2 server. The SGLang arm's remaining + dependency is its own exact-equivalence preflight, not ours. **Gate:** 35B every-axis ≥ vLLM; SGLang preflight equivalence then the binding SGLang + prefix arms. **Size M→L.** **Vehicle:** Qwen3.6-35B-A3B-NVFP4. 6. **`ROAD-V1-D3` ngram + EAGLE3 — DONE 2026-07-27.** `SPEC-NGRAM` (draft-free @@ -164,10 +172,57 @@ gate → size (S/M/L) → vehicle model. `[H]` = user-directed headline. MXFP4/MX; bitsandbytes; KV-quant. **Gate:** e2e correctness + every-axis perf per scheme. **Size M (FP8-generic) → L (breadth).** **Vehicle:** FP8 checkpoints on DGX; AWQ/GPTQ checkpoints for the breadth wiring. -9. **`ROAD-V1-C6` SERVE-ASYNC-LLM + priority prod gates** (shared with A#5). W: land the - depth-2 throughput lever, flip `runner_supports_async` prod-ON, close priority/busy- - loop GPU gates. **Gate:** every-axis no-regression + priority-vs-FCFS token-exact. - **Size M.** **Vehicle:** 27B online serving. +9. **`ROAD-V1-C6` SERVE-ASYNC-LLM + priority prod gates** (shared with A#5). + ~~W: land the depth-2 throughput lever, flip `runner_supports_async` prod-ON, close + priority/busy-loop GPU gates.~~ **SUPERSEDED 2026-08-12 + ([#534](https://github.com/mudler/vllm.cpp/issues/534)) — two of the three clauses + were already stale on the day this spec was committed (`489f7771`, 2026-07-27), and + the first asks for a win the project had already proven does not exist.** The + struck text is kept because it is what the plan said; do not act on it. + - **"land the depth-2 throughput lever" — LANDED 2026-07-16 AND REFUTED.** `6ea7856` + implemented `AsyncOutputPool` (persistent sampled-id/pinned buffers + `Sampler` + greedy scratch, removing every per-step `cudaMalloc`/`cudaFree`/`cudaHostAlloc`/ + event-create from the sampled-id path, incl. the overlap-killing `cudaFree` inside + `get_output`) and its own title records the verdict: `lever REFUTED as tput + unlock` — c16 tput **−0.32%** against a `≥+1.5%` gate, because the removed syncs + are O(10–100 µs) of a ~165 ms step. The next day `CLAIM-W3-ASYNC-DISC` measured + vLLM's OWN async self-A/B at **−0.66 to −0.91%** throughput and found upstream + ships it as the default regardless, so [async-serving.md](async-serving.md) + records *"There is NO depth-2 throughput unlock to find"* and retires the + speed-credit floor. Acting on this clause re-opens a closed negative result. + - **"flip `runner_supports_async` prod-ON" — FLIPPED 2026-07-17,** `a0013a2`, via the + CPU-unit-tested `AsyncRunnerFlagIsOn` predicate + (`include/vllm/v1/worker/gpu/async_runner_flag.h:38`): with no environment set + `runner_supports_async()` is TRUE and `LoadedEngine` resolves an `AsyncScheduler` + + `max_concurrent_batches=2`. DGX-proven token-neutral (27B 235/235 + 35B 315/315, + default and both rollback arms bit-identical). **The item's own title conflates + this with a different row:** `SERVE-ASYNC-LLM` is the AsyncLLM *streaming serving + API*, not the runner's async device path — and it is likewise already the + production path (`src/vllm/entrypoints/openai/server_main.cpp:731-734`, *"the + production server uses AsyncLLM over EngineCoreProc's dedicated engine thread"*, + capacity-fixed HTTP pool by default, `VLLM_CPP_HTTP_FIXED_POOL=0` diagnostic only). + No "prod-ON" flip is left under either reading; `SERVE-ASYNC-LLM` sits at `GATING` + on its declared **broader every-axis parity**, not on a default. + - **"close priority/busy-loop GPU gates" — HALF stale, HALF the one real residual.** + `ENG-CORE-BUSY-LOOP` G1 is covered in substance since `1718bf155` (2026-08-05): + `tests/parity/test_qwen36_async_serving.cpp` drives `LoadedEngine::async_engine()` + → `AsyncLLM` → `EngineCoreProc::step_with_batch_queue` token-exact against the same + pinned oracle continuation the SACRED sync gate uses (RED→GREEN on GB10), joined by + `test_qwen3_dense_async_serving.cpp` (`52d76f3a9`); G4 is covered by the + `SERVE-GATE-ONLINE` binding `9ecd9d0` 114/124, which runs that exact path with + async default ON. **`ENG-PRIORITY-SCHED` is genuinely OPEN:** + `--scheduling-policy priority` is plumbed to the production server + (`server_main.cpp:408-411,672-673`) but **no priority-vs-FCFS gate exists at the + engine/model level** — only scheduler-unit ordering cases + (`test_scheduler.cpp:674,916`, `test_request_queue.cpp`) and C-ABI wire-name + validation (`tests/capi/test_capi.cpp:1159`); nothing drives a real engine with + `policy=kPriority` and compares token streams. Both rows still record *"GPU held by the + `SERVE-GATE-ONLINE` campaign"*, a scheduling excuse that five successive bindings + (`3f256ab`, `246a23c`, `a875397`, `f0fb727`, `9ecd9d0`) have since expired. + **Remaining W (the whole of it):** write the priority-vs-FCFS gate that does not + exist, then run it. **Gate:** priority-vs-FCFS token-exact at equal admission set; + the every-axis no-regression half is the `SERVE-GATE-ONLINE`/`ROAD-V1-A` 35B closure, + not separate C6 work. **Size S→M.** **Vehicle:** 27B online serving. 10. **`ROAD-V1-C2` text-sweep speed close + MoE/SSM breadth** (20 families correctness- complete). W: per-family speed lever to vLLM parity; then Qwen3-Next / Falcon-H1 / GraniteMoe / Mamba-hybrid campaigns; Kimi-Linear-48B (+KDA kernel, disk reclaim). diff --git a/docs/FEATURES.md b/docs/FEATURES.md index c368654a2..cabe99416 100644 --- a/docs/FEATURES.md +++ b/docs/FEATURES.md @@ -37,7 +37,7 @@ are our reading of their documented behavior, not measurements. | Chunked prefill | ✅ | ✅ | ✅ | ☐ | | Automatic prefix caching | ✅ | ✅ | ✅ (radix) | ◐ | | Preemption and recompute | ✅ | ✅ | ✅ | ☐ | -| Priority scheduling | ◐ gating | ✅ | ✅ | ☐ | +| Priority scheduling | ◐ gating (`--scheduling-policy priority` reaches the server; scheduler-unit tests only, no engine-level priority-vs-FCFS gate exists yet, #534) | ✅ | ✅ | ☐ | | LPM cache-aware admission | ✅ | ☐ | ✅ | ☐ | | In-batch prefix de-prioritization | ✅ | ☐ | ✅ | ☐ | | Async / overlap scheduling | ✅ default on (UAF-safe drain; device token-ids mirror on gate + classic-dense; the decode graph declines while the mirror is live (#323 fix, eager fallback); opt-in `VT_ASYNC_EXECUTOR` out-of-capture H2D staging) | ✅ | ✅ | ☐ |