mudler · localai-bot · Aug 12, 2026
diff --git a/.agents/benchmark-record.md b/.agents/benchmark-record.md
@@ -9719,7 +9719,7 @@ M=$HOME/.cache/huggingface/hub/models--deepseek-ai--DeepSeek-V2-Lite/snapshots/6
 | fp8 cuBLASLt plan cache (`KERNEL-GEMM-FP8`) | **DONE / MEASURED NEUTRAL - bit-exact mirror, OPT-IN (`VT_FP8_PLAN_CACHE=1`), premise disproven** | The fp8 dense GEMM (`cuda_matmul.cu`) rebuilt the cuBLASLt descriptor + 3 TN layouts + heuristic every call; vLLM reuses an in-graph plan. Added a per-device `{desc,layouts,algo}` cache keyed on the full shape/config (`fp8_plan_cache.h`); BIT-EXACT (algo process-deterministic per shape - byte-exact cached==fresh `test_ops_fp8_cutlass` + CPU key/flag 4/4; 27B 235/235 + 35B 315/315 both flags). **Same-binary 35B A/B (one flock) is wall-clock NEUTRAL**: prefill TTFT in1024/c8 async-on ON 1491.5/OFF 1496.8, async-off ~1496.7/~1503.2; decode TPOT c1 15.16/15.14, c4 21.79/21.83. **nsys (async-off eager prefill): the pre-fp8-GEMM GPU gap is UNCHANGED - median 210 µs (off) vs 204 µs (on)** ⇒ the grounded "~0.8 ms removable gap" premise is NOT reproduced; the heuristic host cost is negligible/hidden (prefill GPU-bound; decode graph-captured so the heuristic runs at capture, not per replay-step). Kept opt-in for eager/non-graph regimes; default NOT flipped (faster unmet). Evidence `dgx:/tmp/fp8pc-*` + `~/work/prefill-attr-35b` | None - the vLLM-mirror lands opt-in; no default-flip and no speed credit |
 | Serving transport (TCP_NODELAY) | **DONE / MEASURED NEUTRAL on the gate workload** | Mirror landed (`SERVE-HTTP-TRANSPORT`): `set_tcp_nodelay(true)` matches vLLM's uvicorn/asyncio default; behavioral accepted-socket test RED **0** → GREEN **1**, 22/22 cases. The non-binding one-lock localhost A/B (`~/work/vllm.cpp-tcpnodelay-sizing/ff915e8…`, 4a450f9 Nagle-ON vs ff915e8 Nagle-OFF, c1/c2 ×2 reps, identical pinned-client workload; raw-set SHA `f5b52900…2128`) is **neutral within noise** on every ITL/TPOT/throughput metric (c1 mean ITL ~102.7 both arms; c2 ~108-109; first cold-start leg excluded). Mechanism: ~100 ms per-token write cadence vs µs loopback ACKs means Nagle never coalesces - the rescan's rank-1 gain hypothesis is REFUTED for the loopback gate; the mirror stays for real-network parity | None for the gate - the c2/c8 full-step attribution is complete (transport ruled out; c2 gap is GPU-busy kernel glue) |
 | Block-table host-cluster cleanup ([rescan](specs/rescan-lost-lanes-2026-07-16.md) §1,§5,§6) | **CPU-SIDE, BIT-IDENTICAL; `benchmark_binding=false`, no speed credit** - the payoff is measured by the dispatched correct-state c2/c8 full-step probe and the next authorized exact grid, not by these mirrors. **(c) LANDED** - `block_table.compute_slot_mapping` drops the dead tail-pad (`~2×(max_num_batched_tokens−total)` int64 writes/step; the decode graph re-pads via `BuildPaddedDecode`, the only other consumer slices `[0,total)`); test_block_table 11/11, test_prepare_inputs 6/6. **(d) LANDED** - the decode-graph capture-size set is DERIVED from `max_num_seqs` (`include/vllm/model_executor/models/decode_graph_sizes.h`, `DecodeGraphSizes`/`PadToCaptureSize`; mirrors vLLM `_set_cudagraph_sizes` reduced to the full-decode-cudagraph regime): `max_num_seqs=32` → `{1,2,4,8,16,24,32}` (adds the missing 24 bucket, drops the never-reachable 64; batches 17-24 stop over-padding to 32, +1 captured graph). CUDA-only; padding rows inert → token-exact. RED→GREEN `test_decode_graph_sizes` 5 cases/478 asserts (RED = 24 bucket absent under the old fixed set). **(e) LANDED** - `InputBatch::make_sampling_metadata` caches + rebuilds only on batch change (add/remove/condense/swap set a dirty flag), mirroring vLLM `refresh_metadata` (gpu_input_batch.py:812-830); deviation: the penalty-active path rebuilds every step (our port copies output_token_ids where vLLM holds a live ref), so the greedy/no-penalties gate gets the full win bit-identically. `scheduler.cpp:371,377` `std::move`s the `num_scheduled_tokens` map + `finished_req_ids` set (container plumbing, zero policy change; move-then-clear keeps observable behavior identical). RED→GREEN `test_input_batch` (+2 cases; RED = stale cache after a 2nd add returns size 1 != 2); `test_scheduler` 31/31, `test_runner` 228/228, `test_sampling_metadata` 6/6 unchanged. Items (a)-runner + (b) (GDN col-0) are in `runner.cpp` (async/GDN-claim owned) - reported, not touched. Landed `8a717b2`/`81afc36`/`0c4b41c` (merged `e027ad5`); clean full `-Werror` rebuild 0 warnings, CPU battery + tools 164/164 green. **DGX token-exactness gate PASSED** on the `e027ad5` build (GB10, one `flock /tmp/gpu` series, root `dgx:~/work/vllm.cpp-blocktable-gate`): 27B default **235/235**, 27B `VT_GDN_PACKED_DECODE=0` rollback **235/235**, 35B **315/315**, all exit 0 | **CLOSED - claim released.** No A/B by design (`benchmark_binding=false`, no speed credit); payoff measured by the c2/c8 attribution probe + the next authorized exact grid |
-| Host-weight ownership | **Established 27B binding memory axes remain PASS at `246a23c`. Local plain-BF16 4B remains `GATING`.** Current direct ON/OFF/stable-vLLM peak PSS is **2.406/8.592/7.662 GiB**, stable PSS **0.759/8.589/4.029 GiB**, and peak VRAM **12850.7/12843.3/12942.7 MiB**. Direct ON=OFF output IDs match 128/128 in every pair; direct loading cuts peak/stable PSS **72.0%/91.2%** and stays within the +8 MiB ON-vs-OFF VRAM gate. | H32 AOT, plain-BF16 graph and ratio-4 FA2 are complete. Current total throughput is **5769.99/5660.70/5849.80 tok/s** for ON/OFF/vLLM, so ON is **0.9864x** vLLM; TPOT/ITL **43.72 vs 38.55 ms** keeps speed open. Implement a compaction-safe device-resident sampled-token map for discrete CUDA, remove the immediate main-stream wait, then rerun the exact series. [Binding evidence](../docs/bench-evidence/qwen35-4b-main-repair-20260725.md). |
+| Host-weight ownership | **Established 27B binding memory axes remain PASS at `246a23c`. Local plain-BF16 4B remains `GATING`.** Current direct ON/OFF/stable-vLLM peak PSS is **2.406/8.592/7.662 GiB**, stable PSS **0.759/8.589/4.029 GiB**, and peak VRAM **12850.7/12843.3/12942.7 MiB**. Direct ON=OFF output IDs match 128/128 in every pair; direct loading cuts peak/stable PSS **72.0%/91.2%** and stays within the +8 MiB ON-vs-OFF VRAM gate. | H32 AOT, plain-BF16 graph and ratio-4 FA2 are complete. As of 2026-07-25 total throughput was **5769.99/5660.70/5849.80 tok/s** for ON/OFF/vLLM, so ON was **0.9864x** vLLM; TPOT/ITL **43.72 vs 38.55 ms**. **SUPERSEDED 2026-08-12 ([#527](https://github.com/mudler/vllm.cpp/issues/527)):** the "implement a compaction-safe device-resident sampled-token map for discrete CUDA, remove the immediate main-stream wait, then rerun the exact series" next-action is DONE and its premise REFUTED — the map landed `deed7c2a1` (2026-07-27) and is DEFAULT ON since `1718bf155` (2026-08-05), and `deed7c2a1` records that the 497-`cudaStreamSynchronize` attribution motivating the wait removal is wrong for this workload (`vllm-bench` drives the synchronous `LLMEngine::step()`, so `sample_tokens_async` never runs). The series was rerun through 2026-08-09: throughput **6831.71 vs 6643.40 tok/s (1.0283x PASS)**; TPOT/ITL **1.0165x** and peak VRAM **+118.7 MiB** remain the open axes. Next action is same-tool interval profiling to split the residual TTFT into intake vs prefill. [2026-07-25 binding evidence](../docs/bench-evidence/qwen35-4b-main-repair-20260725.md); [current sm_120 baseline](../docs/bench-evidence/qwen35-4b-sm120-main-20260807.md). |
 | Qwen3.6-35B-A3B performance | **BLOCKED / NOT RUN (grid crash FIXED)** | Correctness passes; no current v0.25.0 performance denominator exists. The 35B online-serving **c2+ crash that blocked the grid is root-caused + FIXED** (2026-07-18, `CLAIM-35B-GRAPH-SCRATCH-1`, [spec](specs/decode-graph-scratch-uaf-2026-07-18.md)): at concurrency > 1 the engine died with `cudaEventSynchronize: an illegal memory access`. cuda-gdb pinned the faulting kernel to `marlin_moe_wna16::Marlin<…><<<(144,1,1),(128,1,1)>>>` - its fp32-reduce scratch `c_tmp` (`EnsureCtmp`, `cuda_moe_marlin.cu`) is a grow-on-free per-stream buffer whose pointer is baked into the captured pure-decode CUDA graph; a bigger later prefill/decode `cudaFreeAsync`s it → the next graph replay reads freed memory (single-stream c1 never grows it → never crashed). Differential isolation: needs graphs ON + concurrency > 1 + long context; async/WMMA ruled OUT (async-off + wmma-off still 5/5 crash; graphs-off CLEAN); memcheck masks it. Fix = retire-on-grow (`RetireGraphScratch`, `src/vt/cuda/graph_safe_scratch.h`) across the four decode-graph-reached scratch allocators. Sweep c1-c16 pre-fix 5/5 crash → post-fix 0; 315/315 token-exact preserved. `benchmark_binding=false` (correctness fix, no speed credit) | Run the v0.25.0 performance grid (now unblocked) after 27B reaches 124/124 |
 | 35B FA2-prefill + fused-preamble lever | **LANDED / default-ON (2026-07-18, `CLAIM-35B-FA2-FLIP-1`); `benchmark_binding=false` (offline op A/B, no grid speed credit - the 35B grid re-measures in-situ)** ([spec](specs/qwen36-35b-fa2-prefill-oracle-2026-07-18.md)) | `FuseAttnPreambleOn` flipped default-ON all arches ⇒ the 35B ratio-8 full-attn layers take the exact `flash_fwd_splitkv` kernel (kernel-side `fa2_prefill` admits any GQA ratio at head_dim 256, `cuda_paged_attn.cu:2494`). FULL current-main default-set gate (async + GDN cubin + all fast kernels + flip): **35B `test_qwen36_paged_engine` 315/315 + 27B `test_qwen27_paged_engine` 235/235** (`dgx:/tmp/fa2gates_u.log`); **memcheck 35B prefill 0 errors** (`dgx:/tmp/fa2_memcheck2.log`); `test_ops_attn_preamble` 14/14. Realistic input-1024 TTFT A/B (same binary, FA2 default vs `VT_FA2_PREFILL=0`, conc8/num-prompts32, 3 interleaved on/off pairs + dropped warmup, `dgx:/tmp/fa2_ttft2.log`): **FA2-on Mean TTFT 824.7 ms vs off 874.4 ms = −5.7%** (median 638.6 vs 676.9 = −5.7%; prefill token-throughput 5170.5 vs 4902.5 tok/s = +5.5%; per-arm spread <7 ms, so the ~50 ms gap is well-separated). Below the ~7-9% offline-kernel target because the 1.86× attention-kernel win dilutes across the whole prefill (GEMM/MoE/GDN dominate); no decode/TPOT regression (prefill-only lever, both arms 315/315 token-exact). The "round normed q/k→bf16 before RoPE" tighten (`fused_qk_norm_rope.py:67`) was op-level bit-identical (fused-bf16 == unfused-bf16, q/k 0-mismatch) but flipped the 27B tok6 whitespace near-tie away from the pip-vLLM oracle (233/235) in COMBINATION (RMSNorm-saga) ⇒ NOT shipped; preamble ships UNTIGHTENED, both arches token-exact. CPU gate: clean `-Werror` rebuild 0 errors/0 warnings, full DGX ctest **156/157** (both engine gates green; the sole failure `test_capi` is a KNOWN pre-existing nondeterministic dgx-box detokenizer UTF-8 flake - 3 runs of the same binary give 3 different results - not a regression, unrelated to the attention path), tools unittest 164/164, checkers green | The 35B v0.25.0 perf grid (orchestrator-owned) re-measures the in-situ prefill/TTFT gain from this lever |
 | 35B FA2-decode lever (split-KV, ratio-8) | **LANDED / default-ON (2026-07-19, `CLAIM-35B-FA2-DECODE-1`); `benchmark_binding=false` (focused same-binary in-situ A/B - the 35B grid re-measures)** ([spec §35B ratio-8 extension](specs/fa2-gqa-split-kv-decode.md)) | Extended the ratio-6-only FA2 split-KV DECODE to the 35B ratio-8 (Hq/Hkv=16/2) hd-256 full-attn layers (new env `VT_FA2_DECODE_35B`, default ON). The old ratio-8 decode ran `PagedAttentionDecodeGqaKernel` at **grid=(num_reqs,num_kv_heads) = 2 blocks** at single-request decode (near-zero GB10 occupancy); the vendored `flash_fwd_splitkv` main+combine splits the KV dimension so the grid fills the machine. nsys `--cuda-graph-trace=node` (`test_qwen36_paged_engine`): clean 1:1 decode-kernel swap - OFF `PagedAttentionDecodeGqaKernel<...(int)8...>` ×300 (grid (1,2,1)=2 / (8,2,1)=16, no combine) → ON `flash_fwd_splitkv_kernel` ×300 + `flash_fwd_splitkv_combine_kernel` ×300 (split axis GridZ up to 16), old kernel absent. FULL current-main default-set token gate: 35B `test_qwen36_paged_engine` **315/315** + 27B `test_qwen27_paged_engine` **235/235**; operator `test_ops_paged_attn` **21 cases / 454,358 assertions** (adds ratio-8 parity ladder B∈{1,2,4,8,16} + ratio-4/window/toggle fallback); memcheck 35B decode **0 illegal-access errors** (315/315; `--leak-check full` leaks are engine exit-time model residency). IN-SITU A/B (35B NVFP4, input-1024/output-128, greedy, same binary `VT_FA2_DECODE_35B=1` vs `=0`, one flock, 4 interleaved ON/OFF pairs, first dropped, pooled pairs 2-4): **c1 Mean TPOT 14.96 vs 16.72 ms = −10.5%** (total tput 540.1 vs 489.3 = +10.4%); **c8 Mean TPOT 33.02 vs 34.12 ms = −3.2%** (tput 1835.1 vs 1785.7 = +2.8%); Mean TTFT neutral (c1 ~233, c8 ~826 ms both arms - decode-only lever). Per-arm spread <0.15 ms TPOT, well-separated. Token-exact + faster ⇒ default-ON. Evidence `dgx:~/work/vllm.cpp-35b-fa2-decode/{gpu_series,gates_engine,gates_default,memcheck35,nsys_ON,nsys_OFF}.log`. CPU gate: clean `-Werror` CUDA build 0/0, tools 164/164, checkers green; full DGX ctest `-j8` **157/160** - 3 non-numerics misses: `test_async_llm` (parallel-port flake, passes 1/1 isolated), `test_capi` (documented nondeterministic dgx detokenizer UTF-8 flake, fails even isolated, unrelated to attention), `test_qwen36_gguf_engine` (MEMORY-capacity artifact: loads TWO full 35B GGUF models sequentially; with FA2 decode ON the first runs 16/16 correct through the FA2 decode path then the second model load OOMs; `VT_FA2_DECODE_35B=0` rerun passes 2 cases / 28/28, so the added decode scratch tips an already-marginal ~119 GiB unified-memory box - NOT a correctness regression; the production safetensors NVFP4 35B is 315/315 ON - flagged for a decode-scratch-pool follow-up) | The 35B v0.25.0 perf grid (orchestrator-owned) re-measures the in-situ c1-c4 decode-TPOT gain; this directly targets the 35B c1 decode 0.810× low-batch residual |

diff --git a/.agents/coordination.md b/.agents/coordination.md
@@ -2213,6 +2213,18 @@ list of what the project supports.
   request-compaction-safe device token map, not retune FA2 blindly or delete
   the synchronization without preserving row identity.
   Evidence: [2026-07-25 4B repair](../docs/bench-evidence/qwen35-4b-main-repair-20260725.md).
+  **That next-owner directive is SUPERSEDED, 2026-08-12
+  ([#527](https://github.com/mudler/vllm.cpp/issues/527)) — the checkpoint text
+  above is left as written, but do not act on it.** The
+  request-compaction-safe device token map landed two days later in `deed7c2a1`
+  (2026-07-27, runner-owned device buffers + `LaunchApplyLastSampledOps` replaying
+  condense's row edits on-device) and is DEFAULT ON since `1718bf155`
+  (2026-08-05); the wait it was meant to repair does not exist on the benchmarked
+  path, because `vllm-bench` drives the synchronous `LLMEngine::step()` and never
+  enters `sample_tokens_async` — `deed7c2a1` records that refutation of the
+  497-`cudaStreamSynchronize` attribution itself. The exact series was rerun
+  through 2026-08-09 (**1.0283x PASS** on throughput). The row's live plan is
+  [the sm_120 Pareto campaign](specs/sm120-qwen35-pareto-2026-08-09.md).
 
 **DFlash D14 note (2026-07-27, `CLAIM-DFLASH-D14`, GPU-GATED on dgx, SPEED GATE MET → `SPEC-DFLASH` DONE, NOT pushed — FULL SHA reported to caller).** Closes `SPEC-DFLASH` (engine-matrix row `SPEC-DFLASH` + kernel-matrix `KERNEL-ATTN-DFLASH-PAGED-BLOCK` + model-matrix `MODEL-SPEC-qwen3-dflash-dflash-qwen3-for-causal-lm`) → `DONE`. Base `origin/main` `4657f6f3`, isolated worktree `/home/mudler/_git/wt-dflash-d14` (branch `dflash-d14`); dgx reused tree `~/work/dflash-d12/tree/build-cuda`, one `flock $HOME/gpu.lock`. An nsys of the graphed spec-on step attributed the D13 ~2% residual to the from-scratch draft-attention kernel; ported it to a WARP-scoped online-softmax variant (`DFlashPagedBlockAttentionWarpKernel`, 3.1× faster) → our-ON graphed 29.32 ≥ vLLM-ON graphed 29.240 (non-overlapping 3-rep bands, 1.003×). Correctness UNCHANGED (27/27 graph==eager, acceptance 19/39/29/25, 1629 accepted identical warp-vs-block, CUDA==CPU 795648/795648 + sanitizer 0); inertness SACRED 235/235 + MTP 9/9. **Owned files ONLY:** `src/vt/cuda/cuda_ops.cu` (additive: `DFlashPagedBlockAttentionWarpKernel` + `UseDflashAttnBlockKernel` + a launcher branch; the D12 block kernel retained under `VT_DFLASH_ATTN_BLOCK=1`) + the record surfaces (spec §0 D14 RESULT, engine-matrix, kernel-matrix, model-matrix + checklist rollup, roadmap, ledger, state, this note, README, `docs/BENCHMARKS.md`). **EXPLICIT NON-COLLISION:** does NOT touch the causal `kAttention`/`kPagedAttention`, any non-DFlash op, the runner, or the models — the warp kernel is reached ONLY via `DFlashPagedBlockAttention` (DFlash draft path). Roadmap's FINAL open speed item; SPEC-DFLASH is now correctness-complete AND at/above vLLM throughput.
 

diff --git a/.agents/roadmap_v1.md b/.agents/roadmap_v1.md
@@ -40,6 +40,7 @@ issue is not yet placed. Keyed record: update in place, never append.
 |---:|---|---|---|
 | [#168](https://github.com/mudler/vllm.cpp/issues/168) | `BACKEND-CUDA-SM110` | Jetson AGX Thor (sm_110): 32B NVFP4, Tekken tokenizer blocker | feature |
 | [#433](https://github.com/mudler/vllm.cpp/issues/433) | `BACKEND-GATE-CPU-LLAMACPP` | No x86_64 arm: the closed CPU floor is Arm/i8mm-only and every lever that closed it is Arm-specific | perf |
+| [#529](https://github.com/mudler/vllm.cpp/issues/529) | `BACKEND-GATE-CPU-LLAMACPP` | `test_cpu_x86_llamacpp_floor` fails on a BUSY host, so preflight can fail itself: the contended-leg test asserts exit 2 but the unpinned real-loadavg quiet window exits 4 first | bug |
 | [#199](https://github.com/mudler/vllm.cpp/issues/199) | `BACKEND-METAL-MLX` | macOS MLX build fails on `-Werror` in MLX headers | bug |
 | [#41](https://github.com/mudler/vllm.cpp/issues/41) | `BACKEND-ROCM` | ROCm (AMD GPU) backend | feature |
 | [#132](https://github.com/mudler/vllm.cpp/issues/132) | `BACKEND-ROCM` | ROCm `-O0` RmsNorm CLR HostcallListener teardown deadlock | bug |
@@ -118,6 +119,7 @@ issue is not yet placed. Keyed record: update in place, never append.
 | [#314](https://github.com/mudler/vllm.cpp/issues/314) | `ROAD-V1-C1` | `FUSION-DENSE-MIGRATE` glue half: `glm4`/`phi3` still hand-call add+RMSNorm instead of `vt::FusedChain` (split out of #299, which closed the merged-GEMM half only) | bug |
 | [#337](https://github.com/mudler/vllm.cpp/issues/337) | `ROAD-V1-C1` | `FUSION-DENSE-MIGRATE`: the five dgx SACRED paged-engine gates are OWED after the merged-GEMM fold (`test_{commandr,glm4,minicpm,minicpm3,phi3}_paged_engine` SKIP on a CPU box) | bug |
 | [#338](https://github.com/mudler/vllm.cpp/issues/338) | `ROAD-V1-C1` | MiniCPM/MiniCPM3 hard-code SiLU: upstream `MiniCPMMLP` (`minicpm.py:219-226`) selects `FatreluAndMul` on `hidden_act == "fatrelu"` and raises otherwise; our `parse_config` never reads `hidden_act` | bug |
+| [#527](https://github.com/mudler/vllm.cpp/issues/527) | `ROAD-V1-C2-LOCAL-BF16` | Punch-list item 14 directs work that landed 2026-07-27 (`deed7c2a1`, default-ON `1718bf155`), on a 497-`cudaStreamSynchronize` premise the same commit refuted | bug |
 | [#241](https://github.com/mudler/vllm.cpp/issues/241) | `ROAD-V1-H3` | MiniMax-H3: support the PRUNED (AdaLN timestep-curve) checkpoint variants | feature |
 | [#471](https://github.com/mudler/vllm.cpp/issues/471) | `GATE-PIN-UNPINNED-SNAPSHOTS` | Checkpoint gates resolve their snapshot UNPINNED: 56 of 61 take whatever `directory_iterator` yields, including the three DFlash gates on a repo caching two materially different revisions (spec `specs/gate-pin-unpinned-snapshots.md`) | bug |
 | [#472](https://github.com/mudler/vllm.cpp/issues/472) | `GATE-PIN-UNPINNED-SNAPSHOTS` | 80 of 92 committed goldens record no checkpoint revision, so their gates cannot be pinned from evidence (`q3mxfp4`, DFlash, every `*_greedy` corpus) | bug |