diff --git a/.agents/NOW.md b/.agents/NOW.md index 4c81af9f7..2e502634f 100644 --- a/.agents/NOW.md +++ b/.agents/NOW.md @@ -1,6 +1,6 @@ # NOW — the one-Read resume surface - + Snapshot, not log. History is git; evidence: [parity ledger](parity-ledger.md), and benchmarks. Budget: 100 lines / 6,000 diff --git a/.agents/coordination.md b/.agents/coordination.md index 3dc1288fa..4113e799c 100644 --- a/.agents/coordination.md +++ b/.agents/coordination.md @@ -1993,6 +1993,7 @@ this claim will meet. The tiled row is speed-gatable on dgx. | `CLAIM-ENG-LOAD-DIRECT-UPLOAD` | `ENG-LOAD-DIRECT-UPLOAD` (engine-matrix, NEW row -> `ACTIVE`) | Claude Code (opus-5) | isolated worktree `.claude/worktrees/agent-a122a5ddc3546bf31`; GB10 `dgx.casa` tree `~/work/load150/src` (Vulkan `build-vk` + CUDA `build-cuda`), every GPU stage under `flock $HOME/gpu.lock`, `local-ai-worker` parked and restored | branch `row/ENG-LOAD-DIRECT-UPLOAD`, base `origin/main` `375a471e` | Issue #150. Owns ONLY: the refcounted safetensors mapping + byte counters (`include/vllm/model_executor/model_loader/safetensors_reader.h`, `src/vllm/model_executor/model_loader/safetensors_reader.cpp`), `BorrowStTensorBytes` + the `AdoptDeviceBytesAsHost` mmap branch + `OwnedTensor::mmap_src` (`include/vllm/model_executor/models/qwen3_5_weights.h`, `src/vllm/model_executor/models/qwen3_5_weights.cpp`), the qualifying call sites in `include/vllm/model_executor/models/dense_weight_loaders.h` and `src/vllm/model_executor/models/qwen3_5_dense_weights.cpp`, the upload counter in `include/vllm/model_executor/models/dense_attn_block.h`, the `VT_LOAD_STATS` phase timing in `src/vllm/entrypoints/model_loader.cpp`, NEW `tests/vllm/test_load_direct_upload.cpp` + its `tests/CMakeLists.txt` line, NEW `.agents/specs/load-direct-upload.md`, the NEW engine-matrix row + its Loading/Total rollup, the `#150` roadmap intake cell, two `docs/ENVIRONMENT.md` rows, this claim, `.agents/NOW.md`, and the `docs/STATUS.md`/`docs/BENCHMARKS.md` one-liners. **NON-COLLISION:** no new kernel, no model registry change, no CMake target change beyond one test line; every non-verbatim loader helper is untouched and the lever is same-binary A/B-able (`VT_LOAD_DIRECT_UPLOAD=0`). MUST NOT touch README, Metal/SACRED/apex/darwin, or any model/kernel source | `ACTIVE` | 2026-08-09 — mechanism test 6/6 (77) and RED under two mutations; GB10 Vulkan gates on the changed tree green (`test_vulkan_backend` 35/35·2650, `test_backend_cross_device` 11/11·132, `test_opt_paged_engine` 6/6 prompts token-exact 96/96, 0 declines, device type 3); GB10 CUDA full `ctest` 383/393 with BOTH SACRED gates PASS and all 10 failures reproduced on a clean `origin/main` build; 27B bf16 load 1.54x warm / 1.61x cold, bytes moved 100.196 -> 81.260 GiB; `local-ai-worker` restored (`--restart=always`, Up). Fresh scoped review OWED | | `CLAIM-ENG-RELEASE-CONTAINERS-W1-W7` | `ENG-RELEASE-CONTAINERS` (`INVENTORIED`->`READY`->`ACTIVE`; spike spec + W1-W5/W7) | Claude Code (opus-5), helper role | isolated worktree `/home/mudler/_git/vllm.cpp-containers`; local Docker only -- NO GPU, NO registry push, NO publication | `row/ENG-RELEASE-CONTAINERS`, base `origin/main` `24306364`, PR [#307](https://github.com/mudler/vllm.cpp/pull/307) | Container spike spec, `docker/Dockerfile` (cpu/vulkan/cuda calling the release scripts), container matrix + checker, image validator, workflow + least-privilege guard, and the SIGTERM fix the boot gate surfaced (#312). Excludes W6 matching-hardware runtime evidence and any GHCR publication | `ACTIVE` | 2026-08-10 -- cpu lane built and gated e2e (config+layout+boot, clean SIGTERM 0.25 s, RED 137 before the fix); mutation suites 31/31 and 22/22; cuda/vulkan gated statically but never built here; nothing published | | `CLAIM-ROCM-GDN-KERNELS` | `BACKEND-ROCM` (stays `ACTIVE` — M3 GDN kernel slice, issue #41) | pi (kimi-k3) | isolated worktree `/home/vikash/vllm.cpp-rocm-gdn-kernels`, base `origin/main` `5812b8b6`; 4x RX 7900 XTX gfx1100 discrete, ROCm 7.14, GPU jobs hold `~/gpu.lock` | `row/ROCM-GDN-KERNELS` | The M3 GDN kernel family unblocking Qwen3.5-0.8B M2 on discrete ROCm. Owns ONLY: NEW `.agents/specs/rocm-gdn-kernels.md`; NEW `src/vt/rocm/rocm_gdn.hip` (10 ops: kGdnStateGather/Scatter 77/78, kCausalConv1dFwd/Update 5/6, kRmsNormGated 9, kSigmoidGateBf16 58, kGdnPrefill/Decode 10/11, kGdnPostConv 65, kAttnQkNormRopeGate 67 — hand-translated from `src/vt/cuda/cuda_gdn.cu`/`cuda_ops.cu` donors); ADDITIVE registration entries in `src/vt/rocm/rocm_ops.hip`; ADDITIVE GDN cases + buffer helpers in `tests/vt/test_backend_cross_device.cpp`; this claim + `.agents/NOW.md`. **NON-COLLISION:** every new file is ROCm-only; the two shared-file edits are additive registration/case blocks that change no existing behavior (unregistered-op backends skip; CUDA/CPU/Vulkan paths byte-identical). Disjoint from #234 (sampling), #317 (gemma4 FP8/WMMA), #273 (near-tie docs). | `ACTIVE` | 2026-08-11 — **ALL 5 FAMILIES LANDED AS STACKED DRAFT PRs + M2 E2E COMPLETE.** #334 (spec + state I/O), #336 (conv), #341 (postconv), #343 (recurrence), #345 (norm-gate/preamble + the SupportsCompressedGdnState capability seam). Qwen3.5-0.8B runs e2e all-native on discrete gfx1100, deterministic; 'The sky is' 8/8 CPU parity; open-prompt divergence characterized as not-state-rounding, GDN-path drift analysis named open. Gates per family: cross-device green (162/162 total), focused 3/3, full-suite zero-delta vs base. NEXT: fresh review + operator gate per PR; then the MoeRouterTopK bf16-logits arm (the named MoE-path blocker) and the GDN-path divergence characterization. | +| `CLAIM-ROCM-M4` | `BACKEND-ROCM` (stays `ACTIVE` — M4 correctness gate on gfx1100, issue #41) | pi (kimi-k3) | isolated worktree `/home/vikash/vllm.cpp-rocm-m4`, base `origin/main` `8b00f79f`; 4x RX 7900 XTX gfx1100 discrete, ROCm 7.14, GPU jobs hold `~/gpu.lock`; the pinned vLLM-ROCm oracle runs in the committed container image `vllm-rocm-oracle:555967922-gfx1100` | `row/ROCM-M4-GATE` | The M4 gate: the pinned vLLM-ROCm oracle on gfx1100 + the ROCm device-golden lane in the Qwen3-0.6B near-tie gate. Owns ONLY: NEW `.agents/specs/rocm-m4-oracle.md` (oracle build/capture/gate recipe); ADDITIVE `kROCM` arm in `tests/parity/test_qwen3_paged_engine.cpp` (the existing Metal/Tenstorrent device-golden lane extended, no CUDA/CPU/Metal/TT behaviour change); NEW goldens `tests/parity/goldens/qwen3_greedy_0_6b/{our_ids_rocm.npy,neartie_gap_mnats_rocm.npy,greedy_ids_rocm.npy,greedy_dist_rocm.npy}` (dgx base byte-untouched); this claim + `.agents/NOW.md` + `parity-ledger` row + `docs/STATUS.md`. **NON-COLLISION:** the gate edit is a three-branch ternary extension already established for two other devices; the goldens are additive files; disjoint from the in-flight kernel PRs (#497/#506/#509/#523) which touch no parity-gate file. | `ACTIVE` | 2026-08-12 — **M4 GATE LANDED GREEN.** Oracle built from source at pin `555967922` in `rocm/vllm-dev:base` (wheel `0.23.1rc1.dev1511+g555967922.rocm723`, `_rocm_C` ABI present; runtime identity string verified == the parity-pin block), K=10 per-prompt greedy capture DETERMINISTIC in every (prompt,pos) cell. Our engine vs the SAME-BOX oracle: **16/16 prompts PASS** (STRICT token-exact 11/16, near-tie-band 5/16, max gap 0.125 nats @ prompt[3] tok1, 0 forward-divergent; 125/125 assertions), BACKEND PROOF kPagedAttention selections 7,168 / 0 declines on device type 5. The known p0 France/Italy flip is a literal 0.0000-nat tie in the oracle's own logits (Metal-lane M3b signature, reproduced on ROCm). **0.8B GDN LANE LANDED AS THE RED ANCHOR (stacked branch `row/ROCM-M4-GDN-GATE`):** NEW `tests/parity/test_qwen35_paged_engine.cpp` — the oracle-backed near-tie gate for Qwen3.5-0.8B — **FAILS 13/16 at landing by design** (0/16 strict, 3/16 band, max teacher-forced gap 14.125 nats; first-token divergences on 6 prompts; backend proof GREEN: 15/15 GDN ops dispatch, 0 declines). The known 0.8B CPU/ROCm divergence is now oracle-measured as a REAL forward divergence, not a near-tie; the gate is the committed RED test the forward fix turns GREEN. Goldens carry manifest.json (oracle identity + pinned HF revision 2fc06364 via `parity::HfSnapshot`; check-snapshot-pins clean). NEXT: the 0.8B forward fix (the RED anchor's target), then the 35B keep-quant gate; fresh review + operator gate for both PRs. | **CLOSED same-session claim — `CLAIM-TP-SPIKE-287` (task #287, 2026-08-08, records-only, helper `row/SPIKE-TENSOR-PARALLELISM` draft PR #143, base `b38f78a7`).** Not an active-claims row because every referenced row keeps its diff --git a/.agents/parity-ledger.md b/.agents/parity-ledger.md index 68e733335..71d38d61b 100644 --- a/.agents/parity-ledger.md +++ b/.agents/parity-ledger.md @@ -940,3 +940,5 @@ Columns: | 2026-08-10 (**`FUSION-DENSE-MIGRATE` — the merged-GEMM allowlist's no-blocker half CLOSED**; issue [#299](https://github.com/mudler/vllm.cpp/issues/299), spec `specs/fusion-dense-migrate.md`; `row/FUSION-DENSE-MIGRATE`, base `origin/main` `c70f42b9` (rebased from the `e17e8f8a` pinned at worktree creation, via `688eea12`), isolated worktree `/home/mudler/_git/vllm.cpp-fusion-fold`; CPU-only, `benchmark_binding=false`) | Routes the gate/up of the five plain bf16 dense SwiGLU MLPs that bypassed the MUST-route merged-GEMM seam with no stated blocker — `commandr`, `glm4`, `minicpm`, `minicpm3`, `phi3` — through `layers::UnquantizedMlpGateUpMethod`, and deletes their five `pending FOLD-MIGRATE` entries from `scripts/merged-gemm-consistency-allowlist.txt` (11 → 6). The six survivors all name a blocker that needs the SHARED LAYER extended (f32 activations + rank-1 biases, ggml block-quant vs the explicitly UNQUANTIZED arm, up-front device staging vs `OwnedTensor`/`ResidentWeight` residency, a GeGLU arm on the SwiGLU-only grouped MoE op for `gemma4_moe`, an NVFP4-Marlin-resident arm for `laguna`'s raw-device-pointer w13) and were NOT touched: extending the seam is its own row. The `gemma4_moe` and `laguna` reasons were REWRITTEN in this branch: both previously read as a bare `pending fold`, which is the vague reason this row's own thesis says let the allowlist grow. Direct Unquantized arm, not `MakeMlpGateUpMethod` — no loader for the three `Qwen3DenseMlpWeights` models ever populates `*_fp4`. Shared-layer TUs (`linear.h`/`nvfp4.h`) and every `vt::` op UNTOUCHED. No env flag: the fold is unconditional and bit-exact, so a rollback arm would be dead code. | Upstream expresses all five as ONE `MergedColumnParallelLinear(gate_up_proj, output_sizes=[I]*2)` + `SiluAndMul()` at pin `555967922` (0.26.0.dev0): `commandr.py:91,102-108,116`; `glm4.py:46` → `llama.py:79,92-99,113`; `minicpm.py:193,204-211,219`; `minicpm3.py:186` (inherits `MiniCPMMLP`); `phi3.py:10` → `llama.py:79`. The fold moves us TOWARD that shape; no new upstream code was ported. | **BYTE-EXACT BY CONSTRUCTION, and that construction is what was gated** (a routing change, so the bar is op-sequence identity, never a tolerance). Each replaced body was the seam's own `{ResidentWeight; MatmulBT[2I,H]; SiluAndMul}` with `M` spelled `T`, and every call site passes a `DBuf{T,H}` so `x.shape[0] == T` identically. NEW byte-exact CPU case in `test_linear_method` for the DIRECTLY-constructed arm at BOTH the decode shape (M=1) and a prefill shape (M=4) vs the standalone sequence — RED-first PROVEN (mutating the seam's `SiluAndMul`→`GeluAndMul` fails 2 cases / 11 assertions; GREEN byte-for-byte on restore) → **6 cases / 76 assertions GREEN**. `check-fusion-consistency.py` RC=0 with 0 drift; two NEW mutation cases in `tests/scripts/test_check_fusion_consistency.py` make BOTH regressions RED (re-allowlisting a folded stem; reverting a fold) → **20/20**. MECHANICAL CORRECTION to issue #299: a fully-folded TU has NO hand-call left, so it leaves the detector's DENOMINATOR rather than entering its numerator — the checker reads `15 scanned / 6 routed / 11 allowlisted` → `10 / 6 / 6`, not the `11 routed / 15` the issue predicted; drift is 0 either way, which is what the gate asserts. Full CPU `ctest -j 6` on the final rebased base `60e71a0e`: **369/369, 0 failed**, 1288.75 s. RE-RUN on the MERGED head `4b99cefb` (after merging `origin/main` `5812b8b6` and adding the new coverage binary): **369/370, 1 failed**, 1441.85 s — the single failure is `test_engine_core_proc`, a STARVATION FLAKE and not a regression: serially on the same binary it is **10/10 cases, 93 assertions GREEN**, whereas under `-j 6` at load average ~170 (two other worktrees running their own suites) it failed in 0.06 s on `CHECK(abort_seen)` (`test_engine_core_proc.cpp:345`) having spun to 1089 assertions waiting for the abort. It is one of the four known starvation-prone binaries, loads none of the five folded TUs, and the other three passed in this run. `test_dense_gate_up_seam_forward` passed inside the suite (32.96 s), proving the new binary is registered and wired. An earlier run on the intermediate base read **367/369**, and both numbers are reported rather than the red one deleted: `test_async_llm` and `test_openai_conformance` failed while a SECOND worktree ran its own suite at load average 89-122, and re-run alone on a quiet box (load 12.8) passed in **0.04 s** and **20.76 s** (vs 605 s of `statuses == -1`, the client's 30 s read timeout, under load); `test_openai_conformance` builds a synthetic `Qwen3_5MoeForConditionalGeneration` in-process and loads none of the five folded TUs, so it could not be this row's, and the clean 369/369 settles it. The #274 ASan/UBSan five did not appear in either run — this gate is Release with no sanitizer. **OWED / NOT CLAIMED:** the five paged-engine SACRED gates (`test_{commandr,glm4,minicpm,minicpm3,phi3}_paged_engine`) are checkpoint-gated dgx-only and emit a loud SKIP on a CPU box, so empirical token-exact confirmation is PENDING for the next GPU holder — the same shape A1 carried for Granite/StableLM — and is tracked by [#337](https://github.com/mudler/vllm.cpp/issues/337) so the handle outlives #299. EXECUTED CPU COVERAGE OF THE FOLDED TUs (added on review finding F4, which proved that mutating `phi3`'s `I` to `I - 1` at the call site survived 176 CPU tests): NEW `tests/vllm/models/test_dense_gate_up_seam_forward.cpp` drives the REAL forward of four of the five folded TUs (`commandr`/`glm4`/`minicpm`/`phi3`) over synthetic in-memory weights with no checkpoint and no GPU — **4 cases / 1940 assertions GREEN** — pinning the gate/up split analytically (zeroing the UP half and zeroing `down_proj` both make the MLP contribute exactly zero, so they must be BYTE-IDENTICAL, which holds only at a split of exactly `I`) plus the half order and a vacuity guard. RED-first EXECUTED both ways: `I` → `I - 1` at all four call sites fails all four cases (`phi3` alone fails only its own, so each TU is genuinely reached), and a SHAPE-PRESERVING mutation of `vt::SiluAndMul`'s CPU kernel (`silu * up` → `silu + up`) fails all four on the split assertion itself while `test_linear_method` stays GREEN at 76/76 — a defect class the seam unit test structurally cannot see. Both restored byte-for-byte (md5 verified). This also retires the row's original "no e2e evidence is possible on a CPU box" framing as too strong: ORACLE evidence needs the GPU, SELF-CONSISTENCY evidence does not. `minicpm3` is the one folded TU that harness does not drive (MLA + load-time `kv_b_proj` absorption belong to the DeepSeek-V2 synthetic harness). Also REJECTED as evidence: an object-code A/B — compiling each of the five TUs from main's source and from the folded source with the identical production command and diffing the disassembly gives 3689-6888 differing instruction lines per TU, which is GCC re-allocating registers around a header-inlined method, not a numerical signal in either direction. STATED RATHER THAN HIDDEN: the OP sequence is identical but the ALLOCATION sequence is not — `DBuf` is pooled and returns its block on destruction (`dense_device_glue.h:99`), so the `[T,2I]` gate_up buffer is now released when `Apply` returns instead of at the end of the MLP block and the `[T,H]` output may reuse it. That cannot move a value (`vt::MatmulBT` writes every output element from a fresh f32 accumulator rather than accumulating into `out`) and peak pool usage is unchanged or lower; it is the same shape every earlier fold produced. ALSO REPOINTED: `glm4`/`phi3` sit on the OTHER (glue) allowlist reading `pending FUSION-DENSE-MIGRATE`; closing this row would leave both aimed at closed work, so their reasons now name [#314](https://github.com/mudler/vllm.cpp/issues/314), which owns the glue half — the glue fold itself is NOT done here. | | 2026-08-11 (`ENG-TRAILER-MERGE-ARTIFACTS`; implementation merge `157080c8`, issue [#406](https://github.com/mudler/vllm.cpp/issues/406)) | The trailer gate was rejecting CORRECT commits. `check-commit-trailers.py` reads through `git interpret-trailers --parse`, which treats only the FINAL paragraph as the block, and GitHub appends `Co-authored-by:` as a separate trailing paragraph on squash merge — so a complete block became invisible and 13 of the last 30 commits on `main` failed the check, unnoticed because those runs were cancelled (#274). Fixed by fusing consecutive trailing TRAILER-SHAPED paragraphs before parsing; no rule relaxed, and a prose paragraph still terminates the block. | `test_check_commit_trailers.py` 21/21 — the RED-BEFORE appended-co-author case plus four guards (doubled block, contradictory declarations, no-trailer merge message, prose after the block) green before and after; the five real `main` commits re-verified per commit and the verdict table reproduces. | Closes ONE of five shapes. `f64f2b71` (bot co-author) is a REAL violation the parse had hidden and now correctly fails; `87308dea`, `b8293c88` and `b580452d` stay red by design — merge-method artifacts, not checker defects. A first attempt that also collapsed identical duplicate trailers was REVERTED in full: it relaxed a uniqueness rule an existing test pins. | | 2026-08-12 (**Audio ENCODER TTFT — FA-2 TENSOR CORES for the hd-64 non-causal encoder attention; 5.50x encoder forward / 115.8x kernel, LANDS OPT-IN because it costs precision**; `CLAIM-MM-SPEED-AUDIO-ENC-FA2`, multimodal-speed.md §17, issue [#432](https://github.com/mudler/vllm.cpp/issues/432); worktree `/home/mudler/_git/vllm.cpp-mm-speed` branch `row/MM-SPEED-ENC-FA2` off `dc7a1392`) | New dense hd-64 non-split FA-2 instantiation (`flash_fwd_hdim64_bf16_sm80.cu`, upstream's own `run_mha_fwd_hdim64` via the plain batch entry — both firsts here) + `LaunchDenseFA2Bf16` (b=1, null `cu_seqlens_q`) + additive op `vt::AttentionDenseFa2`, routing the Whisper encoder self-attention to the kernel vLLM itself dispatches (`whisper.py:298-317` -> `flash_attn_varlen_func`). dgx GB10 sm_121a, all three mandatory banners CONFIRMED, `-Werror` 0-warn, GPU under `flock` on a CONTENDED box (3 other agents' jobs queued). **SPEED:** encoder attention 19,278 -> 166.5 us/layer (115.8x, nsys 32 inst both arms, SAME tool); encoder forward 731.7 -> 133.0 ms (5.50x, non-overlapping bands, rep0 dropped) => the ENCODER FORWARD goes from 15.90x to **2.89x** of vLLM's whole TTFT against the PIN `555967922` (TTFT median 46.02 ms, 45.60-46.41, production/graphed, 6 reps rep0 dropped, re-measured 2026-08-12). NOT a TTFT ratio: our projector, merge and prefill are unmeasured (F3). The originally published `~17.1x -> ~3.11x` used the carried-forward 0.25.0 denominator (42.8 ms), which the pin is 7.5% faster than - so the published figure was CONSERVATIVE. Oracle asserted BY COMMIT: the venv's `0.23.1rc1.dev1511+g555967922` version string is a setuptools_scm nearest-ancestor-tag artefact, not a mismatch; `soundfile==0.14.0` had to be installed into `~/venvs/vllm-oracle-next` before the pin could tokenize Voxtral at all (recorded against #375). **CORRECTNESS:** oracle teacher-force (fixture's own stack, asserted live as vLLM 0.25.0 + mistral_common 1.11.5) = 3 divergent, worst 0.1250 nats, over-band 0, **RESULT PASS** — inside the ratified band but the shipping kernel has 0 divergent at gap 0.0. Cause is a HYPOTHESIS, not grounded (corrected 2026-08-12 by the fresh review of PR #439): the stated cause - FA-2 converting P to bf16 before the PV MMA (`flash_fwd_kernel.h:347`) - was REFUTED by mutation M4, which forced exactly that rounding into the shipping scalar kernel, rebuilt clean, and got the SAME token md5 `89923566...`. Five candidate differences remain and none is isolated. **DISPOSITION: default UNCHANGED (byte-exact, `test_voxtral_e2e` 16/16, goldens md5 unchanged); FA-2 opt-in behind `VT_WHISPER_ENC_FA2=1`; ADOPTION = DEVELOPER DECISION.** Not a ceiling, RE-RANKED after M4: (1) attention is now only 5.33 ms of the 133 ms, so the §15.1-deferred device im2col kernel is the top encoder lever; (2) measure our ACTUAL TTFT (projector + merge + prefill); (3) isolate which of the five FA-2/scalar differences flips the tokens, one M4-style mutation at a time; (4) the FA-3-style f32-correction rescale, DEMOTED from #1 because M4 refuted its premise. Repairs landed on the branch: issue [#463](https://github.com/mudler/vllm.cpp/issues/463) for the zero-assertion SUCCESS skip (now `exit 77` + CTest `SKIP_RETURN_CODE`), `tests/vt/test_ops_attention_dense_fa2.cpp` for the op's missing tests, and a `causal` parameter on `LaunchDenseFA2Bf16` that THROWS instead of silently answering non-causally. No mm row advances to DONE. | +| 2026-08-12 (**M4 on gfx1100 — the pinned vLLM-ROCm oracle stood up on the only board that can host it + the ROCm device-golden lane in the Qwen3-0.6B near-tie gate; issue #41; `CLAIM-ROCM-M4`; worktree `/home/vikash/vllm.cpp-rocm-m4` branch `row/ROCM-M4-GATE`, base `8b00f79f`) | The M4 milestone for `BACKEND-ROCM`: build the PINNED vLLM oracle (`555967922`, runtime `0.23.1rc1.dev1511+g555967922` — the parity-pin block's identity, verified) as a ROCm wheel inside `rocm/vllm-dev:base` for `gfx1100` (`vllm-...rocm723-cp312` wheel, 2,696 files incl. `_rocm_C` ABI), commit the image (`vllm-rocm-oracle:555967922-gfx1100`, recipe in `.agents/specs/rocm-m4-oracle.md`), and capture the Qwen3-0.6B goldens FROM THAT ORACLE on this box. The gate test's existing Metal/Tenstorrent device-golden lane gains the `kROCM` arm (`our_ids_rocm.npy` + `neartie_gap_mnats_rocm.npy` + the `*_rocm` greedy/evidence pair); base dgx goldens byte-untouched. | **Oracle = upstream vLLM AT THE PIN ITSELF** — not a port: `scripts/qwen3-oracle-capture.py` + `scripts/qwen3-neartie-gap.py` run against the pinned wheel in the container; the gate lane mirrors the Metal M3b precedent (PR-set around row 632/633) file:line (`test_qwen3_paged_engine.cpp` device-golden branch). | **PASS.** Oracle K=10 per-prompt greedy: deterministic in every (prompt,pos) cell (0 multi-member). Our ROCm engine vs the same-box oracle: **16/16 prompts PASS** — STRICT token-exact 11/16 vs base greedy, 5/16 near-tie-band only, **max teacher-forced gap 0.125 nats** (prompt[3] tok1), 0 forward-divergent, 125/125 assertions; backend proof all 8 Qwen3-dense ops `selections>0 ∧ declines==0` on device 5 (kPagedAttention 7,168 selections; RoPE served via the fused-cache arm, `rope_alt` counted). The 28 token-divergent positions vs oracle greedy are all in-band; the p0 France/Italy flip is a literal 0.0000-nat tie in the oracle's OWN logits (vLLM's teacher-forced argmax on our prefix IS 15344 — the Metal M3b signature reproduced on ROCm). Hardware: 4x RX 7900 XTX gfx1100, ROCm 7.14 host, HIP 7.2.5 userland, `enforce_eager=True`, batch=1, GPU under `flock ~/gpu.lock`. Not a speed change (`benchmark_binding=false`). NEXT: Qwen3.5-0.8B (GDN) + 35B keep-quant gates under the same oracle. | +| 2026-08-12 (**M4 second model — the Qwen3.5-0.8B GDN near-tie gate LANDED AS ITS OWN RED ANCHOR: oracle-measured REAL forward divergence, not a near-tie**; issue #41; `CLAIM-ROCM-M4`; worktree `/home/vikash/vllm.cpp-rocm-m4` branch `row/ROCM-M4-GDN-GATE` (stacked on `row/ROCM-M4-GATE`), commit `a1755d95`) | NEW `tests/parity/test_qwen35_paged_engine.cpp` — the GDN sibling of the Qwen3-dense SACRED gate: the standard 16-prompt battery through the full paged engine (InputProcessor -> Scheduler -> GDN recurrence + paged attention on full-attn layers + Sampler), anchor + <=0.5-nat near-tie band identical to the dense gate, backend proof over the GDN op set (kCausalConv1dFwd/Update, kGdnPrefill/Decode, kGdnPostConv, kRmsNormGated, kSigmoidGateBf16, kAttnQkNormRopeGate, kReshapeAndCache, kPagedAttention, kEmbedding, kMatmulBT, kRmsNorm, kSiluAndMul, kGreedyArgmax). Goldens `qwen35_greedy_0_8b/` captured from the pinned vLLM-ROCm oracle (K=10 deterministic in every cell) with a `manifest.json` (oracle identity + the real HF revision `2fc06364...`, resolved via NEW `parity::Qwen35_08BSnapshot` in hf_snapshot.h — check-snapshot-pins clean; the header also gains a forward declaration fixing the call-before-declaration order hipcc rejects, the bug #555 fixes canonically on main). | Oracle = upstream vLLM at the parity pin itself (the same `vllm-rocm-oracle:555967922-gfx1100` container as the 0.6B lane); the gate mirrors `test_qwen3_paged_engine.cpp`'s methodology (deviation: none — the [[near-tie-distributional-gate]] pattern applied to a GDN model). | **RED BY DESIGN — THE GATE TELLS THE TRUTH ABOUT THE KNOWN 0.8B DIVERGENCE.** Verdict at landing: **13/16 prompts forward-divergent** (0/16 strict token-exact, 3/16 near-tie-band pass, **max gap 14.125 nats** at prompt[8] tok8; SIX prompts diverge AT THE FIRST TOKEN with gaps 0.375–1.188 nats — over the 0.5-nat band, so NOT near-ties). Backend proof GREEN (kPagedAttention 1,536 selections, kGdnDecode 4,320, 0 declines) — dispatch is native; the divergence is NUMERICAL. This is the `Qwen3.5-0.8B CPU/ROCm divergence remains open` item from STATUS.md, now oracle-quantified instead of anecdotal. The commit is deliberately the RED test of the fix lifecycle — it turns GREEN when the 0.8B forward fix lands (the bf16-softmax amplification localized to the attention block in the earlier characterization). Hardware/gates: gfx1100, `enforce_eager`, batch=1, `flock ~/gpu.lock`; the 0.6B gate re-run GREEN in the same tree (16/16) — the new gate is additive. `benchmark_binding=false`. NEXT: the forward fix (attention-block bf16 issue), then the 35B keep-quant gate. | diff --git a/.agents/specs/rocm-m4-oracle.md b/.agents/specs/rocm-m4-oracle.md new file mode 100644 index 000000000..70b08f408 --- /dev/null +++ b/.agents/specs/rocm-m4-oracle.md @@ -0,0 +1,74 @@ +# ROCm M4 — the pinned vLLM-ROCm oracle on gfx1100 and the ROCm near-tie gate lane + +## Goal + +Give `BACKEND-ROCM` (issue #41) the milestone that only this box can host: the M4 +correctness gate — our ROCm paged engine held against a **pinned vLLM-ROCm oracle +running on the same gfx1100 hardware**, not against the dgx (CUDA) capture. + +## The oracle + +- **Pin:** upstream vLLM commit `5559679229bc961848b121ccdeaa8fa5d79bec98` + (runtime identity `0.23.1rc1.dev1511+g555967922`, verified against the + parity-pin block in `.agents/upstream-sync.md`). +- **Base image:** `rocm/vllm-dev:base` (HIP 7.2.5 userland, host ROCm 7.14 + driver). `PYTORCH_ROCM_ARCH=gfx1100` covers the 7900 XTX. +- **Build:** `python3 setup.py bdist_wheel` inside the container after + `apt-get install binutils mold` (the base image ships no linker) and the + pyproject build-system requirements. Wheel: + `vllm-0.23.1rc1.dev1511+g555967922.rocm723-cp312-cp312-linux_x86_64.whl` + (2,696 files; `_C`/`_rocm_C`/`_moe_C` ABI extensions present). +- **Committed image:** `vllm-rocm-oracle:555967922-gfx1100` + (`/home/vikash/oracle/Dockerfile.oracle`), wheel installed with deps. +- **Determinism:** K=10 per-prompt greedy capture on the 16 gate prompts is + **deterministic in every cell** (0 multi-member (prompt,pos) cells) — a + well-posed strict gate on this board, matching the dgx finding for batch=1 + `enforce_eager=True`. + +## The gate lane + +`tests/parity/test_qwen3_paged_engine.cpp` already has device-aware goldens for +Metal and Tenstorrent (`our_ids_.npy` + `neartie_gap_mnats_.npy` under +the shared `qwen3_greedy_0_6b/` dir, with the dgx CUDA pair as base). This spec +adds `kROCM` to that lane, unchanged logic: + +- hard anchor REQUIRE (our deterministic tokens vs the committed ROCm anchor), +- near-tie band ≤ 500 milli-nats (vLLM teacher-forced on OUR exact prefix), +- backend proof (all 8 Qwen3-dense ops `selections>0 ∧ declines==0` on kROCM, + with the fused-RoPE alternative counted as in the existing lane). + +New committed goldens (captured on gfx1100, 4x RX 7900 XTX, ROCm 7.14, +`enforce_eager=True`, batch=1): + +| file | content | +|---|---| +| `our_ids_rocm.npy` | our engine's greedy tokens (16×16) | +| `neartie_gap_mnats_rocm.npy` | oracle teacher-forced gaps in milli-nats (16×16) | +| `greedy_ids_rocm.npy` | the ROCm oracle's own greedy (evidence; the base `greedy_ids.npy` stays the dgx capture) | +| `greedy_dist_rocm.npy` | K=10 run set (evidence) | + +Flow (the three commands, all GPU-locked): +1. oracle capture: `scripts/qwen3-oracle-capture.py --runs 10 --per-prompt` in the + committed container against `/models` (Qwen3-0.6B). +2. bootstrap dump: `VT_DUMP_IDS=1 ./build-hip/tests/test_qwen3_paged_engine` + → `our_ids_rocm.i32`. +3. teacher-forced gaps: `scripts/qwen3-neartie-gap.py --golden-dir ...` in the + container → `our_ids_rocm.npy` + `neartie_gap_mnats_rocm.npy`. + +## Result (Qwen3-0.6B, gfx1100) + +**16/16 prompts PASS** — STRICT token-exact 11/16 vs the base greedy, 5/16 via +the near-tie band, **max gap 0.125 nats** (prompt[3] tok=1), **0 +forward-divergent**; backend proof: kPagedAttention selections 7,168, 0 declines; +125/125 assertions. The 28 token-divergent positions vs the oracle's own greedy +are all within the band; the known p0 France/Italy flip is a literal 0.0000-nat +tie in the oracle's own logits. + +## Boundaries + +- No source/kernel change; the only edited file is the parity gate test. +- The dgx base goldens are byte-untouched; the ROCm additions are purely additive. +- Qwen3-4B (the strict deterministic dense) is not on this box (disk); the 0.6B + near-tie-robust gate is the M4 evidence for the lane. +- The oracle image and `/home/vikash/oracle/` scratch are machine-local, not + committed; this spec is the reproduction recipe. diff --git a/docs/STATUS.md b/docs/STATUS.md index 86b20c444..b204d2006 100644 --- a/docs/STATUS.md +++ b/docs/STATUS.md @@ -445,7 +445,12 @@ recurrences + fused attn preamble; 27B prefill 21.5x, decode [campaign](../.agents/specs/vulkan-full-support.md)), ROCm (W0 community-green on 5 gfx archs; the APU unified-memory fix remains unverified; gfx1200 runs Gemma-3 and Qwen3 all-native, with Gemma-3 strict 48/48 against two vLLM-ROCm -oracles and Qwen3 in a measured near-tie regime; Qwen3.5-0.8B GDN runs all-native +oracles and Qwen3 in a measured near-tie regime; on gfx1100 the M4 gate now +runs against the **pinned vLLM-ROCm oracle built on the same box** +(`555967922` / `0.23.1rc1.dev1511+g555967922`): Qwen3-0.6B **16/16 PASS** +(11/16 strict token-exact, 5/16 near-tie band, max teacher-forced gap 0.125 +nats, 0 forward-divergent; oracle K=10 deterministic in every cell) with the +ROCm device-golden lane in `test_qwen3_paged_engine`; Qwen3.5-0.8B GDN runs all-native but its CPU/ROCm divergence remains open; gfx1201 Gemma-4 FP8 MoE is contributor-measured on 2x R9700 and CPU-link-verified our side; [guide](ROCM.md)), and the full tool-calling template surface. **Muse Glimmer's diff --git a/tests/CMakeLists.txt b/tests/CMakeLists.txt index 77d4fda9a..97983b50c 100644 --- a/tests/CMakeLists.txt +++ b/tests/CMakeLists.txt @@ -1290,6 +1290,16 @@ target_compile_definitions(test_qwen3_paged_engine PRIVATE target_include_directories(test_qwen3_paged_engine PRIVATE ${CMAKE_SOURCE_DIR}/tests/parity ${CMAKE_SOURCE_DIR}/src) +# ROCm M4 — the GDN sibling of the Qwen3-dense gate: Qwen3.5-0.8B greedy decode +# vs the pinned vLLM-ROCm oracle (the ONLY capture of this model — no dgx +# capture exists; the base golden pair is ROCm-captured on gfx1100). +# Checkpoint-gated: skips loudly when the HF snapshot is absent. +vllm_cpp_add_test(test_qwen35_paged_engine parity/test_qwen35_paged_engine.cpp) +target_compile_definitions(test_qwen35_paged_engine PRIVATE + PARITY_GOLDENS_DIR="${CMAKE_SOURCE_DIR}/tests/parity/goldens") +target_include_directories(test_qwen35_paged_engine PRIVATE + ${CMAKE_SOURCE_DIR}/tests/parity ${CMAKE_SOURCE_DIR}/src) + # ROW-SERVE-ASYNC-DENSE-MIRROR — the classic-dense async-serving greedy gate: # Qwen3-0.6B/4B async batch-1 + concurrency reproduce the race-free SYNC engine # continuation token-for-token. RED on VT_ASYNC_DEVICE_MIRROR=0 (dense EmbedInto diff --git a/tests/parity/goldens/qwen35_greedy_0_8b/greedy_dist.npy b/tests/parity/goldens/qwen35_greedy_0_8b/greedy_dist.npy new file mode 100644 index 000000000..5282c5800 Binary files /dev/null and b/tests/parity/goldens/qwen35_greedy_0_8b/greedy_dist.npy differ diff --git a/tests/parity/goldens/qwen35_greedy_0_8b/greedy_ids.npy b/tests/parity/goldens/qwen35_greedy_0_8b/greedy_ids.npy new file mode 100644 index 000000000..b101b8145 Binary files /dev/null and b/tests/parity/goldens/qwen35_greedy_0_8b/greedy_ids.npy differ diff --git a/tests/parity/goldens/qwen35_greedy_0_8b/manifest.json b/tests/parity/goldens/qwen35_greedy_0_8b/manifest.json new file mode 100644 index 000000000..5bb7056c2 --- /dev/null +++ b/tests/parity/goldens/qwen35_greedy_0_8b/manifest.json @@ -0,0 +1,21 @@ +{ + "op": "qwen35_0_8b_greedy", + "oracle": { + "engine": "pinned vLLM-ROCm oracle built from source on the gate host (issue #41 M4)", + "vllm_commit": "5559679229bc961848b121ccdeaa8fa5d79bec98", + "vllm_runtime_version": "0.23.1rc1.dev1511+g555967922", + "wheel": "vllm-0.23.1rc1.dev1511+g555967922.rocm723-cp312-cp312-linux_x86_64.whl", + "container_image": "vllm-rocm-oracle:555967922-gfx1100 (rocm/vllm-dev:base + the wheel)", + "hardware": "4x RX 7900 XTX (gfx1100), ROCm 7.14 host driver, HIP 7.2.5 userland", + "invocation": "scripts/qwen3-oracle-capture.py --model Qwen/Qwen3.5-0.8B --runs 10 --per-prompt --gpu-mem 0.8 (enforce_eager=True, batch=1); teacher-forced gaps via scripts/qwen3-neartie-gap.py", + "date": "2026-08-12", + "note": "There is no dgx/CUDA capture of this model: the 0.8B GDN checkpoint was only ever stood up on the gfx1100 box, which is also the only board hosting a vLLM-ROCm oracle. The base golden pair here is ROCm-captured. K=10 per-prompt greedy is deterministic in every (prompt,pos) cell (see greedy_dist.npy). GATE STATUS AT CAPTURE: the gate FAILS 13/16 (0/16 strict, 3/16 near-tie band, max gap 14.125 nats) -- a REAL forward divergence, not a near-tie; this gate is the committed RED anchor for the 0.8B forward fix work." + }, + "model": { + "repo": "Qwen/Qwen3.5-0.8B", + "revision": "2fc06364715b967f1860aea9cf38778875588b17", + "architectures": ["Qwen3_5ForConditionalGeneration"], + "note": "revision verified via the checkpoint's huggingface_hub download metadata (2026-08-12)" + }, + "prompts": "the standard 16-prompt battery shared with scripts/qwen3-oracle-capture.py / test_qwen3_paged_engine.cpp" +} diff --git a/tests/parity/goldens/qwen35_greedy_0_8b/neartie_gap_mnats.npy b/tests/parity/goldens/qwen35_greedy_0_8b/neartie_gap_mnats.npy new file mode 100644 index 000000000..76e337b50 Binary files /dev/null and b/tests/parity/goldens/qwen35_greedy_0_8b/neartie_gap_mnats.npy differ diff --git a/tests/parity/goldens/qwen35_greedy_0_8b/our_ids.npy b/tests/parity/goldens/qwen35_greedy_0_8b/our_ids.npy new file mode 100644 index 000000000..5db1b2432 Binary files /dev/null and b/tests/parity/goldens/qwen35_greedy_0_8b/our_ids.npy differ diff --git a/tests/parity/goldens/qwen35_greedy_0_8b/p0_prompt.i32 b/tests/parity/goldens/qwen35_greedy_0_8b/p0_prompt.i32 new file mode 100644 index 000000000..191546743 Binary files /dev/null and b/tests/parity/goldens/qwen35_greedy_0_8b/p0_prompt.i32 differ diff --git a/tests/parity/goldens/qwen35_greedy_0_8b/p10_prompt.i32 b/tests/parity/goldens/qwen35_greedy_0_8b/p10_prompt.i32 new file mode 100644 index 000000000..454776427 Binary files /dev/null and b/tests/parity/goldens/qwen35_greedy_0_8b/p10_prompt.i32 differ diff --git a/tests/parity/goldens/qwen35_greedy_0_8b/p11_prompt.i32 b/tests/parity/goldens/qwen35_greedy_0_8b/p11_prompt.i32 new file mode 100644 index 000000000..623cdcfaf Binary files /dev/null and b/tests/parity/goldens/qwen35_greedy_0_8b/p11_prompt.i32 differ diff --git a/tests/parity/goldens/qwen35_greedy_0_8b/p12_prompt.i32 b/tests/parity/goldens/qwen35_greedy_0_8b/p12_prompt.i32 new file mode 100644 index 000000000..229d6e926 Binary files /dev/null and b/tests/parity/goldens/qwen35_greedy_0_8b/p12_prompt.i32 differ diff --git a/tests/parity/goldens/qwen35_greedy_0_8b/p13_prompt.i32 b/tests/parity/goldens/qwen35_greedy_0_8b/p13_prompt.i32 new file mode 100644 index 000000000..a2b1cb6c5 Binary files /dev/null and b/tests/parity/goldens/qwen35_greedy_0_8b/p13_prompt.i32 differ diff --git a/tests/parity/goldens/qwen35_greedy_0_8b/p14_prompt.i32 b/tests/parity/goldens/qwen35_greedy_0_8b/p14_prompt.i32 new file mode 100644 index 000000000..ea74bf755 Binary files /dev/null and b/tests/parity/goldens/qwen35_greedy_0_8b/p14_prompt.i32 differ diff --git a/tests/parity/goldens/qwen35_greedy_0_8b/p15_prompt.i32 b/tests/parity/goldens/qwen35_greedy_0_8b/p15_prompt.i32 new file mode 100644 index 000000000..c72735e10 Binary files /dev/null and b/tests/parity/goldens/qwen35_greedy_0_8b/p15_prompt.i32 differ diff --git a/tests/parity/goldens/qwen35_greedy_0_8b/p1_prompt.i32 b/tests/parity/goldens/qwen35_greedy_0_8b/p1_prompt.i32 new file mode 100644 index 000000000..1130fee1e Binary files /dev/null and b/tests/parity/goldens/qwen35_greedy_0_8b/p1_prompt.i32 differ diff --git a/tests/parity/goldens/qwen35_greedy_0_8b/p2_prompt.i32 b/tests/parity/goldens/qwen35_greedy_0_8b/p2_prompt.i32 new file mode 100644 index 000000000..083e5883d Binary files /dev/null and b/tests/parity/goldens/qwen35_greedy_0_8b/p2_prompt.i32 differ diff --git a/tests/parity/goldens/qwen35_greedy_0_8b/p3_prompt.i32 b/tests/parity/goldens/qwen35_greedy_0_8b/p3_prompt.i32 new file mode 100644 index 000000000..e69ad7489 Binary files /dev/null and b/tests/parity/goldens/qwen35_greedy_0_8b/p3_prompt.i32 differ diff --git a/tests/parity/goldens/qwen35_greedy_0_8b/p4_prompt.i32 b/tests/parity/goldens/qwen35_greedy_0_8b/p4_prompt.i32 new file mode 100644 index 000000000..a1138d117 Binary files /dev/null and b/tests/parity/goldens/qwen35_greedy_0_8b/p4_prompt.i32 differ diff --git a/tests/parity/goldens/qwen35_greedy_0_8b/p5_prompt.i32 b/tests/parity/goldens/qwen35_greedy_0_8b/p5_prompt.i32 new file mode 100644 index 000000000..92664cad2 Binary files /dev/null and b/tests/parity/goldens/qwen35_greedy_0_8b/p5_prompt.i32 differ diff --git a/tests/parity/goldens/qwen35_greedy_0_8b/p6_prompt.i32 b/tests/parity/goldens/qwen35_greedy_0_8b/p6_prompt.i32 new file mode 100644 index 000000000..3561c6f4d Binary files /dev/null and b/tests/parity/goldens/qwen35_greedy_0_8b/p6_prompt.i32 differ diff --git a/tests/parity/goldens/qwen35_greedy_0_8b/p7_prompt.i32 b/tests/parity/goldens/qwen35_greedy_0_8b/p7_prompt.i32 new file mode 100644 index 000000000..b8bc06310 Binary files /dev/null and b/tests/parity/goldens/qwen35_greedy_0_8b/p7_prompt.i32 differ diff --git a/tests/parity/goldens/qwen35_greedy_0_8b/p8_prompt.i32 b/tests/parity/goldens/qwen35_greedy_0_8b/p8_prompt.i32 new file mode 100644 index 000000000..db62992ee Binary files /dev/null and b/tests/parity/goldens/qwen35_greedy_0_8b/p8_prompt.i32 differ diff --git a/tests/parity/goldens/qwen35_greedy_0_8b/p9_prompt.i32 b/tests/parity/goldens/qwen35_greedy_0_8b/p9_prompt.i32 new file mode 100644 index 000000000..be6f068ae Binary files /dev/null and b/tests/parity/goldens/qwen35_greedy_0_8b/p9_prompt.i32 differ diff --git a/tests/parity/goldens/qwen3_greedy_0_6b/greedy_dist_rocm.npy b/tests/parity/goldens/qwen3_greedy_0_6b/greedy_dist_rocm.npy new file mode 100644 index 000000000..ed0bf47c4 Binary files /dev/null and b/tests/parity/goldens/qwen3_greedy_0_6b/greedy_dist_rocm.npy differ diff --git a/tests/parity/goldens/qwen3_greedy_0_6b/greedy_ids_rocm.npy b/tests/parity/goldens/qwen3_greedy_0_6b/greedy_ids_rocm.npy new file mode 100644 index 000000000..b8b49ca47 Binary files /dev/null and b/tests/parity/goldens/qwen3_greedy_0_6b/greedy_ids_rocm.npy differ diff --git a/tests/parity/goldens/qwen3_greedy_0_6b/neartie_gap_mnats_rocm.npy b/tests/parity/goldens/qwen3_greedy_0_6b/neartie_gap_mnats_rocm.npy new file mode 100644 index 000000000..a845e0408 Binary files /dev/null and b/tests/parity/goldens/qwen3_greedy_0_6b/neartie_gap_mnats_rocm.npy differ diff --git a/tests/parity/goldens/qwen3_greedy_0_6b/our_ids_rocm.npy b/tests/parity/goldens/qwen3_greedy_0_6b/our_ids_rocm.npy new file mode 100644 index 000000000..cff630492 Binary files /dev/null and b/tests/parity/goldens/qwen3_greedy_0_6b/our_ids_rocm.npy differ diff --git a/tests/parity/hf_snapshot.h b/tests/parity/hf_snapshot.h index 82f076b8b..831a5b9c6 100644 --- a/tests/parity/hf_snapshot.h +++ b/tests/parity/hf_snapshot.h @@ -178,6 +178,20 @@ inline std::string Qwen27DFlashDraftSnapshot() { return HfSnapshot("models--z-lab--Qwen3.6-27B-DFlash", kQwen27DFlashDraftRevision, "VT_DFLASH_DRAFT_SNAPSHOT");} +// The Qwen3.5-0.8B GDN gate model (issue #41, the ROCm M4 lane). This revision +// is what the committed `goldens/qwen35_greedy_0_8b` were captured against on +// gfx1100 2026-08-12 (verified via the checkpoint's huggingface_hub download +// metadata; `manifest.json` in the golden dir records the oracle identity). +inline constexpr const char* kQwen35_08BRevision = + "2fc06364715b967f1860aea9cf38778875588b17"; + +// The 0.8B GDN gate model, pinned to the goldens' revision. Absent cache => "" +// and the caller emits its loud SKIP (the intended behavior off the gate host). +inline std::string Qwen35_08BSnapshot() { + return HfSnapshot("models--Qwen--Qwen3.5-0.8B", kQwen35_08BRevision, + "VT_QWEN35_08B_SNAPSHOT"); +} + } // namespace parity #endif // VLLM_TESTS_PARITY_HF_SNAPSHOT_H_ diff --git a/tests/parity/test_qwen35_paged_engine.cpp b/tests/parity/test_qwen35_paged_engine.cpp new file mode 100644 index 000000000..7d1dec45e --- /dev/null +++ b/tests/parity/test_qwen35_paged_engine.cpp @@ -0,0 +1,338 @@ +// vllm.cpp original (ADDITIVE-MODEL W4 — THE SACRED correctness gate); no +// upstream mirror. +// +// THE PAGED-ENGINE Qwen3.5-0.8B (GDN) GREEDY CORRECTNESS GATE. The GDN sibling of +// test_qwen3_paged_engine.cpp: drives the standard prompt battery through the FULL +// PAGED LLMEngine stack (InputProcessor -> Scheduler -> GDN recurrence + paged +// attention on the full-attention layers + Sampler -> OutputProcessor) via +// LoadedEngine::FromModelDir, and checks greedy (temperature-0) decode against the +// pinned vLLM oracle (commit 555967922, runtime 0.23.1rc1.dev1511+g555967922). +// +// ORACLE PROVENANCE — DIFFERENT FROM THE QWEN3-DENSE GATE. There is no dgx/CUDA +// capture for this model: the 0.8B GDN checkpoint was only ever stood up on the +// gfx1100 box (issue #41 lane), which is also the only board that hosts a +// vLLM-ROCm oracle. The base golden pair here is therefore ROCm-captured: +// greedy_ids.npy [N,T] i32 the pinned ROCm oracle's per-prompt greedy +// (K=10 per-prompt runs DETERMINISTIC in every +// cell — see greedy_dist.npy). +// our_ids.npy [N,T] i32 OUR ROCm engine's greedy (the anchor). +// neartie_gap_mnats.npy [N,T] oracle teacher-forced gap (milli-nats) for OUR +// token given OUR prefix. +// On kROCM these base files are the gate; the Metal/Tenstorrent device-lane +// mechanism from the Qwen3-dense gate is inherited unchanged for when those +// devices gain this model (their pairs would be our_ids_metal.npy etc.). +// +// METHODOLOGY — identical to the Qwen3-dense gate (see that file's header and +// [[near-tie-distributional-gate]]): STRICT token-exact is reported, but the +// PASS bar is the near-tie band: given OUR EXACT PREFIX, the oracle's OWN logits +// must place OUR token within kNearTieMnats of the oracle argmax. Strict where +// well-posed, near-tie-robust only where the oracle itself cannot separate the +// tokens. +// +// BACKEND PROOF — GDN op set (not the Qwen3-dense list): the GDN layers dispatch +// kCausalConv1dFwd/Update (prefill/decode conv), kGdnPrefill/kGdnDecode (the +// recurrence), kGdnPostConv, kRmsNormGated, kSigmoidGateBf16, +// kAttnQkNormRopeGate (the GDN preamble), and the full-attention layers dispatch +// kReshapeAndCache + kPagedAttention + kAttnQkNormRope; shared: kEmbedding, +// kMatmulBT, kRmsNorm, kSiluAndMul, kGreedyArgmax. All must show selections>0 +// and declines==0 on the running device. +// +// Checkpoint-gated: resolves the HF snapshot under ~/.cache/huggingface/hub/. +// Absent snapshot (CI/CPU) => loud SKIP. +#include + +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include "hf_snapshot.h" +#include "npy.h" +#include "vllm/entrypoints/model_loader.h" +#include "vllm/sampling_params.h" +#include "vt/op_provider.h" // the "which backend actually ran" proof +#include "vt/ops.h" + +namespace fs = std::filesystem; + +namespace { + +// Near-tie acceptance threshold in milli-nats — same bar as the Qwen3-dense +// gate (0.5 nats); see that file for the derivation. +constexpr int32_t kNearTieMnats = 500; + +// The standard prompt battery — MUST match scripts/qwen3-oracle-capture.py (and +// qwen3-neartie-gap.py) exactly; those scripts are model-agnostic and take +// --model/--out-dir/--golden-dir. +const std::vector& Prompts() { + static const std::vector p = { + "The capital of France is", + "Once upon a time,", + "In the beginning God created", + "The quick brown fox jumps over", + "def fibonacci(n):", + "Water boils at a temperature of", + "The theory of relativity was developed by", + "To be or not to be, that is", + "The largest planet in our solar system is", + "Machine learning is a subfield of", + "The mitochondria is the powerhouse of", + "Roses are red, violets are", + "The first president of the United States was", + "E equals m c", + "A journey of a thousand miles begins with", + "The chemical symbol for gold is", + }; + return p; +} + +vllm::SamplingParams Greedy(int max_tokens) { + vllm::SamplingParams sp; + sp.temperature = 0.0; + sp.max_tokens = max_tokens; + sp.PostInit(); + return sp; +} + +const int32_t* AsI32(const parity::NpyArray& a) { + return reinterpret_cast(a.data.data()); +} + +void RunGate(const std::string& golden_subdir, const char* label) { + const std::string snap = parity::Qwen35_08BSnapshot(); + if (snap.empty()) { + MESSAGE(label << " checkpoint absent; skipping — models--Qwen--Qwen3.5-0.8B " + "snapshot at the pinned revision not present"); + return; + } + const fs::path gdir = fs::path(PARITY_GOLDENS_DIR) / golden_subdir; + const bool dump = std::getenv("VT_DUMP_IDS") != nullptr; + const bool have_gap = fs::exists(gdir / "our_ids.npy") && + fs::exists(gdir / "neartie_gap_mnats.npy"); + if (!fs::exists(gdir / "greedy_ids.npy")) { + MESSAGE(label << " greedy golden absent; skipping — capture with the pinned " + "ROCm oracle: qwen3-oracle-capture.py --per-prompt for " + << golden_subdir); + return; + } + // BOOTSTRAP: with VT_DUMP_IDS set and no gap golden yet, generate + dump OUR + // token ids (our_ids.i32) so qwen3-neartie-gap.py can build the gap golden. + if (dump && !have_gap) { + MESSAGE(label << ": BOOTSTRAP dump (gap golden absent) via FromModelDir(" << snap << ")..."); + std::unique_ptr le = + vllm::entrypoints::LoadedEngine::FromModelDir( + snap, vllm::entrypoints::EngineParams{}); + const parity::NpyArray gg = parity::LoadNpy((gdir / "greedy_ids.npy").string()); + const int64_t NN = gg.shape[0], TT = gg.shape[1]; + std::vector buf(static_cast(NN * TT), -1); + for (int64_t i = 0; i < NN; ++i) { + const vllm::RequestOutput out = le->engine().generate( + Prompts()[static_cast(i)], Greedy(static_cast(TT)), + "boot" + std::to_string(i)); + const std::vector& got = out.outputs[0].token_ids; + for (int64_t j = 0; j < TT && j < static_cast(got.size()); ++j) + buf[static_cast(i * TT + j)] = got[static_cast(j)]; + } + const std::string path = (gdir / "our_ids.i32").string(); + std::FILE* f = std::fopen(path.c_str(), "wb"); + if (f != nullptr) { std::fwrite(buf.data(), sizeof(int32_t), buf.size(), f); std::fclose(f); } + MESSAGE(label << " BOOTSTRAP dumped our token ids -> " << path); + return; + } + if (!have_gap) { + MESSAGE(label << " gap goldens absent; skipping — run the gate under " + "VT_DUMP_IDS=1 then qwen3-neartie-gap.py for " << golden_subdir); + return; + } + + const parity::NpyArray g = parity::LoadNpy((gdir / "greedy_ids.npy").string()); + const parity::NpyArray o = parity::LoadNpy((gdir / "our_ids.npy").string()); + const parity::NpyArray gap = parity::LoadNpy((gdir / "neartie_gap_mnats.npy").string()); + REQUIRE(g.dtype == "(N) == Prompts().size()); + const int32_t* gd = AsI32(g); + const int32_t* od = AsI32(o); + const int32_t* gapd = AsI32(gap); + + std::vector our_dump; + if (dump) our_dump.assign(static_cast(N * T), -1); + + MESSAGE(label << ": loading via FromModelDir(" << snap << ")..."); + std::unique_ptr loaded = + vllm::entrypoints::LoadedEngine::FromModelDir( + snap, vllm::entrypoints::EngineParams{}); + + // The base golden pair for this model is ROCm-captured (see the file header). + // The Metal/Tenstorrent device lanes are kept for when those devices run this + // model; their pairs do not exist yet, so those devices skip loudly. + const vt::DeviceType run_dev = loaded->runner().device().type; + const bool metal = run_dev == vt::DeviceType::kMETAL; + const bool tenstorrent = run_dev == vt::DeviceType::kTENSTORRENT; + const bool rocm = run_dev == vt::DeviceType::kROCM; + const bool other_partial = metal || tenstorrent; + if (other_partial) { + const char* ids_name = + metal ? "our_ids_metal.npy" : "our_ids_tenstorrent.npy"; + const char* gap_name = metal ? "neartie_gap_mnats_metal.npy" + : "neartie_gap_mnats_tenstorrent.npy"; + const bool have_dev = fs::exists(gdir / ids_name) && fs::exists(gdir / gap_name); + if (!have_dev) { + MESSAGE(label << ": this model's only oracle-backed goldens are the ROCm " + "base pair; " << ids_name << "/" << gap_name << " absent — skipping"); + return; + } + } + + // The GDN op set this model dispatches — all must be proven on the running + // device (selections > 0, declines == 0; fan-out spike Risk 4). + const std::vector kGdnOps = { + vt::OpId::kEmbedding, vt::OpId::kMatmulBT, + vt::OpId::kRmsNorm, vt::OpId::kRmsNormGated, + vt::OpId::kCausalConv1dFwd, vt::OpId::kCausalConv1dUpdate, + vt::OpId::kGdnPrefill, vt::OpId::kGdnDecode, + vt::OpId::kGdnPostConv, vt::OpId::kSigmoidGateBf16, + vt::OpId::kAttnQkNormRopeGate, + vt::OpId::kReshapeAndCache, vt::OpId::kPagedAttention, + vt::OpId::kSiluAndMul, vt::OpId::kGreedyArgmax}; + if (rocm) { + for (vt::OpId op : kGdnOps) { + CHECK(vt::OpRegistered(op, run_dev)); + vt::ResetOpProviderStats(op, run_dev); + } + vt::EnableOpProviderCallStats(true); + MESSAGE(label << ": running on device type " << static_cast(run_dev) + << " (5=ROCM) — gated against the ROCm oracle-backed golden pair"); + } + + const int32_t* anchor_ids = od; + const int32_t* gap_ids = gapd; + + int strict_exact = 0; + int neartie_only = 0; + int fail = 0; + int32_t worst_gap = 0; + int worst_i = -1, worst_j = -1; + for (int64_t i = 0; i < N; ++i) { + const vllm::RequestOutput out = loaded->engine().generate( + Prompts()[static_cast(i)], Greedy(static_cast(T)), + "gate" + std::to_string(i)); + REQUIRE(out.finished); + REQUIRE(out.outputs.size() == 1); + const std::vector& got = out.outputs[0].token_ids; + REQUIRE(static_cast(got.size()) == T); + if (dump) { + for (int64_t j = 0; j < T; ++j) + our_dump[static_cast(i * T + j)] = got[static_cast(j)]; + } + + // Anchor: the committed anchor is the exact deterministic sequence OUR ROCm + // engine produces. Drift is a hard REQUIRE — no cross-device latitude. + int first_div = -1; + for (int64_t j = 0; j < T; ++j) { + if (got[static_cast(j)] != anchor_ids[i * T + j]) { + first_div = static_cast(j); + break; + } + } + const bool anchor_ok = dump || first_div < 0; + REQUIRE_MESSAGE(anchor_ok, + label << " anchor drift prompt[" << i << "] tok=" << first_div + << " engine=" << (first_div < 0 ? -1 : got[static_cast(first_div)]) + << " committed anchor=" + << (first_div < 0 ? -1 : anchor_ids[i * T + first_div]) + << " — re-run qwen3-neartie-gap.py to refresh the gap golden"); + + bool exact = true; + bool prompt_ok = true; + int first_bad = -1; + for (int64_t j = 0; j < T; ++j) { + if (got[static_cast(j)] != gd[i * T + j]) exact = false; + const int32_t mn = gap_ids[i * T + j]; + if (mn > worst_gap) { + worst_gap = mn; + worst_i = static_cast(i); + worst_j = static_cast(j); + } + if (mn > kNearTieMnats) { + prompt_ok = false; + if (first_bad < 0) first_bad = static_cast(j); + } + } + if (!prompt_ok) { + ++fail; + MESSAGE(label << " FORWARD DIVERGENCE prompt[" << i << "] tok=" << first_bad + << " our=" << got[static_cast(first_bad)] + << " vLLM_greedy=" << gd[i * T + first_bad] + << " gap=" << (gap_ids[i * T + first_bad] / 1000.0) << " nats (> " + << (kNearTieMnats / 1000.0) << ") \"" << out.outputs[0].text << "\""); + } else if (exact) { + ++strict_exact; + } else { + ++neartie_only; + } + CHECK(prompt_ok); + } + + // Backend proof: token equality alone does not prove which device ran. + if (rocm) { + vt::EnableOpProviderCallStats(false); + for (vt::OpId op : kGdnOps) { + const auto st = vt::GetOpProviderStats(op, run_dev); + CHECK_MESSAGE(st.selections > 0, + label << ": op " << static_cast(op) + << " was never dispatched on device type " + << static_cast(run_dev)); + CHECK_MESSAGE(st.declines == 0, + label << ": op " << static_cast(op) + << " DECLINED and fell back"); + } + MESSAGE(label << ": BACKEND PROOF — Qwen3.5 GDN ops on device type " + << static_cast(run_dev) << " with 0 declines (kPagedAttention " + "selections=" + << vt::GetOpProviderStats(vt::OpId::kPagedAttention, run_dev).selections + << ", kGdnDecode selections=" + << vt::GetOpProviderStats(vt::OpId::kGdnDecode, run_dev).selections + << ")"); + } + + if (dump) { + const std::string path = (gdir / "our_ids.i32").string(); + std::FILE* f = std::fopen(path.c_str(), "wb"); + if (f != nullptr) { + std::fwrite(our_dump.data(), sizeof(int32_t), our_dump.size(), f); + std::fclose(f); + MESSAGE(label << " dumped our token ids -> " << path); + } + } + MESSAGE(label << " correctness gate: " << (strict_exact + neartie_only) << "/" << N + << " prompts PASS (STRICT token-exact vs oracle per-prompt greedy: " + << strict_exact << "/" << N << "; near-tie-band only: " << neartie_only + << "/" << N << "; max gap " << (worst_gap / 1000.0) << " nats @ prompt[" + << worst_i << "] tok=" << worst_j << "; " << fail + << " forward-divergent)"); + REQUIRE(fail == 0); +} + +} // namespace + +// Qwen3.5-0.8B (GDN hybrid: linear-attention recurrence + full-attention +// layers) — the first GDN-architecture gate, ROCm-oracle-backed (issue #41 M4). +TEST_CASE("qwen3.5-0.8B GDN paged-engine greedy near-tie correctness gate (ROCm, SACRED)") { + RunGate("qwen35_greedy_0_8b", "qwen3.5-0.8B"); +} diff --git a/tests/parity/test_qwen3_paged_engine.cpp b/tests/parity/test_qwen3_paged_engine.cpp index d0ded7df7..3d11cd1e2 100644 --- a/tests/parity/test_qwen3_paged_engine.cpp +++ b/tests/parity/test_qwen3_paged_engine.cpp @@ -227,8 +227,9 @@ void RunGate(const std::string& repo_dir, const std::string& golden_subdir, // not against CUDA's — identical anchor+band logic, device-appropriate goldens. const vt::DeviceType run_dev = loaded->runner().device().type; const bool metal = run_dev == vt::DeviceType::kMETAL; + const bool rocm = run_dev == vt::DeviceType::kROCM; const bool tenstorrent = run_dev == vt::DeviceType::kTENSTORRENT; - const bool device_golden = metal || tenstorrent; + const bool device_golden = metal || tenstorrent || rocm; // The forward + greedy ops Qwen3-dense dispatches on the DEFAULT // (VT_QWEN3_ROPE_CACHE) path. kRopeCosSinCache + kRopeFromCache are the M3b // additions (build the per-step cos|sin cache, then apply it); the rest are @@ -252,7 +253,7 @@ void RunGate(const std::string& repo_dir, const std::string& golden_subdir, } vt::EnableOpProviderCallStats(true); MESSAGE(label << ": running on device type " << static_cast(run_dev) - << " (2=METAL, 6=TENSTORRENT) — gated against this device's OWN " + << " (2=METAL, 5=ROCM, 6=TENSTORRENT) — gated against this device's OWN " "oracle-backed golden"); } @@ -261,9 +262,13 @@ void RunGate(const std::string& repo_dir, const std::string& golden_subdir, const int32_t* anchor_ids = od; // hard anchor for THIS device const int32_t* gap_ids = gapd; // vLLM teacher-forced gaps for THIS device parity::NpyArray o_dev, gap_dev; // keep device arrays alive for the loop - const char* ids_name = metal ? "our_ids_metal.npy" : "our_ids_tenstorrent.npy"; + const char* ids_name = + metal ? "our_ids_metal.npy" + : (rocm ? "our_ids_rocm.npy" : "our_ids_tenstorrent.npy"); const char* gap_name = - metal ? "neartie_gap_mnats_metal.npy" : "neartie_gap_mnats_tenstorrent.npy"; + metal ? "neartie_gap_mnats_metal.npy" + : (rocm ? "neartie_gap_mnats_rocm.npy" + : "neartie_gap_mnats_tenstorrent.npy"); bool bootstrap_only = false; if (device_golden) { const bool have_dev = fs::exists(gdir / ids_name) && fs::exists(gdir / gap_name); @@ -272,7 +277,7 @@ void RunGate(const std::string& repo_dir, const std::string& golden_subdir, // qwen3-neartie-gap.py then teacher-forces vLLM on that sequence. bootstrap_only = true; MESSAGE(label << ": BOOTSTRAP dump (device golden absent) for " - << (metal ? "Metal" : "Tenstorrent") << "..."); + << (metal ? "Metal" : (rocm ? "ROCm" : "Tenstorrent")) << "..."); } else { REQUIRE_MESSAGE(have_dev, label << ": device oracle golden absent (" << ids_name << " / " @@ -402,7 +407,8 @@ void RunGate(const std::string& repo_dir, const std::string& golden_subdir, if (dump) { const std::string dump_name = tenstorrent ? "our_ids_tenstorrent.i32" - : (metal ? "our_ids_metal.i32" : "our_ids.i32"); + : (metal ? "our_ids_metal.i32" + : (rocm ? "our_ids_rocm.i32" : "our_ids.i32")); const std::string path = (gdir / dump_name).string(); std::FILE* f = std::fopen(path.c_str(), "wb"); if (f != nullptr) {