diff --git a/.agents/benchmark-record.md b/.agents/benchmark-record.md index 0d27114e0..126cb961e 100644 --- a/.agents/benchmark-record.md +++ b/.agents/benchmark-record.md @@ -20353,3 +20353,34 @@ per stream), NOT as a perf fix; C_tmp size is ELIMINATED as an explanation of th Method: third time drift has fooled a before/after here. Pairing caught the first, pinned clocks the second, and only an in-process toggle catches this one. Future perf claims on this row need the toggle, not two runs. + +## SPEC-DSPARK: storage ruled out; ratio stable at ~0.966 across three sessions (2026-08-12) + +Question raised: are the weights on NAS, or not fully resident, distorting the +measurements? + +Weights are on LOCAL NVMe (/dev/nvme0n1p2 ext4); no NAS mount exists on the box. +A run reads 22.06 GB total = one full model read at load. Process RSS during +decode is 4.8 GB, so weights are uploaded and the mapping released, not held. +Decode is stable to 0.5% across 8 warm reps (146.0-147.6), which file-backed +weights could not be. Storage is NOT a factor. + +Operational: that NVMe is 98% full (76 GB free), and this repo has already lost a +gate run to ENOSPC reporting green over work that never ran. + +Within-session ratios, three independent measurements: + +| session | ours | oracle (modal) | ratio | +|---|---|---|---| +| pinned clocks, pre-C_tmp | 135.98 | 139.36 | 0.9757 | +| pinned clocks, post-C_tmp | 139.20 | 144.32 | 0.9646 | +| free clocks, ours->oracle->ours | 140.98 | 147.32 | 0.9569 | + +~0.966 +/- 0.01, consistently below 1.0. Absolute numbers move up to 5% between +sessions for the SAME binary because GB10's memory clock cannot be pinned, so +only the within-session ratio is quotable -- and all three agree. + +Oracle draws remain bimodal (~147.3 and ~155.6), the same one-extra-accepted-token +effect as the fibacc run, so its MODAL draws are the honest denominator. + +Evidence: `dgx:~/work/dspark-w6/iocheck.log`, `final_pair.log`. diff --git a/.agents/specs/dspark-spec-decode.md b/.agents/specs/dspark-spec-decode.md index 3f0c6bc57..b885405a6 100644 --- a/.agents/specs/dspark-spec-decode.md +++ b/.agents/specs/dspark-spec-decode.md @@ -1623,6 +1623,47 @@ in-process A/B switch, because on a machine whose memory clock cannot be pinned, even a pinned-clock before/after across two runs is not a controlled experiment. Any future perf claim on this row needs the toggle, not two runs. +## 6ac. STORAGE RULED OUT, and the ratio is stable at ~0.966 (2026-08-12) + +Developer question: could the weights being on NAS, or not fully resident in +device memory, be distorting these measurements? + +**Tested, and no.** + +| check | result | +|---|---| +| weight location | local NVMe (`/dev/nvme0n1p2`, ext4) -- **no NAS mount exists on this box** | +| total disk read for a run | 22.06 GB, i.e. ONE full model read at load | +| process RSS during decode | 4.8 GB -- the weights are NOT held in host RSS; they are uploaded and the mapping released | +| decode stability | 8 warm reps at 146.0 / 147.6 / 147.2 / 146.9 / 147.2 / 147.3 / 147.4 -- **0.5% spread** | + +If weights were still file-backed, decode would fault pages from NVMe and the +per-rep numbers would be erratic; a 0.5% spread says they are resident and decode +touches no storage. Worth keeping in mind that on GB10 the failure mode would be +severe if it ever regressed -- host pages reach the GPU through ATS, which this +repo measured at a 20-30% per-GEMM penalty. + +**Operational finding worth acting on separately:** that NVMe is **98% full** +(3.4T of 3.6T, 76 GB free). This repo has already lost a gate run to ENOSPC +producing a green-looking report over work that never executed. + +**And the ratio is now measured three times, each WITHIN one session:** + +| session | ours | oracle (modal) | ratio | +|---|---|---|---| +| pinned clocks, pre-C_tmp | 135.98 | 139.36 | 0.9757 | +| pinned clocks, post-C_tmp | 139.20 | 144.32 | 0.9646 | +| free clocks, ours->oracle->ours | 140.98 | 147.32 | **0.9569** | + +**~0.966 +/- 0.01, consistently below 1.0.** The absolute numbers move a lot +between sessions (135.98 to 142.09 for the same binary) because GB10's memory +clock cannot be pinned, which is exactly why only the within-session ratio is +quotable -- and all three agree. + +Also visible in the last run: the oracle's draws are bimodal at ~147.3 and +~155.6, the same one-extra-accepted-token effect as §6q, which is why its MODAL +draws are the honest denominator. + ## 7. Evidence, authority, stop conditions - Evidence root: `dgx:~/work/vllm.cpp-dspark-/`, one `flock`, named tmux. diff --git a/docs/BENCHMARKS.md b/docs/BENCHMARKS.md index 71d3964d0..3a2fa3149 100644 --- a/docs/BENCHMARKS.md +++ b/docs/BENCHMARKS.md @@ -344,7 +344,7 @@ in the tree, default-OFF, for reproducibility; detail in the benchmark record. | MTP | Qwen3.6-27B NVFP4 | token-identical to vLLM MTP, **~4% faster at c1**; on-par at c2-c8 | `DONE` | | DFlash | Qwen3.6-27B NVFP4 | **2.9x over spec-off** (10.16 → 29.32 tok/s), at/above vLLM DFlash-on (**1.003x**, non-overlapping bands) | `DONE` | | n-gram | Qwen3.6-27B NVFP4 | draft-free (`SPEC-NGRAM`); 27B 5/5 STRICT our-ngram-ON == vLLM-ngram-ON, 180/180 drafts accepted (correctness only, no speed row yet) | `DONE` | -| DSpark | 27B NVFP4 dense k=15; 35B-A3B MoE k=8 | MoE **0.975x** code / **1.012x** prose vs the pinned graphed oracle (PINNED CLOCKS, non-overlapping). NOT parity: **0.965x-0.976x** within-session; C_tmp cap landed but is perf-NEUTRAL (+0.03%) (#442) | `ACTIVE` | +| DSpark | 27B NVFP4 dense k=15; 35B-A3B MoE k=8 | MoE **0.975x** code / **1.012x** prose vs the pinned graphed oracle (PINNED CLOCKS, non-overlapping). NOT parity: **~0.966x +/- 0.01** over three within-session pairs; C_tmp cap perf-NEUTRAL; storage refuted (#442) | `ACTIVE` | | Breadth (EAGLE1/3, suffix, ngram-gpu, dynamic-k, ...) | n/a | enumerated from vLLM source + `INVENTORIED` 2026-08-06 (`.agents/specs/spec-decode-inventory.md`), unmeasured | `INVENTORIED` | ## How we measure diff --git a/docs/STATUS.md b/docs/STATUS.md index 86b20c444..6df60e9c4 100644 --- a/docs/STATUS.md +++ b/docs/STATUS.md @@ -529,7 +529,16 @@ memory-system behaviour that no allocation change we can name would alter; upstr ncu in either replay mode. A C_tmp over-allocation (15-30 MB vs upstream's 3.15 MB) was found and fixed, but an in-session A/B shows it is perf-NEUTRAL (+0.03%) -- an apparent +2.9% was machine drift, since GB10 cannot lock memory -clocks. Editing +clocks. The ratio has now been measured WITHIN a single session three times -- +0.9757, 0.9646 and (ours->oracle->ours at free clocks, drift -0.89%) 0.9569 -- +so it is **~0.966 +/- 0.01, consistently below 1.0**, while the absolute numbers +move up to 5% BETWEEN sessions for the same binary. Storage was raised as a +possible distortion and is refuted: the weights are on local NVMe (no NAS mount +exists on the box), a run reads 22.06 GB once at load, decode-time RSS is 4.8 GB +because the mapping is released after upload, and 8 warm reps hold a 0.5% +spread -- decode touches no storage. (That NVMe is 98% full, 76 GB free, which +is its own operational risk given ENOSPC has previously produced a green report +over a gate that never ran.) Editing the kernel, its launch config, layout or flags is NOT indicated: all are proven identical. (The repack kernels that appear to take 40% of a long run are LOAD-TIME.) NOT parity. The Gemma4 `1 + N` layout is coded and unit-tested but has