BENCH-ASSERT-CLOCK-STATE: the SM clock moves 12.8% between boots and nobody recorded it (#543, #545)

.agents/benchmark-record.md

@@ -20354,6 +20354,81 @@ Method: third time drift has fooled a before/after here. Pairing caught the
 first, pinned clocks the second, and only an in-process toggle catches this one.
 Future perf claims on this row need the toggle, not two runs.
 
+## CLOCK PROVENANCE: the SM clock differs BETWEEN BOOTS on dgx.casa, and nothing recorded it (2026-08-12, `row/BENCH-ASSERT-CLOCK-STATE`, #543 / #545, no GPU work)
+
+This entry adds no measurement and withdraws none. It records the **box state**
+that every existing figure was taken at and could not name, and the assertion
+that stops the next one repeating it.
+
+### The observation
+
+Same binary, same argv, same model, driver `580.159.03`, persistence `Enabled`,
+`clocks_throttle_reasons.active = 0x0` on both sides:
+
+| boot | SM clock over the captured window | our ms/step |
+|---|---|---|
+| `f6bbbfc6` | n=61, min 2398 / **med 2470** / max 2489 | **82.1664** |
+| `2fca2b02` | n=50, **flat 2190** (`clocks.max.sm` 3003, applications 2418) | **88.1000** |
+
+A **12.79%** median-clock delta, **+7.22%** step time. The control settles it:
+`marlin::Marlin`, 129 calls/step, byte-identical invocation and **no source
+change** between `a170c81c` and `4064558d0`, moved **45.2845 -> 49.6544
+ms/step = +9.65%**.
+
+### What that retracts
+
+That control drift is **larger than either deficit it was used to rank** —
+`in_proj` +2.97%, `out_proj`/`o_proj` +6.28%. Both are **NOT ESTABLISHED**:
+neither was ever taken against a clock control. The same effect explains a
+same-binary same-arm swing of 382.60 -> 357.59 us/call (-6.5%) across a reboot,
+and two probes disagreeing ~6% uniformly eight minutes apart *within one boot*
+(2398 MHz against 1781).
+
+Nothing else on this page is withdrawn. Everything recorded before today
+predates clock assertion, which is a statement about attribution, not about
+correctness: these figures carry no clock, so a difference of a few percent
+between two of them is not established **by them alone**.
+
+### What now happens instead
+
+`tools/bench/gpu_clock_state.py` samples the SM clock across the measured
+window and records min/median/max/n, `clocks.max.sm`,
+`clocks.applications.graphics`, the union of active throttle reasons,
+persistence mode, and the **boot id**. Idle samples are excluded from the
+statistics and counted, never silently dropped. `online_gate_summary.py` folds
+an arm's three legs, refuses a cross-boot pair, voids a run whose within-window
+spread exceeds **5%** or whose arms' medians differ by more than **1%**, and
+attaches the clock block to every ratio so the clock can be sized against the
+effect without leaving the row. `--allow-cross-boot` stamps a recorded caveat and
+waives **`boot_id` and nothing else** — the GPU, driver, `clocks.max.sm`,
+applications clock and persistence mode are compared across the arms
+unconditionally, because same-boot equality was the only thing standing in for
+"same machine" and the override removes it.
+
+A window must also have been **observed**: at least **30 retained busy samples**
+and a **majority** of the window busy. Without those floors the incentive is
+inverted — `spread_pct` over `n == 1` is definitionally **0.00%**, the best score
+the gate can award, so six legs each holding one busy sample and 300 idle scored
+a clean pass at `+0.00%`. Both counts are now carried in the ratio's clock block
+and printed beside the offset.
+
+All four thresholds are arguments from the table above and from the grid
+definition, derived in `.agents/specs/bench-assert-clock-state.md`. The **5%**
+spread ceiling is deliberately *not* held to the forward criterion the **1%**
+offset was chosen by; the spec says why and states the residual. The transfer
+used to report an estimated effect — **0.7548** points of kernel time per point
+of clock — is `n = 1`, is reported and never gated on, and is owed a second pair
+once #545 allows one; the offset threshold no longer rests on it, holding
+instead at the transfer's physical ceiling of 1.0.
+
+### Live state at the time of writing
+
+A read-only `ssh dgx.casa` probe on 2026-08-12 returned boot id
+`13dc5579-455c-45c8-8e4d-d09c457fa826` — a **third** boot — at the degraded
+2190 MHz with `clocks.max.sm` 3003 and applications 2418. The defect is live,
+not historical. No clock was pinned and no GPU work was queued: another session
+held `$HOME/gpu.lock`, and `nvidia-smi -lgc` would have repriced their
+in-flight measurement.
 ## SPEC-DSPARK: storage ruled out; ratio stable at ~0.966 across three sessions (2026-08-12)
 
 Question raised: are the weights on NAS, or not fully resident, distorting the

.agents/benchmarking.md

@@ -30,6 +30,68 @@ single pair is an anecdote.
 Prefer an instrument that is immune to page-cache effects (GPU-active time per
 step) over wall clock when the host is doing heavy I/O.
 
+## The clock is part of the measurement
+
+**The SM clock differs between boots and does not announce it.** On `dgx.casa`
+one boot ran the timed window at a median 2470 MHz and the next at a flat
+2190 — a 12.79% delta, with `clocks_throttle_reasons.active = 0x0` and
+persistence `Enabled` throughout, so nothing looked wrong. It repriced a
+byte-identical `marlin::Marlin` with no source change by **+9.65%**, which is
+larger than either deficit that comparison was being used to rank (#543). Two
+probes eight minutes apart *inside one boot* disagreed by ~6% uniformly.
+
+So a number is quotable only with the clock it was taken at.
+`tools/bench/gpu_clock_state.py` is the one helper that samples, folds and
+asserts it: the SM clock across the measured window (min/median/max, the
+retained sample count and the idle count), `clocks.max.sm`,
+`clocks.applications.graphics`, the active throttle reasons, persistence mode,
+and the **boot id**.
+
+**Only one harness calls it today.** `scripts/dgx-online-serving.sh` records a
+clock window per leg, and `tools/bench/online_gate_summary.py` asserts it. The
+trace and per-kernel harnesses — `finalize_*_trace.py`,
+`summarize_torch_kernels.py`, `gdn_packed_component.py` — are **not wired**, so
+a `us/call` or per-kernel figure from those paths carries **no clock
+attribution** and cannot be quoted as one. That is not a footnote: it is the
+path both retracted #543 findings came from. Wiring them is owed work tracked in
+[`specs/bench-assert-clock-state.md`](specs/bench-assert-clock-state.md); until
+it lands, import the helper and record a window yourself before ranking anything
+from a trace, or say plainly that the figure is unattributed. Any new harness
+imports this helper rather than rolling its own.
+
+Two arms on **different boots are not comparable**. The summary refuses that
+pair outright; `--allow-cross-boot` waives the **boot id and nothing else**, and
+stamps a recorded caveat rather than passing silently — the GPU, driver, maximum
+SM clock, applications clock and persistence mode are compared across the arms
+unconditionally, because a waived boot is not a waived machine. Within a run the
+SM-clock spread must stay at or below **5%**, and the two arms' medians within
+**1%** of each other. A window must also have been **observed**: at least **30
+retained busy samples** and a **majority** of the window busy, because the
+spread over one sample is definitionally 0.00% — the best score the gate can
+award — so without a floor the window nobody watched outscores the one that was.
+The argument for all four numbers, including why the spread ceiling is
+deliberately *not* held to the criterion the offset was chosen by, is in
+[`specs/bench-assert-clock-state.md`](specs/bench-assert-clock-state.md).
+
+**Pin the clocks before measuring, under the lock.** Passwordless `sudo` is
+available on `dgx`, and `-lgc` is supported:
+
+```sh
+sudo nvidia-smi -lgc 2100        # pin, before the first leg
+sudo nvidia-smi -rgc             # release, after the last one
+```
+
+Pinning is a **shared-host mutation**. Never run `-lgc` or `-rgc` while another
+session holds `$HOME/gpu.lock` — it silently reprices their in-flight
+measurement, which is the very defect this section exists for. Take the lock,
+pin, measure, reset, release. It is a **pre-measurement step**, not a standing
+configuration: leaving the box pinned makes every later run inherit a state
+nobody recorded, which is where this started.
+
+Figures recorded before 2026-08-12 predate clock assertion. They are not
+withdrawn and are not restated — they simply carry no clock attribution, so a
+delta smaller than ~10% between two of them is not established by them alone.
+
 Budget the disk before the run. A production RelWithDebInfo CUDA build tree is
 about **169 GiB** — the build contract claimed ~3 GiB until 2026-08-10, a 56x
 underestimate on the one number that decides whether a grid fits. A full disk
@@ -60,7 +122,8 @@ tool on the same workload and compare what actually ran.
 
 Record the exact build and run recipe, revisions, model hashes, environment,
 clock and contention state, raw output, and the same-binary A/B. Reproduce on an
-idle box before acceptance.
+idle box before acceptance. "Clock state" is the concrete list in §The clock is
+part of the measurement, boot id included, not a prose adjective.
 
 Record every required axis — throughput, latency, memory — as both values and
 ratios. An axis below floor is an open gap, not a rounding error.

.agents/roadmap_v1.md

@@ -49,6 +49,8 @@ issue is not yet placed. Keyed record: update in place, never append.
 | [#125](https://github.com/mudler/vllm.cpp/issues/125) | `BACKEND-VULKAN` | Vulkan on AMD Strix Halo (gfx1151) does not load | bug |
 | [#203](https://github.com/mudler/vllm.cpp/issues/203) | `BACKEND-VULKAN` | Vulkan on unified memory holds TWO copies of the weights: 27B peaks at 100.8 GiB RSS and OOM-reboots a Spark | bug |
 | [#310](https://github.com/mudler/vllm.cpp/issues/310) | `BACKEND-VULKAN` | docs/FEATURES.md understates Vulkan: says decode 4.24 vs 4.35 where the binding figure is 4.36 vs 4.35 | bug |
+| [#543](https://github.com/mudler/vllm.cpp/issues/543) | `BENCH-ASSERT-CLOCK-STATE` | The SM clock differs BETWEEN BOOTS without throttling and no manifest records it: a 12.79% clock delta repriced a byte-identical `marlin::Marlin` by +9.65%, larger than either deficit it was used to rank, spec [`bench-assert-clock-state.md`](specs/bench-assert-clock-state.md) | bug |
+| [#545](https://github.com/mudler/vllm.cpp/issues/545) | `BENCH-ASSERT-CLOCK-STATE` | BLOCKER: `dgx.casa` MTBF is now shorter than one four-leg measurement chain, so same-boot capture cannot be assumed and cross-boot is the normal case | bug |
 | [#520](https://github.com/mudler/vllm.cpp/issues/520) | `BENCH-ORACLE-PIN-RECONCILE` | The benchmark harness ENFORCES the 0.25.0 rollback oracle and would REJECT the recorded pin `555967922`, so nobody could have measured correctly even deliberately, spec [`bench-oracle-pin-reconcile.md`](specs/bench-oracle-pin-reconcile.md) | bug |
 | [#522](https://github.com/mudler/vllm.cpp/issues/522) | `BENCH-ORACLE-PIN-RECONCILE` | BLOCKER: the pinned oracle venv lacks `pandas`, which `vllm bench serve` requires — installing it mutates a shared oracle and needs recorded authority | bug |
 | [#224](https://github.com/mudler/vllm.cpp/issues/224) | `ENG-DOCS-SITE` | Publish `docs/` as a GitHub Pages site without owning a second copy | feature |