BENCH-ASSERT-CLOCK-STATE: the SM clock moves 12.8% between boots and nobody recorded it (#543, #545) - #568
Merged
Merged
Conversation
…d no manifest records it (#543) FOLLOWING_AGENTS_PROTOCOL On dgx.casa, driver 580.159.03, persistence Enabled and clocks_throttle_reasons.active = 0x0 on both sides, one boot ran the timed window at a median 2470 MHz and the next at a flat 2190. That is a 12.79% delta, and it repriced a byte-identical marlin::Marlin -- 129 calls/step, same invocation, NO source change between a170c81 and 4064558 -- from 45.2845 to 49.6544 ms/step, +9.65%. That control drift is LARGER than either deficit it was used to rank (in_proj +2.97%, out_proj +6.28%), so both are NOT ESTABLISHED. The same effect explains a same-binary same-arm swing of 382.60 -> 357.59 us/call across a reboot, and two probes disagreeing ~6% uniformly eight minutes apart inside ONE boot. Nothing in the tree records it. `grep -rn 'clocks\.' tools/ scripts/` returns one hit, in prose. The three harness scripts that touch nvidia-smi write -q -d PERFORMANCE,TEMPERATURE,POWER into a text blob no summary reads. No manifest carries a clock, a boot id, or a throttle state, so no ratio in this repository can be attributed to the clock it was measured at. Same class as #375 and #520: an environment variable nobody wrote down, silently repricing every number. The spec argues both thresholds from the data rather than from taste. Within-run spread 5.0%: it must ACCEPT the only clean window we have, (2489-2398)/2470 = 3.68%, and REJECT the within-boot disagreement at ~26%. Cross-arm median offset 1.0%: at the measured transfer of 0.754 points of kernel time per point of clock that estimates to ~0.75%, under the 2.97% smallest deficit anyone ranks. The transfer is n = 1, is reported and never gated on, and is recorded as owed. No measurement is taken here and none is restated. #545 is linked because the box's MTBF is now shorter than one four-leg chain, which is why a cross-boot override exists at all -- it waives identity, never state. Following-Agents-Protocol: true AI-Assisted: true Assisted-by: Claude:claude-opus-5 [ClaudeCode]
…ross-boot ratio (#543) FOLLOWING_AGENTS_PROTOCOL One helper, tools/bench/gpu_clock_state.py, does the sampling, the folding and the assertions; other harnesses import it. Standard library only, like serve_low_common, so every assertion runs in CPU CI with no GPU and no nvidia-smi -- the 50 helper cases use synthetic manifests and a stub probe. RECORDED per leg, beside memory/, by a background sampler across the timed bench loop: the SM clock over the window (min/median/max and n), clocks.max.sm, clocks.applications.graphics, the union of active throttle reasons, persistence mode, and the boot id. Idle samples are EXCLUDED from the statistics and COUNTED, never silently dropped -- a clock read while the GPU does nothing priced no work, and the window necessarily contains the harness's own gaps between concurrency points. An entirely idle window has nothing to attribute and is refused. ASSERTED in online_gate_summary through the existing `reasons` seam, which is this harness's spelling of NOT ESTABLISHED, so nothing new is invented and no gate is weakened: a cross-boot pair is refused; a within-run spread above 5.0% or a cross-arm median offset above 1.0% voids the ratio; a non-benign throttle bit or persistence-off voids the leg; an arm whose three legs straddle two boots will not fold. Every ratio carries a clock block with both medians, both spreads, the signed offset and the estimated effect, and the report prints it next to the verdict, so a reader can size the clock against the effect without leaving the row. --allow-cross-boot waives IDENTITY, never STATE. It exists because #545 makes same-boot capture of a four-leg chain unreliable and a gate nobody can satisfy is a gate everybody routes around; the spread and offset rules still apply and the comparison is stamped with a recorded caveat rather than passing silently. Consequence worth stating rather than discovering: re-summarizing an existing evidence tree now yields NOT ESTABLISHED, because no tree on disk carries a clock record. That is the correct answer -- those trees genuinely cannot be attributed to a clock -- and it is why the next commit adds a note instead of editing any recorded number. 233 -> 295 tests, 0 removed, proven by a sorted test-name diff. Twelve mutations with count == 1 anchors, restored byte-for-byte by sha256. THREE SURVIVED the first round: both ratio-level clock terms and the missing-record reason. #520's lesson repeating -- the leg-level reason already voided every gate assertion, so each ratio-level site was dominated, and a single gate_pass assertion let the throughput and memory ratio families mask each other's removal. Two cases now assert the two families' binding_eligible SEPARATELY on a pair whose arms are individually clean, and a third pins the text that names the offending arm, which is the only thing the reader has. Twelve of twelve RED. The trace/per-kernel harnesses are NOT wired yet and are recorded as owed in the spec. That is the path #543's retracted findings came from, so it matters that it is named rather than assumed covered. Following-Agents-Protocol: true AI-Assisted: true Assisted-by: Claude:claude-opus-5 [ClaudeCode]
…y which numbers predate the assertion (#543) FOLLOWING_AGENTS_PROTOCOL benchmarking.md gains the method: what a leg records, why two arms on different boots are not comparable, and the operational fix. Clocks can be pinned with `sudo nvidia-smi -lgc <freq>` and released with `-rgc` -- passwordless sudo is available on dgx and -lgc is supported there. Pinning is a SHARED-HOST MUTATION and must not be done while another session holds $HOME/gpu.lock: it would silently reprice their in-flight measurement, which is exactly the defect the section exists for. It is recommended as a pre-measurement step UNDER the lock, not as a standing configuration -- leaving the box pinned makes every later run inherit a state nobody recorded, which is where this started. NO RECORDED NUMBER IS EDITED. AGENTS.md: never delete evidence. Instead each surface that cites the old figures says they predate clock assertion -- BENCHMARKS.md as an at-a-glance row (the prose there is at its paragraph and cell budget, so the note goes where a result belongs, in a table row), and STATUS.md next to the parity-pin note that already explains a different attribution defect on the same runs. Neither withdraws nor restates anything: these numbers simply carry no clock, so a difference of a few percent between two of them is not established BY THEM ALONE. benchmark-record.md, which is the append-only log, carries the forensics: the two-boot table, the marlin control, the two findings it retracts (in_proj +2.97% and out_proj +6.28%, neither ever taken against a clock control), the thresholds and their derivation, and the live state at the time of writing. A read-only probe on 2026-08-12 returned a THIRD boot id, 13dc5579, at the degraded 2190 MHz with clocks.max.sm 3003 -- the defect is live, not historical. No clock was pinned and no GPU work was queued in this row: another session held the lock. Following-Agents-Protocol: true AI-Assisted: true Assisted-by: Claude:claude-opus-5 [ClaudeCode]
Carries fafa16f (Nemotron35LightningSnapshot declaration order, #556) and d861819 (DSA top-k bounds, #505) onto the review-fix branch. No conflict: the merged paths and the clock-state paths are disjoint. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: Claude:claude-opus-5 [ClaudeCode]
…ct 0.00%, and the override waived the machine (#543, #545) Four review findings against row/BENCH-ASSERT-CLOCK-STATE. The design, seam, records discipline and the twelve original mutations stand; these are defects in what the assertions cover. F1. Idle dilution. `validate_clock_record` required only `n >= 1`, and `spread_pct` over `n == 1` is definitionally 0.00% -- the BEST score the gate can award -- so the window the sampler barely observed outscored the one it watched. Six legs each holding ONE busy sample and 300 idle gave `gate_pass: true`, `clock reasons: []` and a report line reading `+0.00%`. `idle_samples_excluded`, the one field that would have betrayed it, was written to `r<N>.summary.json` and then never asserted, never folded into the ratio's `clock` block and never printed. `clock_reasons` now floors both the retained busy count (30) and the busy fraction (50%), and both counts are carried in the clock block and printed on the report's `observed:` line. The floors are derived from the sampling rate and the grid: at `--interval 1` a busy sample is a second of OBSERVED BUSY GPU, and the smallest configured leg (1920 sequential decode steps plus 48 prefills of 1024 tokens) would have to average under ~16 ms/step to finish inside 30 of them, against 82-88 ms/step measured on this box. 30 also sits 40% below the smaller of the two real #543 windows (n=61, n=50), so it accepts both. F2. `--allow-cross-boot` waived hardware identity as a side effect. `compare_clock_records` never compared `gpu_name`, `driver_version`, `clocks_max_sm_mhz`, `clocks_applications_graphics_mhz` or `persistence_mode` ACROSS the arms; `STATIC_FIELDS` was enforced only within a window and between one arm's legs. Same-boot equality was the implicit proxy for "same machine", and the override removed it with nothing in its place: an NVIDIA GB10 / 580.159.03 / max 3003 arm compared CLEAN against an NVIDIA H100 80GB HBM3 / 550.54.15 / max 1980 one, caveated only as "different boots". `STATIC_FIELDS` is now asserted across the arms unconditionally. #545 makes the override the normal path, so this was reachable, not theoretical. F3. The test named "the override does not waive state" was dominated. Rewriting `compare_clock_records` to `reasons = [] if allow_cross_boot else [...]` (R15) survived the full suite GREEN: one test asserts an OFFSET reason, appended outside the mutated expression, and the other writes an over-spread LEG already voided at leg level by `_clock_for_leg`. Exactly one check lives only at the compare site -- the merged-arm spread, an arm whose three legs each sat flat at a different clock -- because `merge_clock_records` raises only on a straddled boot and on a static field. A summary-level case now pins it: three individually-flat legs at 2470 / 2300 / 2190 on ONE boot under `allow_cross_boot=True`, with the vLLM arm at the merged median so the offset stays 0.00% and cannot mask the removal. R15 is now RED. F4. The sampler was orphaned on every abort path. `clock_pid` was `local` to `run_leg` and invisible to the EXIT trap, which reaps only `profiled_pid`, `spid` and `mpid`; under `set -euo pipefail` a failing `online_gate.py bench` exits without reaching `stop_clock_sampler`. The sampler had neither `--pid` nor `--max-duration`, so its only stop condition was a SIGTERM that path skips, and it went on polling `nvidia-smi` once a second forever on a box another session may hold `$HOME/gpu.lock` on. The spec claimed "the same shape as the memory sampler"; it was not, and it now is on BOTH counts -- `clock_pid` is a global reaped by `cleanup_server`, and the sampler is bounded by `--max-duration` (7200 s, 4x the driver's own readiness budget, a bound on an orphan and not a budget on a leg). F5. The two thresholds were argued to different standards. The 1.0% offset was derived forward through the n=1 transfer; the same criterion was never applied to spread, where 5.0 x 0.7548 = 3.77% exceeds the 2.97% smallest deficit ranked. Spread is NOT tightened -- the forward criterion would demand <=3.93%, which sits 0.25 points above our only clean capture (3.68%) and would void it on a noisier-but-healthy day -- so the reason it may exceed is stated instead: the offset bounds a SYSTEMATIC arm-to-arm difference that transfers whole into the ratio, while spread bounds DISPERSION inside one arm's window, most of which is common to both arms and whose residual is the difference of two medians that the offset rule already bounds. The residual is recorded: passing spread establishes that an arm was ONE state, not that a sub-4% deficit is established. The offset argument is also strengthened off the n=1 coefficient entirely: for any kernel whose time scales with clock the transfer is bounded above by 1.0, so a 1.0% offset implies at most a 1.0% effect on PHYSICS, still under 2.97%. The measured 0.7548 sits below that ceiling exactly as a partly memory-bound kernel should. The coefficient is not a gate TERM (proven by the mutation set) but it was a gate PREMISE; the physics bound retires it from that role. F6. `.agents/benchmarking.md` -- the file an agent reads before measuring -- claimed "Every leg records..." and "other harnesses import it..." in the present tense with no caveat. Only `dgx-online-serving.sh` records anything; `finalize_*_trace.py`, `summarize_torch_kernels.py` and `gdn_packed_component.py` are unwired, and that trace path is exactly where both retracted findings came from. The guide now says so, matching the spec's own Scope and Owed. Recorded, not fixed, in the spec's Owed: `all-runs.json` carries no clock block and no `allow_cross_boot` flag (report.md does print `(OVERRIDDEN)`/`CAVEAT:`, so it is not buried for a human, but the JSON a consumer parses is silent); `compare_clock_records` builds keys as `f"{ours_label}_boot_id"`, so identical labels would collapse one arm into the other, unreachable from current call sites; and the busy floors are derived but not yet measured against a real leg. Gates. `python3 -m unittest discover -s tests/tools -t .`: 295 -> 310, exit 0, 0 removed, proven by a sorted test-name diff. 24 mutations (the original 12 plus R13-R24), one at a time, `count == 1` anchors, restored byte-for-byte and verified by sha256: 24/24 RED. R17 survived the first pass and was a genuine gap in the new validation, not a dominated site, so two cases now pin it; M11 first read SURVIVED because the mutation itself was inert (`[] or [...]` is the non-empty list), which is why a byte-level sha256 proves the file changed and not that the behavior did. Both reviewer bypass probes re-run and now CAUGHT. `bash -n` and `shellcheck -S warning` clean on the driver. `scripts/agent-preflight.sh --staged` clean. No GPU work queued and no clock pinned: another session may hold `$HOME/gpu.lock`. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: Claude:claude-opus-5 [ClaudeCode]
Keeps the PR current. This row's own changes are python/docs/shell only and touch no C++; the windows-msvc failures on the stale PR head came from main's side. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: Claude:claude-opus-5 [ClaudeCode] # Conflicts: # .agents/benchmark-record.md
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
merge: BENCH-ASSERT-CLOCK-STATE -- the SM clock moves 12.8% between boots and nobody recorded it (#543, #545)
On this GB10 the SM clock differs BETWEEN BOOTS with no throttling active:
median 2470 MHz on one boot, pinned flat at 2190 on the next (max 3003,
applications 2418, throttle 0x0, persistence Enabled). 12.8% clock -> +7.2% step
time. The control proves it: marlin::Marlin, 129 calls/step, byte-identical
invocation, NO source change between a170c81 and 4064558, moved
45.2845 -> 49.6544 ms/step = +9.65%.
That drift is LARGER than either per-call deficit it was used to rank (in_proj
+2.97%, out_proj/o_proj +6.28%). Those deficits are not refuted -- they were
never taken against a clock control, so they are NOT ESTABLISHED. The same effect
explains a same-binary same-arm swing of 382.60 -> 357.59 us/call across a
reboot, and two probes disagreeing ~6% uniformly eight minutes apart in one boot.
So every leg now RECORDS SM clock min/median/max/n and spread, the max and
applications clocks, the union of throttle reasons, persistence, driver, GPU
name, retained-vs-idle sample counts, and boot_id. And the summary ASSERTS:
boot_id equality between arms, within-run spread, cross-arm median offset, benign
throttle bits, persistence. A ratio that cannot be attributed to a clock state is
voided, and gate_pass requires every ratio eligible.
A first review broke it twice, end to end, and both are fixed here:
over n == 1 the spread is definitionally 0.00%. Three busy samples per arm
out of 903 passed with no reasons. The incentive was inverted: a window
actually observed was MORE likely to be voided. Floors now require 30 busy
samples and a 50% busy fraction, both derived from the sampler rate and the
leg grid rather than from what happens to pass, and both surfaced in the
report.
NVIDIA GB10 on driver 580.159.03 against an H100 80GB on 550.54.15, telling
the reader only "different boots". STATIC_FIELDS are now asserted ACROSS arms
unconditionally, ahead of any override; the override waives boot_id alone.
24 mutations, 24 RED, restored by sha256. Two of them taught something worth
keeping: one survived because a leg-level reason already dominated it -- the
third instance of that shape today -- and one was SEMANTICALLY INERT, since
[] or [...] evaluates to the non-empty list, so a checksum proved the file
changed but not the behaviour. Both are recorded as methodology notes.
Thresholds are argued, not fitted. The 1.0% cross-arm offset rests on physics,
not on n=1: for any kernel whose time scales with clock the transfer is bounded
above by 1.0, so 1.0% offset implies <=1.0% effect, under the 2.97% smallest
deficit ranked; the measured 0.7548 sits below that ceiling as a partly
memory-bound kernel should. The 5.0% spread rule deliberately does NOT meet that
criterion, and the spec says so: passing spread establishes an arm was one state,
not that a sub-4% deficit is established.
Scope is stated where it is read, not only where it is specified: only
dgx-online-serving.sh records. finalize_*_trace.py, summarize_torch_kernels.py
and gdn_packed_component.py do not -- and the trace path is exactly where both
retracted findings came from.
tests/tools 233 -> 310, 0 removed. Preflight is green except
test_cpu_x86_llamacpp_floor, which I reproduced FAILING IDENTICALLY on a clean
origin/main checkout at load 110 and which this branch does not touch: narrowly
waived, environmental.
FOLLOWING_AGENTS_PROTOCOL
Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: Claude:claude-opus-5 [ClaudeCode]