gc-ratchet has measured nothing since 2026-08-05: harness validation fails before the probes run (12_large_live_set spread 6768)

[proggeramlug]

The GC ratchet has measured nothing since 2026-08-05

gc-ratchet fails on main in test_pinned_artifact_retention_is_deterministic:

AssertionError: 6768 != 0 : 12_large_live_set.heap_used_bytes was not
deterministic when pinned; it must not be in a gating family

That assertion lives in the "Harness unit tests and artifact validation" step, which runs before the measurement step. So the probes themselves — all twelve — have not executed on any branch since this broke. The job is red, and the thing it exists to measure is not running at all. That is CLAUDE.md's fourth failure mode ("the gate runs but its subject never did") in its most expensive form, compounded by gc-ratchet not being a required context, so the red blocks nothing either.

It has failed on every completed main run since 2026-08-05T15:07Z, which is the window in which #7446 re-pinned the artifact.

The assertion is right; the artifact is unfit

This is not an over-strict test. benchmarks/gc_ratchet/tolerances.json justifies gating heap_used_bytes on exactly this basis:

"Observed spread 0.000% over 21 runs on all 8 probes: bit-identical. The band is therefore pure anti-brittleness margin … not a noise allowance."

The pinned artifact contradicts its own premise. Every other probe still holds it:

probe	heap_used_bytes spread
01_nursery_churn … 11_collect_at_depth (11 probes)	0
12_large_live_set	6768

and the samples show it is a single outlier, not noise:

[59943824, 59943824, 59950592, 59943824, 59943824, 59943824, 59943824]
                     ^^^^^^^^ one run, +6768 bytes

heap_total_bytes on the same probe is spread 0 across all seven.

Two possible causes, and they need different fixes

1. The re-pin captured a bad sample. gc-ratchet: re-pin baseline on the dedicated bench host (ratchets #7443's retention win) #7446 ran on the dedicated bench host; one run in seven landing 6,768 bytes high could be incidental (a lazily-created runtime global, an interned string) rather than a property of the probe. Fix: re-pin 12_large_live_set, ideally with more repeats.
2. 12_large_live_set genuinely became non-deterministic. 6,768 bytes on a 60 MB heap is 0.011% — small, but the gating premise is bit-identical, and the other eleven probes still are. If retention on the largest live set is now sample-dependent, that is a real finding about the collector and worth knowing on its own.

Distinguishing them is cheap: re-run that probe ~21× on a quiet host and see whether the outlier recurs.

Why this matters right now

Roughly fifteen GC-affecting changes landed today — layout installation at allocation (#7532), the shape-install memo (#7535), write-barrier gating (#7536), the tape leaving the old generation (#7553), plus a spill-path rooting fix (#7546). None of them were checked by the ratchet, because it has not measured anything since before they were written. Several carried their own PERRY_GC_DIAG A/Bs, which is why I merged them, but that is per-PR diligence rather than a standing gate.

Suggested acceptance

• 12_large_live_set.heap_used_bytes deterministic across ≥21 runs on the pinned host, or the probe explicitly moved out of the gating family with the reason recorded next to it (the mechanism the assertion's own message points at).
• The ratchet's measurement step observed green on main at least once, so we know the probes run.
• Consider promoting gc-ratchet to a required context after that first green — a new gate has never been green, and promoting it while red would block every open PR.

[proggeramlug]

Experiment: it is cause 2 — genuinely non-deterministic

Ran 12_large_live_set 21 times on the pinned quiet mini (load 1.8, main @ 52f7dae1f, release build), reading heap_used_bytes from PERRY_GC_DIAG=1:

5 distinct values across 21 runs:
  42612752  42613184  42613328  42614048  42617288
  spread = 4536 bytes

So this is not a bad sample captured by #7446's re-pin. The probe's retention is genuinely sample-dependent, and the pinned artifact's single outlier in seven was simply the first time the harness happened to catch it. The gating premise in tolerances.json — "observed spread 0.000% over 21 runs on all 8 probes: bit-identical" — was written when there were 8 probes; 12_large_live_set is the 12th, and it does not share that property.

Note the assertion is still doing exactly its job: it refused to gate on a metric that cannot carry a gate. The bug is that the artifact was pinned as if it could.

The suggested fix is not currently expressible

tolerances.json is keyed per metric per profile, not per probe:

profiles: shared_ci, pinned_host
each -> heap_used_bytes, heap_total_bytes, minor_cycles, ... wall_ms

So "take 12_large_live_set.heap_used_bytes out of the gating family" — what the assertion's own message instructs — cannot be said today. The only available lever turns heap_used_bytes gating off for all twelve probes, which would discard a genuinely bit-identical signal on the other eleven. That is too blunt to accept.

Three options, in my order of preference:

1. Add a per-probe override (probes: { "12_large_live_set": { "heap_used_bytes": { "gating": false, "rationale": "…" } } }), keeping the other eleven gated. Smallest change that makes the artifact honest and gets the probes running again.
2. Find and remove the non-determinism. 4,536 bytes on a ~42.6 MB heap is 0.011%, and every other probe is bit-identical, so something specific to the largest-live-set shape is sample-dependent. This is the best outcome but a real investigation, and it should not block the ratchet from measuring in the meantime.
3. Re-pin with a tolerance band for this metric — but that contradicts the documented rationale, which is explicit that the band is anti-brittleness margin and not a noise allowance.

One more observation, offered carefully

Today's runs read ~42.6 MB where the pinned artifact records a median of 59.94 MB — about 29% lower retention. I am not claiming that as a win: the artifact was captured on the dedicated bench host and this was the mac mini, and I have not controlled for host. But if a re-pin on the correct host reproduces it, the ratchet has a substantial retention improvement to record from today's GC work (#7532, #7535, #7536, #7546, #7553), and that is worth capturing deliberately rather than discovering by accident later.

[proggeramlug]

#7557 adds the per-probe override and gets the probes running again. Four
things worth having here.

The probes run again, and the gate has real output. Full measure +
check end to end on two machine classes — the pinned quiet host
(perry-macos, load 2.2, pinned_host) and a MacBook Pro under load 21
(shared_ci). Twelve probes each, all compiled, all ran, all passed the Node
oracle diff, and retention plus the counters reproduced bit-for-bit across the
two hosts. Both arms report the same ten gating breaches, invisible since
2026-08-05. Most look like improvements wearing a two-sided band (03 and 04
shed 40–95% of their copy/promote work while retention fell 49% and 22%), but
two are not: 05_closure_capture retains +16.44% with every collector counter
at +0.00% — same work, more retained — and 02_survivor_promotion is +2.77%
on the same shape. The PR deliberately does not re-pin; those two want a look
before they are accepted.

A second hole, found on the way. Six of the twelve probes pin
minor_cycles at 1 and the allowance floor is also 1, so a collapse from 1 to 0
was delta == -allowance and scored ok. A collector that stops running
copying minors — the largest regression this ratchet exists to catch — was
being reported as passing. check now asserts liveness instead of inferring it
from the bands.

The non-determinism is a real finding: filed as #7558. It is not the
collector's steady state, it is the measurement. Every probe reads
process.memoryUsage() after an explicit gc(), and an explicit gc() runs a
full mark-sweep with a forced conservative stack scan
([gc-scan-fallback] site=manual_collect automatic=false, on every run).
Diffing two disagreeing traces shows the minors, tenuring decisions, step cycles
and every copy/promote counter matching exactly, with the sole difference in the
last collection's freed_bytes. Under PERRY_CONSERVATIVE_STACK_SCAN=off the
probe reports 51,668,688 bytes on 8 consecutive runs, bit-identical, against
59,943,080–59,952,824 by default. So the variance is entirely false roots — and
the conservative scan is systematically retaining 8.28 MB, 16% of what that
probe reports as its retained heap. The eleven small probes stay bit-identical
because their live sets are one to two orders of magnitude smaller.

The ~29% retention win does not reproduce, and one premise was off. The
pinned artifact was captured on perry-macos.fritz.box — the same Mac mini as
your 21 runs, not a separate bench host; the artifact's own notes call it "the
dedicated bench host … which replaces the shared MacBook". So there was no host
difference to control for. Measured with the harness protocol:

build	host	12_large_live_set.heap_used_bytes
pinned 5e236e6e2, 2026-08-05	perry-macos	59,943,824
52f7dae1f (the commit you read 42.6 MB at)	MacBook	59,943,224 – 59,949,920
be1fc80f8 (main + #7557)	perry-macos	59,943,896

Retention on that probe has not moved at all across the 2026-08-06 batch.
PERRY_GC_DIAG=1 does not change it (the harness's traced/untraced split still
holds) and neither does the auto-optimizer, so there is no 29% win to record —
whatever produced 42.6 MB was not this probe under the harness protocol at that
commit.

CI has a deep runner backlog and may not report on #7557; the validation above
is local and is what it rests on.