gc: 05_closure_capture retains +16.44% with every collector counter at +0.00% — surfaced when the ratchet started measuring again

[proggeramlug]

05_closure_capture retains +16.44% with every collector counter unchanged

Surfaced the moment the GC ratchet started measuring again (#7557 — its probes had been skipped entirely since 2026-08-05 because a harness-validation step ran before them).

Of the 10 gating breaches that became visible, most are improvements wearing a two-sided band. This one is not. Reported by the agent that restored the ratchet, on two machine classes (the pinned quiet mini and a MacBook), with retention and counters reproducing bit-for-bit across hosts:

probe	retention	collector counters
05_closure_capture	+16.44%	+0.00% on every one
02_survivor_promotion	+2.77%	—

Same minor_cycles, copied_objects, copied_bytes, promoted_objects, promoted_bytes, freed_bytes — identical work, more retained. That combination is the interesting one: it rules out "the collector ran differently" and points at something being kept alive that previously was not. A closure-capture probe holding 16% more is exactly the shape of an over-broad root or a capture cell that stopped being released.

Why it was invisible

The ratchet did not run between 2026-08-05 and #7557. In that window roughly fifteen GC-affecting changes landed: #7532 (typed-shape layout at allocation), #7535 (shape-install memo), #7536 (write-barrier gating), #7546 (spill-path value rooting), #7553 (tape out of the old generation), #7550/#7552 (for-init type propagation), plus the rooting stack #7495/#7516/#7527/#7529.

Several of those add rooting. #7516 alone rooted 14 sites, #7527 more, #7529 three more — and a root that is broader or longer-lived than necessary retains. That is the first place I would look, and it is worth saying plainly: these fixes were correct and must not be reverted (they closed real use-after-frees), but "more correct rooting" and "more retention" are the same lever pulled in opposite directions.

Suggested method

Bisect the probe's retention across that window with the ratchet itself now that it works — gc_ratchet.py measure --probe 05_closure_capture at each candidate commit on the pinned host. The counters being flat means a PERRY_GC_DIAG trace diff between the endpoints should show the difference as a retained-object delta rather than a cycle-behaviour one.

Note the measurement caveat from #7558: every probe reads process.memoryUsage() after an explicit gc(), which forces a conservative stack scan that systematically over-retains (measured at 8.28 MB / 16% on 12_large_live_set). Run the comparison with PERRY_CONSERVATIVE_STACK_SCAN=off as well, so a false-root artifact is not mistaken for a real regression — or confirmed as one.

Acceptance

Either the +16.44% is attributed to a specific commit and fixed, or it is shown to be a measurement artifact of the conservative scan and the probe's methodology is corrected. Do not re-pin it away without an answer — that is the decision the ratchet exists to force.

[proggeramlug]

Answer: nothing extra is being retained. It is a measurement artifact of the conservative scan, and it is exactly one nursery block.

Investigated in #7571 (instrument + docs; no band widened, nothing re-pinned — this issue's decision is still open, see the bottom).

The finding

05_closure_capture's retention measured with PERRY_CONSERVATIVE_STACK_SCAN=off is 5,329,880 bytes at both endpoints, to the byte. The +16.44% is false-root residue going from one 1 MiB nursery block to two.

PERRY_GC_DIAG counts it directly at the probe's measurement collection:

arm	general blocks marked live	heap_used_bytes
5e236e6e2 (the pin), default scan	6	6,378,392
5e236e6e2 (the pin), scan off	5	5,329,880
b5a2954ec (v0.5.1321), default scan	7	7,426,960
b5a2954ec (v0.5.1321), scan off	5	5,329,880

Both minors are bit-identical across the arms (copied_objects 1422 + 1614 = 3036, exactly the pinned value), and with the scan off both arms free exactly 8,124,192 bytes at the final mark-sweep and retain exactly 425 forwarded stubs. The precise root set marks the same five blocks at both endpoints.

So the issue's inference was half right. "Identical work, more retained" does rule out a change in collector behaviour — but the thing kept alive is not the program's data. It is one more stale word on the native Rust stack at the instant the probe calls gc().

Why the measurement does that

Two properties of the measurement point, neither about the collector's retention:

1. The measurement forces the conservative scan. Every probe reads process.memoryUsage() immediately after an explicit gc(), and an explicit gc() is the one site in Perry that forces the conservative native-stack scan (ManualGcScanGuard, GC: explicit gc() (full collect, default auto stack-scan) reclaims live top-level locals — string fields read back as garbage #4977 — production resolves to SkipDisabled). PERRY_GC_DIAG shows the suite's entire conservative-scan census is the probes' own gc() (site=manual_collect automatic=false on all twelve), plus one automatic old_reclaim_alloc_point on 12_large_live_set. The gated number is taken under a root set nothing else in the language uses.
2. js_arena_stats sums block offsets, not live bytes. A bump pointer never moves backwards and a block holding one marked object cannot be reset, so a single stale stack word costs a whole 1 MiB block — ~26,000x amplification. This is the nursery's version of the old-generation accounting problem GC: old-gen fragmentation — scattered survivors pin 105 MB of blocks for a ~1 MB live set #7437/gc: old-generation hole free list — swept holes become reusable capacity (#7437) #7443 fixed for OLD_ARENA by subtracting swept holes.

The whole window, all twelve probes

Both arms built --release -p perry -p perry-runtime-static -p perry-stdlib-static with a cold object cache. The baseline arm reproduces the pinned artifact byte-for-byte on all twelve probes, which is what makes the head arm comparable.

probe	heap_used_bytes base → head	scan-off (precise) base → head
01_nursery_churn	7,325,584 → 6,277,048 (−14.3%)	5,228,512 → 5,228,512 (0)
02_survivor_promotion	9,418,232 → 9,678,792 (+2.77%)	9,418,232 → 9,416,632 (−1,600)
03_cross_gen_writes	2,793,472 → 1,427,664 (−48.9%)	2,465,776 → 1,394,880 (−1,070,896)
04_dead_after_deep_stack	6,257,040 → 4,897,320 (−21.7%)	5,208,472 → 4,891,968 (−316,504)
05_closure_capture	6,378,392 → 7,426,960 (+16.44%)	5,329,880 → 5,329,880 (0)
06_string_retention	7,058,896 → 7,058,896	4,961,800 → 4,961,800 (0)
07_array_grow_evacuate	15,649,104 → 15,649,104	unchanged (0)
08_map_set_sidetables	1,548,960 → 1,548,960	unchanged (0)
09_try_catch_roots	7,068,864 → 6,020,320 (−14.8%)	4,982,528 → 4,982,528 (0)
10_store_receiver_across_alloc	4,666,248 → 4,666,248	unchanged (0)
11_collect_at_depth	7,390,400 → 7,390,400	unchanged (0)
12_large_live_set	59,943,824 → 59,943,896	51,668,568 → 51,668,568 (0)

heap_used_bytes moved on five probes, always by whole blocks, three down and two up. Precise retention moved on two, and both went down.

That cuts both ways, which is the part worth keeping: 01_nursery_churn's −14.3% and 09_try_catch_roots' −14.8% are also false-root residue, not wins. Their precise retention is byte-identical.

02_survivor_promotion (+2.77%) is the same shape one survivor block down: excess 0 → 262,160 B (≈256 KiB), and its precise retention fell by 1,600 bytes. Same cause, and it needs no separate investigation.

And the metric is not reporting the live set at all

05_closure_capture drops everything before it measures, so its live set is ~zero for every workload size. Sweeping BATCHES with one compiler:

BATCHES=690   default=6,812,880    scanoff=4,715,800    excess=2,097,080
BATCHES=700   default=7,426,960    scanoff=5,329,880    excess=2,097,080
BATCHES=702   default=6,501,264    scanoff=4,404,184    excess=2,097,080   <- sawtooth
BATCHES=710   default=6,992,528    scanoff=4,895,448    excess=2,097,080

A 1 MiB sawtooth over a constant zero live set. What is left over is the un-reset tail blocks' bump pointers — arena_reset_empty_blocks never resets the current block or the four before it. Two batches of workload (123 KB) move the gated number by 925,696 bytes.

Which commit?

None, meaningfully. Bisecting further would attribute a coincidence: any codegen or allocation change perturbs the native stack image at the gc() callsite, and the metric quantises the result to whole blocks. The rooting stack (#7495/#7516/#7527/#7529) is exonerated by the scan-off column — a broader or longer-lived root would raise the precise number, and it did not move by one byte on ten of twelve probes.

What #7571 lands

gc_ratchet.py classify — runs every probe under both scan modes, prints conservative / spread / precise / excess and the scan-site census. A row whose excess moved and whose precise did not is a false-root artifact; a row whose precise moved is a real retention change. It refuses to tabulate a probe whose stdout changes when the scan is disabled (then the scan was load-bearing for that probe's correctness and its precise number is not evidence), and refuses to report a non-bit-identical precise reading — while allowing the conservative reading to vary and reporting its spread, because that spread on 12_large_live_set is why #7554 had to stop gating the cell. Plus the finding written into the module docstring, the README, and tolerances.json, and six tests.

The decision that is still yours

Per this issue's own instruction, #7571 does not re-pin and does not widen a band. 05_closure_capture and 02_survivor_promotion stay red. The options, with what I think is wrong with each:

1. Re-pin those two cells, recording "false-root residue moved by one block; precise retention 5,329,880 → 5,329,880 / −1,600" in --notes. Cheapest, honest, and needs the pinned host. It does not stop the next unrelated codegen change from moving another probe by a block.
2. Change what the retention family measures — record retention from a second untraced pass with PERRY_CONSERVATIVE_STACK_SCAN=off and gate that, keeping the conservative reading as an informational row. This makes the number semantic, keeps the band tight (a genuinely rooted extra block still costs 1 MiB and still goes red), and lets gc-ratchet has measured nothing since 2026-08-05: harness validation fails before the probes run (12_large_live_set spread 6768) #7554's 12_large_live_set exclusion be deleted — its scan-off reading is bit-identical at 51,668,568. Costs a full re-pin. The objection to weigh is that the ratchet was built partly to catch the campaign's move toward a conservative scan; my answer is that the campaign's scan would be the automatic production path, not this forced-at-gc() one, but that is a call for whoever owns the campaign.
3. Fix the amplifier: make js_arena_stats subtract swept nursery free space the way GC: old-gen fragmentation — scattered survivors pin 105 MB of blocks for a ~1 MB live set #7437/gc: old-generation hole free list — swept holes become reusable capacity (#7437) #7443 made it subtract old-gen holes. Then one falsely-retained object costs ~40 bytes instead of 1 MiB and the metric becomes a live-set measurement. Biggest change, user-visible in process.memoryUsage().heapUsed, and the nursery has no free-list to count from today — but it is the only option that fixes the 26,000x amplification rather than working around it.

Widening the band is not on the list. heap_used_bytes's churn quantum is one whole block, which is also the unit test_one_falsely_retained_nursery_block_fails asserts must go red. A quantity whose noise quantum equals its signal quantum cannot be gated; the fix is to measure a different quantity, not to loosen the band on this one.

_{Measured on a MacBook Pro (10 cores); every number above is retention or a collector counter, both of which are load-independent and were verified bit-identical across repeats. The baseline arm reproducing the pinned quiet-host artifact byte-for-byte on all twelve probes is the cross-host check.}