gc(): the forced conservative stack scan makes every retained-heap reading nondeterministic, and 16% too large on a 50 MB live set

[proggeramlug]

Split out of #7554, which asked whether 12_large_live_set's retention
non-determinism is benign or a real finding. It is a real finding, and it is
not about the probe.

The metric is measured after a deliberately conservative collection

Every GC-ratchet probe reads process.memoryUsage() after an explicit gc().
An explicit gc() runs a full mark-sweep with a forced conservative stack
scan — PERRY_GC_DIAG=1 prints this on every single run:

[gc-scan-fallback] site=manual_collect automatic=false count=1

A conservative scan retains whatever the native stack happens to look like a
pointer to. Stack residue at that moment differs between runs, so the number
the probe reports differs between runs.

The evidence

Diffing two full PERRY_GC_DIAG traces of the same binary that disagree: the
minors, the tenuring decisions, the step cycles and every copy/promote counter
match exactly. The only difference is in the last collection:

< [gc] blocks: … freed_bytes=87675232 …      < [gc-old-free] reusable_bytes=45397168
> [gc] blocks: … freed_bytes=87677896 …      > [gc-old-free] reusable_bytes=45399760
< #gcmetric heap_used_bytes=59945816         > #gcmetric heap_used_bytes=59943224

And turning the scan off removes the variance completely:

arm	heap_used_bytes
default	59,943,080 … 59,952,824 over 22 runs, 5+ distinct values
PERRY_CONSERVATIVE_STACK_SCAN=off	51,668,688 × 8 runs, bit-identical

Two conclusions:

1. The run-to-run variance is entirely false roots. A few kilobytes —
   bounded by how much a handful of stale stack words can pin.
2. The conservative scan retains 8,275,208 bytes systematically — 16% of what
   this probe reports as its retained heap. That is not noise, it is a
   constant tax on every gc()-then-measure reading.

The eleven other probes stay bit-identical because their live sets are one to
two orders of magnitude smaller, so a stale stack word is far less likely to
alias a plausible heap address at all. This is a size effect, not a property of
what 12_large_live_set does.

Reproduced three times independently on 2026-08-06/07 at be1fc80f8
(0.5.1315): pinned quiet host perry-macos spread 2,304 over 7 harness repeats
(all eleven other probes spread 0); MacBook Pro under load 21 spread 9,072 over
7; 22 ad-hoc runs of the same binary spanning 9,744 bytes.

Why it matters beyond the ratchet

gc() is the measurement instrument for every retained-heap number this project
quotes, and it reports precise retention plus a stack-residue tax. On a ~50 MB
live set that tax was 16%. Any conclusion of the form "this change reduced
retained heap by X%" taken through gc() on a large heap has that term in it.

Worth deciding deliberately:

• Should a manual gc() force a conservative scan at all? The probes' roots
  are precise; the fallback is presumably defensive.
• If it must, should the ratchet measure with PERRY_CONSERVATIVE_STACK_SCAN=off
  so heap_used_bytes on the large-live-set probe becomes gateable again? That
  would let the probe_overrides entry added in test(gc-ratchet): per-probe gating overrides so the probes run again (#7554) #7557 be deleted, which the
  override machinery already forces to be a deliberate act (an entry that
  matches nothing fails).

Current state

#7557 takes 12_large_live_set.heap_used_bytes out of the gating family with
this reason recorded next to it, so the ratchet measures again. Every other
metric on that probe — heap_total_bytes, minor_cycles, copied_objects,
copied_bytes, promoted_*, freed_bytes — is unaffected by the scan and
stays gating, so a real over-retention regression on that probe still goes red.