Found while trying to sabotage-verify the repaired GC ratchet (#7609, for #7554).
The intent was an end-to-end proof that the restored gate can go red: disable an
evacuation knob, re-measure, watch the counters collapse. It did not collapse
anything.
The measurement
Pinned quiet host perry-macos, main 26b9c9d59 (0.5.1346), identical
binaries and protocol in both arms (--repeats 7, warmup 1,
PERRY_NO_AUTO_OPTIMIZE=1, prebuilt static archives), the only difference being
the knob:
arm A: PERRY_RUNTIME_DIR=... PERRY_NO_AUTO_OPTIMIZE=1
arm B: PERRY_RUNTIME_DIR=... PERRY_NO_AUTO_OPTIMIZE=1 PERRY_GEN_GC_EVACUATE=0
Cell-by-cell diff of the two measurements over all 12 probes and
minor_cycles, step_cycles, copied_objects, copied_bytes,
promoted_objects, promoted_bytes, freed_bytes, heap_used_bytes:
Zero. Not "small deltas inside the band" — bit-identical medians on every one
of the 96 cells. gc_ratchet.py check against the pinned baseline returns exit
0 with the knob set.
For contrast, the same procedure with PERRY_GEN_GC=0 moved 79 cells and
returned exit 1 with 90 findings, so the method is sound and the harness is
sensitive — this knob specifically changes nothing the ratchet can see.
Why this matters
CLAUDE.md's GC knob kill-policy is explicit: "Every GC env knob either has a
required CI arm exercising its OFF state, or it is deleted after one release of
soak… A mode that still exists is a decision that hasn't been made." It also
records two precedents of exactly this shape:
This is the same failure mode a third time. The practical consequence: any past
or future claim of the form "the suite still passes with
PERRY_GEN_GC_EVACUATE=0" carries no information about the GC ratchet's twelve
probes, because the knob does not perturb them.
What I did not establish
I have not confirmed the mechanism from the code. The plausible reading is
that the knob gates the C4b policy evacuation of tenured objects into old-gen,
while copied_objects/copied_bytes count the copying minor's scavenge, which
is a different path — so the probes may simply never reach the gated branch. That
is a hypothesis from CLAUDE.md's description, not something these runs prove.
Worth confirming before acting.
Suggested acceptance, in kill-policy terms
Either:
- The knob's OFF state gets an arm that demonstrably exercises it — a workload
where policy evacuation actually fires, with a liveness assertion (moved > 0
or equivalent) so the arm cannot pass having run zero policy evacuations; or
- The OFF state and the branch behind it are deleted, so the losing mode stops
compiling rather than lingering as an untested configuration a future bisect
will trust.
Either way, if a GC probe suite is meant to be evidence about evacuation, at
least one probe should be able to tell whether evacuation happened.
Found while trying to sabotage-verify the repaired GC ratchet (#7609, for #7554).
The intent was an end-to-end proof that the restored gate can go red: disable an
evacuation knob, re-measure, watch the counters collapse. It did not collapse
anything.
The measurement
Pinned quiet host
perry-macos,main26b9c9d59(0.5.1346), identicalbinaries and protocol in both arms (
--repeats 7, warmup 1,PERRY_NO_AUTO_OPTIMIZE=1, prebuilt static archives), the only difference beingthe knob:
Cell-by-cell diff of the two measurements over all 12 probes and
minor_cycles,step_cycles,copied_objects,copied_bytes,promoted_objects,promoted_bytes,freed_bytes,heap_used_bytes:Zero. Not "small deltas inside the band" — bit-identical medians on every one
of the 96 cells.
gc_ratchet.py checkagainst the pinned baseline returns exit0 with the knob set.
For contrast, the same procedure with
PERRY_GEN_GC=0moved 79 cells andreturned exit 1 with 90 findings, so the method is sound and the harness is
sensitive — this knob specifically changes nothing the ratchet can see.
Why this matters
CLAUDE.md's GC knob kill-policy is explicit: "Every GC env knob either has a
required CI arm exercising its OFF state, or it is deleted after one release of
soak… A mode that still exists is a decision that hasn't been made." It also
records two precedents of exactly this shape:
PERRY_GC_FORCE_EVACUATEwas inert for everygc()-driven test(gc: no reachable configuration exercises an evacuating minor with unpinned runtime locals — the #6655/#6935 bug class is untestable #6942/GC testing: PERRY_GC_FORCE_EVACUATE is inert for gc()-driven tests (full mark-sweep + forced conservative scan) — stress claims may be unsupported #6946) — months of "passes under evacuation" meant nothing.
--pressureknob disabled the very path it was measuring(gc-matrix: --pressure disables the very path #7019 added — the 'default' arm runs ZERO copying minors on all 22 corpus rows #7024).
This is the same failure mode a third time. The practical consequence: any past
or future claim of the form "the suite still passes with
PERRY_GEN_GC_EVACUATE=0" carries no information about the GC ratchet's twelveprobes, because the knob does not perturb them.
What I did not establish
I have not confirmed the mechanism from the code. The plausible reading is
that the knob gates the C4b policy evacuation of tenured objects into old-gen,
while
copied_objects/copied_bytescount the copying minor's scavenge, whichis a different path — so the probes may simply never reach the gated branch. That
is a hypothesis from CLAUDE.md's description, not something these runs prove.
Worth confirming before acting.
Suggested acceptance, in kill-policy terms
Either:
where policy evacuation actually fires, with a liveness assertion (
moved > 0or equivalent) so the arm cannot pass having run zero policy evacuations; or
compiling rather than lingering as an untested configuration a future bisect
will trust.
Either way, if a GC probe suite is meant to be evidence about evacuation, at
least one probe should be able to tell whether evacuation happened.