perf(gc): break the survivor-promotion handoff livelock (#7592) - #7594
Conversation
The handoff replaces a minor with a full mark-sweep to make room in old-gen for survivors about to be promoted. But a full mark-sweep is non-moving and promotes nothing, so it cannot relieve the pressure it was scheduled for: the survivor space still holds the same bytes and the predicate is true again at the next minor. Measured on json_pipeline at 200k records: 19 consecutive full collections triggered by survivor_promotion_bytes, each freeing 0.0 MB at ~400 ms -- 7.6 s of an 8.6 s phase, with peak RSS unchanged whether those collections ran or not. Latch it: one handoff per copying minor. The handoff makes room, the copying minor performs the promotion that consumes it. The latch clears only on a copying minor, since a non-moving minor fallback promotes nothing and would reinstate the livelock at half rate.
With an empty heap `copied_minor_promotion_handoff_due` returns false at the survivor-occupancy check anyway, so a bare assertion on the verdict passes with the latch deleted -- a test that cannot fail. Count the suppressions so the test observes the latch branch itself; verified by removing the latch and watching the test go red.
|
Caution Review failedThe pull request is closed. ℹ️ Recent review info⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Plus Run ID: ⛔ Files ignored due to path filters (1)
📒 Files selected for processing (2)
📝 WalkthroughWalkthroughThe GC now records survivor-promotion handoffs, suppresses repeated arena- and malloc-triggered handoffs, counts suppressions, and clears the latch after copying-minor completion. A regression test validates the trigger behavior. ChangesSurvivor promotion handoff
Estimated code review effort: 3 (Moderate) | ~20 minutes Sequence Diagram(s)sequenceDiagram
participant GCTriggerPolicy
participant FullCollection
participant CopyingMinor
GCTriggerPolicy->>FullCollection: record survivor-promotion handoff
FullCollection-->>GCTriggerPolicy: await copying minor
GCTriggerPolicy-->>GCTriggerPolicy: suppress ArenaBytes and MallocCount handoffs
CopyingMinor->>GCTriggerPolicy: note_copying_minor_completed()
GCTriggerPolicy-->>GCTriggerPolicy: clear handoff latch
Possibly related PRs
Suggested reviewers: 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Audit before merge — verified end to end, merged as v0.5.1338Reproduced the livelock and the fix independently, own builds of both arms
The census is the mechanism and it reproduces to the digit — including the The diagnosis is structurally sound, verified in the code, not just the The wiring is right in the two places that decide correctness:
Sabotage re-verified myself: deleting the latch branch turns the test red at Gates re-run here: runtime suite 1,843 / 0, One coverage gap, noted not blockingThe test exercises the latch via direct calls to BookkeepingMerged with a plain |
…gate (#7554) An artifact defect used to abort validation on the first problem, and the artifact-validation step runs BEFORE the measurement step. So one cell — 12_large_live_set.heap_used_bytes, spread 6,768 bytes — meant none of the twelve probes executed on any branch for three days, while two GC pacing changes (#7594, #7596) merged with hand-run both-arms A/Bs standing in for the gate. The defect was a claim about ONE cell. Nothing about it voided the other 143, and nothing about it made the probes unrunnable. Defects now carry a scope. `artifact` (unreadable, tampered, missing metric) stays fatal and stays in preflight. `probe` (pinned without an oracle diff, or with no collection) and `cell` (contradicts the bit-identity premise of its own band) demote their subject out of the gating family and are reported as failures — so `check` still measures everything, still evaluates the other cells, and still names a regression elsewhere in the matrix, while the defect itself keeps the job red. `validate --scope structural` (what CI preflight now runs) fails only on the fatal kind. It cannot suppress: `check` re-derives the same list and fails on it, and a test asserts that coupling per planted defect shape. `assemble` is unchanged — pin time still refuses any defect outright, so this cannot be used to freeze a new unfit artifact.
…ost (#7554) gc-ratchet had not been green on main since 2026-08-01T05:39Z — 179 consecutive red main runs. The 2026-08-05 window where it could not reach its probes at all (#7554, fixed by #7557) was an episode inside that, not the whole of it: after #7557 restored measurement the job stayed red against a 0.5.1280 artifact that no longer described the collector. Re-pinned at origin/main 26b9c9d (0.5.1346) on perry-macos — the same Mac mini and the same rustc/cargo/clang the 2026-08-05 pin used, so this is like-for-like. All 12 probes oracle-pass; heap_used_bytes spread 0 on eleven and 864 B on 12_large_live_set. Full per-cell attribution is in the artifact's own `notes`. Three of the four moved groups are explained: - 03/04's copy and promote counters collapsing 40–99.8% is #7594 + #7596 doing what they said (less futile promotion). Recorded caveat: 03's promoted_* now pin at 0, where the allowance floor and the liveness assert both go quiet. - 02 +2.77% and 05 +16.44% retention are conservative-scan false roots, not retention. `classify` on this host gives 05 precise 5,329,880 — byte-identical to what #7571 measured at both ends of its window — and 02 precise 9,416,632, BELOW the number this baseline previously recorded. That is #7559's answer, reproduced rather than assumed. The fourth is flagged, not explained: 12_large_live_set.wall_ms 3,056 -> 3,471 ms (+13.58%), two non-overlapping 7-sample clusters on one host, while 06 and 11 got 9.6% and 28.4% faster. #7596 reported -7.4% on that cell, so by its own evidence this is not #7596. It is gated on pinned_host only. #7596's accepted 12_large_live_set.heap_total_bytes +36% did NOT reproduce here (110,100,480 -> 110,100,480, +0.00%), so nothing was re-pinned for it.
1 participant
Fixes the
build_outhalf of #7592.The bug is a livelock, and it is not in JSON
#7592 reported
json_pipelineat 500k records being 97.6× bun. Splitting the phases first, as the issue insists:JSON.parsehandles 107 MB in 742 ms and scales linearly.build_out— theout.push({...})loop — is ~100 % GC pause (8,840 ms of traced pause against 8,633 ms of phase at 100k) and grows quadratically.The GC trace says exactly what is happening. At 200k records, of 22 collections:
copied_minor_promotion_handoff_duereplaces a minor with a full mark-sweep to make room in old-gen for survivors that are about to be promoted. But a full mark-sweep is non-moving — it promotes nothing. So it cannot relieve the pressure it was scheduled for: the survivor space still holds the same 108 MB, the reclaim baseline it resets does not count those bytes, and the predicate is true again at the very next minor. The copying minor that would have done the promotion never gets to run.That is 7.6 s of an 8.6 s phase spent on collections that free nothing. Peak RSS is the same whether those 19 collections run or not.
Fix
Latch it: one handoff per copying minor. The handoff makes room; the copying minor performs the promotion that consumes it. The latch clears only on a copying minor, because a non-moving minor fallback promotes nothing and would reinstate the livelock at half rate.
The guard sits before the
copied_minor_promotable_active_survivor_bytes()walk, so a suppressed handoff also skips that O(n) survivor pass.Same workload, after:
survivor_promotionfullsMeasurements
Both arms built from the same package set and linked against a pinned
PERRY_RUNTIME_DIR, interleaved, output hash checked every row:Output hash identical on every row.
The RSS cost is real and I am not hiding it: +17–24 % at the large sizes, because the run now does 6 collections instead of 22. It is a genuine trade, not a free win. It is confined to workloads that actually hit this livelock — see below.
GC ratchet
Checking against the pinned baseline reports 29 regressions, but that baseline is from 0.5.1315 on a different host, so it cannot separate this change from drift. I measured both arms back to back on one host with an identical package set instead — 144 metric medians across all 12 probes:
12_large_live_set.heap_used_bytesat −0.01 % (2,232 B). That cell is the one the harness itself documents as ungated and sample-dependent (conservative-scan residue, observed spread 9,072 B over 36 runs), so 2,232 B is inside its own noise.That is the expected shape: the latch can only fire after a survivor-promotion handoff has occurred, so a workload that never hits the livelock is unaffected.
cargo test -p perry-runtime: 1,843 passed, 0 failed.cargo fmt --checkandscripts/check_file_size.shclean.Test
test_survivor_promotion_handoff_waits_for_the_copying_minor.The obvious version of this test cannot fail: with an empty heap
copied_minor_promotion_handoff_duereturnsfalseat the survivor-occupancy check regardless, so asserting the verdict passes with the latch deleted. The test therefore counts suppressions and asserts the latch branch is what rejected it — and checks that a trigger kind the handoff never applies to is not counted, so the counter cannot pass by incrementing everywhere.Verified by deleting the latch and watching it go red:
What this does not fix
build_outis still ~28,500 ns/record and not yet flat, so #7592 stays open. With a nursery large enough to avoid collecting during the loop the same phase runs at 764 ns/record, so there is roughly another order of magnitude available. That remainder is the collection-budget question — the cap is a constant (16 MB ×NURSERY_CAP_SCALE_MAX), so cadence is independent of live-set size. I tried a live-proportional cap and it is structurally wrong as written: the cap gates from-space occupancy, and here from-space is nearly all of live, socap = liveis a fixed point that stops scavenging entirely. Details are in the issue.Summary by CodeRabbit
Bug Fixes
Performance
Tests
Documentation