gc: diagnostics, two rooting fixes, and the ungated corpus/lowering cell (#7803 investigation) - #8084
Draft
proggeramlug wants to merge 31 commits into
Draft
gc: diagnostics, two rooting fixes, and the ungated corpus/lowering cell (#7803 investigation)#8084proggeramlug wants to merge 31 commits into
proggeramlug wants to merge 31 commits into
Conversation
added 23 commits
August 14, 2026 09:50
PERRY_UNCAUGHT_BACKTRACE=1 emits a symbolicated native backtrace on the uncaught-throw path, reusing the libc backtrace pair arena::quarantine already uses. A #7154-class rooting bug surfaces in a function nowhere near the code that lost the value, and the JS-level stack this path prints reads 'at <anonymous>'. PERRY_KEEP_SYMBOLS=1 skips ONLY the final strip. PERRY_DEBUG_SYMBOLS does that too, but every consumer reads it with is_some(), so it also turns on -g -- and on the #7803 corpus the symbolized build passed 13 seeds that the plain build fails at 44%. Asking for symbols changed the subject. This knob leaves codegen byte-identical to the build that reproduces. Both default off and are parsed BY VALUE, not by presence (#7993).
…l arms rooting/temp_root.rs already decides 'root, re-derive or reuse?' correctly and in one place, and already says module globals and locals must be ROOTED rather than reloaded. The gap is the POSITION it is asked about: that machinery protects call OPERANDS. Three arms lower the CALLEE into a bare register, lower the arguments after it -- each of which can allocate -- and then pass the original register: new_dynamic.rs (both js_new_function_construct arms) call_spread.rs (cb_box, across js_array_like_to_array and the concat) early_branches.rs (recv_box, unmasked into closure_handle after the args) Under the shipping statepoint lowering that register is in no live bundle, so nothing marks it and nothing relocates it. A root and not a reload: JS resolves the callee BEFORE the arguments, so re-reading below them would hand the call whatever an argument assigned. Measured on the dependency-scale corpus under the native lowering: 66 -> 26 hazards, sink=js_new_function_construct 24 -> 0, sink=js_closure_call_apply_with_spread 16 -> 0, live bundles 39073 -> 39140. This does NOT close #7803: the failure rate is unmoved (3/8 -> 5/16 -> 8/16 across the three binaries, all noise at this sample size). Landing it on the static ratchet alone. NOT YET gap-suite tested.
… 8/16) zod compiles a fastpass parser with new Function for every object schema (core/schemas.ts:2028), which on Perry runs through the dyn_eval interpreter -- the frames under #7803's throw. Taking that path out of the workload with zod's own jitless switch takes the failure with it: 0/16 against 8/16 on the same compiler, runtime and zod pin, with the instrument hot (5054-5434 forced collections, ~765k objects moved per run) and the answer byte-identical. Not a clean single-variable A/B and recorded as such: jitless also drops the workload from 6840 to 5056 safepoints. What makes it persuasive is the conjunction with the captured stack, not the sweep alone. Two traps on the way, both recorded: the config must run BEFORE any schema is constructed (jit is captured in the ctor, and the schema modules run at import), and the first attempt still entered interp_thunk through the identical stack -- a clean sweep of it would have been quoted as evidence while the interpreter was still running the parse.
) The interpreter offered the collector NO safepoints. Compiled code polls at loop back-edges; interpreted code polled nowhere, so a collection could only reach it at an allocation point -- and that arm forces a conservative stack scan, which makes the copying minor ineligible. The consequence is not that the interpreter is safe, it is that it is untestable: PERRY_GC_ZEAL forces collection at safepoints and there were none, PERRY_GC_SCHEDULE_SEED selects safepoints and there were none, and gc_root_dominance_check.py reads emitted LLVM IR of which the interpreter has none. The one rooting domain with no static checker also had no dynamic one, so mod.rs's claim that interpreter frames hold EVERY live JSValue in a rooted stack was unfalsifiable by anything in the tree. PERRY_GC_INTERP_SAFEPOINTS=1 calls js_gc_loop_safepoint at every eval_expr node and exec_stmt -- through the shared entry point deliberately, so the entry guards and the seeded-schedule ordinal apply exactly as they do to a compiled back-edge. An interpreter safepoint is the same safepoint, not a second kind. Subject asserted live (seed 2, rate 1, one binary): loop_polls 24029 -> 93210, safepoints 2725 -> 6973, moved 369076 -> 866480. Output byte-identical. Opt-in, not on: if the interpreter's rooting is complete, default-on is strictly better; if it is not, the flip turns a latent hole into a live crash for ajv / fast-json-stringify / find-my-way. Same sequencing PERRY_GC_MOVING_LOOP_POLLS had between #7161 and #7721.
…ve roots gc-root-dominance.yml emitted three of four corpus x lowering combinations. #7280 fixed the POPULATION (curated files lack the shapes a real library produces) and added the dependency corpus; #7452 fixed the LOWERING (statepoints ship; a PERRY_RS4GC=0 corpus contains none of that root form) and added the native corpus. Neither reached the other's cell, so the zod corpus compiled the way shipped binaries are compiled had never been checked. First measurement: 66 unrooted hazards, against a curated arm calibrated to ZERO -- 24 sinking into js_new_function_construct and 39 into the js_closure_call family, which are #7803's two observed messages. 40 of those were the callee-outlives-arguments defect fixed in 95d9fbb9d, leaving 26. Lands as a budget that can only go down, not an allowlist: the residual is a population under triage (19 are the js_box_get_bits mutable-capture-box shape), not a list anyone has adjudicated entry by entry -- the same reasoning the --stale-registers budget records. Carries the same liveness floors and the --seeded-violations 40 arm as its curated sibling, so a corpus that did not exercise the subject cannot read as clean. NOT yet promoted to a required check: a new gate has never been green, so it runs once before anyone depends on it (CLAUDE.md, hazard-4 corollary).
The third fixed arm (early_branches.rs) was never measured statically: the 26 came from a corpus emitted before that fix existed. A from-scratch rebuild reads 3 -- js_new_function_construct 24->0, js_closure_call_apply_with_spread 16->0, js_closure_call1/2 23->0, leaving one each of js_array_concat, js_rel_ge and js_get_string_pointer_unified. The lesson is about the 26. It came from an incremental build and went into a committed ratchet; a ratchet's number has to come from a tree someone else can reproduce. Caught only because the box swept the worktree and forced a clean rebuild.
…preter's own frames Same binary, one variable: safepoints off 6/8 fail, on 2/8. Collecting MORE often inside the interpreter made it fail LESS -- the opposite of what 'interpreted frames hold the unrooted value' predicts. n=8, p~0.13, so it settles nothing alone, but with the jitless result it narrows the position: the failure needs the new-Function PATH, and the interpreter's own locals are not obviously the holder. Next look is the BOUNDARY (bridge.rs, dispatch_with_arity, the interpreted-dispatch caches), not dyn_eval's locals, which §21 audited and found sound.
… gap-suite block The three call arms change the lowering of every new-expression, spread call and closure-typed-local call in the language, and the gap suite has NOT run against them. This box could not give a trustworthy run -- load average 60 with 47 sibling worktrees building, the suite slowing from 25 tests in 3 minutes to 30 in 19 -- so it was stopped rather than finished badly. Partial 30/554 with 0 failures is evidence of nothing except that the first 30 do not crash. run_gap_tests.sh + cargo test -p perry-codegen on a quiet host before that change goes into a PR.
…t first (#7803) js_native_call_method roots its receiver and arguments in a RuntimeHandleScope and #7528 added refreshed_args() so a use below a collection point re-reads them. That fix reached ten sites; several dispatch arms still pass the CALLER's raw args_ptr, which is the caller's memory -- arg_handles is what the collector rewrites, the buffer is not. Two arms verified to have a collection point between entry and dispatch: the dynamic-prop-on-a-closure arm (clone_closure_rebind_this allocates) and the accessor-getter arm (js_closure_call0 runs user code, then the rebind allocates). Both now refresh. Fits #7803's symptom: zod's generated fastpass calls shape[k]._zod.run({ value, issues: [] }, ctx) -- a freshly allocated object literal, the youngest thing on the heap, handed to the callee at its pre-move address -- and _zod is an accessor, which is the second arm. Not yet proven: the rate A/B has not run. The remaining raw-args_ptr arms are deliberately untouched; each needs its own 'can anything above me collect?' argument rather than a uniform guess.
…all four Seed 4 still fails on the argument-buffer fix, so that is a real defect found and fixed and a cause refuted, not a cause established. Adds a scorecard: four separate rooting defects, all real, none of them this bug -- the corpus under a rate-1 unprotected schedule is not a one-defect workload. Notes the pattern worth pulling on next: the two interventions that make it vanish (--debug-symbols, the from-space quarantine) both change memory LAYOUT, while all four that change ROOTING leave it untouched. That fits a stale raw pointer in a runtime-side cache keyed on an address rather than a value on a stack -- the class CLAUDE.md says the static checker cannot see.
…that can never pass Through 68/554 on a quiet host: two known failures and test_gap_4510_enum_ forward_ref, which is NOT a regression -- Perry prints the correct answer and NODE cannot run the file (--experimental-strip-types rejects enum, which is not erasable syntax). It is red rather than skipped because run_parity_tests.sh records node_fail only for an ABNORMAL exit; a clean exit 1 falls through to the output comparison against Node's crash text. So the test can never pass under the pinned Node. That is the mirror of the hazard CLAUDE.md documents for this suite (node-unrunnable tests silently DROPPED); this one is silently RED. Needs an expected-output file or a widened node_fail predicate. Unrelated to #7803.
The two flagged regressions are both cleared: test_gap_specabi_reassign fails byte-identically with the three codegen files reverted to 410dadd (so it is pre-existing on main, and is #6906/#7052's own regression test failing unnoticed because parity is tag-gated), and test_gap_zlib_4917_level's compile_fail was spurious -- I ran a cargo build concurrently with the suite and swapped the perry binary mid-run; recompiled by hand it is clean and byte-matches node. The ten node_fail -> parity_fail flips are all oracle-side: six need npm packages this worktree lacks, four are TypeScript node cannot strip (enum, parameter properties). They read RED rather than skipped because node_fail is recorded only for an abnormal exit. The codegen PR's blocker is cleared, with the caveats stated.
…ACE (#7803) Establishes WHY every existing instrument suppresses this bug. reset_region_to_zero is misleadingly named: it resets block.offset, it does NOT zero the bytes. Retired from-space therefore keeps its dead objects intact until new allocations bump over them, so: unprotected pages recycle into Eden, new objects overwrite the dead ones, and a stale pointer reads A DIFFERENT OBJECT -> property miss -> undefined. The failure. quarantined pages are held out of Eden, nothing overwrites them, a stale pointer reads its own dead object still intact, and the program is CORRECT. The suppression. So the quarantine does not miss #7803 by luck, it hides it by construction -- and --debug-symbols hides it for the same family of reasons. Both interventions that make the bug vanish are LAYOUT interventions; four separate rooting fixes left it untouched. This mode changes no layout: same pages, same order, same addresses, recycled at the same moment, with the retired bytes scribbled first. Only [0, offset) is touched, so pages the allocator has not faulted in stay untouched. A stale read then finds the poison word instead of a plausible object. Control: the unscheduled corpus run is byte-identical with it on, i.e. nothing in a healthy run reads retired from-space.
… blocker is experiment power 0/6 vs 3/6 looks like a fifth suppression and is not supportable: the two arms' schedules differ by ±0.7%, the same magnitude as the fixed-seed run-to-run drift §1 measured, and Fisher gives p~0.09. States the design problem plainly. A ~40% failure rate, ~1-4% schedule drift, and every intervention perturbing the schedule by about that much means no 6-16 run sweep can attribute anything; ~40 runs per arm would be needed, at 3-20 min each. Four of this session's rate comparisons are under-powered; only the jitless result (0/16 vs 8/16) clears the bar. The fix is a deterministic reproducer, not more runs, and the lever has been unused since the first task list: PERRY_GC_SCHEDULE_ALLOC_KB=0 removes the allocation-pacing feedback, leaving the candidate set equal to loop_polls -- which §1 already measured as STABLE at 63,936 across runs. Run in flight.
…the bug LESS likely PERRY_GC_SCHEDULE_ALLOC_KB=0 gives polls_paced=0 and safepoints=63941 (= loop_polls + 5 event-loop boundaries), i.e. the candidate set is now the one quantity §1 measured as stable across runs. 63,941 collections, 9.4x the paced run -- and it passed. That is the third independent observation of the same shape (paced ~40% fail; interpreter safepoints on 2/8 vs 6/8; unpaced passed). More collection pressure makes this bug LESS likely, which is backwards for a value held unrooted across a collection point, and fits four rooting fixes changing nothing. Hypothesis that predicts all of it: moved_objects barely changed (892k vs 862k) despite 9.4x the cycles, so denser collections promote survivors out of the evacuating nursery sooner (two-bit aging tenures after 2 minors, and old-gen objects do not move on a minor). Fewer relocations per object -> safer. The quarantine and --debug-symbols are explained by the same 'the object was not relocated into reused memory' mechanism, and rooting fixes are explained by not changing promotion at all. Next experiment is the promotion boundary itself, not the schedule: force promotion on the first minor (predicts the failure vanishes) and suppress tenuring entirely (predicts it becomes reliable -- which would be the deterministic reproducer this session lacked).
…experimental control Two seed-1 runs: safepoints / scheduled_collections / copying_minors all 63941 exactly, polls_paced 0, moved_objects 892662 vs 892062 (0.07%). Against ~4% schedule drift in the paced config. That fixes the design problem §30 named. With the schedule pinned, an intervention that changes the outcome at a fixed seed has changed something real, and one run per arm can say so instead of forty. Use ALLOC_KB=0 for every A/B from here; the paced config is a rate-survey tool only. Cost is ~9.4x the collections, 30-60 min per run, which is cheap next to forty paced runs that still could not attribute anything.
…iagnostic) Overrides the adaptive threshold (#7432) so the promotion hypothesis can be tested directly rather than through the schedule. Three independent measurements say #7803 gets LESS likely as collections get denser (paced ~30-50%; interpreter safepoints on 2/8 vs 6/8; unpaced, 9.4x the cycles, passing). That is backwards for a value held unrooted across a collection point, and it is what four rooting fixes failing to move the rate looks like. moved_objects explains it: 892k unpaced vs 862k paced despite 9.4x the cycles, so the extra collections promote the same objects SOONER, and an old-gen object is not moved by a minor -- denser collections mean FEWER relocations per object. =1 promote on the first minor -> predicts the failure disappears =255 never promote by age -> every survivor re-evacuated every cycle -> predicts the failure becomes reliable, i.e. the deterministic reproducer this bug has never had Pairs with PERRY_GC_SCHEDULE_ALLOC_KB=0, which pins the schedule exactly (63,941 safepoints, reproduced to the digit), so an outcome change at a fixed seed is attributable to this knob alone. Unset = adaptive, unchanged.
…mpling route is exhausted PERRY_GC_TENURING_SURVIVALS pinned: =255 (most relocations) 0/5, =1 (fewest) 1/5, adaptive ~40%. §31 predicted =255 becomes reliable and =1 disappears; neither happened. The follow-on 'it is the adaptive transitions' story dies with =1's failure -- a pinned threshold has no transitions. The result is non-monotonic and no relocation-count story fits it; at n=5 no cell is significant anyway. Five hypotheses tested, two real defects fixed, bug still standing. Stopping the sampling route deliberately: a ~40% base rate with ~1-4% schedule drift and five-run arms cannot attribute anything, and a sixth hypothesis would be pattern-matching on noise. Next person: either search seeds under the PINNED schedule (ALLOC_KB=0, 63941 safepoints reproduced to the digit) until one fails -- after which every A/B is one run per arm -- or attack the interpreted/compiled boundary statically, where a hazard can be found by reading rather than sampling.
…6 sites, not 10 Let the compiler count instead of eyeballing: shadow args_ptr/args_len to () right after arg_handles is built, and cargo check reports 36 errors -- 36 arms that reach past the rooted handles for the caller's memory. #7528 converted ten; the other 26 were never distinguished from those ten by anything but an author's per-arm judgement. The file's own #7528 rationale is what makes it a defect: the receiver is re-read at every use because 'this function then runs ~1160 more lines across a dozen probes that allocate'. arg_handles is the slot, args_ptr is the copy, and the argument that forces one forces the other. Reverted rather than landed: doing it right needs a per-site refreshed_args() (a single refresh at the top is the exact mistake #7528 documents), i.e. 36 individually-checked edits plus a gap run -- a focused change for a clean host, with the shadowing landed alongside so the population cannot regrow. The hot path is unaffected: try_class_vtable_fast_dispatch returns above the scope, so all 36 are already slow paths.
|
Important Review skippedDraft detected. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Plus Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
added 6 commits
August 14, 2026 09:56
…#7803) schedule_hit short-circuits to true at rate 1, so ALLOC_KB=0 + RATE=1 makes every seed the same 63,941-collection run. Seed 1 already passed that twice. The seed only selects when RATE < 1; pair that with ALLOC_KB=0 so the candidate set stays pinned.
RATE=0.1 + ALLOC_KB=0 makes the seed select. Seed 1 passes the pinned candidate set; seeds 2 and 3 abort the pin-latch on incoherent headers (INTERNED Map, 2 GiB native_pod_view). That is a stale slot, not a real pin. The latch used to print only the garbage; it now prints which walk followed it.
Same class both times (incoherent pinned header), not the same safepoint. Seed 1 still the passing control.
Two of three aborts land on safepoint 21547. Seed 2 is 1/2. Seed 1 passes. The latch is a layout lottery on a pinned schedule.
Seed 3 on the walk-phase binary aborted in mutable_root_slots (safepoints=52836). That walk is three populations. Label each slot shadow_stack / native_stack / global_root so the next abort names which one held the stale pointer.
added 2 commits
August 14, 2026 12:03
Seed 3's stale pointer is in a native stack-map root — an RS4GC live bundle. The collection is at a safepoint; the frames below the copier name the compiled function that held the slot.
…erateFastpass Seed 3 backtrace: js_gc_loop_safepoint in Doc.write, called from generateFastpass (schemas 135), called from $ZodObjectJIT.parse (schemas 138). parse.ts:65 is the victim. jitless 0/16 follows.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Investigation of #7803 plus the defects it turned up. Draft: #7803 is NOT fixed by this branch, and nothing here claims to fix it.
What #7803 is, now
Localized for the first time. The corpus dies at
zod/src/v4/core/parse.ts:65—result.issueswhereschema._zod.run({ value, issues: [] }, ctx)returnedundefined. All three observed messages (…reading 'issues',value is not a function,is not iterable) are one loss seen at different points. The failure needs thenew Functionpath, which on Perry runs through thedyn_evalinterpreter:jitlessgives 0/16 against 8/16 with it.Root cause remains unknown. Five hypotheses tested, refuted or unsupported — the table is in
gc-handoff/ZOD-NOTES.md§32.What this branch lands
Diagnostics (all default-off, all parsed by value, not by presence)
PERRY_UNCAUGHT_BACKTRACEPERRY_KEEP_SYMBOLSPERRY_DEBUG_SYMBOLSalso turns on-g, and--debug-symbolssuppresses this bug (0/13 vs 44%)PERRY_GC_INTERP_SAFEPOINTSdyn_evalcooperative GC safepointsPERRY_GC_POISON_FROMSPACEPERRY_GC_TENURING_SURVIVALSTwo rooting defects fixed
new_dynamic.rs×2,call_spread.rs,early_branches.rs). Each lowered the callee into a bare register, lowered the arguments after it — each of which can allocate — then passed the original register. Under the shipping statepoint lowering that register is in no live bundle. A root, not a reload: JS resolves the callee before the arguments, so re-reading below them would hand the call whatever an argument assigned.js_native_call_method, both with a verified collection point before the dispatch.Static effect, dependency corpus under the shipping lowering: 66 → 3 hazards.
js_new_function_construct24→0,js_closure_call_apply_with_spread16→0,js_closure_call1/223→0.A gate cell that never existed.
gc-root-dominance.ymlemitted three of four corpus × lowering combinations. #7280 fixed the population (curated files lack the shapes libraries produce); #7452 fixed the lowering (statepoints ship; a shadow corpus has none of that root form). Neither reached the other's cell — the zod corpus compiled the way shipped binaries are compiled had never been checked, and it read 66 where the curated arm is calibrated to 0. Now emitted and gated at--max-unrooted 3 --max-stale 0, ratchet-only-down.The interpreter was untestable, not merely unrooted. It offered the collector no safepoints, so
PERRY_GC_ZEALandPERRY_GC_SCHEDULE_SEEDran straight past it and the static checker has no IR to read.dyn_eval/mod.rs's claim that interpreter frames hold every live JSValue in a rooted stack was unfalsifiable by anything in the tree. Subject asserted live:loop_polls24,029 → 93,210 on one binary — the interpreter was ~74% of this workload's potential safepoints.Validation
test_gap_specabi_reassignfails byte-identically with the three codegen files reverted to the base commit (pre-existing on main), and acompile_failwas my own doing (I ran acargo buildconcurrently with the suite and swapped the binary mid-run; recompiled by hand it is clean and byte-matches Node).cargo fmt --all -- --check, YAML parse,gc_runtime_root_holders.py, and the new gate script all pass; the gate was verified end-to-end (81 modules, 52,201 statepoints).early_branches.rs(main renamed the predicate tolocal_type_hint; resolved keeping both sides). It compiles, but the suite has not been re-run post-rebase — hence draft.Two pre-existing failures on main, neither from this branch
crates/perry-runtime/src/timer.rsis 2010 lines onorigin/main(from fix(gc): preserve request graphs across route imports #8044), soscripts/check_file_size.sh— alintgate — is red independent of this branch. I did not touch that file and have not fixed it here.test_gap_specabi_reassignis failing on main and is not inknown_failures.json. It is let binding reassigned from Int32Array to plain array keeps typed-array read lowering (wrong values: 0/undefined instead of elements) #6906/fix: close five confirmed TypeScript parity regressions #7052's own regression test: a reassigned binding still provingTaPtr, so a plain array reads through typed-array lowering as zeros (plain: 0 0 2where Node gives99 101 2). Parity is tag-gated, so nothing per-PR catches it.Separately, ten gap tests the oracle cannot run report as red rather than skipped, because
node_failis recorded only for an abnormal exit and a cleanexit 1falls through to the output comparison.Not done
js_native_call_methodpasses the caller's rawargs_ptrbelow its handle scope at 36 sites; gc: js_native_call_method reuses one copy of its rooted receiver across ~1200 lines of allocating probes #7528 converted 10 and the other 26 were never distinguished from them by anything but per-arm judgement. Measured by shadowing the raws to()and counting compile errors. Fixing them needs a per-siterefreshed_args()(a single refresh at the top is the exact mistake gc: js_native_call_method reuses one copy of its rooted receiver across ~1200 lines of allocating probes #7528 documents) plus the shadowing landed alongside so the population cannot regrow. Worth its own PR.gc-handoff/7803-NEXT-PROMPT.mdis a self-contained brief for that work, including the acceptance bar and the deterministic-reproducer route (PERRY_GC_SCHEDULE_ALLOC_KB=0pins the schedule to the digit — 63,941 safepoints, reproduced exactly).