[Next.js/dylib] Forced-evacuation App Route arm: stale closure from a holder outside the GC heap

[proggeramlug]

Splitting the one unresolved item out of #8082 so the production App Route fixture can land on its own merits. The fixture's normal arms pass end-to-end; its forced-evacuation arms do not, and this is that bug.

Symptom

With the Next 16.3.0 production App Route fixture running under forced evacuation and a seeded GC schedule:

PERRY_GC_FORCE_EVACUATE=1 PERRY_GC_HEAP_LIMIT=8 PERRY_GC_INCREMENTAL=0 \
PERRY_CONSERVATIVE_STACK_SCAN=off PERRY_GC_SCHEDULE_RATE=1 PERRY_GC_SCHEDULE_SEED=8036

the server serves requests and then fails with TypeError: value is not a function, at roughly copying minor #238.

What it is not (each ruled out by measurement)

• Not the statepoint lowering. The same application compiled with PERRY_RS4GC=0 (shadow-stack roots) fails identically — 80 TypeErrors in each arm, same seed, providers and workload. So it is neither fix(gc): index stack maps from loaded provider apps #8081's loaded-image stack maps nor compiled-frame root coverage.
• Not the remembered set. skip_remembering never fires (336/336 cycles report remembering_skipped=false, and in-place promotion never happens), and remembered-set coverage is stable — zero cycles lose dirty pages between collections.
• Not a holder in the GC heap. PERRY_GC_PROTECT_FROMSPACE_HOLDERS=1 (added in fix(next): pass production App Route dylib gate #8082) sweeps the whole live heap at fault time for any word still decoding to the faulting address and finds none.
• Not a slot the rewrite pass enumerates. PERRY_GC_VERIFY_EVACUATION=1 never panics, so every registered scanner rewrites correctly.

Taken together: the stale copy is held by something unregistered and outside the GC heap — a runtime side table, an FFI structure, or a frame slot.

Where the fault lands

Under PERRY_GC_PROTECT_FROMSPACE=1 PERRY_GC_PROTECT_FROMSPACE_DEPTH=800, the stale value is a 48-byte closure consumed by js_value_typeof inside Next's compiled adapter:

static get(e, t, r) { let n = Reflect.get(e, t, r); return "function" == typeof n ? n.bind(e) : n }

reached through js_proxy_get (the adapter is a Proxy get trap over the wrapped route-module object). js_reflect_get itself is fully rooted, so the pre-move copy lives in whatever supplies that property read.

Suggested starting point

scripts/gc_runtime_root_holders.py audits crates/perry-runtime/src and crates/perry-stdlib/src only — pointed at the ext crates it matches 0 declarations, because its patterns do not parse them. Yet every ext crate that invokes closures (perry-ext-http, -net, -ws, -zlib, -fastify) keeps user closures in handle side tables and registers its scanners by hand, with nothing checking that a scanner reaches every table in its own crate. Extending that audit is the obvious next move.

Reproducing in seconds

Build the providers per tests/release/packages/next-app-route/fixture.sh steps 3–5 into a kept build dir (PERRY_NEXT_ROUTE_KEEP_BUILD=1), then run the provider host directly with the env above plus one node verify.mjs. PERRY_GC_FROMSPACE_SCAN=1 mines holders at the retiring minor — but read its counts carefully: it reports dead old-gen residue and raw payload bytes as offenders (impossible obj_types like 47/95/144), which cost this hunt a retracted diagnosis. The protect fault is the trustworthy signal.

The fixture ships this arm behind PERRY_NEXT_ROUTE_FORCED_GC=1, off by default, so it fails loudly for whoever picks this up rather than sitting green.

[proggeramlug]

Retested on 53e8a21e3 (main, #8168 included): STILL REPRODUCES

#8168 (3cc318f69, "prune FUNCTION_CLASS_IDS keys whose closure died") landed fifteen hours after this issue was filed and names the identical seeded configuration, so it was worth checking whether it had already closed this. It has not.

The run

Full tests/release/packages/next-app-route/fixture.sh with PERRY_NEXT_ROUTE_FORCED_GC=1, PERRY_NEXT_ROUTE_COLD_STARTS=2 (index 0 normal, index 1 forced), on a clean origin/main worktree at 53e8a21e3, perry built --release from that tree.

  [1/7] … [5/7] compile untouched production route handler as app-only dylib
  next=16.3.0 mode=dylib providers=823d8b0244620a83fd73e7946b8297df6b9f36a66024cf4689c966193038ca3e
  [6/7] 2 cold processes (alternating normal / FORCED-evacuation), two 21-request verifier runs each
FAIL next-app-route — forced cold verifier 1 failed

The provider images are pristine main. Verified directly rather than assumed: the runtime image contains 2 occurrences of the gc-copy-minor diagnostic string and 0 of a marker string that exists only on a local working branch.

The forced arm ran, and its subject was live

Not "nothing threw" — the numbers, from perry-forced-1.log:

quantity	value
[gc-copy-minor] ran lines	243
Σ copied_objects	117,579
PERRY_NEXT_APP_ROUTE_READY	1 (server came up and served)
PASS: 21 production App Route requests	0
TypeError: value is not a function	2
gc evacuation verification failed panics	0

The normal arm (cold start 0) passed both verifier runs — PASS: 21 production App Route requests twice — so this is the forced-evacuation arm specifically, not a general regression.

It is the same bug

Every discriminator in the issue body still holds: the failure is TypeError: value is not a function, PERRY_GC_VERIFY_EVACUATION=1 never panics (so every registered scanner rewrites correctly), and the copying minor is demonstrably moving objects. #8168 removed one dead key from one rekeyed table; whatever holds the stale closure here is a different holder.

Related work

#8174 (the general rewrite_raw_addr exposure this shares a root cause shape with) is addressed by PR #8196, which refuses a forwarding walk out of, or into, something that is not a heap object, and adds a gate requiring every rekeyed side table to have a declared death story. That narrows the following of a stale key; it does not remove the holder here, and this issue stays open.

The issue's own suggested next move is still undone: scripts/gc_runtime_root_holders.py:109 reads

CRATES = ("crates/perry-runtime/src", "crates/perry-stdlib/src")

so perry-ext-http, -net, -ws, -zlib and -fastify — every ext crate that parks user closures in handle side tables and registers its scanners by hand — are unaudited.

Kept build artifacts (providers, app dylib, both logs) are at /private/tmp/perry-8174-8163/build.

[proggeramlug]

Reproduces under DEFAULT GC — not only the forced arm

Measured 2026-08-16 on the bench mini (idle M1, no other load), clean main 3c95020f8, perry built --profile perry-dev from that tree, providers + app dylib built by the untouched tests/release/packages/next-app-route/fixture.sh (release layout, C host, PERRY_MODULE_JOBS=2), no GC knobs at all except PERRY_GC_DIAG=1 on two of the runs. Only deviation from the fixture: the armed perry-host.js from tests/fixtures/next-app-route/ was dropped in so the routeModule.handle bypass guard is live.

fixture.sh itself passes (10 cold × 2 verifiers, bypassed=0 in all 10 logs). It never sees the bug because it stops at 2 passes per process. Run 100 verifier passes against ONE warm process and it appears:

run	env	pass/100	failing iterations	host TypeError: value is not a function
batch100	—	98	3, 6	4
rerun1	—	100	—	0
rerun2	—	99	19	1
diag1	PERRY_GC_DIAG=1	97	6, 12, 19	4
diag2	PERRY_GC_DIAG=1	98	6, 12	3

492/500 batches (each = 20 concurrent GETs + 1 POST) correct; the 8 failing batches each had one request answer with an empty body (verify.mjs: Unexpected end of JSON input) after the host logged TypeError: value is not a function (+ Next E180 failed to pipe response). Never a wrong ID/header/cookie — the response is lost, not swapped. Bypass guard: 0 fires in 500 batches.

Every failure sits immediately after a [gc-copy-minor] ran … in_place=false line and nowhere else (diag1 host log, in order): minor #1 → E180; minor #2 → 2× TypeError + E180; minor #4 → TypeError; … 21 copying minors in the run, 4 TypeErrors. E180 alone also occurs in fully passing runs (3× in rerun1), so on its own it is benign; the empty body only comes with the TypeError. The periodicity (iterations 6/12/19) is just the nursery step schedule (step=134217728→67108864→33554432→16777216, then a minor every ~6 batches).

Same symptom, same site (Next's compiled Reflect.get/bind adapter, from the trail in the issue body), reached by the shipped default configuration at ~1 broken request per 700 in a warm process. Worth restating for prioritisation: this is production-visible, the forced arm only makes it deterministic.

Cheap reproducer (seconds once the fixture is built)

Kept artifacts on the mini: ~/perry-bench.noindex/build8082/ (providers, app dylib, host) and ~/mini-batch-8040.sh <tag> <N> [ENV=..] (starts one host, loops verify.mjs N times, reports pass/fail, per-iteration TypeError count, RSS). Logs in ~/perry-bench.noindex/logs8040/batch-*-{host,verify}.log.

Also seen, separate from the stale closure

Retention over the same 100 batches (2100 requests): [gc-step] post_in_use rises monotonically 5.4 MB → 39.7 MB across the run's 21 collections while process RSS goes 160 MB → 510–620 MB (5 runs, no plateau) — the JS heap accounts for ~35 MB of that, the rest is off-heap. Noting it here only because it comes out of the same run; it wants its own home (or belongs to whatever fix/async-state-rss-accumulation is tracking).

[proggeramlug]

Independent validation of the candidate fix 3b01efba4 (fetch caches + ext-http once_listeners/EndTail rooting): forced arm FIXED, default-GC arm NOT — a holder remains

Ran on the bench mini 13:29–14:20 UTC today, worktree at 3b01efba4 (parent verified = 3c95020f8), armed perry-host.js, same artifacts pipeline as the earlier default-GC measurement, plus a same-hour main baseline so the comparison is same-host/same-hour.

Fixture (forced arm ON by default on this branch): PASS. PASS next-app-route (with forced-evacuation arm), 10/10 cold starts PASS: 21×2, TypeError=0, bypassed=0, gc evacuation verification failed=0. Each forced cold start: 443 copying minors, Σcopied_objects≈197.7k, liveness assert evacuation live on all five. Pre-fix this arm died deterministically at ~minor #238 — genuinely fixed at fixture scale.

Warm-process batches (one host, N× the 21-request verifier):

run	binary	env	pass/N	failing iters	host TypeError
b1	fix	—	99/100	12	1
b2	fix	—	100/100	—	0
b3	fix	—	99/100	12	1
d1	fix	PERRY_GC_DIAG=1	98/100	3, 12	2
d2	fix	PERRY_GC_DIAG=1	99/100	3	1
f1	fix	forced+seeded (SEED=8036 env set)	29/30	22	1
base	main 3c95020f8	—	96/100	3, 6, 12, 19	6

• Fix: 5 failing batches / 500 default-GC (vs 4/100 on main same-hour; vs 8/500 in yesterday's main measurement). Every failure keeps the exact original signature: empty body + TypeError: value is not a function immediately after a [gc-copy-minor] ran in_place=false (d1: minor Support custom menu bar items #1→TypeError, minor Using the fetch API #3→TypeError; d2: minor Support custom menu bar items #1→TypeError).
• f1: 3712 copying minors, Σcopied≈1.04M, verify-evacuation panics 0, host survived; its one TypeError landed right after copy-minor Add buffer INSPECT_MAX_BYTES to API manifest #2696 — pre-fix the same env died at ~fix(fetch): #234 real Blob with arrayBuffer/text/bytes/slice instance methods #238. ~10× longer survival, not unbounded.
• Rate improvement vs today's baseline is real (≈1.0/100 vs 4/100); vs yesterday's 8/500 it overlaps Poisson noise — treat "reduced" as suggestive.
• Retention unchanged by the fix: post_in_use 5.6→40 MB and RSS 160→~600 MB per 100 batches, same as main.

Conclusion: the two rooted holders were real, but at least one more unregistered holder sits on the default path (same consumer site, surviving copying minors under no GC knobs at all). Recommend landing 3b01efba4 on its merits (forced arm + measured rate reduction) while keeping this issue open for the residual, with the 100-pass warm-process batch as the acceptance instrument — the 10×2 fixture cannot see it.

Artifacts on the mini: worktree ~/perry-bench.noindex/wt-validate-8163, build ~/perry-bench.noindex/build8163 (10 cold-start logs), batch logs ~/perry-bench.noindex/logs8163/, runner ~/mini-batch-8163.sh; baseline logs ~/perry-bench.noindex/logs8040/batch-base-*.log.