[codegen] In-process LLVM backend fails the 5 biggest Next App Route modules on current main (silent per-module failure; regression in 3c95020f8..07c8040bf)

[proggeramlug]

Symptom

On current main-equivalent (07c8040bf, perry-dev profile), compiling the pinned production Next App Route fixture (tests/release/packages/next-app-route/, 104 modules) with the in-process LLVM backend — the default whenever PERRY_LLVM_INPROCESS is unset, i.e. what fixture.sh step [5/7] runs — fails deterministically:

✗ 5 module(s) failed to compile — REFUSING TO LINK
  - .next/server/chunks/2.js
  - .next/server/chunks/430.js
  - node_modules/next/dist/compiled/jsonwebtoken/index.js
  - node_modules/next/dist/compiled/next-server/app-page.runtime.prod.js
  - node_modules/next/dist/compiled/next-server/app-route.runtime.prod.js

Those are the app's 5 biggest modules (the 4 that "run exclusively to cap peak memory" plus jsonwebtoken). Reproduced 5/5 times on the bench mini (macOS 26.5.1 arm64, 8 GB, LLVM 22.1.8): four full fixture.sh runs (with PERRY_MODULE_JOBS=2 and both PERRY_CODEGEN_UNIT_JOBS=2 and =1) and one bare compile:

cd tests/release/packages/next-app-route
PERRY_MODULE_JOBS=2 PERRY_CODEGEN_UNIT_JOBS=1 PERRY_RUNTIME_DIR=<provider release dir> \
  perry compile perry-host.js --output-type dylib --no-auto-optimize --no-cache -o /tmp/app.dylib
# exit 1

Two aggravating details

1. No per-module error text is emitted. The failure summary says "Fix the codegen errors above (search for Error compiling module)" — but no such line exists anywhere in the output (grepped the full logs). The modules visibly make progress (LLVM unit 1/10 finished, froze 3/10 units, the fix(next): pass production App Route dylib gate #8082 RS4GC-optnone fallback fires for perry_closure_…app_page_runtime_prod_js__5118) and then are reported failed with nothing to act on. Whatever fails should say what failed.
2. It is a fresh regression. The untouched fixture.sh compiled the same app clean, in-process, on this same box earlier today at 3c95020f8 (the [Next.js/dylib] Forced-evacuation App Route arm: stale closure from a holder outside the GC heap #8163 validation runs, and [Next.js/dylib] Full production App Route compatibility tracker #8040's DoD audit — 500 serving batches). The window is 3c95020f8..07c8040bf; commits touching codegen in it: perf(codegen): decide scalar parameter descriptors with the typed-abi leaf guards #8201 (scalar parameter descriptors via typed-abi leaf guards), perf(gc, codegen): recover the instruction cost of the 56 B → 48 B header shrink (#8122) #8204 (recover the 56→48 B header-shrink instruction cost), gc: verify GC_STORE_AUDIT claims against emitted IR and source structure, not comments (#8185) #8206 (verify GC_STORE_AUDIT claims against emitted IR).

PERRY_LLVM_INPROCESS=0 (external clang/opt — the configuration tests/test_next_app_route_dylib.sh pins) compiles all 104 modules clean at the same commit on the same box, and the resulting app serves. So the release-tier fixture and anything else on the in-process default is blocked, while the external-backend path is fine.

Evidence on the bench mini

• ~/perry-bench.noindex/tmp-8040-knob/probe-inprocess.log — bare-compile repro (exit 1, full output).
• ~/perry-bench.noindex/tmp-8040-knob/fixture-v2.log, fixture-v10.log, fixture-v10-diag.log, rerun-v2.log — four fixture.sh failures (FAIL next-app-route — Perry dylib compile failed). Note: in three of these the process also died without printing the refusing-to-link summary at all.
• ~/perry-bench.noindex/tmp-8040-knob/ext-v2.log — same fixture, same commit, PERRY_LLVM_INPROCESS=0: compile clean, PASS next-app-route (20 batches, …).

Found while validating #8209/#8210; the serving-side results there are unaffected (validated on the external backend).

[proggeramlug]

Attribution, from hitting this while building the #8163 residual A/B on merged main b8d32ab19.

The offending instruction is the module-init header-image compose, and Perry's own IR dialect parser has no case for it.

Error compiling module '.next/server/chunks/2.js' … with --backend llvm:
native codegen unit 2/3 failed: unit 1: in line:
  %r7 = insertelement <2 x i64> <i64 207232172546, i64 0>, i64 %r6, i32 1:
  bad binary op `insertelement` operands: <2 x i64> <i64 207232172546, i64 0>, i64 %r6, i32 1

• Emitter: crates/perry-codegen/src/function.rs:586 and crates/perry-codegen/src/codegen/string_pool.rs:514, both "{} = insertelement <2 x i64> <i64 {}, i64 0>, i64 {}, i32 1".
• Rejecter: crates/perry-codegen/src/dialect/mod.rs:1014 — bail!("bad binary op \{op}` operands: {rest}"). insertelement` is not in the dialect's instruction table, so it falls through to the binary-op arm.
• Introduced by bf8fd868e — perf(gc, codegen): recover the instruction cost of the 56 B → 48 B header shrink (perf(object): remove the derivable object_type and field_count header words (56 B -> 48 B) [HELD: #8157 refuted; footprint-coupled residual + new #8094 guard cost] #8122) (perf(gc, codegen): recover the instruction cost of the 56 B → 48 B header shrink (#8122) #8204). git grep -c 'insertelement <2 x i64>' 3c95020f8 -- crates/perry-codegen/src returns nothing; the same grep at b8d32ab19 returns four files. That is inside this issue's stated 3c95020f8..07c8040bf range and narrows it to one commit.

All five failures are the same instruction, and all five are modules large enough to be split across native codegen units (unit 2/3, 4/6, 2/2, 8/10, 3/4) — consistent with the compose only being round-tripped through the dialect parser on the multi-unit path.

Two consequences worth recording:

1. The [Next.js/dylib] Add a pinned production App Route parity fixture and CI gate #8034/[Next.js/dylib] Full production App Route compatibility tracker #8040 App Route fixture cannot build its app dylib on merged main. perry compile perry-host.js --output-type dylib --no-auto-optimize --no-cache refuses to link with "5 module(s) failed to compile", and the failed set includes next-server/app-route.runtime.prod.js — the route runtime itself — so PERRY_ALLOW_PARTIAL_CODEGEN=1 is not a usable workaround.

2. An app dylib compiled before bf8fd868e does not run against providers built after it. build8163's dylib (compiled at 3b01efba4, whose parent is 3c95020f8) dies during Next's module init against b8d32ab19 providers with TypeError: Cannot set properties of null or undefined (setting 'id'). A clean-main control with no other change fails identically, so it is the provider/compiler pair, not any local patch. Since perf(gc, codegen): recover the instruction cost of the 56 B → 48 B header shrink (#8122) #8204 is a coordinated codegen+runtime change to the object header image, that is the expected shape of a co-versioning break — worth deciding explicitly whether the shared-provider model is supposed to tolerate a compiler/runtime skew, because the fixture always rebuilds both together and would never notice.

This blocked re-baselining the #8163 residual A/B onto merged main; that measurement was taken on 3b01efba4 instead (where crates/perry-stdlib/src/fetch/headers.rs is byte-identical to b8d32ab19).

[proggeramlug]

Interim workaround, measured: PERRY_LLVM_INPROCESS=off

Confirmed on the bench mini against unfixed main (537f74a461), same fixture, same
runtime archives, only the env var added:

cd tests/release/packages/next-app-route
PERRY_LLVM_INPROCESS=off \
PERRY_MODULE_JOBS=2 PERRY_CODEGEN_UNIT_JOBS=2 PERRY_RUNTIME_DIR=<runtime dir> \
  perry compile perry-host.js --output-type dylib --no-auto-optimize --no-cache -o app.dylib

• unset (default, in-process) — exit 1 in 14m19s, the 5 modules from the report,
  every one bad binary op \insertelement` operands`.
• =off (text transport) — exit 0 in 9m15s, Error compiling module count 0,
  all 104 modules, dylib written.

So anyone blocked on this (#8215 re-validation, #8220 end-to-end numbers, #8210 HEAD
validation) can move today with that one variable. native_units_mode()
(crates/perry-codegen/src/native_emit.rs:61) maps 0|off|false to the text
transport, and the missing insertelement case is only in the dialect reader on the
native path — which is also why the report's own PERRY_LLVM_INPROCESS=0 observation
held.

It is not free. The text path materialized 222.6 MB of IR for a single module and
produced a 206 MB dylib where the native path produces 87 MB — the oversized
modules fall to -O0 there. Expect a slower, larger artifact, and do not compare
performance numbers across the two backends.

Fix

PR #8241 (draft) teaches the dialect reader the closed set of vector forms
perry-codegen actually emits: insertelement, extractelement, shufflevector,
vector-typed add/mul, constant vector literals, and vector
poison/undef/zeroinitializer. That also covers expr/channel.rs's <4 x i32>
SIMD reduction, which was the same latent bug and had simply never landed in a module
big enough to split. With the fix, the same fixture compiles in-process: exit 0,
0 errors, dylib linked.

On the report's aggravating detail #1 — "no per-module error text is emitted"

The per-module Error compiling module … lines are emitted; they were in this
run at lines 500-504, above the refusing-to-link summary. What is genuinely
misleading is the progress output: for a module whose unit fails, the surviving units
still print LLVM unit N/M finished (… ; K complete) and the counter reaches M, so
the log reads exactly like a module that succeeded. .next/server/chunks/2.js printed
"3 complete" out of 3 units and had in fact failed unit 1. That is worth a separate
issue if anyone wants it — it cost me half a run of confusion before the summary
appeared.

Root cause of why this shipped green

The dialect reader's only gate is dialect::tests::corpus_*, which round-trips
frozen .ll snapshots under experiments/llvm-inprocess-spike/. A form the
emitters started producing after those files were captured is invisible to them by
construction — all three corpus tests pass with and without the fix. Combined with the
reader being the default only for split modules (every gap/parity fixture is a single
unit and keeps the text path), nothing per-PR could have caught #8204's new
insertelement. #8241 replaces the snapshot with a live emit → re-parse test.

[proggeramlug]

The PERRY_LLVM_INPROCESS=off workaround: mechanism confirmed, and an operational cost worth knowing before you take it

Two independent measurements, on two machines and two trees, agree on both halves.

It works, and we now know why rather than just that. Counting freeze/LLVM pipeline started in the compile log: 0 occurrences with the flag, 5 without — and those 5 are exactly the failing modules, with their unit splits matching one-to-one:

module	units	fails without the flag
chunks/2.js	3	✔
chunks/430.js	6	✔
next/dist/compiled/jsonwebtoken/index.js	2	✔
next-server/app-page.runtime.prod.js	10	✔
next-server/app-route.runtime.prod.js	4	✔

So the flag works by routing precisely the multi-unit modules off the reader path — mechanism, not correlation. (Credit for the workaround hypothesis and this measurement to the parallel session working #8163.)

Three-arm result, same tree/fixture/runtime, only the compiler and flag differing:

arm	outcome
main, in-process (default)	fails — 5 modules, bad binary op \insertelement`, REFUSING TO LINK`, 14m19s
main + PERRY_LLVM_INPROCESS=off	succeeds, 9m15s, dylib 206 MB
main + #8241, in-process	succeeds, 15m24s, dylib 87 MB, 104/104 modules

⚠️ The workaround is not free, and on a constrained machine it can fail outright. The text path materialises enormous IR for the big modules — 222.6 MB for one module on one host, 171.3 MB for app-page.runtime.prod.js on another, both at -O0 — and then links a ~206 MB dylib. One of the two hosts ran out of disk mid-compile. Budget several GB of free space, or use #8241 instead.

Do not compare performance or footprint across the two backends. Oversized modules drop to -O0 on the text path (15 of 104 modules compile below -O3 in one measured build), so a binary produced with =off differs from the shipped configuration in code quality, size, and plausibly in allocation behaviour and GC scheduling. Any GC or perf measurement taken under the workaround should say so explicitly; it is evidence about the text path only, and the native path is the default — it is what users actually get.

Correction to this issue's description: the per-module Error compiling module … lines are emitted. What misleads is the progress output — a module whose unit failed still prints LLVM unit N/M finished (… complete) all the way to M, so a failed module reads as "3 complete of 3". That deserves its own issue.

Fix is up as #8241 (ready for review).

[proggeramlug]

Addendum: how the disk exhaustion actually presents — it does not say "no space"

I was the host that ran out of disk mid-compile under the workaround. Recording the failure signature because it is actively misleading, and the compile-time cost that produces it.

The scratch is not cleaned up on failure. My failed run left 124 perry_llvm_scratch_* directories totalling 4.9 GB in $TMPDIR. That is one run. The text path materialises 100–200 MB of IR per oversized module and leaves it behind when the module fails, so a couple of failed attempts on a box near its limit will bury it — and the second attempt fails because of the first.

Under ENOSPC, clang -c fails with an EMPTY stderr, so all 105 modules report:

Error compiling module '…' with --backend llvm: clang -c failed (stderr empty)
clang:           /usr/bin/clang
clang --version: Apple clang version 21.0.0 (clang-2100.1.1.101)
requested -target: arm64-apple-macosx15.0.0
…
If the failure is a triple/ABI mismatch, set PERRY_LLVM_CLANG to a clang whose
default Target: matches arm64-apple-macosx15.0.0 (run `perry doctor` to verify).

There is no os error 28 anywhere in the log, and the diagnostic that is printed points at a triple/ABI mismatch — a plausible-sounding wrong answer that would cost real time to chase. The tells that it is disk and not toolchain: the failure hits every module rather than a subset, stderr is empty rather than containing a clang diagnostic, and bad binary op / freeze/LLVM pipeline started counts are both 0 (i.e. the flag was working and #8228 was not the cause).

Worth considering independently of #8241: an empty clang stderr plus a non-zero exit is a case where checking free space and saying so would be strictly better than suggesting a triple mismatch. Filing that thought here rather than as its own issue since it is one line of diagnosis in the same code path — happy to split it out if preferred.

Practical guidance if you take the workaround: budget several GB beyond the ~206 MB dylib, check df first, and clean $TMPDIR/perry_llvm_scratch_* between attempts — but only your own, matched by the pid-derived prefix in the error message (perry_llvm_scratch_<prefix>_N). Another session's compile in the same $TMPDIR looks identical, and there were six live ones on my box when I cleaned mine.