Skip to content

bun test: free test-runner finalizer-owned allocations before exit so LSan lanes don't abort green runs#32180

Open
robobun wants to merge 5 commits into
mainfrom
farm/1b909a84/test-exit-lsan-leak
Open

bun test: free test-runner finalizer-owned allocations before exit so LSan lanes don't abort green runs#32180
robobun wants to merge 5 commits into
mainfrom
farm/1b909a84/test-exit-lsan-leak

Conversation

@robobun

@robobun robobun commented Jun 12, 2026

Copy link
Copy Markdown
Collaborator

Fixes #32176

Problem

On the 13 x64-asan lane, test/regression/issue/30205.test.ts intermittently fails with the inner bun test --isolate subprocess exiting 134 after a fully green run. The CI runner propagates ASAN_OPTIONS=...detect_leaks=1:abort_on_error=1 and LSAN_OPTIONS=...suppressions=test/leaksan.supp into spawned processes via bunEnv, and LeakSanitizer's exit scan reports test-runner allocations that are still malloc-live, turning the report into SIGABRT.

Deterministic repro on main (debug ASAN build, no napi addon needed; also fails without --isolate and with --parallel):

# 4 files of: import { test, expect } from "bun:test";
#             test("t", () => { expect(1 + 1).toBe(2); });
ASAN_OPTIONS="allow_user_segv_handler=1:disable_coredump=0:detect_leaks=1:abort_on_error=1" \
LSAN_OPTIONS="malloc_context_size=30:print_suppressions=0:suppressions=$PWD/test/leaksan.supp" \
bun-debug test --isolate .   # exit 134, 10/10 runs
Direct leak of 48 byte(s): <bun_runtime::test_runner::expect_core::Expect>::call  expect.rs:656
Indirect leak of 448 byte(s): <bun_runtime::test_runner::bun_test::BunTest>::ref_  bun_test.rs:732
Indirect leak of 312 byte(s): RcInner<BunTestCell>, <BunTestRoot>::enter_file  bun_test.rs:530

Cause

Three distinct allocation classes are live when LSan scans at exit:

  1. The final file's expect() wrapper boxes are freed only by their GC finalizers (Expect::finalize, which also derefs the RefData pinning that file's BunTestCell), and no collection runs between the last test and exit(). Under --isolate, earlier files' wrappers are collected by GCs that happen while later files run; the last file has no later GC. Plain runs and --parallel workers have the same hole.
  2. With a napi addon in the fixtures (the 30205 case), sweeping the wrapped objects defers each finalizer as a NapiFinalizerTask. NapiFinalizerTask::schedule() parks them on RareData::cleanup_hooks because bun test's exit paths never call on_exit(), so has_run_cleanup_hooks is false and the list is never drained: every task box leaks (32000 bytes in 1000 allocations for the 30205 fixture). This is also why the lane failure is timing dependent on unpatched main: it needs destructOnExit's conservative collectNow() to actually collect the wraps.
  3. The final global's ScopeFunctions boxes (the native state of its test/describe/expect function wrappers) are genuinely alive at exit, rooted by the still-gcProtected global's bun:test exports. The only pointers to them live in JSC heap cells, which LSan cannot scan, so it reports them as leaks. They are freed whenever a global actually dies (isolation swap, BUN_DESTRUCT_VM_ON_EXIT teardown).

Fix

  • New VirtualMachine::collect_for_leak_check_at_exit(), called from the test runner's exit path (covers plain, --isolate, and the --parallel coordinator) and from the --parallel worker's exit path. No-op in non-ASAN builds, which keep skipping all teardown for exit speed. Under ASAN it sets has_run_cleanup_hooks = true first, so napi finalizers deferred by the sweep take schedule()'s existing drop-immediately branch (the documented shutdown semantics for destructOnExit's collection) instead of being parked forever, then runs one synchronous full collection so the test-runner finalizers free their boxes. The load-bearing lines are the flag set and the garbage_collect(true) call.
  • test/leaksan.supp entry for the ScopeFunctions allocation site (class 3 is a reachable-but-unscannable false positive, the exact thing the suppression file exists for; the file already suppresses the globals themselves).

Verification

New exit is leak-clean under LeakSanitizer cases in test/cli/test/isolation.test.ts (serial, --isolate, --parallel=2), gated on isASAN. They spawn bun test with detect_leaks=1:abort_on_error=1 and BUN_DESTRUCT_VM_ON_EXIT stripped, and assert no leak report, 4 pass, exit 0. All three fail on the unfixed build (the serial and isolate children exit 134; a parallel worker's abort does not change the coordinator's exit code, so that case asserts on the report text in inherited stderr).

Verified on a debug ASAN build:

  • the new tests fail 3/3 on unfixed main, pass 3/3 (x3 runs) with the fix
  • the exact 30205 CI scenario (8 isolate files x 1000 addon.wrap(...), BUN_JSC_collectContinuously=1, BUN_DESTRUCT_VM_ON_EXIT=1, leak checking on) exits 0 with no report, 5/5 runs; on the unfixed build it aborts with the NapiFinalizerTask leak
  • test/regression/issue/30205.test.ts passes (4/4) under the CI lane's env
  • full test/cli/test/isolation.test.ts passes (22/22)

Note on #32146

#32146 fixes the napi half of the same lane flake (issue #32144) by draining the cleanup hooks inside global_exit() with a hooks-only restriction. The two changes are complementary but touch the same shutdown bookkeeping; whichever lands second needs a small rebase: with #32146 in, this PR's method should call napi_internal_set_cleanup_hooks_only() + run_cleanup_hooks() instead of setting has_run_cleanup_hooks directly, so the hooks-only restriction and the env cleanup hooks both still apply before the collection.

Scope note

Exits after failures (--bail and ordinary failing runs) are intentionally excluded: they abort under the same env for additional reasons (a direct Global::exit(1) in the bail-on-test-failure path, and a ParsedSourceMap cache entry from printing the failure that LSan cannot see through), none of which have CI exposure today. Tracked in #32183.

@robobun

robobun commented Jun 12, 2026

Copy link
Copy Markdown
Collaborator Author
Updated 5:47 PM PT - Jun 20th, 2026

@robobun, your commit 57e361b has 2 failures in Build #63682 (All Failures):


🧪   To try this PR locally:

bunx bun-pr 32180

That installs a local version of the PR into your bun-32180 executable, so you can run:

bun-32180 --bun

@coderabbitai

coderabbitai Bot commented Jun 12, 2026

Copy link
Copy Markdown
Contributor

Review Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Pro

Run ID: 2b421c4a-975f-43b5-afb9-138b98401061

📥 Commits

Reviewing files that changed from the base of the PR and between 3347d3f and 57e361b.

📒 Files selected for processing (1)
  • src/jsc/VirtualMachine.rs
💤 Files with no reviewable changes (1)
  • src/jsc/VirtualMachine.rs

Walkthrough

Adds an ASAN-only final garbage-collection step at bun test shutdown, implemented on VirtualMachine, invoked from both main and worker test shutdown paths, and validated with a LeakSanitizer test plus a suppression entry.

Changes

Leak-check collection at test exit

Layer / File(s) Summary
VirtualMachine leak-check collection method
src/jsc/VirtualMachine.rs
VirtualMachine::collect_for_leak_check_at_exit() asserts shutdown state, marks cleanup hooks run to avoid deferred N-API finalizer queuing, and triggers ASAN-only synchronous GC to collect remaining test-runner objects before exit.
Integration into test command shutdown paths
src/runtime/cli/test_command.rs, src/runtime/cli/test/parallel/runner.rs
Main test command and worker shutdown paths call collect_for_leak_check_at_exit() immediately before the final global_exit() sequence.
LeakSanitizer test coverage and suppression
test/cli/test/isolation.test.ts, test/leaksan.supp
New concurrent test runs bun test under LeakSanitizer in serial, isolate, and parallel modes to assert no leak reports; leaksan.supp adds suppression and explanation for leak:scope_functions::ScopeFunctions at test exit.

Possibly related issues

  • #32176: The PR addresses the reported bun test --isolate exit-time leaks of test-runner objects by adding a final GC before exit to deterministically collect those objects before ASAN leak detection runs.
🚥 Pre-merge checks | ✅ 4
✅ Passed checks (4 passed)
Check name Status Explanation
Title check ✅ Passed The title accurately and specifically describes the primary change: fixing a LeakSanitizer issue in bun test exit paths by freeing finalizer-owned allocations before process exit.
Description check ✅ Passed The description comprehensively covers the problem, root causes (three allocation classes), the implemented fix, verification methodology with specific test results, and known dependencies with related PRs.
Linked Issues check ✅ Passed The PR fully addresses #32176's objectives: finalizing test-runner allocations (Expect wrappers, RefData, BunTestCell) via final GC, preventing NapiFinalizerTask leaks by setting has_run_cleanup_hooks before collection, and suppressing ScopeFunctions false-positives in leaksan.supp.
Out of Scope Changes check ✅ Passed All changes directly support the PR's focused objective: the new VirtualMachine method, its invocations in test exit paths, test cases validating the fix, and the leak suppression file entry are all in scope for fixing #32176.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.


Comment @coderabbitai help to get the list of available commands and usage tips.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (1)
src/runtime/cli/test_command.rs (1)

2943-2955: ⚠️ Potential issue | 🟠 Major | ⚡ Quick win

Route the --bail shutdown through this same leak-check path.

This fixes the normal exec() exit, but TestCommand::run() still has a direct bail-out global_exit() path at Lines 3179-3205 that skips collect_for_leak_check_at_exit(). A bun test --bail run can therefore still bypass the final ASAN GC and hit the same LSan abort class on exit.

Based on learnings: "Rust code: fix the whole bug class in the same PR - grep for every sibling site sharing the pattern."

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/runtime/cli/test_command.rs` around lines 2943 - 2955, The direct bailout
path in TestCommand::run() calls global_exit() and skips the leak-check teardown
(reporter.jest.bun_test_root.deinit_for_exit(), unsafe {
jest::Jest::RUNNER.write(None); }, drop(reporter);
vm.collect_for_leak_check_at_exit()), so add routing from the --bail branch to
perform the same shutdown sequence instead of calling global_exit() directly:
locate the early-exit branch in TestCommand::run() that currently calls
global_exit(), and replace or augment it to call the existing exit teardown
sequence (deinit_for_exit(), clear RUNNER, drop reporter, then
vm.collect_for_leak_check_at_exit()) before returning/exiting so the ASAN/LSan
leak check runs on bail as well.

Source: Coding guidelines

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@test/cli/test/isolation.test.ts`:
- Around line 889-922: Replace the manual for-loop parameterization with Bun's
test.each by turning the cases array into a test.each(cases) call so each tuple
[label, args] becomes a separate test; inside the test callback keep the
existing test.skipIf(!isASAN) guard and reuse the same body (calls to tempDir,
FILE_COUNT, bunExe(), bunEnv, the Bun.spawn invocation assigned to proc, and
assertions on stderr and exitCode) so behavior is identical but the matrix is
declarative and idiomatic.

---

Outside diff comments:
In `@src/runtime/cli/test_command.rs`:
- Around line 2943-2955: The direct bailout path in TestCommand::run() calls
global_exit() and skips the leak-check teardown
(reporter.jest.bun_test_root.deinit_for_exit(), unsafe {
jest::Jest::RUNNER.write(None); }, drop(reporter);
vm.collect_for_leak_check_at_exit()), so add routing from the --bail branch to
perform the same shutdown sequence instead of calling global_exit() directly:
locate the early-exit branch in TestCommand::run() that currently calls
global_exit(), and replace or augment it to call the existing exit teardown
sequence (deinit_for_exit(), clear RUNNER, drop reporter, then
vm.collect_for_leak_check_at_exit()) before returning/exiting so the ASAN/LSan
leak check runs on bail as well.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Pro

Run ID: 719ec80f-9fd9-4195-8b0c-a30439f181f5

📥 Commits

Reviewing files that changed from the base of the PR and between 885c44f and 2865219.

📒 Files selected for processing (5)
  • src/jsc/VirtualMachine.rs
  • src/runtime/cli/test/parallel/runner.rs
  • src/runtime/cli/test_command.rs
  • test/cli/test/isolation.test.ts
  • test/leaksan.supp

Comment thread test/cli/test/isolation.test.ts Outdated

@claude claude Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I didn't find any bugs, but this touches VM shutdown ordering (GC, has_run_cleanup_hooks, napi finalizer scheduling) across three exit paths and has a noted interaction with #32146, so it's worth a human look.

Extended reasoning...

Overview

This PR adds VirtualMachine::collect_for_leak_check_at_exit() — an ASAN-only synchronous full GC that first sets has_run_cleanup_hooks = true so napi finalizers deferred by the sweep take NapiFinalizerTask::schedule()'s drop-immediately branch instead of being parked on RareData::cleanup_hooks. It's called from the serial/isolate exit path in test_command.rs and the --parallel worker exit path in runner.rs, both after is_shutting_down = true (so the debug_assert holds). A leaksan.supp entry covers the still-rooted ScopeFunctions boxes, and three new ASAN+Linux-gated tests in isolation.test.ts reproduce the original SIGABRT and assert it's gone.

Security risks

None apparent. The new code path is gated on bun_core::env::ENABLE_ASAN (a compile-time constant), so release builds are byte-for-byte unchanged at the call sites. No new inputs, no auth/crypto/permissions surface.

Level of scrutiny

Medium-high. Although the runtime change is a no-op outside ASAN builds, shutdown/teardown ordering is notoriously subtle: this flips has_run_cleanup_hooks to true on a path that never ran on_exit(), which means any registered napi env cleanup hooks on that list are intentionally never drained (the PR description argues this is the documented destructOnExit semantics for bun test, and notes that #32146 changes the same bookkeeping — whichever lands second needs a small rebase to use napi_internal_set_cleanup_hooks_only() + run_cleanup_hooks() instead). That cross-PR coupling and the GC-during-shutdown ordering are exactly the kind of thing a maintainer who owns the shutdown path should sign off on.

Other factors

  • CI on the head commit shows musl build-cpp failures and assorted infra noise; this PR touches no C++, so those are likely unrelated, but the build isn't fully green.
  • The new tests are well-targeted (they assert on the LSan report text for --parallel since a worker's at-exit abort doesn't change the coordinator's exit code), and the suppression entry is narrowly scoped and well-commented.
  • The bug-hunting system found no issues; CodeRabbit's only note was a trivial test.each style nit.

@robobun

robobun commented Jun 12, 2026

Copy link
Copy Markdown
Collaborator Author

Regarding the review note about the --bail shutdown path (the module-load-failure bail in TestCommand::run): verified, and it is real but insufficient on its own. I prototyped adding collect_for_leak_check_at_exit() there plus a --bail test case; the child still aborts because printing the failure caches a ParsedSourceMap Arc in vm.saved_source_map_table (SavedSourceMap.rs:365 via remap_zig_exception), which LSan cannot scan, and that fires on any failing run, bail or not. The ordinary-test bail is a third case: handle_test_completed calls Global::exit(1) directly mid-execution, where running the exec tail teardown is not obviously safe (the run loop still holds the BunTestCell strong).

None of these have CI exposure today: every test spawning bun test --bail is in test/no-validate-leaksan.txt, and the leak lane sets BUN_DESTRUCT_VM_ON_EXIT=1 whose destroy() frees the source-map table. Making failure exits LSan-clean is a separate change with its own blockers, tracked in #32183; this PR stays scoped to green-run exits (the CI flake from #32176).

Comment thread src/runtime/cli/test_command.rs
Comment thread src/jsc/VirtualMachine.rs Outdated
@robobun

robobun commented Jun 12, 2026

Copy link
Copy Markdown
Collaborator Author

The failing-exit half from the scope note is up at #32187, stacked on this branch. Merging this PR first, then rebasing that one onto main, is the intended order.

robobun added 5 commits June 21, 2026 00:01
… LSan lanes don't abort green runs

At exit of a bun test run, the final file's expect() wrapper boxes are
freed only by GC finalizers (Expect::finalize derefs the RefData that
pins the file's BunTestCell), and no collection runs between the last
test and exit(). The ASAN CI lane runs subprocesses with
ASAN_OPTIONS=detect_leaks=1:abort_on_error=1, so LeakSanitizer reports
those boxes and SIGABRTs (exit 134) an otherwise green run.

Add VirtualMachine::collect_for_leak_check_at_exit, called from the
test runner's exit path and the --parallel worker's exit path. It is a
no-op in non-ASAN builds. Under ASAN it marks the cleanup-hook list as
never-drained (these exit paths never call on_exit(), so napi
finalizers deferred by the sweep would otherwise be parked there
forever) and runs one synchronous full collection so the finalizers
free their Rust boxes before LSan scans.

The final global's ScopeFunctions boxes stay alive at exit (rooted by
the still-protected global's bun:test exports); LSan cannot scan the
JSC heap cells holding the only pointers to them, so suppress that
allocation site in test/leaksan.supp.

Fixes #32176
@robobun robobun force-pushed the farm/1b909a84/test-exit-lsan-leak branch from 3347d3f to 57e361b Compare June 21, 2026 00:08
@robobun

robobun commented Jun 21, 2026

Copy link
Copy Markdown
Collaborator Author

Rebased onto main. The only conflict was in test/leaksan.supp: main added suppression entries for run_scripts_with_filter / multi_run::run while this branch added the scope_functions::ScopeFunctions entry. Both are independent additions, kept both. VirtualMachine.rs auto-merged. Re-verified on a debug ASAN build: the three leak-clean cases pass with the fix and the serial case still aborts with the same 1884-byte Expect/RefData/BunTestCell leak when src is reverted to main.

@claude claude Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM — ASAN-only exit-path GC, no-op in release builds; prior feedback addressed.

Extended reasoning...

Overview

This PR adds VirtualMachine::collect_for_leak_check_at_exit() — a method that sets has_run_cleanup_hooks = true and runs one synchronous full GC — and calls it from the two bun test exit paths (main runner at test_command.rs:2955, parallel worker at runner.rs:713) immediately before global_exit(). It also adds a leaksan.supp entry for scope_functions::ScopeFunctions (a reachable-but-unscannable false positive), and three new isASAN && isLinux-gated test cases in isolation.test.ts that spawn bun test under detect_leaks=1:abort_on_error=1 and assert exit 0 with no LeakSanitizer report.

Security risks

None. The new code path is hard-gated on bun_core::env::ENABLE_ASAN (compile-time constant), so release builds early-return and behavior is unchanged. In ASAN builds it runs a GC at process exit after all test results are collected — no new attack surface, no user-controlled inputs.

Level of scrutiny

Low-to-moderate. While VM shutdown / napi finalizer ordering is subtle territory in general, the entire behavioral change is a no-op outside ASAN debug builds, and within ASAN builds it only fires after is_shutting_down = true on the test runner's exit path. The has_run_cleanup_hooks = true write matches the documented semantics on the field (VirtualMachine.rs:214-217) and mirrors what on_exit() already does before destructOnExit()'s collection. The suppression file addition is well-commented and bounded (per-global). This is CI-infrastructure work to deflake a leak-check lane, not a production behavior change.

Other factors

  • Both of my prior inline comments (the --bail sibling path and the LSAN_OPTIONS→ASAN_OPTIONS doc typo) were resolved: the doc comment was fixed, and the author explained (with a prototype) why the --bail path was intentionally excluded and tracked it in #32183 / #32187. The PR body's scope note documents this.
  • CodeRabbit's test.each nit was applied; the test is now also gated on isLinux to avoid the macOS-arm64 startup error.
  • The PR was rebased onto main today with only a trivial additive conflict in leaksan.supp; the author re-verified the three test cases pass and the serial case still fails on unfixed src.
  • No CODEOWNERS coverage for the touched files.
  • The bug-hunting system found no issues on this run.
  • The noted interaction with #32146 is a small rebase concern for whichever lands second, not a correctness issue with this change.

@robobun

robobun commented Jun 21, 2026

Copy link
Copy Markdown
Collaborator Author

CI (build 63682) is red on one lane, unrelated to this diff:

  • 13 x64-asan: crash in test/js/node/zlib/zlib.test.js with panic: 78 JavaScript functions were called outside of the microtask queue without draining microtasks. This is a pre-existing flake in zlib's async-write path, not from this PR: the identical crash on the same lane appears on build 63681 (unrelated branch farm/ad2450b3/transpiler-cache-version-namespace). This change only adds a GC at global_exit (after the final event-loop tick, and global_exit diverges), while that panic is checked at the top of tick_queue_with_count during the run, so the two are temporally disjoint and this diff cannot produce it.
  • 11 aarch64: test/bake/dev-and-prod.test.ts HMR timeout (warning, passed on retry).
  • 14 x64: test/cli/install/bun-install.test.ts (package install, untouched by this diff).

The diff touches only the bun test exit paths, the leak test, and a leaksan suppression. Locally on a debug ASAN build the three leak-clean cases pass with the fix and the serial case still aborts with the same 1884-byte Expect/RefData/BunTestCell leak when src is reverted to main. Ready for a maintainer; the red lane is unrelated flake.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

bun test --isolate leaks the last file's Expect/RefData at exit, aborting ASAN CI lanes (flaky 30205.test.ts failures)

1 participant