Skip to content

http: fix use-after-free when proxy tunnel closes after response completes mid-read#31959

Merged
cirospaciari merged 7 commits into
mainfrom
farm/06ff7d9a/fix-proxy-tunnel-close-uaf
Jun 17, 2026
Merged

http: fix use-after-free when proxy tunnel closes after response completes mid-read#31959
cirospaciari merged 7 commits into
mainfrom
farm/06ff7d9a/fix-proxy-tunnel-close-uaf

Conversation

@robobun

@robobun robobun commented Jun 7, 2026

Copy link
Copy Markdown
Collaborator

Fixes a use-after-free in the HTTP client's proxy tunnel close path (Sentry BUN-2VY8, ~10 events/day on Windows release builds; reproduces deterministically under ASAN on all platforms).

Repro

fetch() through an HTTP CONNECT proxy to an HTTPS origin, where the origin's final response bytes and its TLS close_notify reach the client in a single TCP batch (origin writes the response and immediately closes). The regression test builds exactly that: a local CONNECT proxy that holds origin-to-client bytes after the handshake and flushes session tickets + response + close_notify in one write.

On an unfixed ASAN build:

ERROR: AddressSanitizer: heap-use-after-free
READ of size 8 thread T11 (HTTP Client)
  #0 Option<RefPtr<ProxyTunnel>>::as_ref
  #1 bun_http::proxy_tunnel::on_close                    src/http/ProxyTunnel.rs:525
  #2 SSLWrapper<*mut HTTPClient>::trigger_close_callback src/uws/lib.rs:802
  #3 SSLWrapper<*mut HTTPClient>::handle_reading         src/uws/lib.rs:1022
  #4 SSLWrapper<*mut HTTPClient>::handle_traffic
  #5 SSLWrapper<*mut HTTPClient>::receive_data
  #6 ProxyTunnel::receive                                src/http/ProxyTunnel.rs:751
freed by:
  AsyncHTTP::on_async_http_callback_raw                  src/http/AsyncHTTP.rs:813
  HTTPClient::send_progress_update_without_stage_check   src/http/lib.rs:3793

Cause

  1. handle_reading processes the batch: SSL_read returns the body bytes, the next SSL_read hits close_notify (SSL_ERROR_ZERO_RETURN), which sets received_ssl_shutdown and sent_ssl_shutdown before flushing the already-decrypted bytes through the data callback.
  2. The data callback completes the response. The done path runs close_proxy_tunnel(true) -> ProxyTunnel::shutdown() -> SSLWrapper::shutdown(true), which hits the already-shut-down early return (sent_ssl_shutdown || fatal_error) and returns without setting closed_notified. The result callback then frees the ThreadlocalAsyncHTTP embedding the HTTPClient, the exact pointer stored in the wrapper's handlers.ctx.
  3. Control returns to handle_reading. Its liveness guard (ssl.is_none() || closed_notified()) passes because neither is set, so trigger_close_callback() invokes on_close(handlers.ctx) on the freed client. When the allocation has been recycled, on_close can ref or close a different request's tunnel instead of faulting.

Fix

src/uws/lib.rs: when SSLWrapper::shutdown(fast_shutdown=true) takes the already-shut-down early return, fire trigger_close_callback() (idempotent via closed_notified) so the wrapper is marked closed before the owner detaches and frees handlers.ctx. A fast shutdown is a full teardown, and the normal fast-shutdown path already fires the close callback unconditionally; this only closes the gap where the SSL-level shutdown had already happened.

Graceful shutdown(false) (node:tls half-close via UpgradedDuplex / WindowsNamedPipe) is unchanged, so reads after a sent close_notify keep working.

Verification

New test in test/js/bun/http/proxy.test.ts (test.skipIf(!isASAN), the UAF is only deterministic under ASAN): fails on an unfixed ASAN debug build with the heap-use-after-free above, passes with the fix. Full proxy.test.ts (46 tests) plus node-tls-connect, node-tls-upgrade, node-tls-duplex-close-throw-uaf, node-tls-socket-allow-half-open-option, node-tls-server, fetch-tls-cert, and node-https-checkServerIdentity suites pass.

Note on the asan-lane CI failure (#32144)

The intermittent LeakSanitizer failure on the x64-asan shard (deferred napi finalizers parked on a never-drained cleanup-hook list at bun test exit) is being fixed in #32146, which carries the same global_exit() drain plus a hooks-only guard that skips pending napi_wrap finalizers on undrained-loop exits. A subset version of that fix was briefly on this branch (e59bc1d) but without the hooks-only guard it made test/js/third_party/duckdb/duckdb-basic-usage.test.ts SEGV at exit on the asan lane (build 62135), exactly the failure mode #32146's guard prevents, so it was reverted (61f9e70). This PR is scoped to the proxy-tunnel UAF; its asan lane can still intermittently hit the pre-existing #32144 leak until #32146 lands.

Related PRs

  • http: fix ProxyTunnel use-after-free when response completes mid-handleReading #30606 addresses the same crash signature but patches only the .zig reference files, which are no longer compiled; this PR fixes the shipping Rust implementation.
  • http: fix use-after-free in proxy-tunnel close dispatch #31952 fixes the same UAF by calling a new mark_close_notified() helper from ProxyTunnel::shutdown (silently setting the flag at one call site, with close_raw exempted). This PR instead closes the gap inside SSLWrapper::shutdown(true) itself, so every fast-shutdown caller (ProxyTunnel::shutdown, ProxyTunnel::close_raw, UpgradedDuplex::close, WebSocketProxyTunnel::shutdown) gets the same "no callbacks after teardown" guarantee without new wrapper API or a shutdown/close_raw asymmetry. The close callback is fired rather than suppressed, so the error teardown path keeps delivering on_close -> close_and_fail exactly once (idempotent via closed_notified). Test here is a deterministic single-shot repro (the test proxy reassembles TLS records and flushes tickets + response + close_notify in one write) rather than an iteration loop.

@github-actions github-actions Bot added the claude label Jun 7, 2026
@robobun

robobun commented Jun 7, 2026

Copy link
Copy Markdown
Collaborator Author
Updated 1:46 PM PT - Jun 17th, 2026

@cirospaciari, your commit 37d8722a6f3b3ae72b12ca90d91408f5d206d54a passed in Build #63180! 🎉


🧪   To try this PR locally:

bunx bun-pr 31959

That installs a local version of the PR into your bun-31959 executable, so you can run:

bun-31959 --bun

@coderabbitai

coderabbitai Bot commented Jun 7, 2026

Copy link
Copy Markdown
Contributor

Review Change Stack

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

  • @coderabbitai resume to resume automatic reviews.
  • @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

  • ▶️ Resume reviews
  • 🔍 Trigger review

Walkthrough

This PR ensures SSLWrapper::shutdown invokes the close callback in the fast-shutdown path, adds an ASAN-only regression test that reproduces response bytes coalesced with TLS close_notify, and makes child-process stderr visible on non-zero exits in an existing regression test.

Changes

SSL/TLS Shutdown and Callback Invocation

Layer / File(s) Summary
SSL wrapper shutdown callback invocation
src/uws/lib.rs
SSLWrapper::shutdown now computes received_ssl_shutdown, invokes trigger_close_callback() in the fast-shutdown path (guarded by existing idempotency), and returns the computed value.
VirtualMachine cleanup hooks drain
src/jsc/VirtualMachine.rs
Adds VirtualMachine::run_cleanup_hooks() and ensures global_exit drains RareData::cleanup_hooks (marking has_run_cleanup_hooks) even when on_exit() was skipped.
ASAN proxy regression test & harness import
test/js/bun/http/proxy.test.ts
Updated harness import to include TLS utilities; added an ASAN-only regression test that spawns a subprocess running an HTTPS origin and a custom CONNECT proxy which coalesces the tunnel tail so final response bytes and TLS close_notify arrive together, asserting the subprocess prints hello\n and exits with code 0.
Regression test stderr visibility
test/regression/issue/30205.test.ts
When spawned subprocesses exit with non-zero exitCode in both --isolate and --parallel variants, the test now prints captured stderr before asserting to ensure teardown-time abort/assertion output is visible.
🚥 Pre-merge checks | ✅ 4
✅ Passed checks (4 passed)
Check name Status Explanation
Title check ✅ Passed The title accurately and specifically describes the main fix: a use-after-free in the HTTP proxy tunnel close path when responses complete mid-read.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Description check ✅ Passed The PR description is comprehensive and covers all required template sections: 'What does this PR do?' with detailed explanation of the fix, and 'How did you verify?' with testing details.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.


Comment @coderabbitai help to get the list of available commands and usage tips.

@github-actions

github-actions Bot commented Jun 7, 2026

Copy link
Copy Markdown
Contributor

This PR may be a duplicate of:

  1. http: fix use-after-free in proxy-tunnel close dispatch #31952 - Fixes the same proxy-tunnel UAF in the same Rust files (ProxyTunnel.rs, uws/lib.rs) where a close callback fires after the HTTPClient is freed during response completion

🤖 Generated with Claude Code

@claude claude Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I didn't find any bugs and the fix looks well-reasoned, but this touches re-entrant callback ordering in the core SSLWrapper used by ProxyTunnel, UpgradedDuplex, WebSocketProxyTunnel, and WindowsNamedPipe — worth a human pass on the memory-safety invariants.

Extended reasoning...

Overview

The PR fixes a use-after-free in SSLWrapper::shutdown() (src/uws/lib.rs). When fast_shutdown=true hits the already-shut-down early return (sent_ssl_shutdown || fatal_error), it now fires trigger_close_callback() so closed_notified is set before the owner frees handlers.ctx. This closes the gap where handle_reading's deferred close-callback would otherwise fire into a freed HTTPClient. A new ASAN-only regression test in proxy.test.ts constructs a CONNECT proxy that coalesces response + close_notify into one TCP write to deterministically reproduce the UAF.

Security risks

No new attack surface is introduced — this is a defensive fix for a UAF in client-side TLS teardown. However, the change runs inside the BoringSSL wrapper's re-entrant callback machinery, so an incorrect fix here could itself introduce UAF or double-close in a different consumer.

Level of scrutiny

High. SSLWrapper::shutdown(true) has four consumers (HTTP ProxyTunnel, UpgradedDuplex, WebSocketProxyTunnel, WindowsNamedPipe), and the new trigger_close_callback() fires their respective on_close handlers from a code path that previously did not. I traced the ProxyTunnel case (the motivating bug): close_proxy_tunnel does proxy_tunnel.take() before tunnel.shutdown(), so the re-entrant on_close sees proxy_tunnel == None and returns early — only closed_notified is set, which is exactly the intent. The UpgradedDuplex/WebSocket paths look consistent with the existing non-early-return fast-shutdown branch (which already fires the callback unconditionally), and idempotency via closed_notified guards double-fire. But the correctness here hinges on subtle aliasing and ordering invariants documented in scattered comments (LaunderedSelf, disjoint field accessors, NLL-dead borrows), and I'd want a human familiar with this subsystem to confirm none of the other consumers' on_close handlers assume they won't fire from inside their own teardown path.

Other factors

The PR description is thorough (ASAN trace, root-cause walkthrough, verification across 7 test suites), the code change itself is ~15 lines with a detailed inline comment, and received is correctly read before the callback to survive re-entrant teardown. The test is well-constructed (TLS-record-aware coalescing proxy) though ASAN-gated. This is a good PR — it's just not a mechanical change I can rubber-stamp.

@robobun

robobun commented Jun 7, 2026

Copy link
Copy Markdown
Collaborator Author

Audited the fourth consumer the review mentions, WebSocketProxyTunnel (src/http_jsc/websocket_client/WebSocketProxyTunnel.rs):

  • Its only shutdown(true) caller is WebSocketProxy::drop (src/http_jsc/websocket_client/WebSocketProxy.rs:72), which already gets a synchronous on_close from the normal fast-shutdown path today whenever the SSL flags are not yet set, so firing it in the already-shut-down case is not a new re-entry shape for that handler.
  • Its handlers.ctx is the intrusively refcounted tunnel itself (guarded by ScopedRef in on_close), not a freeable client pointer, and on_close reads the connected_websocket/upgrade_client backrefs which are cleared before their owners free (documented BACKREF invariant at WebSocketProxyTunnel.rs:406).
  • Recursion is bounded the same way as the other consumers: closed_notified is set before on_close runs, so a re-entrant shutdown(true) hits the early return and the added trigger_close_callback() no-ops.

Same conclusion for UpgradedDuplex (ctx is the duplex that embeds the wrapper) and WindowsNamedPipe (only ever calls shutdown(false), which this change does not touch).

Comment thread test/js/bun/http/proxy.test.ts

@claude claude Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for addressing the stderr nit. The fix itself reads correctly and the consumer audit is convincing, but since this changes callback re-entry timing in shared SSLWrapper::shutdown (TLS lifetime code touching ProxyTunnel/UpgradedDuplex/WebSocketProxyTunnel) and there's a competing approach in #31952, I'd like a human to make the final call.

Extended reasoning...

Overview

The PR adds ~15 lines to SSLWrapper::shutdown() in src/uws/lib.rs: when fast_shutdown=true hits the already-shut-down early return (sent_ssl_shutdown || fatal_error), it now fires trigger_close_callback() so closed_notified is set before the owner frees handlers.ctx. This closes a UAF where handle_reading's deferred on_close fired into a freed HTTPClient after the data callback completed the response and tore down the tunnel mid-read. A new ASAN-gated regression test in proxy.test.ts deterministically reproduces the batched response+close_notify scenario via a record-coalescing CONNECT proxy.

Since my prior review, commit f8b4675 added the if (exitCode !== 0) console.error("stderr:", stderr) line I'd suggested — that nit is resolved.

Security risks

No new attack surface. The change is a memory-safety hardening (UAF fix), not a protocol or auth change. The only behavioral delta is that on_close may now fire synchronously from the early-return branch of a fast shutdown — the author audited all four shutdown(true) callers (ProxyTunnel, UpgradedDuplex, WebSocketProxyTunnel; WindowsNamedPipe only uses shutdown(false)) and showed re-entry is bounded by the closed_notified idempotency guard, matching what the normal fast-shutdown path already does.

Level of scrutiny

High. This is production-critical TLS/HTTP-client code with subtle unsafe lifetime and re-entry semantics (laundered &mut self, handler vtables that can re-enter via the owning socket). The fix is small and the reasoning is well-documented, but the blast radius spans every SSLWrapper consumer, and the "fire vs. suppress the callback" design choice differs from the competing PR #31952. A maintainer familiar with the SSLWrapper consumers should confirm the re-entry analysis and pick between the two approaches.

Other factors

  • The bug-hunting system found no issues.
  • received is correctly captured before trigger_close_callback() (which may re-enter and mutate flags), preserving the original return value.
  • The test is ASAN-only by design (release builds read freed-but-intact memory), follows the file's subprocess conventions, and now surfaces the ASAN report on failure.
  • No CODEOWNERS entry covers these paths.

@robobun

robobun commented Jun 7, 2026

Copy link
Copy Markdown
Collaborator Author

CI status: the diff is green; all remaining failures in build 61324 are unrelated to this change.

The new regression test (ASAN-gated, in test/js/bun/http/proxy.test.ts) passed on the linux x64 ASAN lane in all three builds of this PR. Ready for maintainer review; the Related PRs section in the description compares this fix with #31952.

@robobun

robobun commented Jun 10, 2026

Copy link
Copy Markdown
Collaborator Author

Build 61676 (the merge of main at 1fe7c4e) did not execute: every job expired waiting for an agent, so there are no test results. The previously noted bunx.test.ts breakage is now skipped on main (#32042), so a plain re-run of CI should come back green. The diff itself is unchanged since f8b4675.

robobun added 2 commits June 11, 2026 16:42
…letes mid-read

When the final response bytes and the peer's TLS close_notify arrive in
one TCP batch, SSLWrapper::handle_reading sets sent_ssl_shutdown before
flushing the decrypted bytes through the data callback. If that callback
completes the HTTP response, the done path calls ProxyTunnel::shutdown()
to tear the wrapper down, but SSLWrapper::shutdown()'s already-shut-down
early return skipped trigger_close_callback, leaving closed_notified
unset. The result callback then freed the AsyncHTTP embedding the
HTTPClient, and handle_reading's subsequent trigger_close_callback
invoked on_close on the freed handlers.ctx pointer.

Make the fast_shutdown early return fire trigger_close_callback (a
no-op when already notified) so the wrapper is marked closed before the
owner detaches and frees the ctx. Graceful shutdown(false) is unchanged
so half-open TLS reads keep working.
@cirospaciari cirospaciari force-pushed the farm/06ff7d9a/fix-proxy-tunnel-close-uaf branch from 1fe7c4e to e7a03c0 Compare June 11, 2026 23:42
@cirospaciari

Copy link
Copy Markdown
Member

@robobun adopt

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 2

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@test/js/bun/http/proxy.test.ts`:
- Around line 919-1029: This ASAN subprocess test lacks the standard 60_000ms
timeout and can flake under slow ASAN/debug runs; update the test invocation
that starts with test.skipIf(!isASAN)( "response + close_notify...", async () =>
{ ... } ) to pass a 60_000 timeout (i.e. add the 60_000 ms timeout as the third
argument to the test.skipIf(...) call), so the spawned Bun subprocess (Bun.spawn
/ bunExe usage) has enough time to complete under ASAN.
- Around line 1024-1027: The test currently only prints stderr when exitCode !==
0 which can miss ASAN diagnostics; change the assertions in the block around
proc/stdout/stderr/exited to assert stderr first and fail if it contains
sanitizer diagnostics (e.g., check stderr is empty or does not match
/AddressSanitizer|ASAN|ERROR: AddressSanitizer/), then assert stdout and
exitCode; update the variables referenced (proc, stdout, stderr, exitCode) in
the test to make the stderr check the first expectation so sanitizer output
causes the test to fail immediately.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Pro

Run ID: 0c54676d-7b61-4990-920d-32ff13cc32fa

📥 Commits

Reviewing files that changed from the base of the PR and between f8b4675 and e7a03c0.

📒 Files selected for processing (2)
  • src/uws/lib.rs
  • test/js/bun/http/proxy.test.ts

Comment thread test/js/bun/http/proxy.test.ts
Comment thread test/js/bun/http/proxy.test.ts
@cirospaciari

Copy link
Copy Markdown
Member

@robobun fix the asan failure

…borts at teardown

The --isolate and --parallel subprocesses in 30205.test.ts can abort
after printing their summary line, in which case the toContain
assertions pass and only the exit code differs; the captured stderr
holding the actual abort report was never shown. Print it on nonzero
exit so CI failures are diagnosable.
@robobun

robobun commented Jun 12, 2026

Copy link
Copy Markdown
Collaborator Author

Investigated the ASAN failure (test/regression/issue/30205.test.ts, the --isolate napi finalizer test, exit 134 on one debian x64-asan shard of build 61984):

  • Could not reproduce locally on this branch's ASAN debug build: 40 runs total (7 full-test runs, 15 sequential inner bun test --isolate runs with BUN_JSC_collectContinuously=1, 18 more under 3-way parallel contention), all green.
  • The inner process aborted after printing "Ran 8 tests across 8 files." (the stderr toContain assertions passed; only the exit code differed), so this is a teardown-time abort, and the captured stderr holding the actual ASAN/assertion report was swallowed by the test. Pushed e20464e adding if (exitCode !== 0) console.error("stderr:", stderr) to both subprocess tests in that file so the next occurrence shows the real report.
  • No mechanism connection to this PR's diff: the change is confined to SSLWrapper::shutdown(true) (proxy-tunnel TLS teardown), and the 30205 fixtures do no networking, so the changed code never executes there. The same ASAN lane ran this test green on the three pre-rebase builds of this PR (61302, 61322, 61324), and none of the 16 main commits in the rebase window touch napi, the test runner, or this test.

The push re-runs CI; if that shard fails again the abort report will now be in the log and I'll dig into it from there.

@robobun

robobun commented Jun 12, 2026

Copy link
Copy Markdown
Collaborator Author

The stderr surfacing from e20464e did its job: the asan failure is a LeakSanitizer report at process exit, not a use-after-free.

==5438==ERROR: LeakSanitizer: detected memory leaks
Direct leak of 31904 byte(s) in 997 object(s) allocated from:
    new<bun_runtime::napi::napi_body::NapiFinalizerTask>
    NapiFinalizerTask::init / Finalizer::enqueue (src/runtime/napi/napi_body.rs)
    Zig::NapiRef::callFinalizer
    JSC::WeakBlock::sweep <- Heap::collectNow
    Zig__GlobalObject__destructOnExit (ZigGlobalObject.cpp:3966)
    VirtualMachine::global_exit

The final exit GC sweeps the last fixture file's ~1000 napi-wrapped objects; their deferred finalizer tasks are enqueued into a sink that never drains at that point of teardown, so the boxes are live when LSAN scans. NapiFinalizerTask::schedule() already has is_shutting_down/has_run_cleanup_hooks handling, so one of those flags is not in the expected state during destructOnExit's collect on the release-asan build. It does not reproduce on a local debug ASAN build even with ASAN_OPTIONS=detect_leaks=1.

This is pre-existing and independent of this PR's diff (the 30205 fixtures do no networking, so the SSLWrapper change never executes there); it will intermittently fail any PR whose asan shard runs that file. Filed #32144 with the full analysis; happy to take that fix in a separate PR.

@cirospaciari

Copy link
Copy Markdown
Member

@robobun can you fix CI erros?

…on't leak

bun test's exit paths set is_shutting_down directly and call
global_exit() without on_exit(), so RareData::cleanup_hooks was never
drained and has_run_cleanup_hooks stayed false. NapiFinalizerTask::
schedule() parks finalizers deferred during destructOnExit's final
collectNow() on that list when is_shutting_down && !has_run_cleanup_hooks,
relying on a drain that never came, leaking every parked
Box<NapiFinalizerTask> (LeakSanitizer failures on the x64-asan lane
running test/regression/issue/30205.test.ts, tracked in #32144).

Drain the hooks at the top of global_exit when on_exit() has not run:
parked hooks execute while the heap is still alive, and finalizers
enqueued by the final GC take the existing drop path instead of parking.
The bun run path is unchanged (on_exit() already ran, so the drain
no-ops).
@robobun

robobun commented Jun 12, 2026

Copy link
Copy Markdown
Collaborator Author

Fixed in e59bc1d. Root cause of the asan failure:

bun test's exit paths (test_command.rs) set is_shutting_down directly and call global_exit() without on_exit(), so RareData::cleanup_hooks is never drained and has_run_cleanup_hooks stays false. When destructOnExit's final collectNow() sweeps napi-wrapped objects, NapiFinalizerTask::schedule() sees is_shutting_down && !has_run_cleanup_hooks and parks each deferred finalizer on the cleanup-hook list, expecting a later drain that never happens. Every parked Box<NapiFinalizerTask> is still live when LeakSanitizer scans, and abort_on_error=1 turns that into the exit-134 you saw. bun run never hits this because it calls on_exit() before global_exit().

The fix makes global_exit() drain the cleanup hooks (and set the flag) when on_exit() has not run, so finalizers enqueued by the final GC take the existing drop path instead of parking. The only reader of has_run_cleanup_hooks is NapiFinalizerTask::schedule, and the bun run path no-ops (hooks already drained), so the blast radius is the test-runner exit only.

On a deterministic regression test: the leak needs the exit-time GC to be the first thing that sweeps the wrappers, which depends on JSC sweep scheduling. I could not trigger that window locally (40+ ASAN runs of the 30205 scenario plus three targeted fixture variants), so the proof is the failing shard itself: builds 61984 and 61990 show the NapiFinalizerTask leak on unfixed code, and this push should bring that shard back green. Details in #32144.

@robobun

robobun commented Jun 12, 2026

Copy link
Copy Markdown
Collaborator Author

Update on the asan failure: I reverted my cleanup-hook drain (e59bc1d, reverted in 61f9e70).

Build 62135 showed why: with the drain in place but nothing skipping pending napi_wrap finalizers, test/js/third_party/duckdb/duckdb-basic-usage.test.ts SEGVs at exit on the asan lane (duckdb still has a queued ConnectTask whose destructor Unrefs a Connection wrap that the finalizer pass already deleted).

#32146 is the complete version of this fix and was already in flight: it carries the same global_exit() drain plus a hooks-only guard that skips the wrap-finalizer pass on undrained-loop exits (which prevents exactly that duckdb crash), a deterministic regression test, and it closes #32144. This PR stays scoped to the proxy-tunnel UAF; until #32146 lands, the asan shard here can still intermittently hit the pre-existing #32144 leak, which is unrelated to this diff.

Comment thread test/regression/issue/30205.test.ts
@cirospaciari cirospaciari force-pushed the farm/06ff7d9a/fix-proxy-tunnel-close-uaf branch from 61f9e70 to 1d17b39 Compare June 16, 2026 21:57
@robobun

robobun commented Jun 16, 2026

Copy link
Copy Markdown
Collaborator Author

Build 62154 is green on 61f9e70. The PR is scoped to the SSLWrapper fast-shutdown fix, its deterministic ASAN regression test in proxy.test.ts, and the stderr-surfacing diagnostic in 30205.test.ts; the napi exit leak (#32144) is handled by #32146. Ready for review/merge.

cirospaciari pushed a commit that referenced this pull request Jun 17, 2026
…y socket is dead at ProxyHeaders write (#32462)

## Crash signature

`Panic: Internal assertion failure` inside `on_writable` after the
proxy-tunnel inner TLS handshake completes. Sentry BUN-2V7Z (~1080
lifetime, ~28/24h) and BUN-36JT (sibling). 100% Windows; affected 1.3.x
back to firstSeen 2026-04-11.

Stack (bun 1.3.14):
```
onData                    src/http/HTTPContext.zig
onData                    src/http/http.zig
receive                   src/http/ProxyTunnel.zig
receiveData / handleTraffic / updateHandshakeState / triggerHandshakeCallback
onHandshake               src/http/ProxyTunnel.zig
onWritable                src/http/http.zig   <- assert(!socket.isShutdown()); assert(!socket.isClosed());
```

## Cause

`on_writable` has two sites that asserted the outer socket was live
immediately before writing the initial request:

* `send_initial_request_payload` (Pending/Headers/Opened arm)
* the `ProxyHeaders` arm (proxy tunnel: first HTTP write after the inner
TLS handshake)

When `fetch()` goes through a CONNECT proxy to an `https://` target, the
inner TLS handshake runs inside an in-process `SSLWrapper`.
`ProxyTunnel.on_handshake` fires from `handleTraffic` while draining
buffered bytes delivered by a single `onData` call, then synchronously
calls `HTTPClient.on_writable`. Between that call and the `ProxyHeaders`
write, the outer socket can already be closed or shut down:

* `proxy.on_writable` (run first, line 2849) writes buffered encrypted
bytes with `socket.write()` and then `wrapper.flush()` re-enters
`handleTraffic`; on an HTTPS proxy, a failing outer `SSL_write` sets
`ssl_fatal_error` so `socket.is_shutdown()` flips true, and a buffered
inner `close_notify` fires `on_close -> close_and_fail ->
terminate_socket` so `socket.is_closed()` flips true.

In release builds without debug assertions, both sites fall through to
`write_to_socket` / `ProxyTunnel::write` which return 0 on a dead
socket, leaving the request stuck at `Headers` / `ProxyHeaders` until
the idle timeout. In builds with debug assertions enabled the assert
panicked.

The Sentry cluster is 100% Windows because Windows Zig release builds
used `ReleaseSafe` (see `scripts/build/zig.ts` at the affected
revisions: "since Bun 1.1, Windows builds use ReleaseSafe"), which
enables `Environment.allow_assert`. Other platforms shipped
`ReleaseFast` where the assert compiled out. Since the Rust rewrite the
assert is `debug_assert!`, so release builds on all platforms now stall
instead of panicking; `bun bd` still panics.

## Fix

Convert both assertion sites to real runtime checks that surface
`error.ConnectionClosed` through the existing `close_and_fail` path.
`send_initial_request_payload` already returns `Result<_, Error>` and
its caller already `close_and_fail`s on error; the `ProxyHeaders` arm
calls `close_and_fail` directly (matching the adjacent `write_request`
error branch).

## Verification

New test in `test/js/bun/http/proxy.test.ts` (fixture at
`proxy-handshake-closed-socket-fixture.ts`) runs 10 fetches through an
HTTP and HTTPS CONNECT proxy that relays the backend's TLS
server-handshake flight and immediately RSTs the outer connection. Every
iteration rejects with `ECONNRESET` / `ConnectionClosed` (not
`TimeoutError`, not a panic). Full `proxy.test.ts` (47 tests) passes.

### Note on fail-before

This race is not deterministically reproducible on Linux loopback. A
100-iteration probe against an unfixed `bun bd` build (50 HTTP-proxy +
50 HTTPS-proxy) never tripped the assertion: on POSIX the proxy's RST is
delivered to the client as a separate event loop iteration after
`on_writable` has already returned, so
`socket.is_closed()/is_shutdown()` is still false at the assertion site.
The Windows-specific timing that lands the dead-socket state inside the
synchronous `on_handshake -> on_writable` window (AFD poll semantics
plus ReleaseSafe assertions) cannot be reconstructed without
instrumenting `src/`. The new test guards the code path and the
observable behaviour (clean rejection, no stall) but does not
fail-before on Linux.

## Related

* #31959 fixes a UAF on the *response* side of the same tunnel
(`handle_reading` after the client is freed); this PR is the *request*
side (first write after handshake).
* #23092 is a different crash site in the same function.

---------

Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>
Concatenates #31959's ASAN proxy-tunnel-close UAF regression test with
#31587's proxy tests (both append to the same end-of-file location).
@cirospaciari cirospaciari merged commit 1ecbea9 into main Jun 17, 2026
78 checks passed
@cirospaciari cirospaciari deleted the farm/06ff7d9a/fix-proxy-tunnel-close-uaf branch June 17, 2026 22:00
Jarred-Sumner pushed a commit that referenced this pull request Jun 23, 2026
…AF through tunnel (#32635)

## What

Comprehensive stress testing of the HTTP client proxy code paths (fetch
and WebSocket, both `ProxyTunnel` and `WebSocketProxyTunnel`): **661
tests** across 5 new files plus a shared adversarial proxy/origin helper
and a subprocess memory-probe fixture.

| File | Tests | Covers |
| --- | --- | --- |
| `proxy-stress-helpers.ts` | n/a | Adversarial CONNECT + absolute-form
proxy (http/https outer) with per-stage RST/kill/trickle/split hooks;
adversarial origin with content-length / chunked / close-delimited ×
identity/gzip/deflate/br/zstd, truncation at arbitrary byte offset,
redirect, echo. |
| `proxy-stress-matrix.test.ts` | 335 | `{http,https}` proxy ×
`{http,https}` origin × framing × encoding × body-size × keepalive
response matrix; upload matrix across
string/Uint8Array/Blob/FormData/ReadableStream/async-iterator; streamed
response via `getReader()`; trickled 1-byte-per-tick downstream; split
CONNECT envelope; RFC 9110 §9.3.6 ignored-header handling; hop-by-hop
stripping; method matrix; cross-scheme redirects. |
| `proxy-stress-lifecycle.test.ts` | 93 | Proxy RSTs client at every
tunnel stage (request-received / upstream-connected / connect-replied /
first-client-byte / first-upstream-byte) × both origins × both
keepalive; same during upload (string + stream body); proxy drops
upstream at every stage; origin RST at 0/10/60/200 response bytes;
close-delimited × compression; abort at every stage; abort after
headers; abort churn (200× under ASAN); ASAN-only
TLS-alert-during-handshake UAF repro. |
| `proxy-stress-errors.test.ts` | 51 | CONNECT
400/403/407/500/502/503/504/301/302; proxy unreachable; upstream
unreachable (502 via CONNECT and absolute-form); proxy auth
(missing/wrong/correct, URL and header) for all 4 combos; inner-TLS
verify (CA match, no-CA fail, checkServerIdentity reject); unsupported
scheme (ftp/socks4/socks5/socks5h/ws); h2-capable origin through proxy
stays HTTP/1.1. |
| `proxy-stress-concurrent.test.ts` | 31 | 32× parallel per combo ×
keepalive; tunnel reuse (sequential + auth-keyed); 12 origins through
one proxy; `reject_unauthorized` pool gate (lax → strict forces fresh
CONNECT); 4× concurrent 4MB echo; idle pooled tunnel receiving stray
data is evicted; 12 subprocess memory probes (6 modes × 2 proxy schemes,
300-1200 iterations each, RSS-growth bounded). |
| `proxy-stress-adversarial.test.ts` | 151 | Stacked split-CONNECT ×
chunked × compression × keepalive; origin status 200-503 × all combos;
origin-facing Host/header shape; `checkServerIdentity`
approve/reject/GC-loop; path/query edges; `verbose:true`;
`AbortSignal.timeout`; interleaved proxy/direct to same origin; CONNECT
target shape; **WebSocket proxy matrix** (ws/wss × http/https proxy)
with echo, 256KB binary, RST at CONNECT, 407 without auth, rapid close,
wss-via-https-proxy open/close churn under GC. |

Full suite runs in ~5 min under debug+ASAN.

## Bugs found and fixed

Two pre-existing bugs surfaced by the suite (both reproduce on main
without any test-helper changes). Fixed here in `src/http/lib.rs`.

### 1. Streamed request body through a CONNECT tunnel never sent

`fetch()` to an `https://` origin through any proxy with a
`ReadableStream` / async-iterator body hangs forever: CONNECT succeeds,
inner TLS handshake completes, the `Transfer-Encoding: chunked` request
head is written, then nothing. String/Uint8Array/Blob/FormData bodies
are fine; `http://` origins (absolute-form, no tunnel) are fine.

Cause (two layers):

- The `RequestStage::ProxyHeaders` arm of `on_writable` computes
`has_sent_body = self.request_body().is_empty()`. For
`HTTPRequestBody::Stream` the bytes buffer is always empty here, so the
request jumps straight to `RequestStage::Done` and the
`is_streaming_request_body` signal path a few lines below never runs.
The non-proxy `send_initial_request_payload` already gates this on
`matches!(original_request_body, HTTPRequestBody::Bytes(_))`.
- Even once `ProxyBody` is reached, its `Stream` case calls
`flush_stream` → `write_to_stream_using_buffer` →
`write_to_socket(socket, …)`, which writes plaintext chunked bytes to
the outer proxy socket instead of through the inner TLS session. The
`Bytes` case right next to it correctly routes through
`ProxyTunnel::write`.

Fix: mirror the non-proxy path's `Bytes`-only `has_sent_body` check (and
matching `debug_assert`) in the `ProxyHeaders` arm; in
`write_to_stream_using_buffer`, route through `ProxyTunnel::write` when
`self.proxy_tunnel.is_some()`, treating `WantRead`/`WantWrite` from the
inner SSL as backpressure. Encrypted output reaches the outer socket via
the existing `write_encrypted` callback, same as the `Bytes` path. This
also covers the `HTTPThread` drain loop which calls the same
`flush_stream`.

### 2. `heap-use-after-free` in `HTTPClient::on_writable` when a TLS
alert is buffered with the inner handshake flight

```
READ of size 1 thread T12 (HTTP Client)
  #0 HTTPClient::on_writable::<true, false>    src/http/lib.rs:~2856
  #1 bun_http::proxy_tunnel::on_handshake      src/http/ProxyTunnel.rs:450
freed by:
  AsyncHTTP::on_async_http_callback_raw        src/http/AsyncHTTP.rs:813
  HTTPClient::close_and_fail::<false>
  bun_http::proxy_tunnel::on_close             src/http/ProxyTunnel.rs:577
  SSLWrapper::handle_reading                   src/uws/lib.rs:1053
  SSLWrapper::flush                            src/uws/lib.rs:669
  ProxyTunnel::on_writable::<false>            src/http/ProxyTunnel.rs:740
  HTTPClient::on_writable::<true, false>       src/http/lib.rs:~2850
  bun_http::proxy_tunnel::on_handshake         src/http/ProxyTunnel.rs:450
```

If the origin closes or sends a TLS alert in the same buffer as its
ServerHello flight (origin rejecting client cert, corrupted stream via
misbehaving proxy, origin crash), the client's `on_handshake →
on_writable → proxy.on_writable → SSLWrapper::flush → handle_reading`
chain processes the alert, fires `on_close → close_and_fail`, which runs
the result callback and frees the `ThreadlocalAsyncHTTP` embedding
`*self`. Control returns to `on_writable`, which immediately reads
`self.state.flags.is_waiting_for_cert_check` on freed memory. Same bug
class already documented in `start_proxy_handshake`'s comment.

Fix: `close_and_fail` → `terminate_socket` synchronously marks the outer
socket closed, and the socket handle is owned by the event loop
(outlives the client). Check `socket.is_closed()` immediately after
`proxy.on_writable()` and return before touching `self`.

Deterministic ASAN repro in `proxy-stress-lifecycle.test.ts` ("TLS alert
in same buffer as inner handshake"): a CONNECT proxy that double-writes
every client→upstream byte, making the origin's TLS stack abort with an
alert. Looped 20× with `ASAN_OPTIONS=…:abort_on_error=1` so the
HTTP-thread UAF aborts the subprocess before the main thread's clean
exit wins the race.

## Verification

```
bun bd test test/js/bun/http/proxy-stress-*.test.ts
# 661 pass, 0 fail (debug+ASAN, ~5min)
```

Fail-before (src/ stashed, `bun bd`):
- `proxy-stress-matrix.test.ts -t "https-origin POST
ReadableStream|https-origin POST async-iterator"` → 8 fail
(hang/timeout)
- `proxy-stress-lifecycle.test.ts -t "TLS alert in same buffer"` → 1
fail (subprocess aborts with `heap-use-after-free`)

Existing `test/js/bun/http/proxy.test.ts` (48 tests) still passes.

## Related

- #31959 fixed a sibling UAF on the shutdown side of the same
`SSLWrapper` callback chain.
- #30606 touches the same `ProxyTunnel` close path but only in the
`.zig` reference files.

---------

Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants