fix(web): prevent SIGILL crash under sustained gRPC stream by adding WASM yield and backpressure - #12784
Conversation
…WASM yield and backpressure Fixes rerun-io#12723 The gRPC read loop in re_grpc_client/src/read.rs runs as a tight async loop via wasm_bindgen_futures::spawn_local. Under sustained load, stream.try_next().await resolves instantly (Poll::Ready) on every iteration, starving the browser event loop. Simultaneously, the re_quota_channel WASM send path bypasses backpressure entirely - logging a debug warning but sending into an unbounded crossbeam channel regardless of byte budget. The combination causes unbounded WASM linear memory growth. When memory.grow hits the 2 GiB wasm32 ceiling, the allocator aborts, and Chromium translates the WASM trap to SIGILL, killing the renderer. Firefox survives because SpiderMonkey traps the unreachable instruction at the VM level rather than forwarding it as an OS signal. Changes: 1. Periodic yield in gRPC read loop (WASM only): Every 32 messages, yield to the browser via setTimeout(0). This gives the event loop a tick for GC, rendering, and channel consumer drain. When the channel queue exceeds 128 messages, pause ingestion entirely until the consumer catches up. 2. Hard ceiling on WASM channel: When byte budget is exceeded by 2x, drop the message with a warning instead of sending anyway. This prevents unbounded memory growth while still allowing small bursts.
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
There was a problem hiding this comment.
Hi! Thanks for opening this pull request.
Because this is your first time contributing to this repository, make sure you've read our Contributor Guide and Code of Conduct.
Wumpf
left a comment
Wumpf
left a comment
There was a problem hiding this comment.
apologies for the long delay on this!
Not quite happy with this yielding mechanism, seems a lot of guesswork involved in that. Alternative suggestion: on wasm the quota channel could fail with a "I'm full, try later" kind of message. Upon receiving that, we delegate further pushing into the channel / processing input into another spawn_local call to try again.
(It might be needed to bubble this mechanism up through the internal stream_async to its caller)
| /// On WASM, there is no preemptive scheduler. A tight async loop that never | ||
| /// returns `Poll::Pending` will starve the browser event loop, preventing GC, | ||
| /// rendering, and other tasks from executing. This function creates a minimal | ||
| /// yield point by scheduling a `setTimeout` callback. |
There was a problem hiding this comment.
should be formulated as case where this is needed rather than describing the bug that we want to fix with that elsewhere
| // On WASM, we cannot block. Previously we would log a warning and send | ||
| // anyway, but this caused unbounded memory growth because the consumer | ||
| // (running on the same single thread) could never drain the channel while | ||
| // the producer was busy sending. When WASM linear memory hit the 2 GiB | ||
| // ceiling, `memory.grow` failed and Chromium raised SIGILL. (#12723) | ||
| // | ||
| // Now we enforce a hard ceiling at 2× capacity: messages that would push | ||
| // us past that limit are silently dropped. This is better than crashing. |
There was a problem hiding this comment.
should not describe the history of the method - only care about what it does now and why
| // Drop the message — it is better to lose data than to crash | ||
| // the entire browser tab with SIGILL. | ||
| return Ok(()); | ||
| } |
There was a problem hiding this comment.
the browser tab crashing is a browser bug that should be reported to the respective vendor! (crashing the wasm process ofc is not that, that's just us making mistakes)
However, not being able to send things is definitely an error and should return like that. This gives us potentially the a chance to handle things. Generally I see dataloss as a hard error - if Rerun just drops data you send to it without backpressure or a clear contract of when and when not it can receive data we have a problem that's almost as bad as crashing in my opinion
| /// rendering, and other tasks from executing. This function creates a minimal | ||
| /// yield point by scheduling a `setTimeout` callback. | ||
| #[cfg(target_arch = "wasm32")] | ||
| async fn yield_to_browser(millis: i32) { |
There was a problem hiding this comment.
how is yield 0 here defined exactly? When would one call with 0 and when with a higher value?
| // rather than pushing more data into unbounded memory. | ||
| #[cfg(target_arch = "wasm32")] | ||
| if !tx.is_empty() && tx.len() > 128 { | ||
| yield_to_browser(1).await; |
| // Yield to browser event loop periodically on WASM. | ||
| // Under sustained load, try_next().await resolves instantly | ||
| // (Poll::Ready) every iteration, starving the single-threaded | ||
| // event loop. A setTimeout(0) yield gives the browser one tick | ||
| // for GC, rendering, and channel consumer to drain. (#12723) | ||
| #[cfg(target_arch = "wasm32")] | ||
| { | ||
| msgs_since_yield += 1; | ||
| if msgs_since_yield >= 32 { | ||
| msgs_since_yield = 0; | ||
| yield_to_browser(0).await; | ||
| } |
There was a problem hiding this comment.
similar questions as above, but also why do we need both?
2 participants
Fixes #12723
Root Cause
The gRPC read loop in
e_grpc_client/src/read.rs runs as a tight async loop via wasm_bindgen_futures::spawn_local. Under sustained load, stream.try_next().await resolves instantly (Poll::Ready) on every iteration, starving the browser event loop.
Simultaneously, the
e_quota_channel WASM send path bypasses backpressure entirely — logging a debug warning but sending into an unbounded crossbeam channel regardless of byte budget.
The combination causes unbounded WASM linear memory growth. When memory.grow hits the 2 GiB wasm32 ceiling, the allocator aborts, and Chromium translates the WASM trap to SIGILL, killing the renderer process.
Firefox survives because SpiderMonkey traps the unreachable instruction at the VM level rather than forwarding it as an OS signal.
Changes
1. Periodic yield in gRPC read loop (WASM only)
Every 32 messages, yield to the browser event loop via setTimeout(0). This gives the browser a tick for GC, rendering, and the channel consumer to drain queued messages. When the channel queue exceeds 128 messages, pause ingestion entirely until the consumer catches up.
Uses the same web_sys::Window::set_timeout + js_sys::Promise + wasm_bindgen_futures::JsFuture pattern already established in
e_backoff/src/lib.rs.
2. Hard ceiling on WASM channel send path
When the byte budget is exceeded by 2×, drop the message with a warn_once! instead of sending anyway into the unbounded crossbeam channel. This prevents unbounded memory growth while still allowing small bursts (up to 2× capacity).
Previously, the WASM path logged a debug_once! and sent unconditionally — the byte quota was purely advisory.
Files Changed
Testing
Tested with the reproduction script from #12723:
�ash cargo run --package rerun-cli --no-default-features --features web_viewer -- \ --serve-web --memory-limit 1500MiBWith a sustained 250K point-cloud stream at ~8.75 MB/row. Before fix: Chrome crashes within 30–120 seconds with SIGILL. After fix: Chrome survives indefinitely, with graceful degradation under extreme load.