Under real multi-client load, wisp burns ~2 cores in a worker-pool lock/allocator contention storm. This is separate from the spider TLS busy-spin fixed in v0.5.12 (#145) — it reproduces with the spider fully disabled.
Observed (on-device, v0.5.12)
- ~188% CPU with the spider on; ~116% base with the spider disabled (new pid, no spider threads), serving ~30 clients via the StartOS proxy (
trust_proxy on).
- 2 of 4 httpz worker threads pinned ~99% each, rotating across TIDs; per-thread syscall alternates between
202 (futex FUTEX_WAIT) and running (userspace spin), with transient mmap churn. 3.3M+ voluntary context switches per thread.
- Signature =
std.Io.Mutex/RwLock contention + glibc arena (c_allocator) churn, not poll/read.
- Load-driven ramp (~30% → ~105% over the first 30–60s as clients attach).
Root cause
- Broadcast fan-out allocates per event.
Subscriptions.forEachMatching allocated a fresh AutoHashMap (seen) + ArrayList (pending) via c_allocator on every event → glibc arena-mutex futex + mmap churn on the writer thread.
- Global per-message rate-limiter locks.
ConnectionLimiter and EventRateLimiter each guarded a single process-global std.Io.Mutex+map, taken by every worker on every event/query/connection → worker-pool serialization. RwLock is writer-preferring, so REQ/CLOSE writers also force concurrent readers onto the futex slow path.
Fix
PR #151:
subscriptions.zig: broadcast scratch → threadlocal retained buffers cleared per call (race-free under the concurrent sync=none path); semantics unchanged.rate_limiter.zig: shard both limiters 16-way by IP hash; per-IP semantics exact, API unchanged.
Verified on Zig 0.16.0 (build + test green); security-review + pr-review clean.
Remaining
On-device confirmation after deploy: with the spider still off, re-run top -H + per-thread syscall and confirm the two workers drop off the futex storm.
Under real multi-client load, wisp burns ~2 cores in a worker-pool lock/allocator contention storm. This is separate from the spider TLS busy-spin fixed in v0.5.12 (#145) — it reproduces with the spider fully disabled.
Observed (on-device, v0.5.12)
trust_proxyon).202(futexFUTEX_WAIT) andrunning(userspace spin), with transientmmapchurn. 3.3M+ voluntary context switches per thread.std.Io.Mutex/RwLockcontention + glibc arena (c_allocator) churn, not poll/read.Root cause
Subscriptions.forEachMatchingallocated a freshAutoHashMap(seen) +ArrayList(pending) viac_allocatoron every event → glibc arena-mutex futex +mmapchurn on the writer thread.ConnectionLimiterandEventRateLimitereach guarded a single process-globalstd.Io.Mutex+map, taken by every worker on every event/query/connection → worker-pool serialization.RwLockis writer-preferring, so REQ/CLOSE writers also force concurrent readers onto the futex slow path.Fix
PR #151:
subscriptions.zig: broadcast scratch → threadlocal retained buffers cleared per call (race-free under the concurrent sync=none path); semantics unchanged.rate_limiter.zig: shard both limiters 16-way by IP hash; per-IP semantics exact, API unchanged.Verified on Zig 0.16.0 (build + test green); security-review + pr-review clean.
Remaining
On-device confirmation after deploy: with the spider still off, re-run
top -H+ per-thread syscall and confirm the two workers drop off the futex storm.