Symptom
On HttpArena, io_uring json-comp @ 16384 conns collapses to ~293K rps (CPU ~694%) only when the json-tls listener is co-hosted in the same process. CPU ~694% (≈7 of 64 cores) means the io_uring workers are blocked, not spinning — the rings stall.
run (same vanilla pin b189036) |
json-tls present? |
rps |
CPU |
| leaderboard baseline |
no |
2.01M |
4839% |
| MDA2AV/HttpArena#951 (io_uring static) |
no |
1.97M |
4557% |
| MDA2AV/HttpArena#955 (io_uring json-tls) |
yes |
293K ⚠️ |
694% |
| #952/#956 (epoll json-tls) |
yes |
2.31M |
6056% |
Collapses only with json-tls, only on the io_uring backend, only at the top conn count (16384, the highest lock/ring pressure). The epoll entry — same per-request gz_mu lock, same 2-server layout — is fine, so this is an io_uring-backend-specific interaction, not the lock or thread count alone.
Mechanism (most likely)
The io_uring HttpArena entry hosts two servers in one process: the io_uring server on :8080 + an epoll TLS server on :8081 (json-tls; io_uring has no TLS). Each calls new_server(...) and the backend spawns core.worker_count() (= nr_cpus()) workers, so on a 64-core box that's ~128 worker threads on 64 cores (2× oversubscription).
io_uring workers must drain CQEs continuously to make progress; when the ~64 (idle) TLS-server threads oversubscribe the cores, at 16384 conns the scheduler jitter is enough to stall the rings → cascade → CPU collapses. epoll tolerates worker starvation (it just re-arms on the next epoll_wait; no ring to starve). The per-request shared gz_mu RwMutex rlock on the json-comp hot path (a 7-key, static-after-warmup cache) amplifies it — a worker blocked on the futex stops draining its ring — but isn't the sole trigger (epoll has the same lock and survives). Already visible pre-collapse: io_uring CPU drops 6049%→4839% from 512→16384 conns even without json-tls.
Caveat: it's one CI run; the most-demanding test (max conns + gzip + max mem) could also flake. MDA2AV/HttpArena#957 (io_uring + json-tls) re-runs json-comp-16384 and is the decisive confirmation.
Proposed fixes
- (lib, the real fix) per-server worker count. Add a
workers int field to http_server.ServerConfig (0 = current default of worker_count()), threaded into both backends' worker fan-out. The json-tls listener serves a modest-load profile on :8081 and does not need nr_cpus() workers — capping it (e.g. 4–8) removes the 2× oversubscription so the io_uring :8080 workers keep the cores. Today worker_count() is process-global (VANILLA_WORKERS/nr_cpus), so the two co-hosted servers can't be sized independently.
- (entry, clean cleanup) drop
gz_mu from the json-comp hot path. The json-comp profile sends only 7 fixed (count,m) pairs (requests/json-gzip-{1,5,10,15,25,40,50}.raw), so precompute all 7 gzip responses at boot and read the cache lock-free (it's never written after warmup). Removes a per-request shared-RwMutex acquisition across all workers — a latent high-conn contention point on both backends.
Acceptance
- io_uring
json-comp @ 16384 holds ~2M rps with the json-tls listener co-hosted (CPU back near saturation).
- No regression on the io_uring non-TLS profiles at 512/4096.
Found investigating the MDA2AV/HttpArena#955 benchmark-bot run. Fix #1 is the vanilla lib (http_server.c.v + backend worker fan-out); fix #2 is the HttpArena io_uring entry (consumer).
Symptom
On HttpArena, io_uring
json-comp@ 16384 conns collapses to ~293K rps (CPU ~694%) only when the json-tls listener is co-hosted in the same process. CPU ~694% (≈7 of 64 cores) means the io_uring workers are blocked, not spinning — the rings stall.b189036)Collapses only with json-tls, only on the io_uring backend, only at the top conn count (16384, the highest lock/ring pressure). The epoll entry — same per-request
gz_mulock, same 2-server layout — is fine, so this is an io_uring-backend-specific interaction, not the lock or thread count alone.Mechanism (most likely)
The io_uring HttpArena entry hosts two servers in one process: the io_uring server on :8080 + an epoll TLS server on :8081 (json-tls; io_uring has no TLS). Each calls
new_server(...)and the backend spawnscore.worker_count()(=nr_cpus()) workers, so on a 64-core box that's ~128 worker threads on 64 cores (2× oversubscription).io_uring workers must drain CQEs continuously to make progress; when the ~64 (idle) TLS-server threads oversubscribe the cores, at 16384 conns the scheduler jitter is enough to stall the rings → cascade → CPU collapses. epoll tolerates worker starvation (it just re-arms on the next
epoll_wait; no ring to starve). The per-request sharedgz_muRwMutex rlock on the json-comp hot path (a 7-key, static-after-warmup cache) amplifies it — a worker blocked on the futex stops draining its ring — but isn't the sole trigger (epoll has the same lock and survives). Already visible pre-collapse: io_uring CPU drops 6049%→4839% from 512→16384 conns even without json-tls.Proposed fixes
workers intfield tohttp_server.ServerConfig(0 = current default ofworker_count()), threaded into both backends' worker fan-out. The json-tls listener serves a modest-load profile on :8081 and does not neednr_cpus()workers — capping it (e.g. 4–8) removes the 2× oversubscription so the io_uring :8080 workers keep the cores. Todayworker_count()is process-global (VANILLA_WORKERS/nr_cpus), so the two co-hosted servers can't be sized independently.gz_mufrom the json-comp hot path. The json-comp profile sends only 7 fixed (count,m) pairs (requests/json-gzip-{1,5,10,15,25,40,50}.raw), so precompute all 7 gzip responses at boot and read the cache lock-free (it's never written after warmup). Removes a per-request shared-RwMutex acquisition across all workers — a latent high-conn contention point on both backends.Acceptance
json-comp@ 16384 holds ~2M rps with the json-tls listener co-hosted (CPU back near saturation).Found investigating the MDA2AV/HttpArena#955 benchmark-bot run. Fix #1 is the vanilla lib (
http_server.c.v+ backend worker fan-out); fix #2 is the HttpArena io_uring entry (consumer).