[Tracking] New remote I/O backend: libcurl multi poll-based

[kingcrimsontianyu]

Background

KvikIO's existing remote I/O backend (EASY_THREADPOOL) services each sub-range with a blocking curl_easy_perform() on a dedicated thread-pool worker. One busy worker per in-flight request means concurrency is capped by the thread-pool size, and workers sit idle while blocked on network I/O.

This effort introduces a second backend (MULTI_POLL) built on libcurl's multi interface. A small number of reactor threads each own one CURLM* multi handle and drive many easy handles through a single non-blocking curl_multi_poll() loop. This decouples in-flight concurrency from thread count, removes the one-thread-per-request cost, and lets request batching scale much higher.

Essentially this new backend leverages both multplexing and multithreading.

Design

• Reactor (MultiPollReactor): One I/O thread + one CURLM*, driven by a 4-stage loop: submit (drain inbox, curl_multi_add_handle, gated by the concurrency limiter), perform (curl_multi_perform), drain (curl_multi_info_read, fire completion/failure), poll (curl_multi_poll, waken by curl_multi_wakeup).
• Reactor pool (MultiReactorPool): Process-wide singleton owning N reactors. Routes sub-ranges by dispatch policy (PER_CHUNK round-robin vs PER_PREAD affinity for connection reuse) and tracks pool-wide error.
• Aggregate context (RemoteMultiAggregateContext): One per pread(). Atomically counts down sub-ranges and resolves the std::future<size_t> with total bytes or the first exception.
• Concurrency limiter (ConcurrentRequestLimiter): Per-reactor share of a global request budget. Non-blocking try_acquire().
• Device path: BounceBufferCachePerThreadAndContext (pinned host staging, recycled via cuLaunchHostFunc once the async H2D completes), and CudaEventPool for low-overhead CUDA events.

Configuration

• New configurations are specified via environment variables:

Variable	Default	Meaning
KVIKIO_REMOTE_IO_BACKEND	EASY_THREADPOOL	Choice of backend EASY_THREADPOOL or MULTI_POLL
KVIKIO_REMOTE_IO_NUM_REACTORS	1	Number of reactors, each running on a separate thread
KVIKIO_REMOTE_IO_REACTOR_DISPATCH	PER_CHUNK	Choice of sub-range dispatching pattern PER_CHUNK or PER_PREAD
KVIKIO_REMOTE_IO_MAX_CONCURRENT_REQUESTS	256	Global in-flight request budget

Related PRs

• 1/n Create a new remote I/O backend: libcurl multi poll-based (1/n: host support) #960: Create the new backend (host buffer support). Merged
• 2/n Implement CUDA event pool to minimize runtime resource allocation overhead (libcurl multi poll-based backend 2/n) #919: CUDA event pool to minimize runtime resource-allocation overhead. Merged
• 3/n Implement per-(thread, CUDA context) bounce buffer cache singleton (libcurl multi poll-based backend 3/n) #965: Per-(thread, CUDA context) bounce buffer cache singleton. Merged
• 4/n Limit concurrency to counterintuitively improve overall performance for host buffer (libcurl multi poll-based backend 4/n) #978: Limit concurrency to improve host-buffer throughput. Merged
• 5/n Support device buffer for the new libcurl multi poll-based backend (5/n) #966: Device buffer support. WIP
• 9/n Develop an Nsight System plugin to profile network bandwidth #999: NIC bandwidth monitoring. In review

Together, 1~ 5 deliver host and device buffer support.

Planned future work

6/n: Retry logic for the multi-poll backend

Make the multi-poll backend consistent with the EASY_THREADPOOL on the retry logic. Max attempts (KVIKIO_HTTP_MAX_ATTEMPTS, default 3), retryable status codes (KVIKIO_HTTP_STATUS_CODES, default 429/500/502/503/504), CURLE_OPERATION_TIMEDOUT, and exponential backoff (500 ms base, 4 s cap). The multi-poll backend currently has no retry. Any transient error fails the entire pread.

7/n: Robustness and correctness hardening

Fix the curl_multi_remove_handle potential use-after-free issue (TODO at cpp/src/detail/multi_poll_reactor.cpp:245-249): on remove failure (rarely happen) the easy handle must not be returned to the LibCurl free pool.

8/n: Batch read API (vector of ranges in a single call)

Add an API to submit many (offset, size, dst) ranges in one call, returning a vector of futures (or a single future), all serviced concurrently through the reactor pool. Generalize RemoteMultiAggregateContext (or add a batch aggregate) to span multiple logical reads, and extend the public RemoteHandle API. Python API is a follow-up PR.