Implement CUDA event pool to minimize runtime resource allocation overhead (libcurl multi poll-based backend 2/n) by kingcrimsontianyu · Pull Request #919 · rapidsai/kvikio

kingcrimsontianyu · 2026-02-02T20:18:22Z

Summary

This PR adds a CUDA event pool, a thread-safe singleton that caches reusable CUDA events organized per CUDA context. This eliminates the per-acquisition cuEventCreate / cuEventDestroy cost in hot paths that need short-lived events (e.g. bounce-buffer recycling).

This PR facilitates the implementation of device buffer support for the remote I/O multiplexing backend.

Related PR

Depends on #917
Addresses part of #914

copy-pr-bot · 2026-02-02T20:18:26Z

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

wence- · 2026-02-03T10:17:13Z

+  if (event == nullptr) {
+    // Create an event outside the lock to improve performance.
+    // The pool is not updated here; the returned Event object will automatically return the event
+    // to the pool when it goes out of scope
+    CUDA_DRIVER_TRY(cudaAPI::instance().EventCreate(&event, CU_EVENT_DISABLE_TIMING));
+  }


question: Should we have a max size on this event pool? Since we never destroy events, could the pool unboundedly grow in a long-running application? I suppose it will depend on the maximum concurrency of threads issuing reads?

I think it is probably fine to let it grow unbounded. The idea I'm trying to implement in #921 is that each pread() builds a "pread context" that gets a num_threads number of events from the pool, i.e. a single event for each thread. Each 4-MiB chunked read() originating from a specific pread() performs the following in sequence:

I/O (file->bounce buffer ring ([WIP] Optimize easy-handle remote I/O using bounce buffer ring #916))

H2D copy async (bounce buffer ring->device) on a per-thread, per-context stream

Enqueue the per-pread, per-thread, per-context event on the stream

This event is reused for all the chunks on the same thread originating from the same pread() call. So the space complexity overall is O(num_threads * num_concurrent_pread), which I think is not likely to blow up the RAM in a long running application.

But I do think that in the future a limitation on the resource pools (currently we have bounce buffer pool, this event pool, and libcurl easy handle pool) is a good feature.

wence- · 2026-02-03T10:18:16Z

+    _pools[context].push_back(event);
+  } catch (...) {
+    // If returning to pool fails, destroy the event
+    cudaAPI::instance().EventDestroy(event);


question: Not checking the error code because this is called from ~Event()?

I think we should at least log an error in that case.

Thanks. Agreed. I've added the error logging.

The exception message is now properly logged.

wence- · 2026-02-03T10:19:45Z

+std::size_t EventPool::num_free_events(CUcontext context) const
+{
+  std::lock_guard const lock(_mutex);
+  auto it = _pools.find(context);
+  return (it != _pools.end()) ? it->second.size() : 0;
+}
+
+std::size_t EventPool::total_free_events() const


question: Do you need these for correct usage, or are they just going to be introspection facilities?

They are just going to be introspection facilities. I plan to include them in the unit test.

These utility functions are now being used in the unit test.

Co-authored-by: Lawrence Mitchell <wence@gmx.li>

KyleFromNVIDIA

Approved trivial CMake changes

madsbk

Looks good. Do you see any performance gain? Do you have a sense of the overhead of event creation/destruction?

madsbk · 2026-05-26T06:20:42Z

+ * https://docs.nvidia.com/cuda/cuda-programming-guide/02-basics/intro-to-cuda-cpp.html#runtime-initialization
+ * The OS reclaims process memory at exit regardless.
+ */
+class EventPool {


Consider naming them CudaEventPool and CudaEvent.

Sure! Done!

madsbk · 2026-05-26T06:32:03Z

+
+  try {
+    std::lock_guard const lock(_mutex);
+    _pools[cuda_context].push_back(event);


This note may not hold for pooled events:

/** * @note An event that has never been recorded reports `is_done() == true`, * since CUDA's `cuEventQuery` returns `CUDA_SUCCESS` when there is no * captured work. */

If an Event goes out of scope without synchronize() being called, it is returned to the pool while its recorded work may still be pending on the stream. A subsequent pool.get() can then hand that same event to a new caller, for whom is_done() may report false even though they have never called record() themselves.

Either tighten the note to acknowledge that pooled events may retain pending work from a previous user, or make put() reject (or synchronize) non-idle events before returning them to the pool.

That makes sense. I've updated the comment.

kingcrimsontianyu · 2026-05-26T14:36:09Z

I haven't done a direct comparison between event pool vs on-the-fly event creation/destruction, but Mark (@harrism) did this benchmark before: https://github.com/harrism/cuda_event_benchmark

madsbk

Looks good. Please report back when you get some performance numbers

kingcrimsontianyu · 2026-05-26T19:12:48Z

Sure. Thanks for the review!

kingcrimsontianyu · 2026-05-26T19:12:58Z

/merge

kingcrimsontianyu added 5 commits February 1, 2026 16:40

Implement event pool

08dd7f9

Fix stream race condition

3948400

Make ctor dtor private. Remove inner NVTX

43a38d9

Initial impl of event pool

bf20744

Set a get() overload to private

7e0c71f

kingcrimsontianyu added improvement Improves an existing functionality non-breaking Introduces a non-breaking change c++ Affects the C++ API of KvikIO labels Feb 2, 2026

kingcrimsontianyu added the DO NOT MERGE label Feb 2, 2026

Add Doxygen comments

74e6740

kingcrimsontianyu mentioned this pull request Feb 2, 2026

Unified k-way bounce buffer infrastructure for local and remote read #914

Open

7 tasks

kingcrimsontianyu changed the title ~~Implement CUDA event pool to minimize runtime resource allocation overhead~~ [WIP] Implement CUDA event pool to minimize runtime resource allocation overhead Feb 2, 2026

kingcrimsontianyu mentioned this pull request Feb 2, 2026

Use bounce buffer ring to optimize local pread #921

Draft

wence- reviewed Feb 3, 2026

View reviewed changes

kingcrimsontianyu and others added 9 commits February 3, 2026 09:42

Remove get(ctx, tid) and move its content to get()

812c8a9

Merge branch 'fix-stream-bug' into event-pool

c9d7acd

Log exception msg

d878f79

Update cpp/src/detail/event.cpp

df8a4a8

Co-authored-by: Lawrence Mitchell <wence@gmx.li>

Update name for clarity

9d73727

Update

b71c857

Update

dab118e

Silly bug fixes

3103863

Update

4208e5c

kingcrimsontianyu changed the title ~~[WIP] Implement CUDA event pool to minimize runtime resource allocation overhead~~ Implement CUDA event pool to minimize runtime resource allocation overhead Feb 4, 2026

kingcrimsontianyu and others added 5 commits February 4, 2026 19:44

Merge branch 'main' into event-pool

badc794

Merge branch 'main' into event-pool

075dbd8

Add event query

64ee425

Merge branch 'main' into event-pool

87e8fe2

Merge branch 'main' into event-pool

e741a51

Merge branch 'main' into event-pool

c323302

kingcrimsontianyu changed the title ~~Implement CUDA event pool to minimize runtime resource allocation overhead~~ Implement CUDA event pool to minimize runtime resource allocation overhead (libcurl multi poll-based backend 2/n) May 21, 2026

kingcrimsontianyu added 2 commits May 22, 2026 15:17

Improve event pool impl. Add tests

a97196b

Add cuCtxCreate/Destroy to shim to improve event pool testing

4f820a3

kingcrimsontianyu removed the DO NOT MERGE label May 22, 2026

kingcrimsontianyu marked this pull request as ready for review May 22, 2026 20:49

kingcrimsontianyu requested review from a team as code owners May 22, 2026 20:49

KyleFromNVIDIA approved these changes May 22, 2026

View reviewed changes

kingcrimsontianyu requested a review from wence- May 22, 2026 21:01

Improve naming a bit

d1609a0

This was referenced May 24, 2026

Implement per-(thread, CUDA context) bounce buffer cache singleton (libcurl multi poll-based backend 3/n) #965

Merged

Support device buffer for the new libcurl multi poll-based backend (5/n) #966

Open

madsbk reviewed May 26, 2026

View reviewed changes

kingcrimsontianyu added 2 commits May 26, 2026 10:51

Rename Event* to CudaEvent*

ad87ea0

Update doc for event is_done method

5d22b95

kingcrimsontianyu requested a review from madsbk May 26, 2026 15:31

madsbk approved these changes May 26, 2026

View reviewed changes

rapids-bot Bot merged commit 0da1e74 into rapidsai:main May 26, 2026
64 checks passed

kingcrimsontianyu mentioned this pull request Jun 23, 2026

[Tracking] New remote I/O backend: libcurl multi poll-based #987

Open

6 tasks

Uh oh!

Conversation

kingcrimsontianyu commented Feb 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Related PR

Uh oh!

copy-pr-bot Bot commented Feb 2, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

KyleFromNVIDIA left a comment

Choose a reason for hiding this comment

Uh oh!

madsbk left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kingcrimsontianyu commented May 26, 2026

Uh oh!

madsbk left a comment

Choose a reason for hiding this comment

Uh oh!

kingcrimsontianyu commented May 26, 2026

Uh oh!

kingcrimsontianyu commented May 26, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

kingcrimsontianyu commented Feb 2, 2026 •

edited

Loading