Skip to content

[None][feat] KVCacheManagerV2 C++ translation#14047

Open
lowsfer wants to merge 2 commits into
NVIDIA:mainfrom
lowsfer:kvCacheManagerV2-cpp
Open

[None][feat] KVCacheManagerV2 C++ translation#14047
lowsfer wants to merge 2 commits into
NVIDIA:mainfrom
lowsfer:kvCacheManagerV2-cpp

Conversation

@lowsfer

@lowsfer lowsfer commented May 12, 2026 •
edited by coderabbitai Bot
Loading

Copy link
Copy Markdown
Member

Dev Engineer Review

  • Review the C++ KVCacheManagerV2 translation, including storage tiers, eviction, radix-tree reuse, CUDA virtual memory, copy/event handling, statistics, lifecycle management, and Python bindings for correctness, performance, API consistency, error handling, and regressions.
  • Review SHA-256 vendoring, architecture-specific dispatch, CMake integration, pre-commit exclusions, backend selection, and public API compatibility.
  • Validate configuration checks, memory/resource cleanup, cross-backend behavior, and the MPI shutdown-safety change.
  • CI follow-up is required: recent pipeline runs failed despite successful helper jobs; rerun with fail-fast disabled, including PerfSanity.

QA Engineer Review

  • Test code was added or updated in:
    • kvCacheManagerV2HostMemTest.cpp
    • kvCacheManagerV2StatsTest.cpp
    • kvCacheManagerV2TypedIndexTest.cpp
    • KV-cache event manager, manager behavior, salting, and stats API Python tests.
  • Coverage includes host-memory behavior, typed-index contracts, statistics and pending accounting, event parity, reuse introspection, salting, native C++ bindings, and request-ID handling.
  • No tests/integration/test_lists/, test-db/, or qa/ coverage changes are listed. Test functions should be mapped to CI or manual-QA entries.
  • Verdict: needs follow-up.

@hyukn

hyukn commented May 15, 2026

Copy link
Copy Markdown
Collaborator

/bot run --disable-fail-fast --stage-list "PerfSanity"

@lowsfer
lowsfer force-pushed the kvCacheManagerV2-cpp branch 2 times, most recently from 5bd984d to 456cfa8 Compare May 26, 2026 08:44
@lowsfer
lowsfer force-pushed the kvCacheManagerV2-cpp branch 3 times, most recently from 4f4dd8d to 0802c6c Compare June 15, 2026 10:25
@lowsfer
lowsfer force-pushed the kvCacheManagerV2-cpp branch 6 times, most recently from 4c24d8a to 44c797c Compare June 18, 2026 09:30
@lowsfer lowsfer changed the title Draft: Kv cache manager v2 cpp [None][feat] KVCacheManagerV2 C++ translation Jun 19, 2026
@lowsfer
lowsfer force-pushed the kvCacheManagerV2-cpp branch 5 times, most recently from b4592c1 to 5a1fa77 Compare July 10, 2026 10:10
@lowsfer lowsfer added the api-compatible Accepted LLM API contract change that is backwards-compatible label Jul 10, 2026
@juney-nvidia
juney-nvidia requested a review from thorjohnsen July 14, 2026 06:37
@lowsfer
lowsfer force-pushed the kvCacheManagerV2-cpp branch 3 times, most recently from fd7d167 to 99bfc50 Compare July 20, 2026 03:03
@lowsfer
lowsfer marked this pull request as ready for review July 20, 2026 05:16
@lowsfer
lowsfer requested review from a team as code owners July 20, 2026 05:16
coderabbitai[bot]
coderabbitai Bot reviewed Jul 21, 2026
View reviewed changes

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 6

🤖 Prompt for all review comments with AI agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/copyEngine.cpp`:
- Around line 159-163: Make CopyEngine::getStagingManager thread-safe by
ensuring mStagingManager is initialized eagerly or guarded with std::call_once;
preserve lazy initialization only if the one-time synchronization covers both
creation and publication before returning the manager.
- Around line 205-217: Update CopyEngine::transfer to return immediately when
numBytes is zero, before dispatching any direct or two-hop transfer, while
preserving existing behavior for nonzero transfers and task lists.

In `@cpp/tensorrt_llm/runtime/utils/mpiUtils.cpp`:
- Around line 600-606: Reflow the explanatory comment above the static MpiComm
teardown logic to keep every line within the 120-character limit, without
changing its meaning or the surrounding destructor/communicator-free behavior.

In `@docs/source/features/kvcache.md`:
- Line 83: Update the block-key hash requirement in the cache documentation to
distinguish digest size from generic collision resistance: state that SHA-256
has a 256-bit digest but approximately 128-bit collision strength, and require a
512-bit hash only if the intended requirement is 256-bit collision resistance.
Preserve the existing salt-isolation and digest-equality behavior.

In `@tensorrt_llm/_torch/pyexecutor/kv_cache_manager_v2.py`:
- Around line 1791-1801: Limit the KeyError/IndexError handling in the
role-buffer lookup to only the initial retrieval that determines whether
INDEX_KEY is registered. Move stride, upper-bound, and converter access outside
that try block so their IndexError failures propagate instead of returning None,
while preserving the missing-role-to-None behavior for both backends.

In `@tensorrt_llm/runtime/kv_cache_manager_v2/_storage_manager.py`:
- Line 921: Replace the assert validating max_util_for_resume with an explicit
runtime check in the storage manager initialization flow. Raise ValueError when
max_util_for_resume is less than or equal to 0 or greater than 1, preserving
acceptance of values in the range (0, 1].
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: cf471610-335a-44e6-9c6a-00a5f774f9a5

📥 Commits

Reviewing files that changed from the base of the PR and between 99bfc50 and fda1959.

📒 Files selected for processing (77)
  • .github/CODEOWNERS
  • .pre-commit-config.yaml
  • 3rdparty/sha256/LICENSE
  • 3rdparty/sha256/README.md
  • 3rdparty/sha256/attributes.h
  • 3rdparty/sha256/sha256.cpp
  • 3rdparty/sha256/sha256.h
  • 3rdparty/sha256/sha256_arm_shani.cpp
  • 3rdparty/sha256/sha256_endian.h
  • 3rdparty/sha256/sha256_x86_shani.cpp
  • cpp/tensorrt_llm/batch_manager/CMakeLists.txt
  • cpp/tensorrt_llm/batch_manager/kvCacheManagerV2Utils.cu
  • cpp/tensorrt_llm/batch_manager/kvCacheManagerV2Utils.h
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/CMakeLists.txt
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/blockRadixTree.cpp
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/blockRadixTree.h
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/common.cpp
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/common.h
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/config.cpp
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/config.h
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/copyEngine.cpp
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/copyEngine.h
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/cudaVirtMem.cpp
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/cudaVirtMem.h
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/eventManager.cpp
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/eventManager.h
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/eventSink.h
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/evictionController.cpp
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/evictionController.h
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/exceptions.h
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/introspection.cpp
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/introspection.h
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/kvCache.cpp
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/kvCache.h
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/kvCacheManager.cpp
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/kvCacheManager.h
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/lifeCycleRegistry.cpp
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/lifeCycleRegistry.h
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/movingAverage.h
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/page.cpp
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/page.h
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/pendingStats.h
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/stats.h
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/storage/config.cpp
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/storage/config.h
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/storage/core.cpp
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/storage/core.h
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/storageManager.cpp
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/storageManager.h
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/utils/cudaEvent.cpp
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/utils/cudaEvent.h
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/utils/hostMem.cpp
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/utils/hostMem.h
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/utils/math.h
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/utils/sharedPtr.h
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/utils/typedIndex.h
  • cpp/tensorrt_llm/nanobind/CMakeLists.txt
  • cpp/tensorrt_llm/nanobind/batch_manager/kvCacheManagerV2.cpp
  • cpp/tensorrt_llm/nanobind/batch_manager/kvCacheManagerV2.h
  • cpp/tensorrt_llm/nanobind/bindings.cpp
  • cpp/tensorrt_llm/runtime/utils/mpiUtils.cpp
  • cpp/tests/unit_tests/batch_manager/CMakeLists.txt
  • cpp/tests/unit_tests/batch_manager/kvCacheManagerV2HostMemTest.cpp
  • cpp/tests/unit_tests/batch_manager/kvCacheManagerV2StatsTest.cpp
  • cpp/tests/unit_tests/batch_manager/kvCacheManagerV2TypedIndexTest.cpp
  • docs/source/features/kvcache.md
  • scripts/build_wheel.py
  • scripts/nanobind_stubgen.patterns
  • tensorrt_llm/_torch/pyexecutor/kv_cache_manager_v2.py
  • tensorrt_llm/_torch/pyexecutor/py_executor.py
  • tensorrt_llm/runtime/kv_cache_manager_v2/__init__.py
  • tensorrt_llm/runtime/kv_cache_manager_v2/__init__.pyi
  • tensorrt_llm/runtime/kv_cache_manager_v2/_block_radix_tree.py
  • tensorrt_llm/runtime/kv_cache_manager_v2/_cache_key.py
  • tensorrt_llm/runtime/kv_cache_manager_v2/_core/_kv_cache.py
  • tensorrt_llm/runtime/kv_cache_manager_v2/_introspection.py
  • tensorrt_llm/runtime/kv_cache_manager_v2/_storage_manager.py
🚧 Files skipped from review as they are similar to previous changes (66)
  • 3rdparty/sha256/LICENSE
  • 3rdparty/sha256/attributes.h
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/introspection.h
  • 3rdparty/sha256/sha256_endian.h
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/CMakeLists.txt
  • cpp/tensorrt_llm/nanobind/batch_manager/kvCacheManagerV2.h
  • cpp/tensorrt_llm/batch_manager/CMakeLists.txt
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/movingAverage.h
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/eventSink.h
  • cpp/tensorrt_llm/nanobind/CMakeLists.txt
  • cpp/tensorrt_llm/batch_manager/kvCacheManagerV2Utils.h
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/config.cpp
  • 3rdparty/sha256/sha256.h
  • cpp/tests/unit_tests/batch_manager/CMakeLists.txt
  • cpp/tensorrt_llm/nanobind/bindings.cpp
  • .pre-commit-config.yaml
  • tensorrt_llm/_torch/pyexecutor/py_executor.py
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/lifeCycleRegistry.cpp
  • tensorrt_llm/runtime/kv_cache_manager_v2/_introspection.py
  • scripts/nanobind_stubgen.patterns
  • tensorrt_llm/runtime/kv_cache_manager_v2/init.pyi
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/common.cpp
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/evictionController.h
  • tensorrt_llm/runtime/kv_cache_manager_v2/_cache_key.py
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/exceptions.h
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/introspection.cpp
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/utils/hostMem.h
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/storage/config.cpp
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/utils/cudaEvent.cpp
  • 3rdparty/sha256/sha256_arm_shani.cpp
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/kvCache.h
  • tensorrt_llm/runtime/kv_cache_manager_v2/_core/_kv_cache.py
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/pendingStats.h
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/page.cpp
  • tensorrt_llm/runtime/kv_cache_manager_v2/_block_radix_tree.py
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/stats.h
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/kvCacheManager.h
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/eventManager.h
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/storageManager.h
  • scripts/build_wheel.py
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/copyEngine.h
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/utils/math.h
  • 3rdparty/sha256/README.md
  • 3rdparty/sha256/sha256.cpp
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/cudaVirtMem.cpp
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/utils/sharedPtr.h
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/common.h
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/cudaVirtMem.h
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/blockRadixTree.cpp
  • tensorrt_llm/runtime/kv_cache_manager_v2/init.py
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/evictionController.cpp
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/utils/cudaEvent.h
  • 3rdparty/sha256/sha256_x86_shani.cpp
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/storage/config.h
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/blockRadixTree.h
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/config.h
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/utils/hostMem.cpp
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/lifeCycleRegistry.h
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/storage/core.cpp
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/page.h
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/utils/typedIndex.h
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/eventManager.cpp
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/storage/core.h
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/kvCache.cpp
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/kvCacheManager.cpp
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/storageManager.cpp

Comment thread cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/copyEngine.cpp
Comment thread cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/copyEngine.cpp
Comment thread cpp/tensorrt_llm/runtime/utils/mpiUtils.cpp
Comment thread docs/source/features/kvcache.md Outdated
Comment thread tensorrt_llm/_torch/pyexecutor/kv_cache_manager_v2.py
Comment thread tensorrt_llm/runtime/kv_cache_manager_v2/_storage_manager.py Outdated
@lowsfer
lowsfer force-pushed the kvCacheManagerV2-cpp branch from fda1959 to efe3e0a Compare July 22, 2026 01:47
coderabbitai[bot]
coderabbitai Bot reviewed Jul 22, 2026
View reviewed changes

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 4

🧹 Nitpick comments (2)
cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/utils/cudaEvent.h (1)

307-341: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Missing braces on single-statement control-flow bodies; duplicated dedup logic.

Lines 314-315, 319-320, 332-333, and 337-340 use unbraced single-statement bodies for if/for. As per coding guidelines, **/*.{cpp,cc,cxx,h,hpp,cu,cuh}: "use Allman braces, braced control-flow bodies". Separately, streamWaitEvents and synchronizeAll duplicate the same sort/unique dedup sequence; consider factoring a small shared helper.

♻️ Proposed brace fix (illustrative for one instance) 
     for (auto const* ev : events)
     {
-        if (ev && !ev->isClosed())
-            handles.push_back(ev->handle());
+        if (ev && !ev->isClosed())
+        {
+            handles.push_back(ev->handle());
+        }
     }
🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/utils/cudaEvent.h` around
lines 307 - 341, Update streamWaitEvents and synchronizeAll to use Allman-style
braces for every if and for body, including the null/closed-event checks, handle
waits/synchronization, and event closing loop. Also factor their duplicated
handle sort/unique deduplication into a small shared helper and reuse it from
both functions.

Source: Coding guidelines

cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/blockRadixTree.h (1)

310-330: 🎯 Functional Correctness | 🔵 Trivial | ⚡ Quick win

Make mRoots mutable or drop the const_cast in drainPendingRootErases()
drainPendingRootErases() const mutates mRoots through const_cast, which breaks the no-cv-removal guideline. If this cleanup is meant to stay on a const path, mark mRoots mutable; otherwise make the helper non-const.

🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/blockRadixTree.h` around
lines 310 - 330, Resolve the const-correctness issue in drainPendingRootErases:
either mark mRoots mutable so the const cleanup can erase pending roots without
const_cast, or remove const qualification from drainPendingRootErases and its
callers. Preserve the existing deferred root-erasure behavior and eliminate the
const_cast.

Source: Coding guidelines

🤖 Prompt for all review comments with AI agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/utils/cudaEvent.h`:
- Around line 140-152: Update PoolItem get() so mOutstandingCount is incremented
only after mCreateFn() or popFront() successfully returns an item, ensuring
creation exceptions leave the counter unchanged. Preserve the existing deleter
selection and returned PoolItem behavior for successful allocations.
- Around line 39-71: Update the destructor of FuncGuard so exceptions from
mFunc(), including failures propagated by TemporaryCudaStream::enter() through
mStream.recordEvent(), are not forced into std::terminate; declare ~FuncGuard()
noexcept(false) while preserving the existing active-guard behavior.
- Around line 108-201: Add mutex protection to SimplePool’s shared state:
synchronize all accesses and mutations of mItems and mOutstandingCount in get(),
put(), clear(), outstandingCount(), cachedCount(), and the
constructor/destructor cleanup path as needed. Use scoped locking that does not
hold the mutex while invoking mCreateFn or mDestroyFn, and preserve the existing
pool size and ownership behavior.

In `@tensorrt_llm/_torch/pyexecutor/kv_cache_manager_v2.py`:
- Line 1364: Update both f-strings in the KV-cache logging code to format role
with the explicit conversion syntax `{role!s}` instead of calling `str(role)`,
including the occurrence near lifecycle_id and the matching occurrence in the
adjacent message.

---

Nitpick comments:
In `@cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/blockRadixTree.h`:
- Around line 310-330: Resolve the const-correctness issue in
drainPendingRootErases: either mark mRoots mutable so the const cleanup can
erase pending roots without const_cast, or remove const qualification from
drainPendingRootErases and its callers. Preserve the existing deferred
root-erasure behavior and eliminate the const_cast.

In `@cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/utils/cudaEvent.h`:
- Around line 307-341: Update streamWaitEvents and synchronizeAll to use
Allman-style braces for every if and for body, including the null/closed-event
checks, handle waits/synchronization, and event closing loop. Also factor their
duplicated handle sort/unique deduplication into a small shared helper and reuse
it from both functions.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 658f6547-404e-443a-b974-bb319f39217b

📥 Commits

Reviewing files that changed from the base of the PR and between fda1959 and efe3e0a.

📒 Files selected for processing (83)
  • .github/CODEOWNERS
  • .pre-commit-config.yaml
  • 3rdparty/sha256/LICENSE
  • 3rdparty/sha256/README.md
  • 3rdparty/sha256/attributes.h
  • 3rdparty/sha256/sha256.cpp
  • 3rdparty/sha256/sha256.h
  • 3rdparty/sha256/sha256_arm_shani.cpp
  • 3rdparty/sha256/sha256_endian.h
  • 3rdparty/sha256/sha256_x86_shani.cpp
  • cpp/tensorrt_llm/batch_manager/CMakeLists.txt
  • cpp/tensorrt_llm/batch_manager/kvCacheManagerV2Utils.cu
  • cpp/tensorrt_llm/batch_manager/kvCacheManagerV2Utils.h
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/CMakeLists.txt
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/blockRadixTree.cpp
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/blockRadixTree.h
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/common.cpp
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/common.h
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/config.cpp
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/config.h
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/copyEngine.cpp
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/copyEngine.h
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/cudaVirtMem.cpp
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/cudaVirtMem.h
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/eventManager.cpp
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/eventManager.h
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/eventSink.h
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/evictionController.cpp
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/evictionController.h
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/exceptions.h
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/introspection.cpp
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/introspection.h
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/kvCache.cpp
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/kvCache.h
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/kvCacheManager.cpp
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/kvCacheManager.h
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/lifeCycleRegistry.cpp
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/lifeCycleRegistry.h
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/movingAverage.h
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/page.cpp
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/page.h
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/pendingStats.h
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/stats.h
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/storage/config.cpp
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/storage/config.h
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/storage/core.cpp
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/storage/core.h
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/storageManager.cpp
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/storageManager.h
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/utils/cudaEvent.cpp
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/utils/cudaEvent.h
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/utils/hostMem.cpp
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/utils/hostMem.h
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/utils/math.h
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/utils/sharedPtr.h
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/utils/typedIndex.h
  • cpp/tensorrt_llm/nanobind/CMakeLists.txt
  • cpp/tensorrt_llm/nanobind/batch_manager/kvCacheManagerV2.cpp
  • cpp/tensorrt_llm/nanobind/batch_manager/kvCacheManagerV2.h
  • cpp/tensorrt_llm/nanobind/bindings.cpp
  • cpp/tensorrt_llm/runtime/utils/mpiUtils.cpp
  • cpp/tests/unit_tests/batch_manager/CMakeLists.txt
  • cpp/tests/unit_tests/batch_manager/kvCacheManagerV2HostMemTest.cpp
  • cpp/tests/unit_tests/batch_manager/kvCacheManagerV2StatsTest.cpp
  • cpp/tests/unit_tests/batch_manager/kvCacheManagerV2TypedIndexTest.cpp
  • docs/source/features/kvcache.md
  • scripts/build_wheel.py
  • scripts/nanobind_stubgen.patterns
  • tensorrt_llm/_torch/pyexecutor/kv_cache_manager_v2.py
  • tensorrt_llm/_torch/pyexecutor/py_executor.py
  • tensorrt_llm/runtime/kv_cache_manager_v2/__init__.py
  • tensorrt_llm/runtime/kv_cache_manager_v2/__init__.pyi
  • tensorrt_llm/runtime/kv_cache_manager_v2/_block_radix_tree.py
  • tensorrt_llm/runtime/kv_cache_manager_v2/_cache_key.py
  • tensorrt_llm/runtime/kv_cache_manager_v2/_core/_kv_cache.py
  • tensorrt_llm/runtime/kv_cache_manager_v2/_introspection.py
  • tensorrt_llm/runtime/kv_cache_manager_v2/_storage_manager.py
  • tests/unittest/_torch/executor/test_kv_pool_rebalance.py
  • tests/unittest/disaggregated/test_router.py
  • tests/unittest/kv_cache_manager_v2_tests/test_kv_cache_event_manager.py
  • tests/unittest/kv_cache_manager_v2_tests/test_kv_cache_manager_v2.py
  • tests/unittest/kv_cache_manager_v2_tests/test_kv_cache_salting.py
  • tests/unittest/kv_cache_manager_v2_tests/test_kv_cache_stats_api.py
🚧 Files skipped from review as they are similar to previous changes (73)
  • 3rdparty/sha256/LICENSE
  • scripts/nanobind_stubgen.patterns
  • tests/unittest/_torch/executor/test_kv_pool_rebalance.py
  • cpp/tensorrt_llm/batch_manager/CMakeLists.txt
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/introspection.h
  • docs/source/features/kvcache.md
  • cpp/tensorrt_llm/batch_manager/kvCacheManagerV2Utils.cu
  • tensorrt_llm/runtime/kv_cache_manager_v2/init.pyi
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/CMakeLists.txt
  • cpp/tensorrt_llm/nanobind/CMakeLists.txt
  • cpp/tensorrt_llm/nanobind/batch_manager/kvCacheManagerV2.h
  • cpp/tensorrt_llm/nanobind/bindings.cpp
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/movingAverage.h
  • cpp/tests/unit_tests/batch_manager/CMakeLists.txt
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/config.cpp
  • cpp/tensorrt_llm/runtime/utils/mpiUtils.cpp
  • 3rdparty/sha256/sha256.h
  • 3rdparty/sha256/sha256_endian.h
  • 3rdparty/sha256/attributes.h
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/common.cpp
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/eventSink.h
  • cpp/tensorrt_llm/batch_manager/kvCacheManagerV2Utils.h
  • tensorrt_llm/runtime/kv_cache_manager_v2/_storage_manager.py
  • tensorrt_llm/_torch/pyexecutor/py_executor.py
  • tensorrt_llm/runtime/kv_cache_manager_v2/_cache_key.py
  • tests/unittest/disaggregated/test_router.py
  • .github/CODEOWNERS
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/introspection.cpp
  • 3rdparty/sha256/sha256.cpp
  • tests/unittest/kv_cache_manager_v2_tests/test_kv_cache_salting.py
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/evictionController.h
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/stats.h
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/utils/hostMem.h
  • 3rdparty/sha256/sha256_x86_shani.cpp
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/storage/config.cpp
  • 3rdparty/sha256/sha256_arm_shani.cpp
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/utils/cudaEvent.cpp
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/exceptions.h
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/storage/config.h
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/utils/sharedPtr.h
  • 3rdparty/sha256/README.md
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/pendingStats.h
  • tensorrt_llm/runtime/kv_cache_manager_v2/_core/_kv_cache.py
  • tensorrt_llm/runtime/kv_cache_manager_v2/_block_radix_tree.py
  • tensorrt_llm/runtime/kv_cache_manager_v2/init.py
  • tensorrt_llm/runtime/kv_cache_manager_v2/_introspection.py
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/kvCacheManager.h
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/lifeCycleRegistry.cpp
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/copyEngine.h
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/evictionController.cpp
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/storageManager.h
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/cudaVirtMem.h
  • scripts/build_wheel.py
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/utils/math.h
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/page.cpp
  • tests/unittest/kv_cache_manager_v2_tests/test_kv_cache_stats_api.py
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/lifeCycleRegistry.h
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/config.h
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/common.h
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/utils/hostMem.cpp
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/page.h
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/eventManager.h
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/storage/core.h
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/utils/typedIndex.h
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/kvCacheManager.cpp
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/cudaVirtMem.cpp
  • tests/unittest/kv_cache_manager_v2_tests/test_kv_cache_manager_v2.py
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/kvCache.h
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/kvCache.cpp
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/blockRadixTree.cpp
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/eventManager.cpp
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/storageManager.cpp
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/storage/core.cpp

Comment thread cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/utils/cudaEvent.h
Comment thread cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/utils/cudaEvent.h
Comment thread cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/utils/cudaEvent.h
Comment thread tensorrt_llm/_torch/pyexecutor/kv_cache_manager_v2.py Outdated
@lowsfer
lowsfer force-pushed the kvCacheManagerV2-cpp branch from efe3e0a to e1a92e6 Compare July 22, 2026 02:44
@lowsfer
lowsfer requested review from a team as code owners July 22, 2026 02:44
@lowsfer
lowsfer force-pushed the kvCacheManagerV2-cpp branch from e1a92e6 to 2e1f0f3 Compare July 22, 2026 02:52
coderabbitai[bot]
coderabbitai Bot reviewed Jul 22, 2026
View reviewed changes

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 2

🧹 Nitpick comments (3)
cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/kvCacheManager.cpp (1)

398-407: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low value

Broad catch (std::exception) swallows all failures (and e is unused).

This converts every exception — including std::bad_alloc and logic errors that likely indicate a real bug — into a silent resize() == false, making failures hard to diagnose. Consider narrowing to the expected capacity-adjustment exception(s) and/or logging before returning false.

🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/kvCacheManager.cpp` around
lines 398 - 407, Update the exception handling in the resize flow around
_adjustLevel so it catches only the expected capacity-adjustment exception
types, allowing allocation and logic errors to propagate. If a fallback catch
remains necessary, log the caught exception details before returning false, and
remove the unused exception variable otherwise.
cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/utils/cudaEvent.cpp (2)

28-37: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Replace pool-size magic literals with named constants.

Line 37 and Line 56 hard-code pool capacities. Define kEventPoolInitialSize and kStreamPoolInitialSize instead.

Proposed fix 
+constexpr std::size_t kEventPoolInitialSize = 1024;
+constexpr std::size_t kStreamPoolInitialSize = 128;
+
 CudaEventPool::CudaEventPool()
...
-        /*initSize=*/1024)
+        /*initSize=*/kEventPoolInitialSize)
...
-        /*initSize=*/128)
+        /*initSize=*/kStreamPoolInitialSize)

Also applies to: 47-56

🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/utils/cudaEvent.cpp`
around lines 28 - 37, Replace the hard-coded initial pool capacities in
CudaEventPool and the corresponding stream pool constructor with named constants
kEventPoolInitialSize and kStreamPoolInitialSize, defining them in the
appropriate scope and preserving the existing capacity values.

Source: Coding guidelines

162-180: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Use braced control-flow bodies in mergeEvents().

Line 168, Line 171, Line 173, and Line 178 omit braces, contrary to the required Allman/braced-body style.

Proposed fix 
     for (auto& ev : events)
     {
-        if (!ev.isClosed())
+        if (!ev.isClosed())
+        {
             live.push_back(&ev);
+        }
     }
-    if (live.empty())
+    if (live.empty())
+    {
         return CachedCudaEvent::makeNull();
-    if (live.size() == 1)
+    }
+    if (live.size() == 1)
+    {
         return std::move(*live[0]);
+    }
...
-    for (auto* ev : live)
+    for (auto* ev : live)
+    {
         priors.push_back(ev);
+    }
🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/utils/cudaEvent.cpp`
around lines 162 - 180, Update mergeEvents() to wrap every single-statement if
and loop body in braces, including the live-event filtering, empty and
single-event returns, and priors population, preserving the existing control
flow and Allman formatting style.

Source: Coding guidelines

🤖 Prompt for all review comments with AI agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/storage/core.cpp`:
- Around line 551-555: Update the CUDA memory query block to wrap both
cuCtxGetDevice and cuDeviceTotalMem with cuCheck, preserving the existing dev
and totalGpuMem flow while propagating either driver API failure instead of
continuing with an invalid zero value.
- Around line 685-687: Update the physMemSize calculation near kPageSize to
compute the quota ratio before calling std::log2, and handle a zero ratio
explicitly by using the lower exponent bound. Only call std::log2 when the ratio
is nonzero, while preserving the existing min/max clamping for positive ratios.

---

Nitpick comments:
In `@cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/kvCacheManager.cpp`:
- Around line 398-407: Update the exception handling in the resize flow around
_adjustLevel so it catches only the expected capacity-adjustment exception
types, allowing allocation and logic errors to propagate. If a fallback catch
remains necessary, log the caught exception details before returning false, and
remove the unused exception variable otherwise.

In `@cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/utils/cudaEvent.cpp`:
- Around line 28-37: Replace the hard-coded initial pool capacities in
CudaEventPool and the corresponding stream pool constructor with named constants
kEventPoolInitialSize and kStreamPoolInitialSize, defining them in the
appropriate scope and preserving the existing capacity values.
- Around line 162-180: Update mergeEvents() to wrap every single-statement if
and loop body in braces, including the live-event filtering, empty and
single-event returns, and priors population, preserving the existing control
flow and Allman formatting style.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: f4a32390-4114-4c91-bb84-11e4cb2b9b2f

📥 Commits

Reviewing files that changed from the base of the PR and between efe3e0a and e1a92e6.

📒 Files selected for processing (84)
  • .github/CODEOWNERS
  • .pre-commit-config.yaml
  • 3rdparty/sha256/LICENSE
  • 3rdparty/sha256/README.md
  • 3rdparty/sha256/attributes.h
  • 3rdparty/sha256/sha256.cpp
  • 3rdparty/sha256/sha256.h
  • 3rdparty/sha256/sha256_arm_shani.cpp
  • 3rdparty/sha256/sha256_endian.h
  • 3rdparty/sha256/sha256_x86_shani.cpp
  • cpp/tensorrt_llm/batch_manager/CMakeLists.txt
  • cpp/tensorrt_llm/batch_manager/kvCacheManagerV2Utils.cu
  • cpp/tensorrt_llm/batch_manager/kvCacheManagerV2Utils.h
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/CMakeLists.txt
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/blockRadixTree.cpp
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/blockRadixTree.h
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/common.cpp
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/common.h
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/config.cpp
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/config.h
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/copyEngine.cpp
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/copyEngine.h
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/cudaVirtMem.cpp
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/cudaVirtMem.h
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/eventManager.cpp
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/eventManager.h
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/eventSink.h
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/evictionController.cpp
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/evictionController.h
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/exceptions.h
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/introspection.cpp
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/introspection.h
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/kvCache.cpp
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/kvCache.h
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/kvCacheManager.cpp
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/kvCacheManager.h
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/lifeCycleRegistry.cpp
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/lifeCycleRegistry.h
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/movingAverage.h
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/page.cpp
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/page.h
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/pendingStats.h
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/stats.h
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/storage/config.cpp
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/storage/config.h
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/storage/core.cpp
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/storage/core.h
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/storageManager.cpp
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/storageManager.h
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/utils/cudaEvent.cpp
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/utils/cudaEvent.h
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/utils/hostMem.cpp
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/utils/hostMem.h
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/utils/math.h
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/utils/sharedPtr.h
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/utils/typedIndex.h
  • cpp/tensorrt_llm/nanobind/CMakeLists.txt
  • cpp/tensorrt_llm/nanobind/batch_manager/kvCacheManagerV2.cpp
  • cpp/tensorrt_llm/nanobind/batch_manager/kvCacheManagerV2.h
  • cpp/tensorrt_llm/nanobind/bindings.cpp
  • cpp/tensorrt_llm/runtime/utils/mpiUtils.cpp
  • cpp/tests/unit_tests/batch_manager/CMakeLists.txt
  • cpp/tests/unit_tests/batch_manager/kvCacheManagerV2HostMemTest.cpp
  • cpp/tests/unit_tests/batch_manager/kvCacheManagerV2StatsTest.cpp
  • cpp/tests/unit_tests/batch_manager/kvCacheManagerV2TypedIndexTest.cpp
  • docs/source/features/kvcache.md
  • scripts/build_wheel.py
  • scripts/nanobind_stubgen.patterns
  • tensorrt_llm/_torch/pyexecutor/kv_cache_manager_v2.py
  • tensorrt_llm/_torch/pyexecutor/py_executor.py
  • tensorrt_llm/runtime/kv_cache_manager_v2/__init__.py
  • tensorrt_llm/runtime/kv_cache_manager_v2/__init__.pyi
  • tensorrt_llm/runtime/kv_cache_manager_v2/_block_radix_tree.py
  • tensorrt_llm/runtime/kv_cache_manager_v2/_cache_key.py
  • tensorrt_llm/runtime/kv_cache_manager_v2/_core/_kv_cache.py
  • tensorrt_llm/runtime/kv_cache_manager_v2/_introspection.py
  • tensorrt_llm/runtime/kv_cache_manager_v2/_storage_manager.py
  • tests/integration/test_lists/waives.txt
  • tests/unittest/_torch/executor/test_kv_pool_rebalance.py
  • tests/unittest/disaggregated/test_router.py
  • tests/unittest/kv_cache_manager_v2_tests/test_kv_cache_event_manager.py
  • tests/unittest/kv_cache_manager_v2_tests/test_kv_cache_manager_v2.py
  • tests/unittest/kv_cache_manager_v2_tests/test_kv_cache_salting.py
  • tests/unittest/kv_cache_manager_v2_tests/test_kv_cache_stats_api.py
🚧 Files skipped from review as they are similar to previous changes (69)
  • 3rdparty/sha256/LICENSE
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/CMakeLists.txt
  • .pre-commit-config.yaml
  • scripts/nanobind_stubgen.patterns
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/common.cpp
  • tests/unittest/_torch/executor/test_kv_pool_rebalance.py
  • cpp/tensorrt_llm/nanobind/batch_manager/kvCacheManagerV2.h
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/introspection.h
  • 3rdparty/sha256/sha256.h
  • 3rdparty/sha256/sha256_endian.h
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/movingAverage.h
  • tensorrt_llm/_torch/pyexecutor/py_executor.py
  • 3rdparty/sha256/attributes.h
  • docs/source/features/kvcache.md
  • cpp/tensorrt_llm/nanobind/bindings.cpp
  • cpp/tests/unit_tests/batch_manager/CMakeLists.txt
  • cpp/tensorrt_llm/batch_manager/kvCacheManagerV2Utils.h
  • .github/CODEOWNERS
  • tensorrt_llm/runtime/kv_cache_manager_v2/init.pyi
  • tests/unittest/disaggregated/test_router.py
  • cpp/tensorrt_llm/batch_manager/CMakeLists.txt
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/config.cpp
  • cpp/tensorrt_llm/nanobind/CMakeLists.txt
  • cpp/tensorrt_llm/runtime/utils/mpiUtils.cpp
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/storage/config.cpp
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/introspection.cpp
  • 3rdparty/sha256/sha256.cpp
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/evictionController.h
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/utils/sharedPtr.h
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/copyEngine.h
  • tensorrt_llm/runtime/kv_cache_manager_v2/_core/_kv_cache.py
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/lifeCycleRegistry.cpp
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/stats.h
  • tensorrt_llm/runtime/kv_cache_manager_v2/_cache_key.py
  • tests/unittest/kv_cache_manager_v2_tests/test_kv_cache_stats_api.py
  • 3rdparty/sha256/sha256_x86_shani.cpp
  • tests/unittest/kv_cache_manager_v2_tests/test_kv_cache_salting.py
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/blockRadixTree.h
  • 3rdparty/sha256/README.md
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/storage/config.h
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/common.h
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/pendingStats.h
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/cudaVirtMem.cpp
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/utils/hostMem.h
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/exceptions.h
  • tensorrt_llm/runtime/kv_cache_manager_v2/_block_radix_tree.py
  • tensorrt_llm/runtime/kv_cache_manager_v2/_introspection.py
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/utils/math.h
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/page.cpp
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/kvCacheManager.h
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/cudaVirtMem.h
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/kvCache.h
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/config.h
  • 3rdparty/sha256/sha256_arm_shani.cpp
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/utils/cudaEvent.h
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/eventManager.h
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/utils/hostMem.cpp
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/storage/core.h
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/blockRadixTree.cpp
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/storageManager.h
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/lifeCycleRegistry.h
  • tests/unittest/kv_cache_manager_v2_tests/test_kv_cache_manager_v2.py
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/evictionController.cpp
  • tensorrt_llm/runtime/kv_cache_manager_v2/init.py
  • scripts/build_wheel.py
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/eventManager.cpp
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/kvCache.cpp
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/storageManager.cpp
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/eventSink.h

Comment thread cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/storage/core.cpp
Comment thread cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/storage/core.cpp Outdated
coderabbitai[bot]
coderabbitai Bot reviewed Jul 22, 2026
View reviewed changes

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🧹 Nitpick comments (1)
cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/copyEngine.h (1)

33-40: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Use Doxygen documentation for the new public interfaces.

  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/copyEngine.h#L33-L40: Document CopyTask fields and address-direction semantics with Doxygen.
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/copyEngine.h#L69-L104: Document staging-buffer ownership, locking, constructor arguments, and stream lifetime.
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/copyEngine.h#L110-L158: Document allocation bounds and concurrency guarantees.
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/copyEngine.h#L166-L197: Document transfer preconditions, async behavior, singleton lifetime, and shutdown requirements.

As per coding guidelines, “use Doxygen comments for new interfaces.”

🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/copyEngine.h` around lines
33 - 40, Add Doxygen documentation to the public interfaces in
cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/copyEngine.h: document
CopyTask fields and source/destination address semantics at lines 33-40;
document staging-buffer ownership, locking, constructor arguments, and stream
lifetime at lines 69-104; document allocation bounds and concurrency guarantees
at lines 110-158; and document transfer preconditions, asynchronous behavior,
singleton lifetime, and shutdown requirements at lines 166-197.

Source: Coding guidelines

🤖 Prompt for all review comments with AI agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Nitpick comments:
In `@cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/copyEngine.h`:
- Around line 33-40: Add Doxygen documentation to the public interfaces in
cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/copyEngine.h: document
CopyTask fields and source/destination address semantics at lines 33-40;
document staging-buffer ownership, locking, constructor arguments, and stream
lifetime at lines 69-104; document allocation bounds and concurrency guarantees
at lines 110-158; and document transfer preconditions, asynchronous behavior,
singleton lifetime, and shutdown requirements at lines 166-197.
ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 3498fc47-b3d6-4d5a-b50e-f49d7393f756

📥 Commits

Reviewing files that changed from the base of the PR and between e1a92e6 and 2e1f0f3.

📒 Files selected for processing (83)
  • .github/CODEOWNERS
  • .pre-commit-config.yaml
  • 3rdparty/sha256/LICENSE
  • 3rdparty/sha256/README.md
  • 3rdparty/sha256/attributes.h
  • 3rdparty/sha256/sha256.cpp
  • 3rdparty/sha256/sha256.h
  • 3rdparty/sha256/sha256_arm_shani.cpp
  • 3rdparty/sha256/sha256_endian.h
  • 3rdparty/sha256/sha256_x86_shani.cpp
  • cpp/tensorrt_llm/batch_manager/CMakeLists.txt
  • cpp/tensorrt_llm/batch_manager/kvCacheManagerV2Utils.cu
  • cpp/tensorrt_llm/batch_manager/kvCacheManagerV2Utils.h
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/CMakeLists.txt
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/blockRadixTree.cpp
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/blockRadixTree.h
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/common.cpp
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/common.h
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/config.cpp
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/config.h
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/copyEngine.cpp
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/copyEngine.h
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/cudaVirtMem.cpp
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/cudaVirtMem.h
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/eventManager.cpp
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/eventManager.h
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/eventSink.h
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/evictionController.cpp
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/evictionController.h
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/exceptions.h
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/introspection.cpp
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/introspection.h
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/kvCache.cpp
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/kvCache.h
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/kvCacheManager.cpp
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/kvCacheManager.h
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/lifeCycleRegistry.cpp
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/lifeCycleRegistry.h
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/movingAverage.h
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/page.cpp
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/page.h
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/pendingStats.h
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/stats.h
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/storage/config.cpp
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/storage/config.h
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/storage/core.cpp
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/storage/core.h
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/storageManager.cpp
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/storageManager.h
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/utils/cudaEvent.cpp
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/utils/cudaEvent.h
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/utils/hostMem.cpp
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/utils/hostMem.h
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/utils/math.h
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/utils/sharedPtr.h
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/utils/typedIndex.h
  • cpp/tensorrt_llm/nanobind/CMakeLists.txt
  • cpp/tensorrt_llm/nanobind/batch_manager/kvCacheManagerV2.cpp
  • cpp/tensorrt_llm/nanobind/batch_manager/kvCacheManagerV2.h
  • cpp/tensorrt_llm/nanobind/bindings.cpp
  • cpp/tensorrt_llm/runtime/utils/mpiUtils.cpp
  • cpp/tests/unit_tests/batch_manager/CMakeLists.txt
  • cpp/tests/unit_tests/batch_manager/kvCacheManagerV2HostMemTest.cpp
  • cpp/tests/unit_tests/batch_manager/kvCacheManagerV2StatsTest.cpp
  • cpp/tests/unit_tests/batch_manager/kvCacheManagerV2TypedIndexTest.cpp
  • docs/source/features/kvcache.md
  • scripts/build_wheel.py
  • scripts/nanobind_stubgen.patterns
  • tensorrt_llm/_torch/pyexecutor/kv_cache_manager_v2.py
  • tensorrt_llm/_torch/pyexecutor/py_executor.py
  • tensorrt_llm/runtime/kv_cache_manager_v2/__init__.py
  • tensorrt_llm/runtime/kv_cache_manager_v2/__init__.pyi
  • tensorrt_llm/runtime/kv_cache_manager_v2/_block_radix_tree.py
  • tensorrt_llm/runtime/kv_cache_manager_v2/_cache_key.py
  • tensorrt_llm/runtime/kv_cache_manager_v2/_core/_kv_cache.py
  • tensorrt_llm/runtime/kv_cache_manager_v2/_introspection.py
  • tensorrt_llm/runtime/kv_cache_manager_v2/_storage_manager.py
  • tests/unittest/_torch/executor/test_kv_pool_rebalance.py
  • tests/unittest/disaggregated/test_router.py
  • tests/unittest/kv_cache_manager_v2_tests/test_kv_cache_event_manager.py
  • tests/unittest/kv_cache_manager_v2_tests/test_kv_cache_manager_v2.py
  • tests/unittest/kv_cache_manager_v2_tests/test_kv_cache_salting.py
  • tests/unittest/kv_cache_manager_v2_tests/test_kv_cache_stats_api.py
🚧 Files skipped from review as they are similar to previous changes (70)
  • scripts/nanobind_stubgen.patterns
  • 3rdparty/sha256/LICENSE
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/introspection.h
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/common.cpp
  • 3rdparty/sha256/sha256.h
  • cpp/tests/unit_tests/batch_manager/CMakeLists.txt
  • tensorrt_llm/_torch/pyexecutor/py_executor.py
  • 3rdparty/sha256/attributes.h
  • 3rdparty/sha256/sha256_endian.h
  • cpp/tensorrt_llm/nanobind/CMakeLists.txt
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/eventSink.h
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/config.cpp
  • cpp/tensorrt_llm/batch_manager/kvCacheManagerV2Utils.h
  • .github/CODEOWNERS
  • cpp/tensorrt_llm/batch_manager/CMakeLists.txt
  • cpp/tensorrt_llm/batch_manager/kvCacheManagerV2Utils.cu
  • tests/unittest/disaggregated/test_router.py
  • 3rdparty/sha256/README.md
  • cpp/tensorrt_llm/nanobind/batch_manager/kvCacheManagerV2.h
  • cpp/tensorrt_llm/runtime/utils/mpiUtils.cpp
  • tests/unittest/_torch/executor/test_kv_pool_rebalance.py
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/CMakeLists.txt
  • tensorrt_llm/runtime/kv_cache_manager_v2/_cache_key.py
  • cpp/tensorrt_llm/nanobind/bindings.cpp
  • tensorrt_llm/runtime/kv_cache_manager_v2/_storage_manager.py
  • tensorrt_llm/runtime/kv_cache_manager_v2/init.pyi
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/stats.h
  • 3rdparty/sha256/sha256_x86_shani.cpp
  • 3rdparty/sha256/sha256.cpp
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/utils/hostMem.h
  • docs/source/features/kvcache.md
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/movingAverage.h
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/lifeCycleRegistry.cpp
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/cudaVirtMem.h
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/evictionController.h
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/exceptions.h
  • scripts/build_wheel.py
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/pendingStats.h
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/introspection.cpp
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/eventManager.cpp
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/page.cpp
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/common.h
  • tests/unittest/kv_cache_manager_v2_tests/test_kv_cache_salting.py
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/page.h
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/storage/config.h
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/utils/sharedPtr.h
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/utils/hostMem.cpp
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/utils/cudaEvent.cpp
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/config.h
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/kvCacheManager.h
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/storage/config.cpp
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/utils/math.h
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/cudaVirtMem.cpp
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/evictionController.cpp
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/eventManager.h
  • tests/unittest/kv_cache_manager_v2_tests/test_kv_cache_manager_v2.py
  • tensorrt_llm/runtime/kv_cache_manager_v2/init.py
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/lifeCycleRegistry.h
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/storageManager.h
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/utils/cudaEvent.h
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/blockRadixTree.cpp
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/utils/typedIndex.h
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/blockRadixTree.h
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/storage/core.h
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/kvCache.h
  • tensorrt_llm/runtime/kv_cache_manager_v2/_block_radix_tree.py
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/kvCache.cpp
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/storage/core.cpp
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/kvCacheManager.cpp
  • cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/storageManager.cpp

@lowsfer
lowsfer force-pushed the kvCacheManagerV2-cpp branch from 2e1f0f3 to 1d2226d Compare July 22, 2026 04:48
chzblych
chzblych approved these changes Jul 22, 2026
View reviewed changes
Comment thread 3rdparty/sha256/attributes.h
@@ -0,0 +1,27 @@
// Copyright (c) 2009-2010 Satoshi Nakamoto

@chzblych chzblych Jul 22, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@yuanjingx87 @tburt-nv Is this a proper way to import the MIT licensed codes?

@tburt-nv tburt-nv Jul 22, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Almost, I'll put together a patch to tweak it.

@lowsfer
lowsfer force-pushed the kvCacheManagerV2-cpp branch 8 times, most recently from e5a1453 to a4efc5c Compare July 23, 2026 16:13
lowsfer added 2 commits July 24, 2026 06:26
@lowsfer
[None][feat] KVCacheManagerV2: Python→C++ migration with nanobind bin…
fc7bba2 
…dings

Migrates tensorrt_llm/runtime/kv_cache_manager_v2 from pure Python to a C++
implementation under cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/ with
nanobind bindings compiled into bindings.so, preserving the same public API.
The dispatcher __init__.py selects the backend via
TLLM_KV_CACHE_MANAGER_V2_BACKEND (default "cpp"); both backends pass the shared
test suite. CODEOWNERS assigns the new C++ tree to trt-llm-kv-cache-manager-devs.

Includes ports of subsequent main features: commit-min-snapshot + SWA-slot
reservation, SHA-256 block-key hashing, CUDA-graph request IDs, event manager
and stats API to C++, uint64 ReuseScope salt/lora_id, resume-utilization KV
constraints, per-conversation KV cache block reuse (PlannedDropHandle), and an
MPI teardown fix for the unittest/bindings CI shard.

Migration planning docs (TODO.md, MIGRATION_PLAN_CPP.md,
CPP_MIGRATION_PLAN_MAIN_15633.md) are kept on a separate branch.

Signed-off-by: Yao Yao <lowsfer@users.noreply.github.com>
@lowsfer
[None][fix] executor tests: seed __init__ attrs bypassed by __new__
bbf42bd 
Base-branch (main) regressions surfaced by rebasing onto latest main; none
are in the KVCacheManagerV2 change set. Two distinct patterns, both fixed by
matching the existing sibling conventions.

1) tensorrt_llm import break (nvbugs/6442074): SpecWorkerBase.__init_subclass__
   forbids subclasses from overriding forward() -- they must implement
   _forward_impl so the base forward() wrapper can guarantee spec-dec
   attn-metadata cleanup. MTPEagleDynamicTreeWorker still overrode forward(),
   so 'import tensorrt_llm' raised TypeError at class-definition time,
   breaking all test collection. Rename its forward() to _forward_impl(),
   matching every sibling worker; callers use the base forward() wrapper so no
   caller changes.

2) executor tests build objects via __new__ (bypassing __init__) then exercise
   teardown/lifecycle paths. main NVIDIA#16523 (multi-process HTTP frontends) added
   attributes to BaseWorker/GenerationExecutorProxy __init__ that teardown
   reads unguarded, so the __new__-built test objects raised AttributeError:
   - test_proxy_fast_death.py _bare_proxy: seed _multi_frontend_ipc_dir/_hmac.
   - test_event_loop_error_broadcast.py _WorkerStub: seed frontend_result_queues
     (responses_handler read it unguarded -> 4/6 tests failed).
   - test_proxy_postproc_terminate.py _make_proxy: seed workers_started +
     _multi_frontend_ipc_dir/_hmac (GC __del__ -> shutdown teardown).
   - test_base_worker.py __new__ shell: seed doing_shutdown (GC __del__).
   Each seeds only what that object's teardown reads, matching the _bare_proxy
   convention.

Signed-off-by: Yao Yao <lowsfer@users.noreply.github.com>
@lowsfer

lowsfer commented Jul 24, 2026

Copy link
Copy Markdown
Member Author

/bot run --disable-fail-fast

@tensorrt-cicd

tensorrt-cicd commented Jul 24, 2026

Copy link
Copy Markdown
Collaborator

PR_Github #61522 [ run ] triggered by Bot. Commit: bbf42bd Link to invocation

@tensorrt-cicd

tensorrt-cicd commented Jul 24, 2026

Copy link
Copy Markdown
Collaborator

PR_Github #61522 [ run ] completed with state FAILURE. Commit: bbf42bd
/LLM/main/L0_MergeRequest_PR pipeline #49738 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

  • Please check the failed tests and fix your PR
  • If you cannot view the failures, ask the CI triggerer to share details
  • Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

@lowsfer

lowsfer commented Jul 24, 2026

Copy link
Copy Markdown
Member Author

/bot run --disable-fail-fast

@tensorrt-cicd

tensorrt-cicd commented Jul 24, 2026

Copy link
Copy Markdown
Collaborator

PR_Github #61570 [ run ] triggered by Bot. Commit: bbf42bd Link to invocation

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@coderabbitai coderabbitai[bot] coderabbitai[bot] left review comments
@tburt-nv tburt-nv tburt-nv left review comments
@chzblych chzblych chzblych approved these changes
@yizhang-nv yizhang-nv yizhang-nv approved these changes
@thorjohnsen thorjohnsen Awaiting requested review from thorjohnsen
@tongyuantongyu tongyuantongyu Awaiting requested review from tongyuantongyu tongyuantongyu is a code owner automatically assigned from NVIDIA/trt-llm-runtime-devs
@asfiyab-nvidia asfiyab-nvidia Awaiting requested review from asfiyab-nvidia asfiyab-nvidia is a code owner automatically assigned from NVIDIA/trt-llm-runtime-devs
@chuangz0 chuangz0 Awaiting requested review from chuangz0 chuangz0 is a code owner automatically assigned from NVIDIA/trt-llm-disagg-devs
@bo-nv bo-nv Awaiting requested review from bo-nv bo-nv is a code owner automatically assigned from NVIDIA/trt-llm-disagg-devs
@eopXD eopXD Awaiting requested review from eopXD eopXD is a code owner automatically assigned from NVIDIA/trt-llm-kv-cache-manager-devs
@niukuo niukuo Awaiting requested review from niukuo niukuo is a code owner automatically assigned from NVIDIA/trt-llm-infra-devs
@dpitman-nvda dpitman-nvda Awaiting requested review from dpitman-nvda dpitman-nvda is a code owner automatically assigned from NVIDIA/trt-llm-infra-devs
@chang-l chang-l Awaiting requested review from chang-l chang-l is a code owner automatically assigned from NVIDIA/trt-llm-doc-owners
@laikhtewari laikhtewari Awaiting requested review from laikhtewari laikhtewari is a code owner automatically assigned from NVIDIA/trt-llm-doc-owners
@BowenFu BowenFu Awaiting requested review from BowenFu BowenFu was automatically assigned from NVIDIA/trt-llm-devs
@brnguyen2 brnguyen2 Awaiting requested review from brnguyen2 brnguyen2 was automatically assigned from NVIDIA/trt-llm-devs
@StanleySun639 StanleySun639 Awaiting requested review from StanleySun639 StanleySun639 was automatically assigned from NVIDIA/trt-llm-qa
@LarryXFly LarryXFly Awaiting requested review from LarryXFly LarryXFly was automatically assigned from NVIDIA/trt-llm-qa

At least 1 approving review is required to merge this pull request.

Assignees

@lowsfer lowsfer

Labels

api-compatible Accepted LLM API contract change that is backwards-compatible

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

6 participants

@lowsfer @hyukn @tensorrt-cicd @chzblych @yizhang-nv @tburt-nv