[None][feat] KVCacheManagerV2 C++ translation by lowsfer · Pull Request #14047 · NVIDIA/TensorRT-LLM

lowsfer · 2026-05-12T11:44:10Z

Dev Engineer Review

Review the C++ KVCacheManagerV2 translation, including storage tiers, eviction, radix-tree reuse, CUDA virtual memory, copy/event handling, statistics, lifecycle management, and Python bindings for correctness, performance, API consistency, error handling, and regressions.
Review SHA-256 vendoring, architecture-specific dispatch, CMake integration, pre-commit exclusions, backend selection, and public API compatibility.
Validate configuration checks, memory/resource cleanup, cross-backend behavior, and the MPI shutdown-safety change.
CI follow-up is required: recent pipeline runs failed despite successful helper jobs; rerun with fail-fast disabled, including PerfSanity.

QA Engineer Review

Test code was added or updated in:
- kvCacheManagerV2HostMemTest.cpp
- kvCacheManagerV2StatsTest.cpp
- kvCacheManagerV2TypedIndexTest.cpp
- KV-cache event manager, manager behavior, salting, and stats API Python tests.
Coverage includes host-memory behavior, typed-index contracts, statistics and pending accounting, event parity, reuse introspection, salting, native C++ bindings, and request-ID handling.
No tests/integration/test_lists/, test-db/, or qa/ coverage changes are listed. Test functions should be mapped to CI or manual-QA entries.
Verdict: needs follow-up.

lowsfer · 2026-05-12T11:44:34Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-05-12T11:49:49Z

PR_Github #47960 [ run ] triggered by Bot. Commit: 5a3ffd7 Link to invocation

tensorrt-cicd · 2026-05-12T12:08:40Z

PR_Github #47960 [ run ] completed with state FAILURE. Commit: 5a3ffd7
/LLM/main/L0_MergeRequest_PR pipeline #37801 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

lowsfer · 2026-05-14T06:48:16Z

/bot run --disable-fail-fast

hyukn · 2026-05-15T09:51:10Z

/bot run --disable-fail-fast --stage-list "PerfSanity"

lowsfer · 2026-05-26T08:50:42Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-05-26T08:56:33Z

PR_Github #50329 [ run ] triggered by Bot. Commit: 456cfa8 Link to invocation

tensorrt-cicd · 2026-05-26T09:23:30Z

PR_Github #50329 [ run ] completed with state FAILURE. Commit: 456cfa8
/LLM/main/L0_MergeRequest_PR pipeline #39856 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

lowsfer · 2026-06-09T07:27:02Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-06-09T07:33:32Z

PR_Github #52999 [ run ] triggered by Bot. Commit: ae8517a Link to invocation

tensorrt-cicd · 2026-06-09T07:55:19Z

PR_Github #52999 [ run ] completed with state FAILURE. Commit: ae8517a
/LLM/main/L0_MergeRequest_PR pipeline #42226 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

lowsfer · 2026-06-10T16:19:12Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-06-10T16:25:21Z

PR_Github #53338 [ run ] triggered by Bot. Commit: 2cdb452 Link to invocation

tensorrt-cicd · 2026-06-10T16:56:53Z

PR_Github #53338 [ run ] completed with state FAILURE. Commit: 2cdb452
/LLM/main/L0_MergeRequest_PR pipeline #42518 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

lowsfer · 2026-06-11T03:07:41Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-06-11T03:13:24Z

PR_Github #53454 [ run ] triggered by Bot. Commit: cd30bda Link to invocation

tensorrt-cicd · 2026-06-11T03:42:21Z

PR_Github #53454 [ run ] completed with state FAILURE. Commit: cd30bda
/LLM/main/L0_MergeRequest_PR pipeline #42620 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

lowsfer · 2026-06-11T14:34:33Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-06-11T14:40:12Z

PR_Github #53588 [ run ] triggered by Bot. Commit: 86a1752 Link to invocation

tensorrt-cicd · 2026-06-11T15:11:24Z

PR_Github #53588 [ run ] completed with state FAILURE. Commit: 86a1752
/LLM/main/L0_MergeRequest_PR pipeline #42734 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

lowsfer · 2026-06-12T01:01:43Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-06-12T01:08:15Z

PR_Github #53735 [ run ] triggered by Bot. Commit: 3724448 Link to invocation

tensorrt-cicd · 2026-06-12T08:03:33Z

PR_Github #53735 [ run ] completed with state SUCCESS. Commit: 3724448
/LLM/main/L0_MergeRequest_PR pipeline #42861 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

lowsfer · 2026-06-15T07:02:29Z

/bot run --disable-fail-fast

lowsfer · 2026-07-22T16:35:21Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-07-22T16:43:20Z

PR_Github #61028 [ run ] triggered by Bot. Commit: 2ca5340 Link to invocation

tensorrt-cicd · 2026-07-22T16:44:58Z

PR_Github #61012 [ run ] completed with state ABORTED. Commit: 6f646e5

Link to invocation

tensorrt-cicd · 2026-07-22T20:07:22Z

PR_Github #61028 [ run ] completed with state FAILURE. Commit: 2ca5340
/LLM/main/L0_MergeRequest_PR pipeline #49285 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

lowsfer · 2026-07-23T02:55:31Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-07-23T03:02:28Z

PR_Github #61167 [ run ] triggered by Bot. Commit: a747ad3 Link to invocation

lowsfer · 2026-07-23T08:37:32Z

/bot run --disable-fail-fast

lowsfer · 2026-07-23T08:42:08Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-07-23T08:48:09Z

PR_Github #61253 [ run ] triggered by Bot. Commit: af58079 Link to invocation

tensorrt-cicd · 2026-07-23T08:48:13Z

PR_Github #61254 [ ] completed with state ABORTED. Commit: ``

Link to invocation

tensorrt-cicd · 2026-07-23T08:48:24Z

PR_Github #61167 [ run ] completed with state ABORTED. Commit: a747ad3
/LLM/main/L0_MergeRequest_PR pipeline #49417 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

lowsfer · 2026-07-23T11:00:04Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-07-23T11:05:37Z

PR_Github #61280 [ run ] triggered by Bot. Commit: e5a1453 Link to invocation

tensorrt-cicd · 2026-07-23T11:07:38Z

PR_Github #61253 [ run ] completed with state ABORTED. Commit: af58079

Link to invocation

tensorrt-cicd · 2026-07-23T14:43:57Z

PR_Github #61280 [ run ] completed with state FAILURE. Commit: e5a1453
/LLM/main/L0_MergeRequest_PR pipeline #49514 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

lowsfer · 2026-07-23T16:14:36Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-07-23T16:21:52Z

PR_Github #61330 [ run ] triggered by Bot. Commit: a4efc5c Link to invocation

tensorrt-cicd · 2026-07-23T19:47:17Z

PR_Github #61330 [ run ] completed with state FAILURE. Commit: a4efc5c
/LLM/main/L0_MergeRequest_PR pipeline #49559 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

lowsfer · 2026-07-24T03:29:53Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-07-24T03:35:42Z

PR_Github #61489 [ run ] triggered by Bot. Commit: 5fc96d4 Link to invocation

…dings Migrates tensorrt_llm/runtime/kv_cache_manager_v2 from pure Python to a C++ implementation under cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/ with nanobind bindings compiled into bindings.so, preserving the same public API. The dispatcher __init__.py selects the backend via TLLM_KV_CACHE_MANAGER_V2_BACKEND (default "cpp"); both backends pass the shared test suite. CODEOWNERS assigns the new C++ tree to trt-llm-kv-cache-manager-devs. Includes ports of subsequent main features: commit-min-snapshot + SWA-slot reservation, SHA-256 block-key hashing, CUDA-graph request IDs, event manager and stats API to C++, uint64 ReuseScope salt/lora_id, resume-utilization KV constraints, per-conversation KV cache block reuse (PlannedDropHandle), and an MPI teardown fix for the unittest/bindings CI shard. Migration planning docs (TODO.md, MIGRATION_PLAN_CPP.md, CPP_MIGRATION_PLAN_MAIN_15633.md) are kept on a separate branch. Signed-off-by: Yao Yao <lowsfer@users.noreply.github.com>

Base-branch (main) regressions surfaced by rebasing onto latest main; none are in the KVCacheManagerV2 change set. Two distinct patterns, both fixed by matching the existing sibling conventions. 1) tensorrt_llm import break (nvbugs/6442074): SpecWorkerBase.__init_subclass__ forbids subclasses from overriding forward() -- they must implement _forward_impl so the base forward() wrapper can guarantee spec-dec attn-metadata cleanup. MTPEagleDynamicTreeWorker still overrode forward(), so 'import tensorrt_llm' raised TypeError at class-definition time, breaking all test collection. Rename its forward() to _forward_impl(), matching every sibling worker; callers use the base forward() wrapper so no caller changes. 2) executor tests build objects via __new__ (bypassing __init__) then exercise teardown/lifecycle paths. main NVIDIA#16523 (multi-process HTTP frontends) added attributes to BaseWorker/GenerationExecutorProxy __init__ that teardown reads unguarded, so the __new__-built test objects raised AttributeError: - test_proxy_fast_death.py _bare_proxy: seed _multi_frontend_ipc_dir/_hmac. - test_event_loop_error_broadcast.py _WorkerStub: seed frontend_result_queues (responses_handler read it unguarded -> 4/6 tests failed). - test_proxy_postproc_terminate.py _make_proxy: seed workers_started + _multi_frontend_ipc_dir/_hmac (GC __del__ -> shutdown teardown). - test_base_worker.py __new__ shell: seed doing_shutdown (GC __del__). Each seeds only what that object's teardown reads, matching the _bare_proxy convention. Signed-off-by: Yao Yao <lowsfer@users.noreply.github.com>

lowsfer · 2026-07-24T06:32:53Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-07-24T06:38:37Z

PR_Github #61522 [ run ] triggered by Bot. Commit: bbf42bd Link to invocation

tensorrt-cicd · 2026-07-24T06:40:19Z

PR_Github #61489 [ run ] completed with state ABORTED. Commit: 5fc96d4

Link to invocation

github-actions Bot assigned lowsfer May 12, 2026

lowsfer force-pushed the kvCacheManagerV2-cpp branch 2 times, most recently from 5bd984d to 456cfa8 Compare May 26, 2026 08:44

lowsfer force-pushed the kvCacheManagerV2-cpp branch from 456cfa8 to 17d71e2 Compare June 9, 2026 03:19

lowsfer force-pushed the kvCacheManagerV2-cpp branch from ae8517a to 2cdb452 Compare June 10, 2026 16:18

lowsfer force-pushed the kvCacheManagerV2-cpp branch from 4f4dd8d to 0802c6c Compare June 15, 2026 10:25

lowsfer force-pushed the kvCacheManagerV2-cpp branch from 2ca5340 to a747ad3 Compare July 23, 2026 02:40

lowsfer force-pushed the kvCacheManagerV2-cpp branch from 0fc1375 to af58079 Compare July 23, 2026 08:41

lowsfer force-pushed the kvCacheManagerV2-cpp branch 2 times, most recently from af58079 to e5a1453 Compare July 23, 2026 10:57

lowsfer force-pushed the kvCacheManagerV2-cpp branch from e5a1453 to a4efc5c Compare July 23, 2026 16:13

lowsfer added 2 commits July 24, 2026 06:26

Uh oh!

Conversation

lowsfer commented May 12, 2026 • edited by coderabbitai Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Dev Engineer Review

QA Engineer Review

Uh oh!

lowsfer commented May 12, 2026

Uh oh!

tensorrt-cicd commented May 12, 2026

Uh oh!

tensorrt-cicd commented May 12, 2026

Uh oh!

lowsfer commented May 14, 2026

Uh oh!

hyukn commented May 15, 2026

Uh oh!

lowsfer commented May 26, 2026

Uh oh!

tensorrt-cicd commented May 26, 2026

Uh oh!

tensorrt-cicd commented May 26, 2026

Uh oh!

lowsfer commented Jun 9, 2026

Uh oh!

tensorrt-cicd commented Jun 9, 2026

Uh oh!

tensorrt-cicd commented Jun 9, 2026

Uh oh!

lowsfer commented Jun 10, 2026

Uh oh!

tensorrt-cicd commented Jun 10, 2026

Uh oh!

tensorrt-cicd commented Jun 10, 2026

Uh oh!

lowsfer commented Jun 11, 2026

Uh oh!

tensorrt-cicd commented Jun 11, 2026

Uh oh!

tensorrt-cicd commented Jun 11, 2026

Uh oh!

lowsfer commented Jun 11, 2026

Uh oh!

tensorrt-cicd commented Jun 11, 2026

Uh oh!

tensorrt-cicd commented Jun 11, 2026

Uh oh!

lowsfer commented Jun 12, 2026

Uh oh!

tensorrt-cicd commented Jun 12, 2026

Uh oh!

tensorrt-cicd commented Jun 12, 2026

Uh oh!

lowsfer commented Jun 15, 2026

Uh oh!

lowsfer commented Jul 22, 2026

Uh oh!

tensorrt-cicd commented Jul 22, 2026

Uh oh!

tensorrt-cicd commented Jul 22, 2026

Uh oh!

tensorrt-cicd commented Jul 22, 2026

Uh oh!

lowsfer commented Jul 23, 2026

Uh oh!

tensorrt-cicd commented Jul 23, 2026

Uh oh!

lowsfer commented Jul 23, 2026

Uh oh!

lowsfer commented Jul 23, 2026

Uh oh!

tensorrt-cicd commented Jul 23, 2026

Uh oh!

tensorrt-cicd commented Jul 23, 2026

Uh oh!

tensorrt-cicd commented Jul 23, 2026

Uh oh!

lowsfer commented Jul 23, 2026

Uh oh!

tensorrt-cicd commented Jul 23, 2026

lowsfer commented May 12, 2026 •

edited by coderabbitai Bot

Loading