Feature Idea: Prompt KV-Caching

[atyshka]

If you have a large prompt that you use repeatedly for generation tasks, it's inefficient to re-compute the embedding of the entire prompt every time. Llama.cpp currently has a feature where you can store the KV cache to disk, run a diff against the new prompt for each run, and only re-compute the KV cache for the last part of the prompt that might have changed. Adding such a feature to TensorRT-LLM could significantly reduce latencies in such scenarios where only the end of the prompt changes between runs. The easiest solution would be caching in-memory, I think this would mainly just require changes to handle_per_step. A more extensive solution could also feature disk caching, like Llama.cpp does. I'm willing to help contribute this feature, first I just need to figure out my other issues with getting a quantized model that'll fit on my rig 😄.

[kevinlu1248]

Might be possible to do this using swap.

[michaelfeil]

This is currently enabled / with the flag kv cache reuse.

[kinthaiofficial]

Prompt KV-caching is one of the highest-leverage optimizations for agent workloads — agents with long system prompts or stable context prefixes can cut their effective compute cost significantly.

For agent-specific patterns, KV-cache matters most in these scenarios:

System prompt caching — agent system prompts (role description, tool definitions, rules) are often 500-2000 tokens and identical across all calls from the same agent. Caching the KV for this prefix means you only compute it once per model load/restart, not once per request.

Document context caching — when an agent is working through a long document (code review, research paper analysis), the document stays in context across many calls. Cache the document's KV; only compute the KV for the changing query portion.

Multi-agent shared prefix — when multiple agents are configured with the same base system prompt (a common pattern in agent fleets), a shared KV cache pool across agents saves N× the compute for the shared prefix.

The implementation challenge for TensorRT-LLM: KV cache sharing requires matching the cache key (prefix token sequence) exactly. Minor variations (different whitespace, timestamp injections) prevent cache hits. For agent systems, it's worth standardizing prompt construction to maximize cache reuse.

Prefix tree (radix cache) is the data structure that enables efficient prefix matching across requests. Requests are sorted by their prefix; cache pages are shared when prefixes match. SGLang and some vLLM configurations implement this.

For the economics of KV-cache in multi-agent systems: https://blog.kinthai.ai/openclaw-multi-tenancy-why-vm-per-user-doesnt-scale covers how caching affects the multi-tenant cost model.

Is the caching feature request primarily for throughput improvement or for latency (TTFT) reduction?