[Performance]: MAX_UTILIZATION causes a 40.6% output-throughput drop and 9.1x TPOT p99 under KV-cache oversubscription (Llama-3.3-70B-FP8, H200, v1.2.1)

[Smallfu666]

Proposal to improve performance

N/A — this is a measured performance report and a request for guidance on the intended operating regime and observability.

Report of performance regression

Summary

While running a controlled three-engine benchmark study, we swept scheduler_config.capacity_scheduler_policy on TensorRT-LLM 1.2.1 (PyTorch backend, trtllm-serve) and found that switching from the default GUARANTEED_NO_EVICT to MAX_UTILIZATION causes a severe performance drop — but only when the workload's KV working set exceeds the KV-cache pool:

• Under KV pressure (long-context workload, details below): output-token throughput 161.0 → 95.7 tok/s (−40.6%), TPOT p99 122 → 1,111 ms (~9.1×), end-to-end p99 150 → 563 s.
• Off pressure (short-context workload, same server config, not pool-bound): the two policies are equivalent within noise — mean output-token throughput 574.7 vs 576.1 tok/s (+0.24%). Across the two repeats, per-run throughput was 574.4–575.1 tok/s (GUARANTEED_NO_EVICT) and 574.5–577.6 tok/s (MAX_UTILIZATION).

We understand MAX_UTILIZATION is documented as the more aggressive policy that may pause/evict requests, so this may well be a by-design trade-off rather than a bug. We are reporting it because (a) the magnitude is large: under KV pressure, the policy was worse on every listed metric except TTFT p50; off pressure, the two policies were effectively equivalent in the two repeats; and (b) the failure mode was not visible to us through any counter we could find: no warning, just a 9× p99 blowup. Questions for the team are at the end.

Environment

• TensorRT-LLM 1.2.1 (latest stable release at time of writing), NGC container nvcr.io/nvidia/tensorrt-llm/release:1.2.1, run via Apptainer 1.4.3, PyTorch backend, trtllm-serve.
• Container image digest: sha256:b5f9b403c414d663fbdb7be754b90a063785d28827ca418184fa2497b644ec04 (linux/amd64 manifest recorded in the local pull cache at build time); NGC build id 245654590.
• Apptainer image (SIF) SHA256: 4a4c6fc242c818b20ea8346c0680440c64efaa6c7d2f7a34034a7464ccd455f0.
• tensorrt_llm Python package version inside the container: 1.2.1 (pip show tensorrt_llm).
• Model: nvidia/Llama-3.3-70B-Instruct-FP8, Hugging Face revision fde04ee76a27704c88f569542ef023b57d4d0362 (ModelOpt 0.27.1 unified checkpoint, quant_algo: FP8, kv_cache_quant_algo: FP8); model.safetensors.index.json SHA256 6398071a55c63422e6607f3a2e5b9a28a90f51090aca7a2313386f42a3d957ec.
• GPU: 1× H200 (141 GB), single node (HPC cluster, Slurm, cgroup-isolated one-job-per-GPU), TP=1.
• Driver: 580.65.06 (host nvidia-smi reports CUDA 13.0).
• OS / Python inside container: Ubuntu 24.04.3 LTS, Python 3.12.3; in-container CUDA toolkit 13.1 (nvcc V13.1.80), cuDNN 9.17.0.29, NCCL 2.28.9, TensorRT 10.14.1.48, PyTorch 2.10.0a0+b4e4ee81d3.nv25.12.

Server launch

trtllm-serve serve $MODEL --backend pytorch --tp_size 1 \
  --max_batch_size 64 --max_num_tokens 8192 --max_seq_len 20480 \
  --kv_cache_free_gpu_memory_fraction 0.90 \
  --config extra.yaml \
  --host 127.0.0.1 --port $PORT

extra.yaml — baseline arm:

kv_cache_config:
  enable_block_reuse: false   # prefix caching off (cross-engine parity in our study)
  dtype: fp8
  max_tokens: 368640          # pool pinned to 368,640 KV token slots (see note below)
enable_chunked_prefill: true

extra.yaml — MAX_UTILIZATION arm (identical plus the policy):

kv_cache_config:
  enable_block_reuse: false
  dtype: fp8
  max_tokens: 368640
enable_chunked_prefill: true
scheduler_config:
  capacity_scheduler_policy: MAX_UTILIZATION

The server-side resolved config confirms the policy took effect: the launch log prints scheduler_config=SchedulerConfig(capacity_scheduler_policy=<CapacitySchedulerPolicy.MAX_UTILIZATION: 'MAX_UTILIZATION'>, ...) for the MAX_UTILIZATION arm and <CapacitySchedulerPolicy.GUARANTEED_NO_EVICT> for the baseline; both arms allocated the identical KV pool (Allocated 56.25 GiB for max tokens in paged KV cache (368640)). Excerpts are included in the repro bundle.

Note on the pinned pool: max_tokens: 368640 pins the KV pool to the same size across the three engines in our study (fraction-derived would have been ~403,680 tokens / 61.6 GiB on this GPU, so the cap binds). The pressure regime below is defined relative to this pool; with a larger pool the crossover point moves but the regime definition is the same.

Workload

• W-C (KV-pressure arm): 128 requests, each exactly 16,384 input tokens and up to 512 output tokens (ignore_eos on), closed-loop concurrency 32, greedy temperature=0, streaming /v1/completions, two repeats with shuffled submission order (the request set is fixed; the repeat index seeds only the order shuffle). With ignore_eos, output lengths were verified two ways: by re-tokenizing the generated text and by the count of streamed content chunks (our client records both per request; it does not record finish_reason, and usage is not present in the streamed responses). GUARANTEED_NO_EVICT — every request in both repeats produced exactly 512 chunks re-tokenizing to exactly 512 tokens (totals 65,536/65,536). MAX_UTILIZATION — 3 of 256 request-runs (all in one repeat) received fewer than 512 content chunks from the server (509, 395, 329), i.e., those streams genuinely ended early server-side; because finish_reason was not recorded, we cannot determine the server-side stop cause post hoc. Two further request-runs (one per repeat) received the full 512 chunks but re-tokenize to 284 tokens — a decode→re-encode artifact of the re-tokenization, not early termination. Re-tokenized totals: 65,005/65,308 vs the nominal 65,536 (0.8% and 0.3% shortfall; see the counting-sensitivity note under the results table). The nominal full-residency demand of 32 concurrent sequences is 32 × 16,896 = 540,672 KV-token slots, 1.47× the configured pool. The pool therefore cannot keep all 32 sequences resident through their full requested lengths simultaneously.
• W-B (control, not pool-bound): 512 requests × 2,048 in / 256 out, same client, concurrency 32; observed output lengths were exactly 256 for every request in all four runs.
• Warmup 32 requests @ concurrency 8 before measurement; fresh server per config; output tokens re-counted with the model tokenizer.

Metrics (definitions)

All metrics are client-side measurements taken by our benchmark client over streaming /v1/completions:

• Throughput = aggregate re-tokenized generated-text throughput: the sum of output tokens across all completed requests in a run divided by the wall-clock duration of the whole run, where each request's output tokens are counted by re-tokenizing its generated text with the model tokenizer. The client does not record server-side token counts (usage) or finish_reason; it does record the number of streamed content chunks per request, which we use as a server-side cross-check (see the counting-sensitivity note below).
• TTFT (per request) = time from sending the request to client receipt of the first non-empty content chunk.
• TPOT (per request) = (t_last − t_first) / (n_out − 1), where t_first/t_last are client receipt times of the request's first/last content chunks — i.e., the request's mean inter-token time, not a percentile over individual inter-token gaps.
• End-to-end latency (per request) = time from send to the last received chunk.
• Reported p50/p99 are percentiles (linear interpolation) of these per-request values — percentiles over the 128 returned request records for W-C (512 for W-B).

Observed vs expected

Mean of the two repeats per cell. For aggregate output throughput, the largest difference between the two repeats of a cell was 0.7% (worst cell across our whole three-engine sweep); the observed 40.6% throughput delta is therefore much larger than the repeat-to-repeat variation seen in this experiment. Tail percentiles vary more between repeats; per-repeat TPOT p99 values are listed under the table.

W-C, concurrency 32 — KV-pressure regime

policy	output tput (tok/s)	TTFT p50 (ms)	TPOT p50 (ms)	TPOT p99 (ms)	e2e p99 (s)
GUARANTEED_NO_EVICT (default)	161.0	35,774	118.5	122.2	150.3
MAX_UTILIZATION	95.7 (−40.6%)	27,145	122.6	1,110.7 (9.1×)	563.1 (3.7×)

Per-repeat TPOT p99 (W-C): GUARANTEED_NO_EVICT 122.0 / 122.5 ms; MAX_UTILIZATION 1,169.0 / 1,052.4 ms (the table shows the mean of the two repeats). Per-request raw values are included in the repro bundle for independent recomputation.

W-B, concurrency 32 — same server config, not pool-bound

policy	output tput (tok/s)	TTFT p50 (ms)	TPOT p50 (ms)
GUARANTEED_NO_EVICT	574.7	2,468	44.49
MAX_UTILIZATION	576.1	2,496	44.38

• TTFT p50 actually improves under MAX_UTILIZATION (27.1 s vs 35.8 s) on W-C — consistent with requests being admitted earlier; the aggregate token clock then slows ~40% and the tail explodes.
• Expected (naively, from the name and docs): equal-or-better utilization of the KV pool, at worst a modest tail cost. Observed: under KV pressure, output throughput, TPOT p50/p99, and end-to-end p99 were worse, while TTFT p50 improved. Off pressure, the two policies were effectively equivalent across the two repeats.
• Token-counting sensitivity: the −40.6% delta uses re-tokenized counts (see Metrics). Recomputing throughput from the same wall-clock durations with nominal counts (128 × 512 tokens per run) gives 161.0 → 96.2 tok/s (−40.2%); with server-emitted content-chunk counts, 161.0 → 96.0 tok/s (−40.4%). The conclusion is insensitive to the counting method.
• Our working hypothesis is that under oversubscription the policy admits more prefills than the pool sustains and then repeatedly pauses/evicts running requests, potentially including context recomputation for evicted requests, burning compute on repeated prefill. We have not confirmed the mechanism because we did not have a version-appropriate exposed counter for pause/evict/recompute activity in our setup — trtllm-serve 1.2.1 with the PyTorch backend exposed no /metrics endpoint we could poll, and we did not find an equivalent counter.

Reproducibility notes

• Both arms ran in the same Slurm job on the same GPU/node, fresh server per arm, identical warmup; the baseline arm reproduces the same config measured in a different job on a different node two days earlier within 0.6%.
• All request tasks reached a terminal stream state with no client-side or transport exceptions. However, 3 of 256 MAX_UTILIZATION W-C request-runs returned fewer than the requested 512 content chunks. Because finish_reason was not recorded, we classify these as early-terminated outputs with unknown server-side cause, not as confirmed full-length completions. All 512 W-B requests returned the requested 256 chunks in every run. The server accepted the policy (verified in the server-side resolved-config dump, quoted above).
• A self-contained repro bundle (launch script, both YAMLs, workload generator, benchmark client, per-repeat raw results with per-request rows, resolved-config excerpts): https://gist.github.com/Smallfu666/c896b3461e055b3bd59a986cb06d6815 (16 files: launch script, both YAMLs, deterministic workload generator, benchmark client, all 8 per-repeat raw result sets with per-request rows, and both server resolved-config excerpts)

Misc discussion on performance

Questions for the team:

1. Is this magnitude of degradation under KV-cache oversubscription known/expected for MAX_UTILIZATION? The docs frame it as maximizing batch utilization with possible eviction; if a ~40% throughput and ~9× TPOT-p99 penalty in oversubscribed regimes is the intended trade-off, a warning in the scheduler_config docs would save users from discovering it in production.
2. Is there a regime where MAX_UTILIZATION is expected to win? In the two concurrency-32 regimes tested it was neutral off-pressure and worse under pressure on the metrics listed; we may simply not have hit the regime the policy is designed for.
3. Observability: what is the supported way to observe pause/evict/recompute activity from trtllm-serve with the PyTorch backend in 1.2.1? Specifically: do iter_stats_max_iterations / request_stats_max_iterations or return_perf_metrics in the LLM-API config apply to trtllm-serve on the PyTorch backend in 1.2.1, and is the Prometheus /metrics endpoint documented for other frontends available in this configuration? In our runs the resolved config showed return_perf_metrics=False and request_stats_max_iterations=None (defaults), and the server exposed no /metrics endpoint.

Related prior report: the closest issue we found is #9931, where enabling MAX_UTILIZATION under KV-cache exhaustion produced a crash on 1.2.0rc builds. That issue is now closed. We did not identify a linked fix commit. Our v1.2.1 runs did not crash; all requests completed, but they showed severe performance degradation in the same broad policy-plus-KV-pressure regime. We do not know whether the mechanisms are related.

Happy to run additional configurations (different pool sizes, concurrencies, or a 1.3.0 RC) if that helps.

Your current environment

System Information:

• OS: Ubuntu 24.04.3 LTS (inside NGC container); host cluster runs Slurm with cgroup GPU isolation
• Python version: 3.12.3
• CUDA version: 13.1 (in-container nvcc V13.1.80; host driver reports CUDA 13.0)
• GPU model(s): 1× NVIDIA H200 (141 GB)
• Driver version: 580.65.06
• TensorRT version: 10.14.1.48
• PyTorch version: 2.10.0a0+b4e4ee81d3.nv25.12
• TensorRT-LLM version: 1.2.1
• Container: nvcr.io/nvidia/tensorrt-llm/release:1.2.1, image manifest digest sha256:b5f9b403c414d663fbdb7be754b90a063785d28827ca418184fa2497b644ec04 (linux/amd64), NGC build id 245654590; run via Apptainer 1.4.3; SIF SHA256 4a4c6fc242c818b20ea8346c0680440c64efaa6c7d2f7a34034a7464ccd455f0
• Model checkpoint: nvidia/Llama-3.3-70B-Instruct-FP8 @ revision fde04ee76a27704c88f569542ef023b57d4d0362; model.safetensors.index.json SHA256 6398071a55c63422e6607f3a2e5b9a28a90f51090aca7a2313386f42a3d957ec

Detailed output:

$ nvidia-smi
NVIDIA-SMI 580.65.06    Driver Version: 580.65.06    CUDA Version: 13.0
GPU 0: NVIDIA H200 (143771 MiB), Persistence-M: On, ECC: 0 (enabled)

$ nvcc --version   # inside container
Cuda compilation tools, release 13.1, V13.1.80

$ python3 --version   # inside container
Python 3.12.3

$ pip show tensorrt torch tensorrt_llm   # inside container
tensorrt      10.14.1.48
torch         2.10.0a0+b4e4ee81d3.nv25.12
tensorrt_llm  1.2.1

# selected container labels (apptainer inspect, NGC release:1.2.1, build id 245654590):
cuDNN 9.17.0.29, NCCL 2.28.9+cuda13.0, cuBLAS 13.2.0.9, flashinfer-python 0.6.4
base image: nvcr.io/nvidia/tensorrt-llm/release:1.2.1 (Ubuntu 24.04)

• Make sure you already searched for relevant issues, and checked the documentation and examples for answers to frequently asked questions.

[karljang]

@Smallfu666, thank you for providing detailed information!
I noticed that the 1.2.1 version is being used in this report. While it’s the latest stable version, most development has been focused on the 1.3.0rc versions. Would you be able to try the latest 1.3.0rc version as well? Thanks!