For TensorRT-LLM benchmarks ..whats the difference between batch_size and max_batch_size ?

[prasad-nair-amd]

In trtllm-build tool there is an attribute --max_batch_size . What this attribute represent. Is this the same attribute as batch_size seen in other industry standard benchmarks. How to specify batch_size for a benchmark ?

[Advait251206]

--max_batch_size and the benchmark batch_size refer to two related but different concepts.

• --max_batch_size (during trtllm-build) specifies the maximum batch size that the engine is built to support. It determines the upper bound for engine optimization and resource allocation. If you build an engine with --max_batch_size=8, you cannot later run inference with a batch size greater than 8 without rebuilding the engine.
• batch_size (during benchmarking/inference) is the actual number of requests processed together during execution. This is the value that affects the measured throughput and latency.

For example:

trtllm-build --max_batch_size=16

This creates an engine capable of handling batches up to 16. During benchmarking, you could run with batch sizes of 1, 2, 4, 8, or 16, but not 32.

Depending on which benchmarking tool you're using, the runtime batch size is typically controlled by the benchmark itself (e.g., through the dataset, request configuration, or concurrency settings) rather than by trtllm-build. The engine simply defines the maximum supported limit.

If you're using the inflight batching scheduler, it's also worth noting that the actual batch seen by the GPU is dynamic. Multiple incoming requests may be merged into a batch up to the engine's max_batch_size, so the effective batch size can vary throughout execution.

Could you also share which benchmark you're referring to (e.g., gptManagerBenchmark, benchmark.py, trtllm-bench, or another benchmarking script)? The mechanism for specifying the runtime batch size differs slightly between these tools, and I can point you to the exact parameter.

---

If this answer helped or pointed you in the right direction, I'd appreciate it if you could mark it as the accepted answer so it's easier for others with the same issue to find.

Also, if you found my contribution useful, I'd appreciate it if you could check out my GitHub profile, follow me, and star any repositories you find interesting.

GitHub: https://github.com/Advait251206