LLM Serving Trace Replay + Capacity Planner

A trace-driven discrete-event simulator for LLM serving with SLA-aware cluster sizing and Prefill/Decode resource split optimization.

Read the report:

---

Docs (start here)

• Results / analysis:
• Engine (C++20): src/
• Trace generator (Python): traces/
• Planner / sweeps (Python): planner/

---

Overview

This project models realistic LLM serving workloads and answers:

• How many GPUs are required to satisfy a P99 latency SLA?
• How should GPUs be split between Prefill and Decode pools?
• Where is the saturation knee?
• How does prompt heterogeneity (long prompts) impact cluster size?

The simulator is implemented in C++20 and includes a Python-based planning and analysis layer.

---

Architecture

Each request follows:

Arrival → Prefill → Decode → Completion

Prefill stage

• Exclusive GPU usage (Prefill GPU pool)
• Service time proportional to prompt tokens
  T_prefill = 0.08 ms × prompt_tokens

Decode stage

• Continuous micro-step scheduling (Decode GPU pool)
• Step interval: 0.1 ms
• Token budget: 64 tokens per decode GPU per step
  • Total decode budget per step: k_decode × 64 tokens/step
• Round-robin token distribution across active sequences
• Decode length fixed at 128 tokens (controlled experiments)

---

Workload model

Arrivals follow a Poisson process:

Δt = -ln(U) / λ

Where:

• λ = arrival rate (req/s)
• U = uniform random number in (0,1)

Prompt heterogeneity:

• A fraction long_fraction has 2048 prompt tokens
• Otherwise prompt tokens are 512 / 256 (configurable)

---

Capacity estimation example (10% long prompts)

E[prompt] ≈ 614 tokens
T_prefill ≈ 49 ms → μ_prefill ≈ 20 req/s per GPU

With 6 prefill GPUs:

• Total capacity ≈ 120 req/s
• Saturation knee appears near: λ ≈ 100–120 req/s

---

Headline results (example configuration)

• With prompt heterogeneity (10% long prompts), prefill capacity is ~20 req/s per GPU (back-of-the-envelope).
• Under lambda=30 req/s and SLA=500 ms, the minimal feasible cluster in this sweep was K_min=4, with optimal split (k_prefill=3, k_decode=1) achieving P99 ≈ 439 ms.
• Even small long-prompt fractions significantly increase required cluster size (tail dominated by prefill variance).

See full details in .

---

Usage

1) Generate a workload trace

python3 traces/generate_trace.py <duration_s> <arrival_rate_lambda>

Example:
python3 traces/generate_trace.py 30 0.1

2) Build (Ubuntu/Linux)
mkdir -p build
cd build
cmake .. -DCMAKE_BUILD_TYPE=Release
cmake --build . -j

3) Run replay simulation
./replay <trace_file> <k_prefill> <k_decode>

Example:
./replay ../traces/poisson_trace.csv 3 1

Note: If the binary expects a different argument order (e.g., duration), update this section to match the implementation.

Engineering challenges solved

Strict event ordering with stable comparator (discrete-event correctness)
State reset between runs
Warm-up filtering for steady-state P99 measurement
Avoiding drain-phase bias
Resource split optimization under SLA constraints


Project structure
llm-serving-trace-replay/
├── src/
│   ├── replay_engine.hpp
│   ├── event.hpp
│   ├── request.hpp
│   └── main.cpp
├── traces/
│   └── generate_trace.py
├── planner/
│   ├── split_optimizer.py
│   ├── cluster_sizing_sla.py
│   ├── heterogeneity_sweep.py
│   └── global_cluster_sizing.py
├── results/
├── report.md
├── LICENSE
└── README.md

Limitations
No KV-cache modeling
No multi-tenant fairness
No admission control
No batching efficiency curve
Prefill and decode treated as separate GPU pools
Future work
KV-cache capacity modeling
Admission control policies
Erlang-C analytical comparison
Halfin–Whitt (QED) regime modeling
Multi-region cost optimization

License
This project is licensed under the MIT License — see the LICENSE file for details.