LLM Serving Scheduler Engine

A discrete-event simulation engine modeling LLM inference serving under stochastic load.

This project implements a queueing-theoretic simulation of:

• M/M/k serving systems
• Dynamic GPU batching
• SLA-aware admission control
• Deadline-based scheduling (EDF)
• Horizontal scaling across multiple GPUs
• Saturation knee behavior
• Tail latency stabilization under overload

---

1. Motivation

Modern LLM serving systems must:

• Handle stochastic (Poisson-like) request arrivals
• Batch requests to maximize GPU throughput
• Enforce SLA deadlines
• Remain stable under overload
• Scale horizontally across multiple GPUs

This project models those dynamics analytically and via discrete-event simulation.

---

2. System Model

Arrival Process

Poisson process:

[ t_{next} = -\frac{\ln(U)}{\lambda} ]

Simulation runs for fixed time horizon ( T_{sim} ).

---

Service Model (GPU Batching)

Batch service time:

[ T_{batch} = \alpha + \beta \cdot B ]

Where:

• ( \alpha ): fixed overhead (kernel launch, sync)
• ( \beta ): per-request compute cost
• ( B ): batch size (≤ BATCH_MAX)

Maximum per-GPU capacity:

[ \mu_{max} = \frac{B_{max}}{\alpha + \beta B_{max}} ]

---

Multi-GPU (M/M/k)

Each GPU maintains:

• Independent next-free time
• Independent batching
• Shared global request queue

Cluster capacity:

[ \mu_{cluster} \approx k \cdot \mu_{max} ]

---

Admission Control

Upon arrival:

[ estimated_finish = now + (batches_ahead + 1) \cdot T_{batch} ]

If:

[ estimated_finish > deadline ]

→ request is rejected.

This stabilizes the system under overload.

---

3. Key Observations

✅ Saturation Knee

For 1 GPU:

• Throughput saturates at ~1880 req/s
• Rejections grow linearly beyond that

For 2 GPUs:

• Knee ≈ 3760 req/s

For 4 GPUs:

• Knee ≈ 7500 req/s

Linear horizontal scaling is observed.

---

✅ Tail Latency Behavior

Under overload:

• Throughput plateaus
• Rejections increase
• P99 stabilizes instead of diverging

This demonstrates tail protection via admission control.

---

4. Experimental Results

1 GPU

λ	Throughput
1000	~1000
2000	~1880
5000	~1880

2 GPUs

λ	Throughput
2000	~2000
4000	~3760
8000	~3760

4 GPUs

λ	Throughput
4000	~4000
8000	~7450
12000	~7450

---

5. What This Demonstrates

• Compute-bound → queue-bound transition
• Tail amplification in unconstrained systems
• Stabilization via deadline-aware shedding
• Linear cluster scaling under dynamic batching
• Realistic modeling of LLM inference serving

---

6. Build & Run

mkdir build
cd build
cmake ..
make
./scheduler_engine

7. Future Extensions
Adaptive control loop (PID-like)
Heterogeneous GPU modeling
Multi-SLA priority classes
Load balancing policies
Integration with real GPU benchmarks

---

## Author

**João Felipe De Souza**  
AI Infrastructure Engineer  
Specialization: CUDA Runtime Optimization, LLM Serving Systems, Queueing-Theoretic Modeling

---

## License

This project is licensed under the MIT License — see the `LICENSE` file for details.