Accuracy boundary report: throughput & queue depth diverge under severe overload

[2534933103a-code]

Accuracy boundary report: throughput & queue depth diverge under severe overload

Summary

First of all — LLMServingSim is a really impressive piece of work. The cycle-level approach with profiled kernel latencies is elegant, and in the normal operating regime the fidelity is excellent. This report is less a bug and more an attempt to map out where the accuracy envelope currently sits.

I ran a pair of benchmarks at opposite ends of the load spectrum (SPS=3 vs SPS=15 on a single RTX 3090). Under light load the simulator tracks real vLLM almost perfectly. Under heavy overload — the SPS=15 case pushes the system well past its saturation point, with up to 300+ requests waiting — the qualitative trends remain clearly preserved, but absolute throughput and queue-depth values begin to diverge. The divergence onset coincides with the moment the waiting queue becomes non-zero.

To be clear: SPS=15 on a single 3090 is an intentionally extreme stress test. No production deployment would sustain 300+ queued requests without scaling out. The fact that the simulator still gets the trends right under these conditions is actually encouraging — it suggests the core modeling (profiled latencies, cycle-level execution) is fundamentally sound, and what drifts is the scheduling dynamics under deep queue pressure.

Reproduction

Hardware: Single NVIDIA RTX 3090
Model: meta-llama/Llama-2-7b-hf

Small workload — sim ≈ vLLM ✅

python -m workloads.generators sharegpt \
  --max-input-toks 2048 --max-output-toks 2048 --max-kv-toks 4096 \
  --pulse --pulse-poisson --pulse-n 10 --pulse-delay-sec 3 \
  --max-sessions 50000 --model meta-llama/Llama-2-7b-hf \
  --num-reqs 1000 --sps 3 \
  --output workloads/myllama-2-7b.jsonl

Large workload — trends preserved, absolute values drift

python -m workloads.generators sharegpt \
  --max-input-toks 2048 --max-output-toks 2048 --max-kv-toks 4096 \
  --pulse --pulse-poisson --pulse-n 20 --pulse-delay-sec 4 \
  --max-sessions 50000 --model meta-llama/Llama-2-7b-hf \
  --num-reqs 1000 --sps 15 \
  --output workloads/myllama-2-7b.jsonl

Observations

Metric	Small workload (SPS=3)	Large workload (SPS=15)
Prompt throughput	sim ≈ vLLM	qualitative trend matches; absolute values drift
Generation throughput	sim ≈ vLLM	qualitative trend matches; absolute values drift
Running requests curve	matches well	overall shape preserved; peak timing shifts
Waiting requests curve	both at 0 (no queue)	sim and vLLM both show deep queues, but depth and ramp differ

Main takeaway: The divergence begins when the waiting queue becomes non-zero. Below saturation the simulator is remarkably accurate. Above saturation the trends hold but the numbers drift — which is arguably the right place for a "known limitation" rather than a blocking bug.

Possible contributing factors

A. Running-request oscillation under memory pressure (most likely candidate)

I noticed a recurring pattern in the simulator's per-second logs that may be related:

[77.0s] Running 32, Waiting 49, Memory 99.986%
[78.0s] Running  6, Waiting 74, Memory 99.986%   ← 26 requests evicted at once
[79.0s] Running  3, Waiting 76, Memory 99.986%   ← another 3 evicted
[80.0s] Running 75, Waiting  9, Memory 99.465%   ← ~70 new prefills flood in

The number of running requests oscillates dramatically (32 → 6 → 3 → 75) within a few seconds, while memory hovers at ~99.99%. This pattern repeats throughout the overloaded portion of the run.

I took a look at the relevant code path in serving/core/scheduler.py (the eviction loop in schedule() / schedule_with_prefix(), roughly the while temp_len == 0 block). The current logic appears to work as follows when NPU memory is nearly full:

1. Assemble a candidate batch from all waiting requests
2. Compute the total KV cache needed
3. If it doesn't fit, evict decode requests one by one from the tail
4. Keep evicting — in a tight while-loop — until the entire remaining candidate batch fits

I compared this against vLLM v1's scheduler. A few differences stood out that might explain the oscillation:

Aspect	vLLM v1	Sim (current)
Scheduling approach	Incremental: add requests one at a time; break on first allocation failure	Batch: assemble all candidates upfront, then trim/evict in bulk
Memory pre-check	can_fit_full_sequence() checks whether the request's full KV cache growth would fit before accepting it	No pre-check; accepts the request, then evicts if it doesn't fit
Preemption granularity	Preempts 1 request, retries allocation, stops if still insufficient	Evicts N requests in a tight while-loop until everything fits
Running cap	Hard break when len(running) == max_num_running_reqs	Soft cap; eviction can free slots that get immediately filled by new prefills
Overload behavior	Running count stays stable near the cap; waiting queue grows naturally	Running count oscillates: mass eviction → mass admission → memory full → repeat

Hypothesis: the "batch-then-evict-in-bulk" approach may create a boom-bust cycle — when memory is tight, many decodes get evicted at once to make room, the freed memory is immediately consumed by admitting a large number of prefills, and the cycle repeats. In contrast, vLLM's incremental scheduling with pre-checks naturally dampens this oscillation by refusing new work when memory is under pressure.

I want to emphasize that this is just a pattern I observed, not a confirmed root cause. The scheduler code is nuanced (especially the schedule_with_prefix path with prefix locking), and I may well be misreading the logic or missing a compensating mechanism elsewhere.

B. Other factors (lower confidence)

These may contribute but seem less likely to be the primary driver:

1. Iteration granularity: the simulator advances in discrete ASTRA-Sim cycles (batch → execute → complete), while real vLLM interleaves phases. The gap is negligible under light load but could matter more when queue depth feeds back into scheduling latency.

2. Arrival-time quantization (router.route_arrived_requests(current)): requests are routed at cycle boundaries. At SPS=15 (~67ms inter-arrival) this may introduce timing skew that does not exist at SPS=3 (~333ms).

3. Token budget allocation order: the decode-first-then-prefill policy in schedule() may differ from vLLM v1's exact allocation order in edge cases, shifting which requests land in each batch when the budget is fully contested.

Suggestions

• Investigate the eviction oscillation: compare running-request time-series between sim and vLLM at 1-second granularity and check whether the sim shows the 32→6→3→75 pattern while vLLM stays relatively stable near max_num_seqs
• If the oscillation is confirmed, one possible direction: experiment with capping evictions per scheduling step (e.g., evict at most 1–2 requests before giving up and returning None), which would more closely mirror vLLM's incremental preemption behavior
• Document the accuracy envelope: regardless of code changes, it may be helpful to note in the docs that accuracy has been validated up to a certain SPS range and that under severe overload trends are preserved but absolute values have known drift
• Overlay latency CDFs from bench validate for the large workload to quantify per-request drift magnitude
• Compare batch composition (prefill/decode ratio, avg batch size) between sim and vLLM in 5-second windows under overload

Environment

• vLLM version: v0.19.0
• GPU: NVIDIA RTX 3090 (24 GB)

---

Thanks again for building this — the profiler pipeline alone is a substantial contribution, and the fact that the simulator gets the trends right even at 5× overload says a lot about the core approach. I've tried to surface what I found with enough detail to be actionable, but please take the scheduler comparison with a grain of salt — I may well be missing context or misreading the logic. Happy to provide raw bench/sim output or run additional experiments if that would help narrow things down.

---

light load

---

heavy load