Multi-LoRA Serving Benchmark

This repository contains a small benchmark for serving many LoRA adapters with one shared base model.

The experiment uses vLLM to serve meta-llama/Llama-3.1-8B with synthetic LoRA adapters. The adapters have the same shape and file format as real adapters, but their weights are random. That means the benchmark measures serving behavior only: throughput, latency, memory use, and the effect of adapter traffic patterns. It does not measure model quality.

This is the companion repository for a two-part article series:

• The mechanics of LoRA: adapters, rank, and multi-tenant serving
• Multi-LoRA at scale: an empirical map of vLLM's operating range

What This Tests

The benchmark asks a practical question:

If one GPU is serving many LoRA adapters, what actually limits throughput?

The tested variables were:

• total adapters loaded into the server
• request concurrency
• request distribution across adapters, either uniform or skewed
• vLLM's max_loras setting, which limits how many different adapters can be active in one batch
• CPU and GPU memory use while adapters are loaded

The saved results came from a single 80 GB GPU on Vast.ai, LLaMA 3.1 8B in fp16, 1,000 synthetic rank-16 adapters, 1,000 measured requests per throughput case, 50 warmup requests, a fixed prompt, and a 128-token output limit.

This was not a full-factorial sweep. The throughput study used three focused sweeps:

• 21 uniform-traffic runs varying adapter count and concurrency with max_loras=32
• 6 skewed-traffic runs varying adapter count at concurrency 32 with max_loras=32
• 36 skewed-traffic runs varying adapter count and max_loras at concurrency 32

That is 63 executed throughput runs, plus a separate 13-case memory decomposition run.

Main Findings

Adapter count alone was not the real bottleneck. Traffic shape and max_loras mattered much more.

Under uniform traffic, high-concurrency throughput dropped earlier than expected. With concurrency 32 and max_loras=32, throughput fell from 1,935 tokens/sec with one adapter to 1,489 tokens/sec with only ten adapters, then to 884 tokens/sec with 1,000 adapters.

Under skewed traffic, where some adapters are requested much more often than others, throughput was much better. At 1,000 adapters, skewed traffic reached 2,167 tokens/sec, about 2.45x the uniform result at the same setting.

max_loras was not a "set it as high as possible" knob. In this benchmark, max_loras=16 gave the best throughput for 10 to 500 adapters under skewed traffic. Smaller values caused large tail-latency spikes, while larger values reduced throughput.

Adapter storage was mostly a CPU memory cost, not a GPU memory cost. Loading 1,000 adapters added about 13 GB of process memory, but only about 25 MB of GPU memory after server startup.

How To Read These Results

The most defensible result is the relative behavior across configurations, not the absolute tokens/sec number. The benchmark counts streamed completion chunks and uses one fixed prompt with a 128-token output limit. A different model, prompt length, output length, vLLM version, GPU, or request mix can move the absolute numbers.

The main lesson is still stable: multi-LoRA serving behaves like a working-set problem. If traffic concentrates on a small hot set of adapters, throughput is much higher than a uniform worst-case benchmark suggests. If max_loras is too small, tail latency suffers. If it is much larger than the active working set, throughput can fall because the server pays extra per-batch coordination overhead.

Article Alignment

This repository is the companion experiment for the two-part LoRA article series:

• Part 1: The mechanics of LoRA: adapters, rank, and multi-tenant serving
• Part 2: Multi-LoRA at scale: an empirical map of vLLM's operating range

It supports the serving-side claims:

• LoRA adapters are small relative to a full base model.
• Serving many adapters is not just a memory-fit question.
• Traffic distribution and max_loras determine the active serving cost.
• Adapter inventory mostly consumes host memory; GPU memory is bounded by the active adapter slots.

The experiment does not support claims about adapter quality, training effectiveness, or which rank is best for a real task. The adapters here are random-weight synthetic adapters with valid shapes.

Repository Layout

src/
  configs.py              Experiment settings and adapter shape definitions
  benchmark_lib.py        vLLM server control, adapter loading, request client, metrics
  run_benchmark.py        Main benchmark runner
  analyze.py              Summarizes throughput benchmark JSON into CSV
  analyze_memory.py       Summarizes memory benchmark JSON into CSV
  plot_results.py         Regenerates the figures from summary CSVs
results/                  Compact result summaries used by the README and plots
figures/                  Generated figures

Result Files

The important result summaries are:

• results/uniform_throughput_summary.csv: uniform traffic, varying adapter count and concurrency
• results/skewed_throughput_summary.csv: skewed traffic, varying adapter count at concurrency 32
• results/max_loras_sweep_summary.csv: skewed traffic, varying max_loras
• results/memory_decomposition_h100_summary.csv: memory measurements while loading adapters

Raw per-request JSON files are not kept in the clean repo. They are large and are only needed if you want to re-run custom analyses.

Reproducing The Benchmark

Install dependencies on a GPU machine with vLLM support:

pip install -r requirements.txt

By default, generated adapters and raw results are written under this repository. Set LORA_BENCH_WORK_DIR if you want a different working directory, for example a larger mounted disk.

Generate synthetic adapters:

python src/run_benchmark.py generate --count 1000

Run the uniform throughput sweep:

python src/run_benchmark.py benchmark --adapter-counts 1,10,50,100,250,500,1000 --result-dir uniform_throughput_raw

Run a skewed-traffic sweep:

python src/run_benchmark.py benchmark --adapter-counts 10,50,100,250,500,1000 --distribution zipf --max-loras 32 --concurrency 32 --result-dir skewed_throughput_raw

Run the max_loras sweep:

python src/run_benchmark.py max-loras-sweep --adapter-counts 1,10,50,100,250,500 --max-loras-values 4,8,16,32,64,128 --result-dir max_loras_sweep_raw

Run the memory benchmark:

python src/run_benchmark.py memory --result-dir memory_decomposition_h100_raw

Summarize raw results:

python src/analyze.py --input results/uniform_throughput_raw --output results/uniform_throughput_summary.csv
python src/analyze.py --input results/skewed_throughput_raw --output results/skewed_throughput_summary.csv
python src/analyze.py --input results/max_loras_sweep_raw --output results/max_loras_sweep_summary.csv
python src/analyze_memory.py --input results/memory_decomposition_h100_raw --output results/memory_decomposition_h100_summary.csv

Regenerate figures:

python src/plot_results.py

Notes

• The adapters are synthetic. The results say nothing about answer quality.
• The saved throughput runs should be described as running on a single 80 GB GPU on Vast.ai. The memory decomposition run is the only one labeled H100 in the saved artifacts.
• The benchmark uses one fixed prompt and a fixed output-token limit. Different workloads can move the bottleneck.
• Throughput is counted from streamed completion chunks. Treat the absolute tokens/sec numbers as benchmark-specific, and the relative comparisons as the main result.
• The CSV field called cache_hit_rate is a rough latency-based heuristic, not an internal vLLM cache metric.