Mini-vLLM

A lightweight inference and quantization engine for studying LLMs.

Decode Throughput

Measured on RTX 4050 6 GB · greedy · 128 decode tokens

Model	HF Transformers	mini-vllm	Speedup
Qwen3-0.6B	23.0 tok/s	114.5 tok/s (megakernel)	5.0x
Qwen3.5-0.8B	15.3 tok/s	90.9 tok/s (CUDA Graph + flash-attn + fla)	6.0x

Streaming Inference & Megakernel

_{Qwen3-0.6B · bf16 · greedy · same prompt · 6 GB RTX 4050
Left: HF Transformers · Right: mini-vllm megakernel}

Features

• Continuous Batching -- dynamic scheduler with FIFO / SPF / LJF / random admission and chunked prefill
• Flash Attention -- varlen FA2 prefill, flash_attn_with_kvcache decode, SDPA fallback
• Fused CUDA Megakernel -- single-kernel decode fusing all transformer layers (bs=1)
• CUDA Graph Decode -- bucketed graph capture to reduce CPU launch overhead (bs=1)
• KV Cache Quantization (KIVI) -- 2/4-bit asymmetric quantization with full-precision residual window
• AWQ 4-bit Quantization -- weight-only quantization with Triton/CUDA kernels
• SwanLab Monitoring -- throughput and memory tracking (docs)
• lm-eval Benchmark -- evaluation harness adapter (docs)

Supported Models

Model	Model Backend	Attention Backend
Qwen3	default, megakernel_cuda	sdpa, flash_attn, naive
Qwen3.5	default	sdpa, flash_attn, naive, fla

Quick Start

1. Install

git clone https://github.com/BoundlessWindMoon/minivllm.git
cd mini-vllm

uv venv .venv --python 3.12
source .venv/bin/activate

# PyTorch (match your CUDA version)
uv pip install torch==2.9.0 --index-url https://download.pytorch.org/whl/cu128

# Core dependencies
uv pip install -e .

Optional extras:

# Qwen3.5: flash-attn and flash-linear-attention are required
pip install flash-attn --no-build-isolation
pip install flash-linear-attention --no-build-isolation

uv pip install -e ".[benchmark]"   # lm-eval
uv pip install -e ".[monitor]"     # SwanLab
uv pip install -e ".[quant]"       # AWQ

2. Download Model

HF_ENDPOINT=https://hf-mirror.com huggingface-cli download Qwen/Qwen3-0.6B \
  --local-dir ~/huggingface/Qwen3-0.6B/

# Qwen3.5 requires flash-attn + flash-linear-attention
HF_ENDPOINT=https://hf-mirror.com huggingface-cli download Qwen/Qwen3.5-0.8B \
  --local-dir ~/huggingface/Qwen3.5-0.8B/

3. Run

# Qwen3, megakernel backend
python main.py

# Qwen3.5, CUDA Graph (requires flash-attn + flash-linear-attention)
python main.py --config configs/qwen3_5.yaml

# Continuous batching
python batch_main.py
python batch_main.py --sweep-policies fifo spf ljf random
python batch_main.py --repeat 5

4. Benchmark

python -m eval.run --config configs/qwen3_5.yaml --tasks arc_easy --limit 50

See docs/benchmark.md for task customization and log output.

Documentation

• Configuration Reference -- all YAML parameters
• Benchmark Evaluation -- lm-eval integration
• Profiling & Monitoring -- PyTorch profiler, Perfetto, SwanLab
• AWQ Quantization -- calibration and quantized inference

Project Structure

mini-vllm/
├── configs/            # YAML configs
├── engine/             # Inference loop, model runner, scheduler, batched runner, KV pool
├── kernels/            # Triton & CUDA kernels (AWQ, KIVI)
├── layers/             # Attention, MLP, RMSNorm, Rotary, KV cache backends
├── model/              # Qwen3 / Qwen3.5 architectures
├── quantization/       # AWQ calibration and quantized layers
├── eval/               # lm-eval adapter
├── scripts/            # Benchmark, verify, profile tools
├── test/               # Functional test suite (pytest)
├── assets/prompts/     # JSONL prompt workloads for batch testing
├── main.py             # Entry point: single-request inference
├── batch_main.py       # Entry point: continuous batching
└── quant_cli.py        # Entry point: AWQ calibration

Acknowledgements

• nano-vllm -- minimalist inference architecture
• AutoAWQ -- AWQ quantization
• mega-qwen -- fused megakernel design

License

MIT