Mini-SGLang

A lightweight yet high-performance inference framework for Large Language Models.

---

Mini-SGLang is a compact implementation of SGLang, designed to demystify the complexities of modern LLM serving systems. With a compact codebase of ~5,000 lines of Python, it serves as both a capable inference engine and a transparent reference for researchers and developers.

Speculative Decoding (Educational)

python/minisgl/engine/speculative.py implements speculative decoding as a standalone educational component — intentionally separate from the Scheduler and continuous batching.

How it works: A small draft model generates k tokens, the large target model verifies all k tokens in one parallel prefill, and accepted tokens are kept via rejection sampling. The output distribution is mathematically identical to running the target model alone.

What's supported:

• Separate draft/target models with independent KV caches and attention backends (FlashInfer or torch)
• Parallel target verification (one prefill, not k sequential decodes)
• Rejection sampling with corrected distribution + bonus token

What's not supported (by design):

• Batched requests (single request only)
• Dynamic KV cache / Radix Cache prefix sharing
• CUDA graph, Tensor Parallelism

Benchmark (RTX 3090 24G, Qwen3-0.6B draft + Qwen3-4B target, k=5):

Mode	tok/s	Acceptance rate
Speculative (0.6B + 4B)	25.6	2.43/6 (40%)
Baseline (4B only)	67.2	—

Speculative decoding is slower here because the acceptance rate is too low — 0.6B and 4B have too large a distribution gap. In practice, speedup requires ~65%+ acceptance rate, which needs a draft model from the same family with closer parameter count (e.g. 1.5B + 7B), or a purpose-trained draft model (e.g. EAGLE).

# Run speculative decoding
python benchmark/test_speculative.py --speculative-only

# Run target-only baseline
python benchmark/test_speculative.py --baseline-only

# Custom models
python benchmark/test_speculative.py --draft Qwen/Qwen3-0.6B --target Qwen/Qwen3-4B --k 5 --speculative-only

Reference Attention Backends

In addition to the production backends (FlashInfer, FlashAttention, TensorRT-LLM), Mini-SGLang ships two readable reference implementations intended for learning:

• torch backend (python/minisgl/attention/torch_backend.py): Pure PyTorch. Implements paged KV gather, GQA via repeat_interleave, and causal attention with prefix-cache offset in plain Python loops. No external dependencies — easy to step through with a debugger.

• triton backend (python/minisgl/attention/triton_backend.py): Custom Triton decode kernel. Each (request, head) pair runs as an independent GPU program, doing paged KV gather and online softmax (Flash-Attention style) directly in the kernel. Prefill falls back to the torch backend — writing a Triton prefill kernel is essentially reimplementing FlashAttention forward and adds complexity without new insight.

To use either backend:

python -m minisgl --model "Qwen/Qwen3-0.6B" --attention-backend torch --cuda-graph-max-bs 0
python -m minisgl --model "Qwen/Qwen3-0.6B" --attention-backend triton --cuda-graph-max-bs 0

✨ Key Features

• High Performance: Achieves state-of-the-art throughput and latency with advanced optimizations.
• Lightweight & Readable: A clean, modular, and fully type-annotated codebase that is easy to understand and modify.
• Advanced Optimizations:
  • Radix Cache: Reuses KV cache for shared prefixes across requests.
  • Chunked Prefill: Reduces peak memory usage for long-context serving.
  • Overlap Scheduling: Hides CPU scheduling overhead with GPU computation.
  • Tensor Parallelism: Scales inference across multiple GPUs.
  • Optimized Kernels: Integrates FlashAttention and FlashInfer for maximum efficiency.
  • ...

🚀 Quick Start

⚠️ Platform Support: Mini-SGLang currently supports Linux only (x86_64 and aarch64). Windows and macOS are not supported due to dependencies on Linux-specific CUDA kernels (sgl-kernel, flashinfer). We recommend using WSL2 on Windows or Docker for cross-platform compatibility.

1. Environment Setup

We recommend using uv for a fast and reliable installation (note that uv does not conflict with conda).

# Create a virtual environment (Python 3.10+ recommended)
uv venv --python=3.12
source .venv/bin/activate

Prerequisites: Mini-SGLang relies on CUDA kernels that are JIT-compiled. Ensure you have the NVIDIA CUDA Toolkit installed and that its version matches your driver's version. You can check your driver's CUDA capability with nvidia-smi.

2. Installation

Install Mini-SGLang directly from the source:

git clone https://github.com/sgl-project/mini-sglang.git
cd mini-sglang && uv venv --python=3.12 && source .venv/bin/activate
uv pip install -e .

💡 Installing on Windows (WSL2)

Since Mini-SGLang requires Linux-specific dependencies, Windows users should use WSL2:

1. Install WSL2 (if not already installed):

   # In PowerShell (as Administrator)
   wsl --install

2. Install CUDA on WSL2:

   • Follow NVIDIA's WSL2 CUDA guide
   • Ensure your Windows GPU drivers support WSL2

3. Install Mini-SGLang in WSL2:

   # Inside WSL2 terminal
   git clone https://github.com/sgl-project/mini-sglang.git
   cd mini-sglang && uv venv --python=3.12 && source .venv/bin/activate
   uv pip install -e .

4. Access from Windows: The server will be accessible at http://localhost:8000 from Windows browsers and applications.

🐳 Running with Docker

Prerequisites:

• Docker
• NVIDIA Container Toolkit

1. Build the Docker image:

   docker build -t minisgl .

2. Run the server:

   docker run --gpus all -p 1919:1919 \
       minisgl --model Qwen/Qwen3-0.6B --host 0.0.0.0

3. Run in interactive shell mode:

   docker run -it --gpus all \
       minisgl --model Qwen/Qwen3-0.6B --shell

4. Using Docker Volumes for persistent caches (recommended for faster subsequent startups):

   docker run --gpus all -p 1919:1919 \
       -v huggingface_cache:/app/.cache/huggingface \
       -v tvm_cache:/app/.cache/tvm-ffi \
       -v flashinfer_cache:/app/.cache/flashinfer \
       minisgl --model Qwen/Qwen3-0.6B --host 0.0.0.0

3. Online Serving

Launch an OpenAI-compatible API server with a single command.

# Deploy Qwen/Qwen3-0.6B on a single GPU
python -m minisgl --model "Qwen/Qwen3-0.6B"

# Deploy meta-llama/Llama-3.1-70B-Instruct on 4 GPUs with Tensor Parallelism, on port 30000
python -m minisgl --model "meta-llama/Llama-3.1-70B-Instruct" --tp 4 --port 30000

Once the server is running, you can send requests using standard tools like curl or any OpenAI-compatible client.

4. Interactive Shell

Chat with your model directly in the terminal by adding the --shell flag.

python -m minisgl --model "Qwen/Qwen3-0.6B" --shell

You can also use /reset to clear the chat history.

Benchmark

Offline inference

See bench.py for more details. Set MINISGL_DISABLE_OVERLAP_SCHEDULING=1 for ablation study on overlap scheduling.

Test Configuration:

• Hardware: 1xH200 GPU.
• Model: Qwen3-0.6B, Qwen3-14B
• Total Requests: 256 sequences
• Input Length: Randomly sampled between 100-1024 tokens
• Output Length: Randomly sampled between 100-1024 tokens

Online inference

See benchmark_qwen.py for more details.

Test Configuration:

• Hardware: 4xH200 GPU, connected by NVLink.
• Model: Qwen3-32B
• Dataset: Qwen trace, replaying first 1000 requests.

Launch command:

# Mini-SGLang
python -m minisgl --model "Qwen/Qwen3-32B" --tp 4 --cache naive

# SGLang
python3 -m sglang.launch_server --model "Qwen/Qwen3-32B" --tp 4 \
    --disable-radix --port 1919 --decode-attention flashinfer

Note: If you encounter network issues when downloading models from HuggingFace, try using --model-source modelscope to download from ModelScope instead:

python -m minisgl --model "Qwen/Qwen3-32B" --tp 4 --model-source modelscope

📚 Learn More

• Detailed Features: Explore all available features and command-line arguments.
• System Architecture: Dive deep into the design and data flow of Mini-SGLang.