inf: one model, three inference engines, one machine

This repo benchmarks Qwen2.5 on an Apple M4 Pro with 24 GB of unified memory. It compares three common inference stacks:

Engine	Why it is here
Hugging Face transformers on MPS, fp16	The simple baseline most people start with
llama.cpp via Metal, GGUF q8_0 and q4_K_M	Hand-tuned C++ and Metal kernels with quantized weights
MLX-LM, bf16 and 4-bit	Apple's framework built for Apple Silicon unified memory

Each engine runs the same workload: 5 prompts, greedy decoding, and up to 200 new tokens. A shared harness in benchmarks/common.py measures:

• TTFT: time to first token, mostly prefill latency
• Decode tokens/sec: streaming speed after the first token
• Peak memory: process RSS plus engine-reported GPU memory when available

There is also a Colab notebook for a separate vLLM batching experiment on a T4 GPU.

Results

Environment: M4 Pro 24 GB, macOS 27, greedy decoding, 200 max new tokens, mean over 5 prompts after 1 warmup run.

Qwen2.5-0.5B-Instruct

Engine	Weights	Decode tok/s	TTFT (ms)	Peak mem (GB)
transformers (MPS)	fp16	52.3	46	1.92
llama.cpp (Metal)	q8_0	140.8	19	1.40
llama.cpp (Metal)	q4_K_M	179.5	16	1.10
MLX-LM	bf16	204.3	75	1.41
MLX-LM	4-bit	422.8	60	0.74

Qwen2.5-7B-Instruct

Engine	Weights	Decode tok/s	TTFT (ms)	Peak mem (GB)
transformers (MPS)	fp16	10.8	248	15.18
llama.cpp (Metal)	q4_K_M	42.3	94	4.53
MLX-LM	4-bit	57.5	228	4.16

The 7B fp16 run barely fits on a 24 GB machine. macOS moved about 5 GB into swap during loading. The 4-bit runs use about 4 GB, which leaves much more room for context, other apps, or larger experiments.

What the benchmark shows

MLX is much faster than transformers on the 0.5B model even when both use 16-bit weights. That points to engine overhead, not quantization. Small models do lots of small GPU operations, and Python plus PyTorch MPS launch overhead becomes a major part of each generated token. MLX and llama.cpp use more fused, precompiled execution paths.

For 7B, memory bandwidth becomes the main limit. During decode, each generated token has to stream the model weights from memory. The M4 Pro has about 273 GB/s of memory bandwidth. A 15 GB fp16 model has a rough ceiling near 18 tokens/sec, and transformers reaches 10.8. A 4.2 GB 4-bit model has a rough ceiling near 65 tokens/sec, and MLX reaches 57.5.

Quantization helps speed, but memory is where it helps most. In this benchmark, 4-bit weights are about 2x faster than 16-bit in MLX, not 4x faster. The reason is that activations, KV cache, dequantization, and per-token overhead still cost time. The memory reduction is much closer to the full 4x.

TTFT and decode speed measure different parts of generation. TTFT is the prompt prefill pass. Decode speed is the one-token-at-a-time loop after that. Keeping those measurements separate makes the results easier to explain.

One important measurement fix: process RSS alone is misleading on Apple Silicon. PyTorch MPS and MLX can allocate Metal GPU buffers outside the process resident set, so a 15 GB model once reported only 0.24 GB of memory. The harness now asks each engine for memory when possible and takes the max with RSS.

Reproduce

Install dependencies:

uv sync

Run the 0.5B benchmarks:

uv run python benchmarks/bench_transformers.py
uv run python benchmarks/bench_llamacpp.py
uv run python benchmarks/bench_mlx.py

Run the 7B benchmarks:

uv run python benchmarks/bench_transformers.py Qwen/Qwen2.5-7B-Instruct
uv run python benchmarks/bench_llamacpp.py q4_k_m 7B
uv run python benchmarks/bench_mlx.py 4bit 7B

Regenerate charts:

uv run python plots/plot_results.py 0.5B
uv run python plots/plot_results.py 7B

On Nix-based macOS setups, build llama-cpp-python with Apple clang:

CC=/usr/bin/clang CXX=/usr/bin/clang++ uv sync

vLLM notebook

The laptop benchmarks are single-stream results. vLLM is designed for many concurrent requests, so the batching test lives in colab/vllm_benchmark.ipynb. Run it on a free Colab T4 to compare raw transformers and vLLM at batch sizes 1, 8, and 32.

Files

• benchmarks/common.py: shared timing and memory harness
• benchmarks/bench_transformers.py: Hugging Face MPS baseline
• benchmarks/bench_llamacpp.py: llama.cpp Metal runs
• benchmarks/bench_mlx.py: MLX-LM runs
• plots/plot_results.py: chart generation
• GUIDE.md: deeper explanation of the design choices and debugging notes