sasori

Turn any full-precision LLM into a multi-plane ternary model that runs GPU-free on a commodity CPU — no retraining, no calibration data.

sasori post-hoc quantizes the weights of a dense or MoE language model to K ternary trit-planes (w ∈ {-1,0,+1}), writes a GGUF, and ships a small llama.cpp/Ollama kernel so the model runs with additions instead of weight multiplies. It is a torch-only tool; the whole pipeline runs on CPU, and quantization parallelizes across cheap GPUs.

What it's actually good at (and what it isn't)

The point is access, not speed. sasori fills one specific gap: taking an arbitrary open FP checkpoint and making it a high-quality, GPU-free, retrain-free ternary model. That is something native-only tooling (bitnet.cpp) and single-plane ternary types (TQ1_0/TQ2_0, which collapse quality) cannot produce after the fact.

Being honest about the trade: at the reasoning-preserving operating point, sasori is heavier and slower than a 4-bit integer GGUF (Q4_K_M ≈ 4.8 bpw is smaller and faster). Decode is memory-bound, and K=3 reads more bytes than 4-bit. The edge is the characteristic — any model, no GPU, no retrain, multiplication-free — not a footprint or throughput record. If a 4-bit integer quant fits your deployment, use it.

Install

pip install -e .          # torch-only; extras: .[gguf] for GGUF output, .[test] for tests

Quickstart — quantize

python -m sasori quantize <hf-model-dir | model.f16.gguf> -o model.tq3p.gguf --K 3

An HF directory also needs --llama-cpp <path> (reused for the arch + tokenizer conversion). A .gguf input skips conversion and goes straight to quantization.

K and g — the quality dials

K = number of trit-planes (coarse dial); g = scale group-size (fine dial — smaller g means more per-group f16 scales, hence more bits and a closer fit). Both run GPU-free: --K <1..4> --group <32|64|128|256> (default --K 3 --group 256). The free per-matrix reconstruction-gain signal flags which layers benefit most.

K	type	bpw (g=256)	when
1	TQ1P	2.06	skip it — one plane is too coarse for post-hoc quant; output degrades badly. Deployable, but for experiments only.
2	TQ2P	4.125	edge / knowledge / chat; collapses multi-step reasoning at g=256 (GSM8K 3–77% of FP by model scale) — finer g rescues it on large models (see below)
3	TQ3P	6.1875	the default and sweet spot — ≈ full precision on reasoning, code, agentic use
4	TQ4P	8.25	saturating — the extra plane over K=3 gives diminishing returns (only the hardest benchmarks, e.g. competition math, still close at K=4). Above ~6.6 bpw an 8-bit integer quant is usually the better call; prefer K=3 or a stronger base model.

g (group-size) — mostly a K=2 knob

K=3 recovers ≈FP at any g, so keep g=256 there — finer g just adds bits. K=2 is very g-sensitive, and finer g rescues it, but fully only on large models. Measured GSM8K retention (score / FP; lm-eval N=100 fake-quant, which the kernel reproduces bit-exactly, so these are the deployed numbers):

K=2 GSM8K retention	g=256 (4.1 bpw)	g=64 (4.5 bpw)	g=32 (5.0 bpw)
Qwen3-1.7B	3%	4%	19%
Qwen2.5-7B	20%	69%	84%
Qwen3-14B	29%	91%	100%
Qwen3-30B-A3B	77%	89%	99%

At matched bpw (K2,g32 = 5.0 bpw vs K3,g256 = 6.19), K2 + fine g wins on ≥14B (14B 100% vs 83%, 30B 99% vs 94% — with fewer bits) but loses on smaller models (use K=3 there). Cost: finer g decodes slower (more sub-scales per block). K=3/K=4 stay ≈flat across g — the fine dial only moves K=2.

Not sure which K? suggest-k

python -m sasori suggest-k <hf-model-dir> --task math   # prints a K suggestion; does not quantize

Probes late-layer activation SQNR and prints a calibrated K guess. It is a heuristic, not a guarantee — SQNR is a proxy for (not a measurement of) downstream accuracy, the calibration is study-fit, MoE experts aren't probed, and it suggests K only (not g). Use it as a starting point, then verify on your task; when in doubt, K=3.

Deploy — GPU-free CPU inference

The TQ{K}P GGUF uses custom ggml type ids, so it needs a kernel-patched llama.cpp/Ollama. The kernel ships as a self-contained patch against pinned llama.cpp@b9509; the end-to-end build+run guide is llama-cpp/MANUAL.md.

git -C <llama.cpp@b9509> apply <sasori>/llama-cpp/tq-llamacpp-b9509.patch
cmake -B build <llama.cpp> && cmake --build build -j
./build/bin/llama-cli -m model.tq3p.gguf -ngl 0 -p "..."

Without the patch the TQ{K}P GGUF won't load; you can instead export a standard-type GGUF of the reconstructed weights (keeps the quality, not the sub-4-bit size).

Measured results (honest)

• Qwen3-30B-A3B (MoE) → TQ3P, 24.4 GB, sha256-verified — runs on a pure-CPU box at ~14 tok/s, no GPU. HumanEval 0.86 ≈ FP 0.88 at K=3, vs 0.46 at K=2 → K is the quality lever. This 30B line is a single pilot run (HumanEval N=50); its throughput and HumanEval numbers are a demonstration, not a committed benchmark — no log is shipped for them.
• Qwen3-1.7B on CPU: TQ3P decode 12.6 tok/s vs F16 6.7 (~1.9×) (fewer bytes than F16), ppl 25.3 vs 20.1.

Numbers are single-run and hardware-dependent; treat them as order-of-magnitude, not benchmarks.

Capability retention

K=3 recovers ≈ full precision across the board; K=2 trades footprint for reasoning and long-context — knowledge, code and instruction hold up better. Retention = score(K) / score(FP), averaged over the k-plane study's open checkpoints:

Retention at g=256 (the K=2 column is what finer g lifts — see above):

capability	K=2	K=3
Knowledge (MMLU / ARC)	51%	97%
Instruction-following (IFEval, "agentic")	largely holds	≈ FP
Code (HumanEval)	partial (~40–95%, model-dep.)	≈ FP
Math (GSM8K)	11% — a cliff	90%
Competition math (MATH-500)	≤ 31%	≈ FP
Long-context retrieval (NIAH @ 32k)	33%	100%

The K=2 drops are dominated by small models; larger models tolerate K=2 better and finer g closes the gap (GSM8K retention reaches ~100% on 14B/30B at g=32 — see the K/g section). K=3 is the no-caveat operating point at any g. Full per-model, per-scale grid in the k-plane paper.

How it works & paper

Each weight matrix (and MoE expert) is fit independently into a sum of K per-group-scaled trit-planes — layer-local, weight-only, no calibration or forward pass — which is why it streams and parallelizes. The method, the capability-vs-cost analysis, and the systems details are in:

• sasori: No-Retrain Multi-Plane Ternarization for GPU-Free LLM Inference — 10.5281/zenodo.21247534

Background on the K lever and reasoning recovery: k-planes · distillation.

Tests

python -m pytest tests -q        # CPU, no GPU, free

License

MIT — see LICENSE. The llama-cpp/ kernel patches derive from llama.cpp (MIT); its copyright is preserved.

---

Footnote. Tried MoE expert-offload and ternary-draft speculative decoding for low-RAM CPU inference — both negative-to-marginal (native mmap already wins; spec-decode only ~1.1–2.2×, no rescue for big dense models). Picking K/g to fit RAM is what works.