colibrì-nano

Run Colibri-Nano locally — a 2.7B-parameter Mixture-of-Experts with ~0.34B active per token, in ~4000 lines of C with no runtime dependencies. No Python at inference time, no GPU, no framework: one gcc invocation produces a binary that loads a ~1.4 GB int4 container and chats.

git clone https://github.com/view321/nanocolibri-engine && cd nanocolibri-engine/c
./setup.sh                                                    # build + self-test (32/32)
pip install numpy huggingface_hub
./coli convert --repo vovaRL/NanoColibri-Instruct --model ./nano_i4
./coli chat --model ./nano_i4

The engine is a slimmed colibrì derivative: same disk-streaming MoE design that runs 300B-class models off an NVMe, aimed at the other end of the scale. At Nano's size streaming is no longer a necessity — the whole container fits in RAM on any machine — so what you get is a very small, very portable runtime.

model	Colibri-Nano — HYV3ForCausalLM, model_type: hy_v3
total / active params	2.7B / 0.34B (top-2 of 64 experts + a 2048-wide shared expert)
layers / hidden	24 (1 dense + 23 MoE) / 1024, GQA 16Q/4KV, per-head QK-norm
container on disk	~1.4 GB (int4 experts, int8 dense)
always resident	~0.3 GB
needs	a C compiler. AVX2 helps; a GPU is optional and rarely worth it here

Trained from scratch by NanoColibri — that repo is the training half, this one is the runtime half.

The model

repo	what
vovaRL/NanoColibri	base — 20 000 steps, 5.24B tokens of fineweb-edu. Completes text; does not chat.
vovaRL/NanoColibri-Instruct	chat — the base plus a 1 500-step SFT leg on smol-smoltalk. Use this one for coli chat.

Each carries coli_i4/, a ready-made int4 container — download that instead of converting:

hf download vovaRL/NanoColibri-Instruct --include 'coli_i4/*' --local-dir ./nano
./coli chat --model ./nano/coli_i4

If coli_i4/ is not there yet, convert the bf16 weights yourself (next section) — it takes a couple of minutes.

The container is not GGUF / AWQ / GPTQ. It only loads in this engine.

Converting a checkpoint yourself

tools/convert_nano.py turns any HuggingFace hy_v3 checkpoint into a container. It needs numpy only (huggingface_hub too if you pass --repo) — safetensors is read and written directly and bf16 is widened without torch.

./coli convert --repo vovaRL/NanoColibri-Instruct --model ./nano_i4   # from the Hub
./coli convert --repo ./checkpoints/nano_chat    --model ./nano_i4    # from a local dir

# or call the tool directly, e.g. to keep the dense stack at int4 as well
python3 tools/convert_nano.py --indir checkpoints/nano_chat --outdir nano_i4 \
    --ebits 4 --dbits 8 --io-bits 8

Layout: out-00000.safetensors holds the always-resident dense stack (embeddings, attention, norms, routers, shared experts); out-000{L+1}.safetensors holds layer L's 64 routed experts, so the engine can stream one layer's experts at a time. Quantization is per-row symmetric with f32 scales in a sibling .qs tensor; norms, router weights and the router bias stay f32.

The training repo's python -m scripts.export_coli writes the identical format — use whichever is closer to hand.

Chat format

Nano was SFT'd on the SmolLM2-Instruct ChatML template, and the engine picks the wire format by probing the tokenizer's added-token list rather than trusting model_type: if <|im_start|> is present it renders ChatML, byte-identically to what the SFT leg trained on, and registers <|im_end|> as a stop token (the exported config.json names only <|endoftext|>, which the chat data essentially never contains). Override the default system turn with SYSTEM_PROMPT.

Point the engine at the base model instead and use coli run --raw, which feeds the prompt verbatim (CHAT_TEMPLATE=0): a base checkpoint is a continuation model, and wrapping it in chat markers it never saw costs quality for nothing.

./coli run --raw --model ./nano_base_i4 "Photosynthesis is the process by which plants"

Running it

./coli chat  --model ./nano_i4                 # interactive
./coli run   --model ./nano_i4 "Explain photosynthesis in two sentences."
./coli serve --model ./nano_i4 --port 8000     # OpenAI-compatible API, model id: colibri-nano
./coli info  --model ./nano_i4                 # shapes, container size, free RAM
./coli plan  --model ./nano_i4 --ram 4         # what the engine would do with that budget
./coli doctor --model ./nano_i4                # diagnostics; --json for CI

coli serve speaks /v1/chat/completions, /v1/completions and /v1/models, streaming included, so any OpenAI client works:

curl localhost:8000/v1/chat/completions -H 'Content-Type: application/json' \
  -d '{"model":"colibri-nano","messages":[{"role":"user","content":"hi"}]}'

Knobs

Everything is an environment variable; the common ones have flags.

Flag / env	Effect
--ram N	Engine memory budget in GB. Nano's container is ~1.4 GB, so 2–4 pins essentially everything. 0 (default) = auto.
--ctx N	Max context, default 4096 (the model's trained window). Lower it to shrink the KV cache.
--temp T	0 = greedy, default 0.7. --topp / --topk for nucleus / fixed-k sampling.
--ngen N	Response-length cap (default 1024). The real end is the stop token; :more continues.
SYSTEM_PROMPT=…	Replace the default ChatML system turn.
PERF=1	Every 100 tokens, print attn/disk/expert_mm/head percentages on stderr.
IDOT=0	Disable the int8-activation integer matmul kernels (on by default: avx512-vnni / avx-vnni / avx2 / neon). Selects the exact f32 dequant path.
KV_I8=1	int8 KV cache — smaller, marginal here at 4k context.
PIPE=1 / PIPE=2	Async expert prefetch via a thread pool / io_uring (make hy3 IOURING=1). Matters for containers larger than RAM.
PREDICT=N	Cross-layer expert prediction, default 2 layers ahead: future routers run on the current residual and readahead-hint their experts. Hint-only — output is bit-identical. Mostly a no-op for Nano (nothing is cold).
AUTOPIN=0, PIN_GB=N, REPIN=N	Control the learned RAM pin built from .coli_usage.
--gpu 0 --vram N	Use CUDA (requires make hy3 CUDA=1). See below.
--verbose	Pass engine stderr through, or COLI_VERBOSE=1.

Build with ARCH=native to unlock VNNI int8 kernels on CPUs that have them (Zen 4+, Sapphire Rapids+, Alder Lake+); the default x86-64-v3 build is portable AVX2.

Speed, honestly

The only numbers measured so far are on a rented 27-vCPU Linux server, which outmuscles every device this model targets — treat them as an engine characterization, not as laptop speeds:

engine budget	tok/s
0.25–4 GB (streaming)	35–40
8 GB (everything pinned)	~59

Speed is flat below the container size and jumps ~1.6× once everything is pinned, which puts the engine's streaming bookkeeping at ~35–40% overhead. No target-device numbers exist yet — no MacBook, gamer-laptop or mini-PC sweep has been run. Disk streaming is also effectively unmeasured at this size: the container is small enough that the OS page cache holds it on any machine with real RAM. Details and the failed cgroup-capped attempt are in the training repo's docs/RESULTS.md.

Optional CUDA

make hy3 CUDA=1                  # Linux, needs nvcc
COLI_MODEL=./nano_i4 ./coli chat --gpu 0 --vram 4

The GPU is a cache + compute tier for routed experts (disk → RAM → VRAM). That design pays off when experts do not fit in RAM; at Nano's 1.4 GB it usually does not, and the CPU path is competitive. On Windows, build the backend as a runtime DLL instead: make CUDA_DLL=1 cuda-dll from an MSVC-enabled shell (see backend_loader.c).

Quality

Zero-shot, measured on the bf16 chat checkpoint with an lm-evaluation-harness-compatible harness (full tables in the training repo's docs/RESULTS.md):

model	active params	tokens	lambada	piqa	wino	arc-e	arc-c	obqa	hswag
Colibri-Nano (chat)	341M	5.4B	26.3	62.7	49.2	43.4	22.8	22.0	31.1
Pythia-410M @ step3000	405M	6.3B	26.3	59.8	50.9	41.3	18.8	15.6	27.0

At ~5.4B tokens Nano performs like a good dense model in the 300–600M class trained on the same budget, with fewer active parameters. It is a small model trained on a small budget: expect short, simple answers and frequent factual errors. The int4 container adds quantization noise on top of that; the quantization itself is exactness-tested at the kernel level (tests/test_idot_hy3.c) but its end-to-end quality cost on Nano has not been benchmarked.

Build and test

cd c
make hy3                 # or: make portable   (ARCH=x86-64-v3, redistributable)
make oracle              # teacher-forcing self-test on the tiny fixture: 32/32
make test                # C kernel tests + Python tests for coli / server / doctor

hy3_tiny/ + ref_hy3.json are a committed 5-layer random hy_v3 checkpoint (~3 MB) and the HF reference run's token ids and logits. They validate the engine without downloading anything; regenerate with python3 tools/make_hy3_oracle.py (needs torch + transformers>=5.14).

Converter round-trip on the same fixture:

python3 tools/convert_nano.py --indir hy3_tiny --outdir hy3_tiny_i4
SNAP=./hy3_tiny_i4 TF=1 IDOT=0 ./hy3 64 4 8

Expect ~22/32 there, not 32/32, and a non-zero exit: 4-bit quantization of a random 5-layer model flips argmaxes. It exercises the converter and the int4 load path, nothing more — what would signal a real regression is missing tensor or no oracle line at all.

What was left out

This is a slim, single-target cut of a larger engine. Dropped: the GLM-5.2 and OLMoE engines, the Metal backend, the Tauri desktop app and web UI, the benchmark harness, and the checkpoint surgery tooling (expert pruning, int2 cold tails, co-activation disk reordering) — all of which exist to make 300B-class models fit, and none of which Nano needs. hy3.c itself is unchanged, so Hunyuan-Hy3 and Qwen3.6 containers still load.

Provenance & license

• Engine: fork of colibrì (MIT), via the Hy3 port.
• Model: Colibri-Nano, trained from scratch with NanoColibri (Apache-2.0).
• This repository: Apache-2.0 (see LICENSE).