From ec087f6bcaabd397edffc1354ae16937b56d51dd Mon Sep 17 00:00:00 2001 From: JustVugg Date: Mon, 20 Jul 2026 17:36:06 +0200 Subject: [PATCH 1/5] docs: bring the Quick Start guide (incl. Windows prebuilt steps) to main MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit main shipped v1.0.0 with prebuilt Windows binaries but no docs on main explaining what the .exe is or how to run it (#450) — the quickstart lived only on dev. Bring docs/quickstart.md to main and add a Windows-prebuilt pointer to the README's Get started section, so users on the default branch find the steps. Docs-only. Refs #450 Co-Authored-By: Claude Fable 5 --- README.md | 9 +++ docs/quickstart.md | 184 +++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 193 insertions(+) create mode 100644 docs/quickstart.md diff --git a/README.md b/README.md index 797f1ca15..085e4e926 100644 --- a/README.md +++ b/README.md @@ -151,6 +151,15 @@ scale-granularity/rotation ablations live in ## Get started +> **New here, or on Windows?** The [Quick Start guide](docs/quickstart.md) walks +> through install → build → model → first chat step by step for Linux, Windows, +> and macOS. On **Windows** you don't even need to build: download the +> `colibri--windows-x86_64.zip` from +> [Releases](https://github.com/JustVugg/colibri/releases), unzip it, rename +> `colibri-*-windows-x86_64.exe` → `glm.exe`, install +> [Python 3](https://www.python.org/downloads/), and run `coli chat` — full +> details in the [Windows section](docs/quickstart.md#windows). + ### 1. Get the model A pre-converted **GLM-5.2 int4** container is on Hugging Face — **use the diff --git a/docs/quickstart.md b/docs/quickstart.md new file mode 100644 index 000000000..60d48777a --- /dev/null +++ b/docs/quickstart.md @@ -0,0 +1,184 @@ +# Quick Start — from zero to a running model + +A step-by-step guide for first-time users on **Linux**, **Windows**, and **macOS**. +No prior experience with C, CUDA, or model conversion is assumed. If you get +stuck, `./coli doctor` (below) tells you exactly what's missing. + +> **What you're setting up:** colibrì runs a very large Mixture-of-Experts model +> (e.g. GLM-5.2, 744B parameters) on a normal machine by streaming the model's +> experts from disk instead of needing them all in RAM. The engine is a single +> C program; Python is only used once, to prepare the model files. + +--- + +## 0. What you need first (prerequisites) + +| | Minimum | Recommended | +|---|---|---| +| **RAM** | ~16 GB | 24 GB+ | +| **Free disk** | ~380 GB for the int4 model | a fast NVMe SSD (streaming speed = your token speed) | +| **OS** | Linux, Windows 10/11, or macOS | any | +| **Tools** | a C compiler + `make` + `git` + `python3` | — | + +You do **not** need a GPU. A GPU only helps if you have one; the engine runs +CPU-only by default. + +--- + +## 1. Install the build tools + +### Linux (Ubuntu / Debian) + +```bash +sudo apt update +sudo apt install -y build-essential git python3 +``` + +`build-essential` gives you `gcc`, `make`, and OpenMP (libgomp) — everything the +engine needs. + +### Windows + +You have two options. + +**Option A — download a prebuilt binary (no compiler needed).** +Grab `colibri--windows-x86_64.zip` from the +[Releases page](https://github.com/JustVugg/colibri/releases) and unzip it. +Inside you'll find: + +| File | What it is | +|---|---| +| `colibri--windows-x86_64.exe` | **the engine** — the C program that actually runs the model | +| `coli` | the command-line launcher (`chat`, `serve`, `convert`, `doctor`, …) | +| `openai_server.py`, `resource_plan.py`, `doctor.py` | Python support for the API server and placement planner | + +Two setup steps: + +1. **Rename the engine to `glm.exe`** so the launcher can find it (it looks for a + binary named `glm`): + ```powershell + Rename-Item colibri-*-windows-x86_64.exe glm.exe + ``` +2. **Install Python 3** from [python.org](https://www.python.org/downloads/) — the + `coli` launcher and the API gateway are Python scripts (the engine itself is + pure C and needs nothing). + +Then continue to [step 3](#3-get-the-model). Prefer to skip the launcher? You can +run the engine directly — `.\glm.exe` reads the model path from the `SNAP` +environment variable (see [docs/windows.md](windows.md)) — but `coli chat` is the +easy path. + +**Option B — build from source with MSYS2.** +Install [MSYS2](https://www.msys2.org/), open the **UCRT64** shell, and run: + +```bash +pacman -S --needed mingw-w64-ucrt-x86_64-gcc make git python +``` + +### macOS + +```bash +xcode-select --install # C compiler (clang) +brew install libomp git python # OpenMP for multithreading +``` + +--- + +## 2. Get the code and build the engine + +```bash +git clone https://github.com/JustVugg/colibri.git +cd colibri/c +./setup.sh +``` + +`setup.sh` checks your compiler and OpenMP, builds the engine, and runs a tiny +self-test. When it prints: + +``` +engine self-test: 32/32 (expected 32/32) +``` + +the engine is working correctly. (On Windows Option A you already have the +binary — you can skip this step.) + +--- + +## 3. Get the model + +You have two paths. + +### Easiest — download a ready-made int4 container + +A pre-converted **GLM-5.2 int4** model is on Hugging Face. **Use the version +with the int8 MTP heads** (the plain int4 heads disable speculative decoding — +see [#8](https://github.com/JustVugg/colibri/issues/8)): + +**https://huggingface.co/mateogrgic/GLM-5.2-colibri-int4-with-int8-mtp** + +Download it into a folder on a fast disk, e.g. `/nvme/glm52_i4` (Linux/macOS) or +`D:\glm52_i4` (Windows). It is about **372 GB**, so make sure you have the space. + +### Or convert it yourself from the FP8 source + +One resumable command downloads and converts the model shard by shard, so it +never needs the full ~756 GB on disk at once: + +```bash +./coli convert --model /nvme/glm52_i4 +``` + +This step uses Python and runs only once. Safe to interrupt and re-run — it +resumes where it left off. + +--- + +## 4. Run it + +Point `COLI_MODEL` at the folder from step 3 and start chatting: + +```bash +# Linux / macOS +COLI_MODEL=/nvme/glm52_i4 ./coli chat + +# Windows (UCRT64 shell) +COLI_MODEL=/d/glm52_i4 ./coli chat +``` + +Useful first commands: + +```bash +COLI_MODEL=/nvme/glm52_i4 ./coli doctor # read-only check: is everything ready? +COLI_MODEL=/nvme/glm52_i4 ./coli plan # shows where the model will live (RAM/disk/GPU) +COLI_MODEL=/nvme/glm52_i4 ./coli chat --topp 0.85 # faster: reads less from disk, same quality +``` + +> **Tip:** `--topp 0.85` is worth adding on a disk-bound machine — it reads +> fewer expert bytes per token with no quality loss, which directly means more +> tokens per second. + +--- + +## 5. What to expect + +- **First launch loads the resident weights** (~10 GB) — this takes a moment. +- **Speed depends on your disk.** The experts stream from storage, so a fast + NVMe SSD is the single biggest factor in tokens/second. On a slow or shared + disk, generation can be well under 1 token/second — that's expected, and it's + the honest cost of running a 744B model on a small machine. +- **It's still the full model.** Placement only changes speed, never the model's + answers or precision. + +If something doesn't work, run `./coli doctor` — it reports exactly what's +missing (compiler, model files, permissions) and how to fix it. + +--- + +## Where to go next + +| Topic | Doc | +|---|---| +| Windows native build (and CUDA DLL) | [docs/windows.md](windows.md) | +| Tuning: cache, prefetch, speculation | [docs/tuning.md](tuning.md) | +| OpenAI-compatible API + web dashboard | [docs/api.md](api.md) | +| Every environment variable | [docs/ENVIRONMENT.md](ENVIRONMENT.md) | From c6801451bdf1a9ad8b88043ac2963b80b8bca286 Mon Sep 17 00:00:00 2001 From: JustVugg Date: Mon, 20 Jul 2026 17:43:46 +0200 Subject: [PATCH 2/5] site: bring the official website + Pages deploy workflow to main (#424) The site.yml workflow deploys on push to main, so the site only goes live once site/ is on main. Bring site/index.html + icons + the workflow so a GitHub Pages deploy (Settings -> Pages -> Source: GitHub Actions) publishes it at justvugg.github.io/colibri. Refs #424 Co-Authored-By: Claude Fable 5 --- .github/workflows/site.yml | 36 ++ site/colibri-icon.svg | 50 +++ site/colibri.svg | 55 +++ site/index.html | 698 +++++++++++++++++++++++++++++++++++++ 4 files changed, 839 insertions(+) create mode 100644 .github/workflows/site.yml create mode 100644 site/colibri-icon.svg create mode 100644 site/colibri.svg create mode 100644 site/index.html diff --git a/.github/workflows/site.yml b/.github/workflows/site.yml new file mode 100644 index 000000000..160cd6166 --- /dev/null +++ b/.github/workflows/site.yml @@ -0,0 +1,36 @@ +name: Deploy website + +# Publishes site/ to GitHub Pages. One-time repo setup: +# Settings → Pages → Build and deployment → Source: "GitHub Actions". +# Custom domain later: add site/CNAME with the bare domain, point DNS +# (A/AAAA to GitHub Pages IPs or CNAME to .github.io), done. + +on: + push: + branches: [main] + paths: ['site/**', '.github/workflows/site.yml'] + workflow_dispatch: + +permissions: + contents: read + pages: write + id-token: write + +concurrency: + group: pages + cancel-in-progress: true + +jobs: + deploy: + runs-on: ubuntu-latest + environment: + name: github-pages + url: ${{ steps.deployment.outputs.page_url }} + steps: + - uses: actions/checkout@v4 + - uses: actions/configure-pages@v5 + - uses: actions/upload-pages-artifact@v3 + with: + path: site + - id: deployment + uses: actions/deploy-pages@v4 diff --git a/site/colibri-icon.svg b/site/colibri-icon.svg new file mode 100644 index 000000000..76838b903 --- /dev/null +++ b/site/colibri-icon.svg @@ -0,0 +1,50 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/site/colibri.svg b/site/colibri.svg new file mode 100644 index 000000000..4aed0bb3a --- /dev/null +++ b/site/colibri.svg @@ -0,0 +1,55 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + colibrì + tiny engine, immense model + GLM-5.2 · 744B MoE · int4 · streaming CPU + diff --git a/site/index.html b/site/index.html new file mode 100644 index 000000000..dd6dac1ea --- /dev/null +++ b/site/index.html @@ -0,0 +1,698 @@ + + + + + +colibrì — tiny engine, immense model + + + + + + + + + + + + +
+ +
tiny engine · immense model
+

colibrì

+
a 744-billion-parameter mind, on your desk
+

Run GLM-5.2 — 744B Mixture-of-Experts — on hardware you already own. + Pure C, zero dependencies, experts staged just-in-time across VRAM, RAM and disk. + The galaxy behind this text is real: the model's measured expert atlas.

+
+ ★ Star on GitHub + Quick start → +
+
+
744B
parameters
+
19,456
experts
+
0
dependencies
+
25 GB
min RAM
+
9.2
tok/s measured
+
+
scroll
+
+ +
+ +
+
the vision
+

Frontier models should not be sealed inside datacenters.
+ Anyone curious enough should be able to open one up —
+ run it, watch every expert fire, and make it better.

+
+
⚡

Run it

+

A 744B frontier-class MoE answering on a 25 GB machine — the proven floor — up to + 9+ tok/s on a workstation. No cluster, no cloud bill, no API key.

+
🔬

Study it

+

The engine is a microscope: live routing telemetry, per-expert heat, and the first + published expert atlas of a 700B-class model — measured on machines like yours.

+
🛠

Improve it

+

One C file, readable in an afternoon. Every optimisation this project found came from + someone measuring something — the next one can come from you.

+
+
+ +
+
01 — the algorithm
+

A JIT, but for weights

+
+
+

A compiler JIT never compiles the whole program — it watches what actually runs + and compiles the hot paths, just in time. colibrì applies the same bet to a + 744B parameter space: a token activates only ~40B parameters, and just + ~11 GB of those change from token to token — the routed experts.

+
So parameters are not resident state to be held — they are + data to be staged, exactly when the router proves they are needed.
+

The engine keeps the dense weights resident and treats the 19,456 experts as a + heterogeneous storage hierarchy: measured routing heat decides which experts earn + VRAM, which earn pinned RAM, and which stream from NVMe. The router runs a layer + ahead, so prefetch hides the staging latency — and because routing has + measurable structure (the atlas below), the hierarchy keeps getting the answer right. + Like a JIT, it learns your workload: the more you run, the hotter the right experts get.

+

Everything is validated token-exact against the reference transformers + implementation — speed never buys drift.

+
⚗ Active research: the next algorithmic step is under way — + smarter placement and scheduling beyond LRU + pin, overlap of CPU and GPU expert execution, + and routing-aware speculation. Measured, reviewed, and merged in the open.
+
+
+
+ VRAMhottest experts · grouped GPU matmuls · + six 5090s hold the entire routed set — disk reads hit zero
+
↑ promote by measured heat ↓ demote
+
+ RAMwarm set · pinned & wired, learned from + your usage · AVX-512/NEON int4 kernels + LRU behind it
+
↑ promote ↓ demote
+
+ NVMecold tail · streamed on demand with io_uring, + prefetched a layer ahead — how 744B fits a 25 GB machine at all
+
token → router picks 8 of 256 experts × 75 layers → + gather across tiers → int4 matmul → next token
+
+
+
+ +
+
02 — the engine, live
+

Watch it think

+

A replay of the engine at work, paced to measured decode speeds from real + community hardware. The grid is all 19,456 experts (colour = storage tier, brightness = + routing heat, routed = white flash); the galaxy below it is the measured atlas — + watch different questions light different regions of the mind.

+
+
+
+
+ coli chat —
+
+
+
+
+ the brain — 75 MoE layers × 256 experts + MTP
+
+ +
+
0.0 tok/s
decode
+
—
ttft (measured)
+
—
resident hit
+
+
+ VRAM tier + RAM tier + disk tier + routed now +
+
+
+
+
+ the atlas, live — measured specialists routed by the current topic
+
+ +
routing: —
+
+
+
+

Simulation replays a fixed transcript at each profile's measured decode rate + (#82, #389, + #387, #273); + “—” means that machine's report did not publish the number — nothing here is invented. + Atlas flashes sample real measured affinity (atlas v1) for the active topic.

+
+ +
+
+
03 — the atlas
+

Measured, not assumed

+

Every point below is a real measured expert from the published + canonical atlas v1: + 721 canonical specialists confirmed across three independent kernel families + (AVX-512 VNNI · NEON · AVX2, engine-pinned, replication-gated) plus 637 gate-sensitive ones. + Position is the measured 10-topic affinity vector — a specialist sits at its topic's anchor, + a generalist drifts to the centre. Colour = top topic, large points = canonical. Drag to spin.

+
+
+ +
+
drag to spin · scroll past to release
+
+
+
+ +
+
04 — the ladder
+

Same engine, any hardware

+

Measured decode on real community machines — the hardware only changes + where the experts live. Every row links to its public report.

+ + + + + + + + + + + + + + + + + + + + +
hardwaredecodewhere the experts live
6× RTX 5090 · full residency + selective NUMA (#82)9.0–9.2 tok/s
all in VRAM+RAM · disk 0
2× Xeon Gold 6430 · 1 TB DDR5 · CPU-only (#389)5.42 tok/s
every expert pinned in RAM
DGX Spark GB10 · 121 GB unified (#161)3.33 tok/s
unified memory + CACHE_ROUTE
MacBook Pro M5 Max · 128 GB · Metal (#387)2.0 tok/s
Metal tier + warm pin
Ryzen AI Max+ 395 · 128 GB (#200)1.83 tok/s
hot set pinned in RAM
Ryzen 9 9950X3D2 · RTX 5090 · Gen5 NVMe (#120)1.23 tok/s
28 GB VRAM tier + 11.5 GB/s disk
single RTX 5070 Ti laptop-class (#273)1.07 tok/s
GPU-resident pipeline
25 GB dev box · cold — the proven floor0.05–0.1 tok/s
streamed from NVMe
+

…and 15+ more community machines, from an M4 Pro Mac Mini to a 430 GB EPYC server. + Full tables, methodology and quality ablations: docs/benchmarks.md

+
+ +
+
05 — the models
+

One engine, more minds coming

+

The tiering algorithm is model-agnostic: any Mixture-of-Experts with routed + experts can be staged the same way. GLM-5.2 is live today; support for more open-weight + families is on the roadmap.

+
+
live

GLM-5.2

Z.ai · 744B MoE
+

The flagship target: int4 container, token-exact vs reference, full atlas published.

+
live

OLMoE

Allen AI · 7B MoE
+

The small research workhorse — quantization A/Bs and quality ablations run here first.

+
planned

Kimi K2

Moonshot AI · 1T MoE
+

The next scale step: a trillion-parameter mind on the same tiered engine.

+
planned

Qwen3 MoE

Alibaba Qwen
+

The most widely-deployed open family — broad hardware coverage meets broad adoption.

+
planned

MiniMax

MiniMax · MoE
+

Long-context specialists — a different routing profile for the atlas to map.

+
+
+

Thank you to the teams whose open weights make this project possible — + Z.ai (GLM), Moonshot AI (Kimi), Alibaba Qwen, MiniMax, + Allen AI (OLMoE), and everyone releasing frontier-class models in the open. + An engine is nothing without a mind to run.

+
+
+ +
+
06 — join in
+

Come build the microscope

+

Every optimisation in this project started with someone measuring something on + their own machine — a disk swap, a NUMA experiment, an atlas replication on a Mac Mini. + You don't need a datacenter to move frontier-model research. That is the whole point.

+
+

▸ run a benchmark

+

Any hardware is a datapoint — the community table grows one issue at a time.

 +

▸ pick an issue

+

Kernels, schedulers, the atlas, the dashboard — measured problems, waiting.

 +

▸ read the engine

+

One C file plus headers, zero dependencies. Readable in an afternoon — really.

 +
+
+ +
+ +
+ 🐦 colibrì — MIT license + GitHub + API + Tuning + the hummingbird: tiny, fast, precise. +
+ + + + From 00ed8d5aa4ac63bb818e4348388ee87d9f6ec528 Mon Sep 17 00:00:00 2001 From: JustVugg Date: Mon, 20 Jul 2026 18:49:16 +0200 Subject: [PATCH 3/5] docs: surface the website (badge + header link) in README Site (justvugg.github.io/colibri) is live but linked nowhere. Add website + latest-release badges and a Website header link. README-only, no code change. Co-Authored-By: Claude Opus 4.8 --- README.md | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/README.md b/README.md index 085e4e926..d4fe52170 100644 --- a/README.md +++ b/README.md @@ -3,6 +3,12 @@

+ Website + Latest release +

+ +

+ Website · English · 繁體中文

From e942f62cef3a214675938d0f17560c4dc5ac10a5 Mon Sep 17 00:00:00 2001 From: ZacharyZcR Date: Tue, 21 Jul 2026 03:10:27 +0800 Subject: [PATCH 4/5] cuda: reject fmt=4 at the per-row-only entry points (#334 follow-up to #298) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit #298 fixed the dense-resident path's callers after they fed grouped-scale tensors to kernels that read scales per row (CUDA_DENSE=1 produced garbage). The entry points themselves still accept them: coli_cuda_matmul uploads via coli_cuda_tensor_upload (no gs), so scale_count collapses to O and the [O, ceil(I/gs)] buffer is silently truncated, then quant_matmul reads scales[o] per row. coli_cuda_expert_mlp same kernels, same assumption. Return 0 for fmt=4 at both, so a grouped tensor takes the expert-group path (#451, which carries per-group scales) or stays on the CPU — a fallback, never a wrong answer. The caller-side guard in colibri.c already excludes fmt=5 the same way; this makes the ABI safe regardless of who calls it next. make check 105/105, zero warnings. --- c/backend_cuda.cu | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/c/backend_cuda.cu b/c/backend_cuda.cu index a0083f401..298fcee5d 100644 --- a/c/backend_cuda.cu +++ b/c/backend_cuda.cu @@ -638,6 +638,12 @@ extern "C" int coli_cuda_matmul(ColiCudaTensor **tensor, const void *weights, const float *scales, int fmt, int S, int I, int O, int device) { if (fault_injected()) return 0; + /* fmt=4 carries [O, ceil(I/gs)] scales and quant_matmul reads them per row: + * the plain upload would also truncate the buffer to O floats. #298 fixed the + * dense-resident path's callers; this closes the entry itself so no future + * caller can reach the kernel with grouped scales. Grouped tensors take the + * expert-group path (#451) or stay on the CPU. */ + if (fmt == 4) return 0; if (S < 1 || !coli_cuda_tensor_upload(tensor, weights, scales, fmt, I, O, device)) return 0; ColiCudaTensor *t = *tensor; DeviceContext *ctx = find_ctx(t->device); @@ -657,6 +663,8 @@ extern "C" int coli_cuda_expert_mlp(ColiCudaTensor *gate, ColiCudaTensor *up, ColiCudaTensor *down, float *y, const float *x, int S) { if (fault_injected()) return 0; + /* same reason as coli_cuda_matmul: this path's kernels are per-row only. */ + if (gate && (gate->fmt == 4 || (up && up->fmt == 4) || (down && down->fmt == 4))) return 0; if (!gate || !up || !down || !x || !y || S < 1 || gate->device != up->device || gate->device != down->device || gate->I != up->I || gate->O != up->O || From 153e6f4c898cf9adcec9a1681199eab532984cc7 Mon Sep 17 00:00:00 2001 From: JustVugg Date: Mon, 20 Jul 2026 23:40:20 +0200 Subject: [PATCH 5/5] =?UTF-8?q?site:=20fix=20license=20=E2=80=94=20Apache?= =?UTF-8?q?=202.0,=20not=20MIT?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Footer said 'MIT license'; repo is Apache 2.0 (see LICENSE). Fix + link. Co-Authored-By: Claude Opus 4.8 --- site/index.html | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/site/index.html b/site/index.html index dd6dac1ea..8070bee95 100644 --- a/site/index.html +++ b/site/index.html @@ -454,7 +454,7 @@

Come build the microscope