QLoRA training path: fine-tune GLM-5.2 (744B) in 64 GB RAM

[pavolbauer]

Summary

This branch adds a memory-bounded LoRA/SFT training path to colibrì and has
fine-tuned the real 744B GLM-5.2 end-to-end on a 64 GB M4 Max: 100
optimizer steps in 27.8 h, peak RSS 34.9 GB, loss 5.05 → 0.37, adapter
demonstrably changes generation (persona learned, held-out prompt generalizes),
and unsetting ADAPTER restores base behavior. The same streamed-experts idea
that makes colibrì's inference possible carries over to training: dense state
resident, experts streamed per layer pass, no gradients or optimizer state for
any frozen weight, activation checkpointing with routing replay.

Mergeable against current dev: the work was developed on pre-v1.0 dev
and has been ported to the colibri.c + modules layout — 4 clean commits on
top of dev HEAD. All gates re-run on this base: full make test (197 OK),
metal-test (incl. the new r_top8_par suites), tiny oracle TF 32/32, PyTorch
fixture parity, and a real-model probe that reproduces the original run's
first-step loss exactly (3.5764, 239 s/step, identical expert I/O counts).
Opening as a draft to ask how you'd prefer it landed before flipping to
ready-for-review.

Evidence (M8 acceptance run, 2026-07-26)

• 100 steps, seq 128 × accum 4, LoRA rank 8 on o_proj, lr 1e-3, RAM budget 44 GB
• loss 5.05 → 0.37 over 10 epochs of a 38-sample persona set (deliberate overfit)
• peak footprint 34.9 GB; the budget manager shed cache slots on swap growth
  three times and recovered; 104 TB total expert I/O, zero crashes
• greedy on/off eval:

Prompt	Base	Adapter
introduce yourself (held-out)	verbose persona analysis	Kolibrík. Small footprint, big model. 🐦 …
who are you (train)	"The user is asking… Let me craft…"	Kolibrík. Small footprint, big model. 🐦 …
capital of Slovakia (train)	verbose CoT analysis	Bratislava. 🐦 🐦 …

Full logs + generations are attached to the fork's release. A single fwd+bwd
pass reproduces the same loss to 4 decimals before/after the perf work
(784 s → 239 s per pass on the M4 Max).

What's in the diff (+11k lines, almost entirely additive)

Area	Files	Notes
Adapter runtime	c/lora.h	safetensors colibri-lora-v1, fingerprint-gated load, ADAPTER=<dir> in inference; without it output is bit-identical to upstream
Training primitives	c/train/qlora_ops.h	streaming quantized-transpose dx = Q(W)ᵀ dy (int8/int4/int4-grouped, never materializes W), AdamW; validated vs PyTorch float64
Model backward	c/train/train_model.h	manual backward through the full GLM block (MLA, partial RoPE, SwiGLU, frozen sigmoid top-k routing with differentiable gates); activation ckpt stashes x_in + routing ids, recompute is bitwise-equal to retention; experts fetched once per layer pass, row-batched
Memory manager	c/train/budget.h	plan + hard ceiling + swap response (shed expert cache before the OS swaps), per-step telemetry
Data + ckpt	c/train/dataset.h, c/train/checkpoint.h, c/tools/prepare_sft.py	pre-tokenized masked-CE SFT format; atomic checkpoints, bitwise resume, SIGINT-safe
CLI + runners	c/train/train_main.c, c/scripts/m7_smoke.sh, c/scripts/m8_overfit.sh	staged milestone runners with on/off eval
Metal	c/backend_metal.mm/.h	training kernels (t_tmul, LoRA fwd/bwd) with CPU parity tests; dispatch is opt-in (COLI_METAL=1)
Tests	c/tests/test_lora.c, test_train_linear.c, test_train_tiny.c, test_train_resume.c	all in make check; PyTorch fixtures via c/tools/make_train_oracle.py

Everything was built milestone-gated per the brief in AGENTS.md (M0–M8, each
with tests/oracle parity; commit history follows the milestones).

Review pointers

• Start with c/train/train_model.h (the backward + checkpointing core) and
  c/train/qlora_ops.h (the frozen-weight transpose kernel).
• make check runs every training gate incl. PyTorch-fixture parity and
  bitwise resume; make metal-test METAL=1 covers the Metal kernels.
• Upstream's README is untouched by this branch; the training docs live in
  the runbook under docs/plans/ and the brief in AGENTS.md (happy to
  relocate/trim both to your taste).

Port notes

The trainer #includes the engine wholesale; the port from glm.c amounted
to the include swap, a 54-line engine-side patch (adapter load + o_proj
residual + config fingerprint), re-merging the Metal training kernels around
your new r_top8_par work, and adapting the scripts to the colibri binary
name. The original M7/M8 evidence was produced on the pre-v1.0 base; the
real-model probe above ties the two bases together (identical first-step
loss and I/O profile).

Questions for you

1. Would you want this as one feature PR (as here), or split (adapter runtime
   / trainer / scripts+docs)?
2. Any objection to the c/train/ layout and the coli_train binary name?
3. The Metal training path is off by default because its f32 GEMM numerics
   differ from the CPU idot path (both valid; documented in the commit log) —
   happy to align on whichever default you prefer.

Validation

• make -C c check — full pass on this branch (197 tests OK, 13 env-dependent skips), macOS/M4 Max
• CUDA changes were tested with make -C c cuda-test — n/a, no CUDA code touched
• Performance claims include hardware, commands, and repeatable measurements — all numbers are from an M4 Max 64 GB / NVMe, produced by the committed c/scripts/m7_smoke.sh and c/scripts/m8_overfit.sh (invocations in the PR body); raw logs attached to the fork's release

Compatibility

• The default CPU build remains dependency-free — make portable passes in check; Python/PyTorch are needed only to generate optional test fixtures, and the tests self-skip without them
• No model files, generated binaries, or benchmark artifacts are included — compiled binaries are untracked; one caveat declared: ~56 KB of pre-tokenized SFT fixtures under c/data/ (persona set + eval prompts) so the milestone scripts run fully offline — happy to replace with a prepare_sft.py step if you'd rather not carry them

[JustVugg]

This is genuinely exciting — fine-tuning the real 744B on 64GB by carrying the streamed-experts idea into the backward pass is exactly the kind of thing that fits colibrì's mission (frontier AI on consumer hardware), and the numbers you quote (loss 5.05 → 0.37, 34.9GB peak, adapter learned + base restored on unset) are compelling. We'd like to evaluate it seriously as a direction, not rush it in, so a few things would help us dig in:

Reproducible evidence

1. A minimal, runnable example anyone can try end to end — the exact command(s) to fine-tune a small adapter (even a handful of steps on a tiny dataset) and then run inference with ADAPTER set vs unset. Ideally something that finishes in minutes, not the full 27.8h run, so reviewers can reproduce the mechanism.
2. The before/after outputs you measured: the held-out prompt, the base generation, the adapted generation, and the base-restored-on-unset generation, pasted in. That's the proof the adapter actually changed behavior and is reversible.

Quality / correctness
3. A quality check that the base model isn't degraded when you're not training — is the inference path byte-identical with ADAPTER unset (token-exact vs current dev)? That's the thing we can't compromise: training must not touch the inference hot path when off.
4. Any signal on catastrophic forgetting — does the adapter preserve general ability, or only fit the persona? Even a rough MMLU/hellaswag delta on base vs adapted would help.

Scope / integration
5. A short note on the blast radius: which of the +4582 lines are training-only (compiled/gated out of a normal inference build) vs shared with inference, and how colibri.c / the Metal backend are affected.

No rush — we'd rather land this well than fast. If you can add the minimal example + the before/after transcripts + the "inference unchanged when off" confirmation, that gives us what we need to decide on the direction. Really nice work.

[pavolbauer]

Hi there, thank you very much for the kind words and the thorough look — "land it well rather than fast" is exactly the spirit this was built in, so the five asks are a pleasure to address. First things referenced are on the branch 1bc1c21.

1. Minimal runnable example (~2–3 minutes, CPU only)

cd c && ./scripts/quickstart_tiny.sh

It builds, generates the tiny oracle model (tools/make_glm_oracle.py — the one
place Python/torch is needed), synthesizes a coli-sft-v1 set from the oracle's
own prompt ids (tools/make_tiny_sft.py, no tokenizer involved), and then runs
the whole mechanism with pass/fail checks at each step. Output from a clean run:

== 1. BASE: adapter off — must match the oracle reference exactly ==
   Matching tokens: 20/20
== 2. TRAIN: overfit a rank-16 LoRA onto a different continuation ==
[step 800] loss 0.5796 | 20 tok 178.87 tok/s | 0.11s ...
[train] done in 88s
== 3. ADAPTER on: generation should now emit the TRAINED continuation ==
   trained continuation reproduced: 20/20 tokens (vs 0/20 still matching the base reference)
== 4. UNSET: adapter off again — base behavior must be restored ==
   Matching tokens: 20/20
   identical to step 1: base restored ✓

Mechanism, adapter effect, and reversibility — each token-exact.

2. Before/after transcripts (real 744B, held-out prompt)

System prompt in rows marked ✦: You are Kolibrík, the Colibri hummingbird assistant.
User prompt in all four cells: Introduce yourself in one sentence. (never in the
training set). Greedy, 48 tokens, verbatim:

	adapter OFF	adapter ON
✦ persona in system prompt	<think>1. **Analyze the Request:** * **User:** Asks for a one-sentence introduction. * **Persona:** Kolibrík, the hummingbird assistant. * **Constraint** … (budget exhausted mid-think)	</think>Kolibrík. Small footprint, big model. 🐦 🐦 …
no persona in system prompt	<think>The user asked me to introduce myself… I am GLM, a large language model developed by Z.ai… Actually, I need to (budget exhausted mid-think)	</think>I am an expert system. I keep my footprint small and my throughput large. 🐦 🐦 …

The adapter-OFF generations are the base-restored-on-unset case: they were
produced with the trained adapter present on disk but ADAPTER unset. Raw logs
(these four plus a fifth prompt) are attached to the fork's release:
https://github.com/pavolbauer/colibri-qlora/releases/tag/0.1-m8

3. Inference path unchanged when off — token-exact vs current dev

Strongest check I could construct: built pristine dev HEAD (21e7a35) and
this branch, ran both on the real 744B snapshot with the same prompt,
TEMP=0 DRAFT=0 (per #100's byte-exactness rule) and TOKENS=1:

dev 21e7a35 : [TOKENS] 32 generated: 785 1196 374 10156 369 264 825 1331 18351 16782 13 6771 752 1744 911 1246 311 19095 7037 34727 382 40 1079 5588 44 320 15403 11429 4903 5007 701 7881
this branch : [TOKENS] 32 generated: 785 1196 374 10156 369 264 825 1331 18351 16782 13 6771 752 1744 911 1246 311 19095 7037 34727 382 40 1079 5588 44 320 15403 11429 4903 5007 701 7881

Byte-identical. Corroborating: tiny oracle TF is 32/32 and greedy 20/20 on this
branch (steps 1/4 of the quickstart re-verify this on every run), and the full
make -C c check passes (197 tests). Code-wise the off-path is one pointer
check: without ADAPTER set, m->lora stays NULL and the only addition the
hot path ever sees is if (m->lora) … per o_proj call.

4. Catastrophic forgetting

Fair ask — coming as a follow-up comment later: base-vs-adapter delta on
a sampled MMLU subset (teacher-forced choice logprobs, so it's tractable at CPU
speeds). Expectation to set now: the M8 adapter is a deliberate overfit
(lr 1e-3, 10 epochs on 38 samples — that's what the milestone tests), so some
measurable degradation is expected and honest.

5. Blast radius (vs merge-base 21e7a35)

Area	Lines	Reaches a normal inference build?
c/train/*, train tests, tools, scripts, data, docs	+4 077 (31 files)	No — compiled only into coli_train and the four tests/test_train_*/test_lora binaries
c/lora.h + c/colibri.c	+288 / −3 (colibri.c itself: +54/−3)	Yes, but env-gated: adapter loads only when ADAPTER is set; off-path is a NULL check (see #3)
Metal (backend_metal.mm/.h, its test)	+328	Kernels linked into the engine build but dispatched only from trainer code; inference dispatch sites untouched
Makefile + gitignores	+34	New targets/test wiring only

Happy to split the PR along exactly those lines (adapter runtime / trainer /
metal / scripts) if that's easier to review — and to relocate AGENTS.md and
the README material wherever you prefer docs to live.

[ZacharyZcR]

That's a great idea. I'll look into it.

[pavolbauer]

Follow-up on point 4 (catastrophic forgetting) — numbers as promised.

Protocol (harness is on the branch: c/tools/eval_mmlu_subset.py, 43a1a2b):
25 MMLU test questions sampled across 7 subjects with a fixed seed; one forward
per question — think block force-closed, prompt ends Answer:, 2 greedy tokens
(DRAFT=0), first A–D letter scored. Identical protocol both sides:

python3 tools/eval_mmlu_subset.py --model <snapshot> --binary ./colibri                      # base
python3 tools/eval_mmlu_subset.py --model <snapshot> --binary ./colibri --adapter adapters/m8-persona

Result (M4 Max, ~30 min per side):

	accuracy	valid letter emitted
base (adapter off)	20/25 = 80%	25/25
m8-persona adapter	18/25 = 72%	25/25

Read on the delta: on the 6 questions where the runs disagree, the adapter
flips 4 right→wrong and 2 wrong→right — McNemar p ≈ 0.69, i.e. statistically
indistinguishable from no effect at n=25. Directionally it's the mild
degradation you'd expect from this adapter, which (to repeat the framing from
above) is the M8 acceptance test: a deliberate overfit at lr 1e-3 for 10
epochs on a 38-sample persona set. Notable negatives that did NOT happen:

• no collapse — the adapted model still reasons across subjects (world history,
  abstract algebra, EE all answered correctly);
• no persona bleed — all 25 adapter answers were clean letters, zero 🐦: the
  style conditioning stays out of the way when the context demands an answer;
• and (from point 2's transcripts) the persona itself remains intact where it
  was trained to appear.

So: general ability substantially preserved even under a worst-case training
recipe. A production-tuned adapter (earlier stop, lower lr) should sit closer
to base; happy to run the same probe on such an adapter if useful for the
review. Raw per-question JSONs available on request.

[JustVugg]

Rebase request — and an apology for how many of these you have had.

dev moved a lot in the last day: 19 PRs landed, including the Vulkan backend (#418), the Kimi K3 GPU tier (#705), Metal grouped-int4 (#457), the shared routing telemetry (#716/#719), and several CUDA and launcher fixes. This PR now conflicts.

Before asking, three things changed on our side so that this is the last one of these you should need for a while.

1. The main source of these conflicts is closed. c/Makefile had a single hand-written TEST_BINS line listing every test gate. Every PR that added a test appended to that same line, so any two such PRs conflicted by construction, even when they touched entirely unrelated code — c/Makefile appeared in 26 of 40 open PRs. #386 hit it twice while being rebased, and said so, which is what sent us looking.

Gates are now derived from the build rules (#733). Adding a test means adding your .c and its own rule, which land in different places in the file. There is no shared list left to conflict on.

2. We resolved what we could ourselves instead of asking. Eight PRs were unblocked by a maintainer pushing the merge to the contributor's branch rather than requesting a rebase — including three where both sides carried a real change and had to be merged rather than picked. No commits were rewritten. Yours is here because its conflict is in engine code (c/colibri.c or a backend), where guessing your intent would be worse than asking.

3. Merge order is oldest-clean-first from now on. A PR that is green and unconflicted merges ahead of anything opened after it. The reason some of you rebased many times is that newer, smaller PRs kept jumping the queue and resetting you — a starvation loop we built, not bad luck on your side.

What we need: one rebase onto current dev. If your only conflict was the TEST_BINS line, just drop your entry — your test is picked up by its own rule now.

If you would rather not, say so and we will close it with thanks and the branch stays yours to reopen. No pressure either way, and no hard feelings — several of these have been open a while through no fault of the author.

[jazir555]

Anyway this can be optimized for 32 GB ram systems?

[pavolbauer]

@jazir555 yeah there is a RAM parameter you can set.

[pavolbauer]

@JustVugg Heads-up unrelated to this PR: make -C c metal-test METAL=1 fails on current dev — quant.h's fp8_nblk (int64_t) collides with the local int fp8_nblk in tests/test_backend_metal.mm:99. Renaming the test-local helper fixes it; I've left your file untouched here.