[Bug]: Loading Kimi K3 layers is slow, and then Colibri stops doing anything.

[brad-evony]

Commit or version

release 1.4.0

Environment

system: core i9 10940x 256gb ram 4tb nvme ssd, windows 10 pro
running colibri in command prompt.

Reproduction steps

set OMP_NUM_THREADS=14
set COLI_MODEL=E:\AI-MODELS\KIMI-K3-MXFP4
python coli web --ram 242 --gpu none --ngen 512

Expected behavior

It is supposed to load the 93 layers within a reasonable time, then load the tokenizer, then open the browser page to begin chat functionality.

Actual behavior and logs

It is slow to load taking 40 minutes because it only uses one CPU thread out of 14. Then after it loads the tokenizer, it stops using the CPU and SSD, and just doesn't do anything. It never begins the chat or opens the browser window. It doesn't give any errors it just stops doing anything.

dashboard: http://127.0.0.1:8000/  (opens automatically when the engine is ready)
[K3] loaded layer 1/93 (57.0s, RSS 1.4 GB)
[K3] loaded layer 9/93 (270.8s, RSS 4.1 GB)
[K3] loaded layer 17/93 (483.2s, RSS 6.9 GB)
[K3] loaded layer 25/93 (694.8s, RSS 9.8 GB)
[K3] loaded layer 33/93 (906.6s, RSS 12.6 GB)
[K3] loaded layer 41/93 (1117.8s, RSS 15.5 GB)
[K3] loaded layer 49/93 (1329.3s, RSS 18.4 GB)
[K3] loaded layer 57/93 (1540.7s, RSS 21.2 GB)
[K3] loaded layer 65/93 (1752.1s, RSS 24.1 GB)
[K3] loaded layer 73/93 (1963.6s, RSS 26.9 GB)
[K3] loaded layer 81/93 (2175.4s, RSS 29.8 GB)
[K3] loaded layer 89/93 (2386.9s, RSS 32.6 GB)
[K3] init done in 2538.7s | 93 layers | expert cache 4/layer (17.5 MB/slot) | RSS 35.1 GB
[K3] tokenizer.json loaded (family=kimi)

Pressing CTRL-C gives the following message:

Traceback (most recent call last):
File "E:\AIPROGS2\colibri-v1.4.0-windows-x86_64\coli", line 1249, in
main()
File "E:\AIPROGS2\colibri-v1.4.0-windows-x86_64\coli", line 1244, in main
if handler: sys.exit(handler(a) or 0)
File "E:\AIPROGS2\colibri-v1.4.0-windows-x86_64\coli", line 1063, in cmd_web
cmd_serve(a)
File "E:\AIPROGS2\colibri-v1.4.0-windows-x86_64\coli", line 990, in cmd_serve
openai_server.serve(a.model,a.host,a.port,model_id,a.api_key,
File "E:\AIPROGS2\colibri-v1.4.0-windows-x86_64\openai_server.py", line 2374, in serve
runtime = Engine(engine,model,cap,max_tokens,env,kv_slots)
File "E:\AIPROGS2\colibri-v1.4.0-windows-x86_64\openai_server.py", line 1223, in init
read_engine_turn(self.process.stdout, READY, lambda _: None)
File "E:\AIPROGS2\colibri-v1.4.0-windows-x86_64\openai_server.py", line 1128, in read_engine_turn
byte = stream.read(1)
KeyboardInterrupt

[JustVugg]

Both symptoms reproduce in the source, and they are two separate bugs. Thank you for the full log with the KeyboardInterrupt traceback — that stack is what identified the second one immediately, because it shows exactly where it is stuck.

The hang: a Windows-only handshake bug, and it is our fault twice over

coli and the engine speak a byte protocol. The engine prints a sentinel when it is loaded:

\x01\x01READY\x01\x01\n

and the gateway blocks in read_engine_turn until it sees those exact bytes — which is precisely where your traceback stops.

On Windows the CRT opens stdout in text mode, which rewrites \n as \r\n. So kimi_k3.exe emits \x01\x01READY\x01\x01\r\n, the gateway never matches it, and it waits forever. No error, because nothing failed — one side is waiting for a byte the other side already mangled.

colibri.c fixed this long ago and its own comment describes your symptom exactly:

on Windows the CRT opens both handles in TEXT mode: stdout translates '\n'->'\r\n' (so the READY sentinel never matches and chat hangs), and stdin translates '\r\n'->'\n' and rejects writes of raw bytes with EINVAL

The fix is _setmode(_fileno(stdout), _O_BINARY) on both handles. Counting them:

engine	_O_BINARY
colibri.c	✅
inkling.c	❌
kimi_k3.c	❌
olmoe.c	❌

The three newer engines never inherited it. Same shape as the fd_set/select bug fixed in #736 last week: Windows-specific logic that lives in one engine and that the others were written without.

This is not a v1.4.0 regression — the bug has been there since the engine was written. It is v1.4.0-visible because before v1.4.0 the archives did not contain kimi_k3.exe at all (#720), so nobody on Windows could reach it.

Workaround until the fix ships: run the engine directly, where no handshake is involved.

set SNAP=E:\AI-MODELS\KIMI-K3-MXFP4
kimi_k3.exe E:\AI-MODELS\KIMI-K3-MXFP4 "Explain what NVMe is in two sentences."

That gives you generation but not the web UI. coli chat, coli web and coli serve all go through the same handshake, so all three are affected equally.

The slow load is a second, unrelated bug

35.1 GB in 2538 s is 13.8 MB/s. On a 4 TB NVMe that is roughly two orders of magnitude below the drive, so this is not your hardware and not OMP_NUM_THREADS.

The layer loader in kimi_k3.c is a plain serial loop — one w_load after another, 93 layers, no parallelism. The count tells the story: kimi_k3.c has 4 #pragma omp directives against 16 in both colibri.c and inkling.c, and none of them is on the load path. Your 14 threads have nothing to do because the loading code never asks for them.

Two things are worth separating, and your report gives us the numbers to do it: 13.8 MB/s is slow even for a serial reader, so there is likely a small-read problem underneath the missing parallelism — colibri.c reads each expert's matrices adjacently in one pread for exactly this reason. We will look at both rather than just adding a parallel for and declaring it fixed.

Where this goes

Both are ours to fix and neither needs anything from you:

1. _O_BINARY in inkling.c, kimi_k3.c, olmoe.c — small, and it should go into compat.h alongside coli_stdin_readable() rather than being pasted a third and fourth time, which is how it went missing in the first place.
2. The load path — parallel loading plus a look at the read pattern.

(1) unblocks you and is the one we will ship first. I will post here when it is in dev so you can try it before the next release if you want; the Windows exe is downloadable from any green check run under Actions → Artifacts.

Two useful things in your report, so you know they were not wasted: 256 GB of RAM with --ram 242 is the largest Kimi K3 configuration anyone has reported, and once loading is fixed that machine should hold a very large share of the expert bank resident. And the fact that it got all the way to tokenizer.json loaded before hanging is what proves the model itself is fine — this is purely a plumbing bug between the engine and the launcher.

[MichaelFomenko]

set OMP_NUM_THREADS=14
set COLI_MODEL=E:\AI-MODELS\KIMI-K3-MXFP4
python coli web --ram 242 --gpu none --ngen 512

are you sure that this is how to use colibri with this model?
I mean:
python coli web --ram 242 --gpu none --ngen 512
looks wrong to me, but maybe this is how is is used under Windows.

[JustVugg]

@MichaelFomenko — fair question to ask, and worth answering precisely so @brad-evony does not spend time changing a command that is already right.

The invocation is correct. Checked against the CLI on dev:

set OMP_NUM_THREADS=14
set COLI_MODEL=E:\AI-MODELS\KIMI-K3-MXFP4
python coli web --ram 242 --gpu none --ngen 512

• COLI_MODEL=<dir> is the documented way to point at the model (--model <dir> is the equivalent flag).
• coli web, --ram, --gpu, --ngen all exist and all accept those values — coli web --help lists them.
• python coli is right on Windows, where coli is a Python script without an executable bit; ./coli is the Linux/macOS form.

So nothing here is user error, and that matters for the diagnosis: the hang happens after the model has fully loaded and the tokenizer is read. A wrong flag would have failed at argument parsing, not 42 minutes later at the handshake.

What actually goes wrong is in our code, on both counts — kimi_k3.exe never gets its stdout into binary mode, so the READY sentinel arrives as ...\x01\x01\r\n and the gateway waits for a byte that was mangled on the way out; and the layer loader is serial, which is why 14 threads sat idle. Details in the comment above.

The one thing I would change in that command line is OMP_NUM_THREADS=14 — not because it is wrong, but because it will not help until the load path is parallel. It is a Core i9 10940X, so 14 physical cores is exactly the right number to set once it does.

Thanks for looking at it. A second pair of eyes on a report is useful even when the answer turns out to be "the command was fine" — that eliminated a whole branch of the investigation.

[JustVugg]

thanks @MichaelFomenko

[JustVugg]

Fixed on dev — #749 is merged.

coli_serve_binary_mode() now lives in compat.h and both kimi_k3.c and inkling.c call it before emitting the handshake, so the READY sentinel goes out as bare bytes and coli matches it. coli chat, coli web and coli serve should all work for you now.

It also ships tests/test_serve_sentinel.c, which writes the handshake through a real stream and asserts there is no \r in the bytes. Nothing in this repo had ever checked that, which is exactly why this reached you instead of CI.

To try it before the next release, the Windows exe is built by every green CI run: Actions → any recent check run on dev → Artifacts. Otherwise it will be in the next tag.

The slow load is still open and is the more interesting half. I did not want to bundle a guess with a fix that unblocks you, so #749 deliberately does not touch it. The plan there, stated so you can hold us to it:

35.1 GB in 2538 s is 13.8 MB/s. A serial reader on an NVMe should still manage hundreds of MB/s, so the missing parallelism cannot be the whole story. The three candidates, in order:

1. No parallelism — the 93-layer loop is serial and kimi_k3.c has 4 #pragma omp against colibri.c's 16, none on the load path. Explains your idle 14 threads, not the 13.8 MB/s.
2. Small reads — each tensor is fetched on its own. colibri.c stores an expert's three matrices adjacently and reads them in one pread precisely to avoid this; thousands of small scattered reads would produce numbers like yours.
3. On-the-fly expansion — if MXFP4 is being widened during load, the bottleneck is CPU rather than disk, and single-threaded would explain everything.

We will measure before patching. Adding a parallel for when the real problem is read size would just give you 14 threads issuing small reads instead of one, and the number would barely move.

If you are willing, one thing from your machine would help: run the engine directly with PROF=1 and stop it after a couple of minutes — that reports where the load time actually goes, and it would tell us which of the three it is without us guessing from a different disk.

Thanks for a report that was complete enough to diagnose from. The Ctrl-C traceback is what made the hang a five-minute problem instead of a fishing expedition.

[JustVugg]

Closing this — #749 is merged and the fix is confirmed.

For anyone arriving from a search: the symptom was not slow loading. coli waits for an exact READY sentinel, and on Windows stdout was in text mode, so the handshake went out with a \r inserted and the match never fired. The engine had finished loading and was sitting there ready; the launcher was waiting for bytes that could not arrive. coli_serve_binary_mode() in compat.h now puts the stream in binary mode before the handshake, in both kimi_k3.c and inkling.c.

tests/test_serve_sentinel.c writes the handshake through a real stream and asserts there is no \r in it. Nothing here had ever checked that — which is why it reached you rather than CI.

@brad-evony — separately, your line in this thread is the sharpest criticism the project has had:

Colibri does not support any models that fit into 256GB

You are right, and it is now tracked as a gap rather than an opinion. OLMoE 4 GB, GLM-5.2 372 GB, Inkling 469 GB, Kimi K3 1.6 TB — nothing between 4 and 372. The machine most likely to want this engine has nothing that fits it, gets pushed into the streaming regime, which is our worst case, and reasonably concludes we are slow. Two things are moving on it: DeepSeek V4 Flash (#727, ~140 GB at int4) and a from-scratch 24–28B being planned in #683.