[Bug]: Kimi-K3 won't start

[MichaelFomenko]

Commit or version

1.3.0

Environment

OS: Debian 13
CPU: AMD 7950X
RAM: 64
GPU: 2 x GT1030
SSD: Samsung 990 Pro 2TB

Reproduction steps

I downloaded the colibri-v1.3.0-linux-x86_64 and run the binary written in the documentation of the Release.

user@Cartelion:/media/user/6f1e9b86-52a9-4ef6-a5fd-8965ef1db31f/LLM/colibri-v1.3.0-linux-x86_64$ COLI_MODEL=/media/user/6f1e9b86-52a9-4ef6-a5fd-8965ef1db31f/LLM/Kimi-K3/ ./coli chat

tokenizer.json is missing from /media/user/6f1e9b86-52a9-4ef6-a5fd-8965ef1db31f/LLM/Kimi-K3/

I dont know why it asking for the tokenizer.json file, the tokenizer.json don't exists in the Original Huggingface Repo.

Maybe I done something wrong.

Expected behavior

maybe I need to tell the app which model it is?

Actual behavior and logs

user@Cartelion:/media/user/6f1e9b86-52a9-4ef6-a5fd-8965ef1db31f/LLM/colibri-v1.3.0-linux-x86_64$ COLI_MODEL=/media/user/6f1e9b86-52a9-4ef6-a5fd-8965ef1db31f/LLM/Kimi-K3/ ./coli chat

tokenizer.json is missing from /media/user/6f1e9b86-52a9-4ef6-a5fd-8965ef1db31f/LLM/Kimi-K3/

[terrizoaguimor]

You did nothing wrong, and you are right about the repo: Kimi-K3 ships tiktoken.model, not
tokenizer.json. That is stated in the engine's own header comment (c/kimi_k3.c:35):

tokenizer.json is synthesized once by tools/k3_tokenizer.py (the HF repo ships only
tiktoken.model); without it the engine still runs on raw ids.

There are two separate problems here, and the one you hit is not the one blocking you.

The message you got is a launcher bug

c/coli's need_model() checks for tokenizer.json unconditionally, before it resolves
which engine to use:

def need_model(model, engine=None):
    if not os.path.isdir(model): sys.exit(...)
    if not os.path.exists(os.path.join(model,"tokenizer.json")):
        sys.exit(f"tokenizer.json is missing from {model}")   # ← runs for every engine
    engine = engine or engine_for(model)                       # ← engine resolved after

But kimi_k3.c treats that file as optional — it probes for it, sets has_tok=0 if absent,
and carries on; only serve mode bails:

FILE *f=fopen(tp,"rb"); if(f){ ... has_tok=1; }
if(serving){ if(!has_tok){ fprintf(stderr,"serve mode needs tokenizer.json\n"); return 1; } }

So the launcher refuses a model directory that the engine would have accepted. chat without
tokenizer.json should reach the engine and run on raw ids.

What is actually blocking you

The v1.3.0 binary release does not contain a Kimi-K3 engine. I downloaded
colibri-v1.3.0-linux-x86_64.tar.gz and listed it:

LICENSE  coli  colibri  doctor.py  openai_server.py  resource_plan.py  version.py  web/

colibri is the GLM engine. There is no kimi_k3 binary, no inkling binary, and no
tools/k3_tokenizer.py. Even with a tokenizer.json in place you would have hit the next
error — kimi_k3 engine is not built. Run: make -C c kimi_k3 — which a binary release cannot
satisfy, since it ships no c/ directory.

So: Kimi-K3 cannot be run from the v1.3.0 release at all. Nothing you could have typed would
have worked.

Getting it running today

From a source checkout:

git clone https://github.com/JustVugg/colibri && cd colibri
make -C c kimi_k3
python3 c/tools/k3_tokenizer.py /path/to/Kimi-K3      # synthesizes tokenizer.json from tiktoken.model
./c/kimi_k3 /path/to/Kimi-K3 "Explain what NVMe is in two sentences."

Note the engine takes the model directory as its first argument, not via COLI_MODEL — that
env var is the launcher's. Running the engine directly also sidesteps the launcher check above,
so if you want to try it before building the tokenizer, drop the third line; you will get raw
ids rather than text, which is enough to confirm the weights load.

COLI_ENGINE=/path/to/kimi_k3 ./coli chat does correctly redirect the launcher, but it will
still fail on the tokenizer.json check, so it is not a way around this today.

One heads-up on the hardware

Nothing wrong with it, but so you are not surprised: colibrì's CUDA kernels are built for
sm_80 and up, and the GT 1030 is sm_61, so your GPUs will not be used regardless. That is
fine — CPU plus NVMe streaming is colibrì's actual design point, and your 990 Pro is the right
part for it. Do check free space though: K3's weights are large enough that a 2 TB drive holding
both them and a quantised copy gets tight.

Suggested fixes, if useful

1. Make the tokenizer.json check engine-aware — resolve the engine first, and for kimi_k3
   treat it as optional outside serve mode. The error should also name
   tools/k3_tokenizer.py, since the file is synthesizable and the message currently reads as
   "your download is broken."
2. Either ship kimi_k3 (and inkling) in the release archives, or have the launcher say
   plainly that only the GLM engine is included in binary releases and the others need a source
   build. Right now the failure mode points the user at make -C c kimi_k3 from a tree that
   does not exist locally.

@JustVugg I am already in this area for #700 / #716 / #719 — happy to send a PR for (1), and for
the wording half of (2), if you want it. Say the word and I will keep it to the launcher.

[MichaelFomenko]

./install-coli.sh:

rm colibri
git clone --branch dev --single-branch https://github.com/JustVugg/colibri.git
cd colibri
make -C c kimi_k3
python3 c/tools/k3_tokenizer.py /media/user/6f1e9b86-52a9-4ef6-a5fd-8965ef1db31f/LLM/Kimi-K3
./c/kimi_k3 /media/user/6f1e9b86-52a9-4ef6-a5fd-8965ef1db31f/LLM/Kimi-K3 "Explain what NVMe is in two sentences."

than I run it:

user@Cartelion:/media/user/6f1e9b86-52a9-4ef6-a5fd-8965ef1db31f/LLM$ ./install-coli.sh
rm: das Entfernen von 'colibri' ist nicht möglich: Ist ein Verzeichnis
Schwerwiegend: Zielpfad 'colibri' existiert bereits und ist kein leeres Verzeichnis.
make: Verzeichnis „/media/user/6f1e9b86-52a9-4ef6-a5fd-8965ef1db31f/LLM/colibri/c“ wird betreten
make: „kimi_k3“ ist bereits aktuell.
make: Verzeichnis „/media/user/6f1e9b86-52a9-4ef6-a5fd-8965ef1db31f/LLM/colibri/c“ wird verlassen
tiktoken vocab: 163584 tokens
wrote /media/user/6f1e9b86-52a9-4ef6-a5fd-8965ef1db31f/LLM/Kimi-K3/tokenizer.json (3.6 MB)
[K3] loaded layer 1/93 (6.3s, RSS 1.4 GB)
[K3] loaded layer 9/93 (27.0s, RSS 4.2 GB)
[K3] loaded layer 17/93 (48.3s, RSS 7.1 GB)
[K3] loaded layer 25/93 (69.8s, RSS 10.0 GB)
[K3] loaded layer 33/93 (91.3s, RSS 12.9 GB)
[K3] loaded layer 41/93 (112.9s, RSS 15.8 GB)
[K3] loaded layer 49/93 (134.3s, RSS 18.7 GB)
[K3] loaded layer 57/93 (155.7s, RSS 21.6 GB)
[K3] loaded layer 65/93 (177.1s, RSS 24.5 GB)
[K3] loaded layer 73/93 (198.7s, RSS 27.4 GB)
[K3] loaded layer 81/93 (220.1s, RSS 30.3 GB)
[K3] loaded layer 89/93 (240.7s, RSS 33.2 GB)
[K3] init done in 254.9s | 93 layers | expert cache 4/layer (17.5 MB/slot) | RSS 35.7 GB
[K3] tokenizer.json loaded (family=kimi)
[K3] prompt: 10 tokens | ngen 32 | temp 0.00
[K3] prefill 10/10 (56.8s)
[K3] prefill done in 56.8s (0.18 tok/s)
 What  [tok 1: 12.6s/tok, hit 0%, 179.9 GB read]
 is  [tok 2: 12.1s/tok, hit 2%, 202.9 GB read]
 the  [tok 3: 11.0s/tok, hit 2%, 226.7 GB read]
 difference  [tok 4: 10.6s/tok, hit 3%, 250.9 GB read]
 between  [tok 5: 10.4s/tok, hit 3%, 274.6 GB read]
 NV  [tok 6: 10.1s/tok, hit 4%, 298.5 GB read]
Me  [tok 7: 9.8s/tok, hit 4%, 321.6 GB read]
 and  [tok 8: 9.9s/tok, hit 5%, 344.7 GB read]
 M  [tok 9: 9.8s/tok, hit 5%, 368.4 GB read]
.  [tok 10: 9.9s/tok, hit 5%, 391.3 GB read]
2  [tok 11: 10.1s/tok, hit 5%, 415.1 GB read]
?

  [tok 12: 10.2s/tok, hit 6%, 438.7 GB read]
NV  [tok 13: 10.2s/tok, hit 6%, 462.4 GB read]
Me  [tok 14: 10.1s/tok, hit 6%, 486.5 GB read]
 is  [tok 15: 10.1s/tok, hit 6%, 509.9 GB read]
 a  [tok 16: 10.1s/tok, hit 6%, 533.6 GB read]
 protocol  [tok 17: 10.1s/tok, hit 6%, 558.3 GB read]
 that  [tok 18: 10.0s/tok, hit 6%, 581.2 GB read]
 is  [tok 19: 9.9s/tok, hit 6%, 604.8 GB read]
 used  [tok 20: 9.8s/tok, hit 6%, 629.1 GB read]
 to  [tok 21: 9.7s/tok, hit 6%, 652.1 GB read]
 access  [tok 22: 9.8s/tok, hit 7%, 675.8 GB read]
 the  [tok 23: 9.8s/tok, hit 6%, 700.3 GB read]
 flash  [tok 24: 9.8s/tok, hit 6%, 725.2 GB read]
 storage  [tok 25: 9.8s/tok, hit 7%, 748.0 GB read]
.  [tok 26: 9.8s/tok, hit 7%, 772.1 GB read]
 It  [tok 27: 9.8s/tok, hit 7%, 796.5 GB read]
 is  [tok 28: 9.8s/tok, hit 7%, 820.9 GB read]
 a  [tok 29: 9.9s/tok, hit 6%, 845.3 GB read]
 new  [tok 30: 9.9s/tok, hit 7%, 868.5 GB read]
 protocol  [tok 31: 9.8s/tok, hit 7%, 891.3 GB read]
 that
[K3] context full

[K3] decode 32 tokens in 304.9s (0.10 tok/s) | expert hit 6.7% (3658/54455) | 891.3 GB streamed
[K3] time: attn 47.0s moe 312.9s (eload 44.9s) head 1.3s | RSS 42.7 GB
user@Cartelion:/media/user/6f1e9b86-52a9-4ef6-a5fd-8965ef1db31f/LLM$ 

[terrizoaguimor]

It works — that output is correct, and you are the first person I have seen run K3 end to end
from a clean checkout. Now let us make it not take ten seconds a token.

Your cache is 4 experts per layer, and it does not have to be

From your log:

[K3] init done in 254.9s | 93 layers | expert cache 4/layer (17.5 MB/slot) | RSS 35.7 GB

That 4 is not adaptive — it falls straight out of a default (c/kimi_k3.c:616):

double egb = getenv("K3_EXPERT_GB") ? atof(getenv("K3_EXPERT_GB")) : 8.0;
int cap = (int)((egb*1e9) / (m->e_slot * nmoe));

With your numbers: 8e9 / (17.5e6 × 93) = 4.9 → 4 slots per layer. The default budget is
8 GB, which on a 93-layer model is four slots. You have 64 GB of RAM and are using 35.7, so
roughly 28 GB is sitting idle.

The relationship is linear and you can aim it yourself:

slots/layer ≈ K3_EXPERT_GB × 0.61        (on this model: 17.5 MB/slot, 93 layers)
RSS ≈ 29 GB + K3_EXPERT_GB

K3_EXPERT_GB	slots/layer	approx RSS
8 (default)	4	36 GB
20	12	49 GB
24	14	53 GB
30	18	59 GB

I would start at 24 and watch RSS:

K3_EXPERT_GB=24 ./c/kimi_k3 /path/to/Kimi-K3 "Explain what NVMe is in two sentences."

Your hit rate was still climbing when the log ends — 0% at token 1, 7% by token 28 — because the
cache starts empty and 4 slots take a long time to become useful. With 14 slots it should both
climb faster and settle higher. Your read volume is the thing to watch: 820 GB over 28 tokens is
about 24 GB per token, and that is what the 990 Pro is spending ten seconds on.

The deeper reason it starts cold every time

Even with a bigger cache, every run you do will start from zero, because Kimi-K3 has no
learning cache at all. The GLM engine accumulates a .coli_usage history next to the weights and
uses it to pre-pin the hottest experts at startup (PIN=auto), so it gets faster the more you
use it. kimi_k3.c writes no history and reads none — I checked, there is no PIN, no
.coli_usage, no pin path in that file.

That gap is #700, and I have a PR open for it (#719, mine, not yet merged and awaiting review) —
it adds the five call sites that give K3 the same history every other engine writes. Once that
lands, your second run on the same model directory would start with the hot experts already
pinned rather than discovering them again. I mention it because it is directly the thing you are
feeling, not to push you to test anything.

One expectation to set

At 93 layers with experts streamed from a single SSD, this workload is genuinely disk-bound —
that is colibrì's design point rather than a fault. A bigger cache raises the hit rate and cuts
reads, but K3 is a very large model for one drive. If you ever add a second NVMe, K3_DIRS lets
you split shards across drives, which is the other lever that actually moves this number.

Also worth knowing: [K3] loaded layer .../93 took 254.9s at startup. That is a one-off per
process, so prefer one long session over many short invocations.

[MichaelFomenko]

./coli chat --model /media/user/6f1e9b86-52a9-4ef6-a5fd-8965ef1db31f/LLM/Kimi-K3 --ram 50

user@Cartelion:/media/user/6f1e9b86-52a9-4ef6-a5fd-8965ef1db31f/LLM$ ./run-coli.sh

     ▄▀▀▀▄  ▄        colibri v1.3.0
  ▄▄▄▄▀▀▀▀▄▀▀        tiny engine, immense model
      ▀▀▀▀▀▀▀        GLM-5.2 · 744B MoE · int4 · streaming CPU
        ▀▀▀▀         chat · kimi-k3-colibri · local server
          ▀          
  ──────────────────────────────────────────────────────────
  ⠏ loading kimi-k3-colibri… 6s[K3] loaded layer 1/93 (5.4s, RSS 1.4 GB)
  ⠴ loading kimi-k3-colibri… 27s[K3] loaded layer 9/93 (26.4s, RSS 4.2 GB)
  ⠋ loading kimi-k3-colibri… 48s[K3] loaded layer 17/93 (47.5s, RSS 7.1 GB)
  ⠧ loading kimi-k3-colibri… 69s[K3] loaded layer 25/93 (68.7s, RSS 10.0 GB)
  ⠴ loading kimi-k3-colibri… 91s[K3] loaded layer 33/93 (90.1s, RSS 12.9 GB)
  ⠴ loading kimi-k3-colibri… 112s[K3] loaded layer 41/93 (111.8s, RSS 15.8 GB)
  ⠦ loading kimi-k3-colibri… 134s[K3] loaded layer 49/93 (133.5s, RSS 18.7 GB)
  ⠴ loading kimi-k3-colibri… 156s[K3] loaded layer 57/93 (155.0s, RSS 21.6 GB)
  ⠹ loading kimi-k3-colibri… 177s[K3] loaded layer 65/93 (176.2s, RSS 24.5 GB)
  ⠙ loading kimi-k3-colibri… 198s[K3] loaded layer 73/93 (197.8s, RSS 27.4 GB)
  ⠏ loading kimi-k3-colibri… 220s[K3] loaded layer 81/93 (219.1s, RSS 30.3 GB)
  ⠸ loading kimi-k3-colibri… 239s[K3] loaded layer 89/93 (238.8s, RSS 33.2 GB)
  ⠧ loading kimi-k3-colibri… 253s[K3] init done in 252.5s | 93 layers | expert cache 4/layer (17.5 MB/slot) | RSS 35.7 GB
  ⠇ loading kimi-k3-colibri… 253s[K3] tokenizer.json loaded (family=kimi)
OpenAI-compatible API listening on http://127.0.0.1:8000/v1
[api] 127.0.0.1 - "GET /health HTTP/1.1" 200 -
[api] 127.0.0.1 - "GET /health HTTP/1.1" 200 -
----------------------------------------
[api] 127.0.0.1 - "GET /health HTTP/1.1" 200 -
Exception occurred during processing of request from ('127.0.0.1', 46128)
----------------------------------------
[api] 127.0.0.1 - "GET /health HTTP/1.1" 200 -
[api] 127.0.0.1 - "GET /health HTTP/1.1" 200 -
----------------------------------------
----------------------------------------
Exception occurred during processing of request from ('127.0.0.1', 45654)
[api] 127.0.0.1 - "GET /health HTTP/1.1" 200 -
Exception occurred during processing of request from ('127.0.0.1', 46136)
----------------------------------------
Exception occurred during processing of request from ('127.0.0.1', 46140)
Exception occurred during processing of request from ('127.0.0.1', 45660)
----------------------------------------
Exception occurred during processing of request from ('127.0.0.1', 45664)
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
  File "/usr/lib/python3.13/socketserver.py", line 697, in process_request_thread
    self.finish_request(request, client_address)
    ~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.13/socketserver.py", line 697, in process_request_thread
    self.finish_request(request, client_address)
    ~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.13/socketserver.py", line 362, in finish_request
    self.RequestHandlerClass(request, client_address, self)
    ~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.13/socketserver.py", line 697, in process_request_thread
    self.finish_request(request, client_address)
    ~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.13/socketserver.py", line 362, in finish_request
    self.RequestHandlerClass(request, client_address, self)
    ~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.13/socketserver.py", line 697, in process_request_thread
    self.finish_request(request, client_address)
    ~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.13/socketserver.py", line 362, in finish_request
    self.RequestHandlerClass(request, client_address, self)
    ~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.13/socketserver.py", line 697, in process_request_thread
    self.finish_request(request, client_address)
    ~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.13/socketserver.py", line 697, in process_request_thread
    self.finish_request(request, client_address)
    ~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.13/socketserver.py", line 766, in __init__
    self.handle()
    ~~~~~~~~~~~^^
  File "/usr/lib/python3.13/http/server.py", line 436, in handle
    self.handle_one_request()
    ~~~~~~~~~~~~~~~~~~~~~~~^^
  File "/usr/lib/python3.13/socketserver.py", line 766, in __init__
    self.handle()
    ~~~~~~~~~~~^^
  File "/usr/lib/python3.13/socketserver.py", line 362, in finish_request
    self.RequestHandlerClass(request, client_address, self)
    ~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.13/socketserver.py", line 766, in __init__
    self.handle()
    ~~~~~~~~~~~^^
  File "/media/user/6f1e9b86-52a9-4ef6-a5fd-8965ef1db31f/LLM/colibri/c/openai_server.py", line 1448, in handle_one_request
    super().handle_one_request()
    ~~~~~~~~~~~~~~~~~~~~~~~~~~^^
  File "/usr/lib/python3.13/socketserver.py", line 362, in finish_request
    self.RequestHandlerClass(request, client_address, self)
    ~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.13/socketserver.py", line 766, in __init__
    self.handle()
    ~~~~~~~~~~~^^
  File "/usr/lib/python3.13/http/server.py", line 436, in handle
    self.handle_one_request()
    ~~~~~~~~~~~~~~~~~~~~~~~^^
  File "/usr/lib/python3.13/http/server.py", line 424, in handle_one_request
    method()
    ~~~~~~^^
  File "/media/user/6f1e9b86-52a9-4ef6-a5fd-8965ef1db31f/LLM/colibri/c/openai_server.py", line 1632, in do_GET
    self.send_json(200, payload, request_id)
    ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.13/socketserver.py", line 362, in finish_request
    self.RequestHandlerClass(request, client_address, self)
    ~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.13/socketserver.py", line 766, in __init__
    self.handle()
    ~~~~~~~~~~~^^
  File "/usr/lib/python3.13/socketserver.py", line 766, in __init__
    self.handle()
    ~~~~~~~~~~~^^
  File "/usr/lib/python3.13/http/server.py", line 436, in handle
    self.handle_one_request()
    ~~~~~~~~~~~~~~~~~~~~~~~^^
  File "/media/user/6f1e9b86-52a9-4ef6-a5fd-8965ef1db31f/LLM/colibri/c/openai_server.py", line 1448, in handle_one_request
    super().handle_one_request()
    ~~~~~~~~~~~~~~~~~~~~~~~~~~^^
  File "/usr/lib/python3.13/http/server.py", line 436, in handle
    self.handle_one_request()
    ~~~~~~~~~~~~~~~~~~~~~~~^^
  File "/media/user/6f1e9b86-52a9-4ef6-a5fd-8965ef1db31f/LLM/colibri/c/openai_server.py", line 1494, in send_json
    self.wfile.write(data)
    ~~~~~~~~~~~~~~~~^^^^^^
  File "/usr/lib/python3.13/http/server.py", line 436, in handle
    self.handle_one_request()
    ~~~~~~~~~~~~~~~~~~~~~~~^^
  File "/media/user/6f1e9b86-52a9-4ef6-a5fd-8965ef1db31f/LLM/colibri/c/openai_server.py", line 1448, in handle_one_request
    super().handle_one_request()
    ~~~~~~~~~~~~~~~~~~~~~~~~~~^^
  File "/media/user/6f1e9b86-52a9-4ef6-a5fd-8965ef1db31f/LLM/colibri/c/openai_server.py", line 1448, in handle_one_request
    super().handle_one_request()
    ~~~~~~~~~~~~~~~~~~~~~~~~~~^^
  File "/usr/lib/python3.13/socketserver.py", line 845, in write
    self._sock.sendall(b)
    ~~~~~~~~~~~~~~~~~~^^^
  File "/usr/lib/python3.13/http/server.py", line 424, in handle_one_request
    method()
    ~~~~~~^^
  File "/media/user/6f1e9b86-52a9-4ef6-a5fd-8965ef1db31f/LLM/colibri/c/openai_server.py", line 1632, in do_GET
    self.send_json(200, payload, request_id)
    ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/media/user/6f1e9b86-52a9-4ef6-a5fd-8965ef1db31f/LLM/colibri/c/openai_server.py", line 1448, in handle_one_request
    super().handle_one_request()
    ~~~~~~~~~~~~~~~~~~~~~~~~~~^^
  File "/usr/lib/python3.13/http/server.py", line 424, in handle_one_request
    method()
    ~~~~~~^^
  File "/media/user/6f1e9b86-52a9-4ef6-a5fd-8965ef1db31f/LLM/colibri/c/openai_server.py", line 1494, in send_json
    self.wfile.write(data)
    ~~~~~~~~~~~~~~~~^^^^^^
  File "/usr/lib/python3.13/socketserver.py", line 845, in write
    self._sock.sendall(b)
    ~~~~~~~~~~~~~~~~~~^^^
  File "/usr/lib/python3.13/http/server.py", line 424, in handle_one_request
    method()
    ~~~~~~^^
  File "/media/user/6f1e9b86-52a9-4ef6-a5fd-8965ef1db31f/LLM/colibri/c/openai_server.py", line 1632, in do_GET
    self.send_json(200, payload, request_id)
    ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/media/user/6f1e9b86-52a9-4ef6-a5fd-8965ef1db31f/LLM/colibri/c/openai_server.py", line 1494, in send_json
    self.wfile.write(data)
    ~~~~~~~~~~~~~~~~^^^^^^
BrokenPipeError: [Errno 32] Broken pipe
  File "/media/user/6f1e9b86-52a9-4ef6-a5fd-8965ef1db31f/LLM/colibri/c/openai_server.py", line 1632, in do_GET
    self.send_json(200, payload, request_id)
    ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^
BrokenPipeError: [Errno 32] Broken pipe
----------------------------------------
  File "/usr/lib/python3.13/http/server.py", line 424, in handle_one_request
    method()
    ~~~~~~^^
  File "/media/user/6f1e9b86-52a9-4ef6-a5fd-8965ef1db31f/LLM/colibri/c/openai_server.py", line 1632, in do_GET
    self.send_json(200, payload, request_id)
    ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/media/user/6f1e9b86-52a9-4ef6-a5fd-8965ef1db31f/LLM/colibri/c/openai_server.py", line 1494, in send_json
    self.wfile.write(data)
    ~~~~~~~~~~~~~~~~^^^^^^
  File "/usr/lib/python3.13/http/server.py", line 436, in handle
    self.handle_one_request()
    ~~~~~~~~~~~~~~~~~~~~~~~^^
  File "/usr/lib/python3.13/socketserver.py", line 845, in write
    self._sock.sendall(b)
    ~~~~~~~~~~~~~~~~~~^^^
  File "/media/user/6f1e9b86-52a9-4ef6-a5fd-8965ef1db31f/LLM/colibri/c/openai_server.py", line 1494, in send_json
    self.wfile.write(data)
    ~~~~~~~~~~~~~~~~^^^^^^
----------------------------------------
BrokenPipeError: [Errno 32] Broken pipe
  File "/usr/lib/python3.13/socketserver.py", line 845, in write
    self._sock.sendall(b)
    ~~~~~~~~~~~~~~~~~~^^^
  File "/usr/lib/python3.13/socketserver.py", line 845, in write
    self._sock.sendall(b)
    ~~~~~~~~~~~~~~~~~~^^^
----------------------------------------
  File "/media/user/6f1e9b86-52a9-4ef6-a5fd-8965ef1db31f/LLM/colibri/c/openai_server.py", line 1448, in handle_one_request
    super().handle_one_request()
    ~~~~~~~~~~~~~~~~~~~~~~~~~~^^
  File "/usr/lib/python3.13/http/server.py", line 424, in handle_one_request
    method()
    ~~~~~~^^
BrokenPipeError: [Errno 32] Broken pipe
  File "/media/user/6f1e9b86-52a9-4ef6-a5fd-8965ef1db31f/LLM/colibri/c/openai_server.py", line 1632, in do_GET
    self.send_json(200, payload, request_id)
    ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^
----------------------------------------
  File "/media/user/6f1e9b86-52a9-4ef6-a5fd-8965ef1db31f/LLM/colibri/c/openai_server.py", line 1494, in send_json
    self.wfile.write(data)
    ~~~~~~~~~~~~~~~~^^^^^^
BrokenPipeError: [Errno 32] Broken pipe
  File "/usr/lib/python3.13/socketserver.py", line 845, in write
    self._sock.sendall(b)
    ~~~~~~~~~~~~~~~~~~^^^
----------------------------------------
BrokenPipeError: [Errno 32] Broken pipe
----------------------------------------
[api] 127.0.0.1 - "GET /health HTTP/1.1" 200 -
[api] 127.0.0.1 - "GET /v1/models HTTP/1.1" 200 -
  ✦ attached to http://127.0.0.1:8000 · model kimi-k3-colibri · the engine stays warm after you quit
  type and press Enter · Ctrl-C stops the answer · :reset starts a new conversation · :q exits

  ╭────────────────────────────────────────────────────────────────────────────────────────────────╮
  │ › test
  ╰────────────────────────────────────────────────────────────────────────────────────────────────╯

  ◆ colibri
  ⠋ thinking… 0s[api] 127.0.0.1 - "POST /v1/chat/completions HTTP/1.1" 200 -
  Hi! It looks like you're testing the connection. I'm here and ready to help.
  

  └─ ~26 tok · 394s

K3_EXPERT_GB=12

[K3] loaded layer 1/93 (4.5s, RSS 1.4 GB)
[K3] loaded layer 9/93 (21.4s, RSS 4.2 GB)
[K3] loaded layer 17/93 (37.5s, RSS 7.1 GB)
[K3] loaded layer 25/93 (53.6s, RSS 10.0 GB)
[K3] loaded layer 33/93 (69.8s, RSS 12.9 GB)
[K3] loaded layer 41/93 (86.1s, RSS 15.8 GB)
[K3] loaded layer 49/93 (102.8s, RSS 18.7 GB)
[K3] loaded layer 57/93 (119.5s, RSS 21.6 GB)
[K3] loaded layer 65/93 (136.3s, RSS 24.5 GB)
[K3] loaded layer 73/93 (153.3s, RSS 27.4 GB)
[K3] loaded layer 81/93 (170.1s, RSS 30.3 GB)
[K3] loaded layer 89/93 (186.4s, RSS 33.2 GB)
[K3] init done in 198.2s | 93 layers | expert cache 7/layer (17.5 MB/slot) | RSS 35.7 GB
[K3] tokenizer.json loaded (family=kimi)
[K3] prompt: 10 tokens | ngen 32 | temp 0.00
[K3] prefill 10/10 (54.1s)
[K3] prefill done in 54.1s (0.18 tok/s)
 What  [tok 1: 13.1s/tok, hit 1%, 179.4 GB read]
 is  [tok 2: 12.3s/tok, hit 3%, 200.0 GB read]
 the  [tok 3: 11.3s/tok, hit 4%, 222.5 GB read]
 difference  [tok 4: 10.7s/tok, hit 5%, 245.8 GB read]
 between  [tok 5: 10.0s/tok, hit 6%, 267.8 GB read]
 NV  [tok 6: 9.8s/tok, hit 6%, 290.6 GB read]
Me  [tok 7: 9.5s/tok, hit 7%, 311.4 GB read]
 and  [tok 8: 9.5s/tok, hit 8%, 332.5 GB read]
 M  [tok 9: 9.7s/tok, hit 8%, 354.9 GB read]
.  [tok 10: 9.5s/tok, hit 9%, 375.6 GB read]
2  [tok 11: 9.4s/tok, hit 9%, 398.0 GB read]
?

  [tok 12: 9.4s/tok, hit 10%, 419.9 GB read]
NV  [tok 13: 9.5s/tok, hit 10%, 442.2 GB read]
Me  [tok 14: 9.2s/tok, hit 10%, 464.6 GB read]
 is  [tok 15: 9.3s/tok, hit 10%, 486.1 GB read]
 a  [tok 16: 9.2s/tok, hit 11%, 508.3 GB read]
 protocol  [tok 17: 9.2s/tok, hit 10%, 532.1 GB read]
 that  [tok 18: 9.1s/tok, hit 11%, 552.9 GB read]
 is  [tok 19: 9.2s/tok, hit 11%, 574.7 GB read]
 used  [tok 20: 9.1s/tok, hit 11%, 597.6 GB read]
 to  [tok 21: 9.2s/tok, hit 11%, 618.7 GB read]
 access  [tok 22: 9.2s/tok, hit 11%, 640.7 GB read]
 the  [tok 23: 9.2s/tok, hit 11%, 664.0 GB read]
 flash  [tok 24: 9.3s/tok, hit 11%, 687.9 GB read]
 storage  [tok 25: 9.3s/tok, hit 11%, 708.6 GB read]
.  [tok 26: 9.6s/tok, hit 11%, 731.5 GB read]
 It  [tok 27: 9.5s/tok, hit 11%, 754.6 GB read]
 is  [tok 28: 9.5s/tok, hit 11%, 777.8 GB read]
 a  [tok 29: 9.6s/tok, hit 11%, 800.7 GB read]
 new  [tok 30: 9.5s/tok, hit 12%, 821.9 GB read]
 protocol  [tok 31: 9.4s/tok, hit 12%, 843.0 GB read]
 that
[K3] context full

[K3] decode 32 tokens in 292.6s (0.11 tok/s) | expert hit 11.8% (6415/54455) | 843.0 GB streamed
[K3] time: attn 54.7s moe 290.3s (eload 50.9s) head 1.2s | RSS 47.2 GB

K3_EXPERT_GB=18

[K3] loaded layer 1/93 (4.2s, RSS 1.4 GB)
[K3] loaded layer 9/93 (20.4s, RSS 4.2 GB)
[K3] loaded layer 17/93 (36.4s, RSS 7.1 GB)
[K3] loaded layer 25/93 (52.4s, RSS 10.0 GB)
[K3] loaded layer 33/93 (68.6s, RSS 12.9 GB)
[K3] loaded layer 41/93 (84.9s, RSS 15.8 GB)
[K3] loaded layer 49/93 (101.6s, RSS 18.7 GB)
[K3] loaded layer 57/93 (118.3s, RSS 21.6 GB)
[K3] loaded layer 65/93 (135.1s, RSS 24.5 GB)
[K3] loaded layer 73/93 (152.0s, RSS 27.4 GB)
[K3] loaded layer 81/93 (169.0s, RSS 30.3 GB)
[K3] loaded layer 89/93 (185.4s, RSS 33.2 GB)
[K3] init done in 197.3s | 93 layers | expert cache 11/layer (17.5 MB/slot) | RSS 35.7 GB
[K3] tokenizer.json loaded (family=kimi)
[K3] prompt: 10 tokens | ngen 32 | temp 0.00
[K3] prefill 10/10 (29.0s)
[K3] prefill done in 29.0s (0.34 tok/s)
 What  [tok 1: 5.9s/tok, hit 1%, 178.8 GB read]
 is  [tok 2: 5.2s/tok, hit 5%, 195.9 GB read]
 the  [tok 3: 4.8s/tok, hit 7%, 216.1 GB read]
 difference  [tok 4: 4.7s/tok, hit 8%, 238.0 GB read]
 between  [tok 5: 4.5s/tok, hit 9%, 257.5 GB read]
 NV  [tok 6: 4.5s/tok, hit 10%, 278.3 GB read]
Me  [tok 7: 4.5s/tok, hit 12%, 296.1 GB read]
 and  [tok 8: 4.4s/tok, hit 13%, 314.4 GB read]
 M  [tok 9: 4.7s/tok, hit 13%, 335.0 GB read]
.  [tok 10: 4.7s/tok, hit 15%, 352.7 GB read]
2  [tok 11: 4.6s/tok, hit 15%, 372.3 GB read]
?

  [tok 12: 4.6s/tok, hit 16%, 391.8 GB read]
NV  [tok 13: 4.6s/tok, hit 16%, 412.0 GB read]
Me  [tok 14: 4.6s/tok, hit 16%, 432.3 GB read]
 is  [tok 15: 4.5s/tok, hit 17%, 451.3 GB read]
 a  [tok 16: 4.5s/tok, hit 17%, 471.5 GB read]
 protocol  [tok 17: 4.6s/tok, hit 17%, 494.3 GB read]
 that  [tok 18: 4.7s/tok, hit 17%, 512.0 GB read]
 is  [tok 19: 4.7s/tok, hit 18%, 531.3 GB read]
 used  [tok 20: 4.7s/tok, hit 18%, 552.9 GB read]
 to  [tok 21: 4.7s/tok, hit 18%, 571.3 GB read]
 access  [tok 22: 4.7s/tok, hit 18%, 590.9 GB read]
 the  [tok 23: 4.7s/tok, hit 18%, 613.0 GB read]
 flash  [tok 24: 4.7s/tok, hit 18%, 635.7 GB read]
 storage  [tok 25: 4.7s/tok, hit 18%, 653.3 GB read]
.  [tok 26: 4.7s/tok, hit 18%, 674.6 GB read]
 It  [tok 27: 4.7s/tok, hit 18%, 696.1 GB read]
 is  [tok 28: 4.7s/tok, hit 18%, 718.0 GB read]
 a  [tok 29: 4.7s/tok, hit 18%, 739.4 GB read]
 new  [tok 30: 4.7s/tok, hit 18%, 757.9 GB read]
 protocol  [tok 31: 4.7s/tok, hit 19%, 775.7 GB read]
 that
[K3] context full

[K3] decode 32 tokens in 144.8s (0.22 tok/s) | expert hit 18.8% (10246/54455) | 775.7 GB streamed
[K3] time: attn 19.7s moe 153.1s (eload 65.2s) head 0.6s | RSS 53.2 GB

K3_EXPERT_GB=20

user@Cartelion:/media/user/6f1e9b86-52a9-4ef6-a5fd-8965ef1db31f/LLM$ ./install-coli.sh
[K3] loaded layer 1/93 (4.4s, RSS 1.4 GB)
[K3] loaded layer 9/93 (20.8s, RSS 4.2 GB)
[K3] loaded layer 17/93 (36.9s, RSS 7.1 GB)
[K3] loaded layer 25/93 (53.1s, RSS 10.0 GB)
[K3] loaded layer 33/93 (69.2s, RSS 12.9 GB)
[K3] loaded layer 41/93 (85.5s, RSS 15.8 GB)
[K3] loaded layer 49/93 (102.2s, RSS 18.7 GB)
[K3] loaded layer 57/93 (119.1s, RSS 21.6 GB)
[K3] loaded layer 65/93 (135.9s, RSS 24.5 GB)
[K3] loaded layer 73/93 (154.0s, RSS 27.4 GB)
[K3] loaded layer 81/93 (175.8s, RSS 30.3 GB)
[K3] loaded layer 89/93 (196.1s, RSS 33.2 GB)
[K3] init done in 210.6s | 93 layers | expert cache 12/layer (17.5 MB/slot) | RSS 35.7 GB
[K3] tokenizer.json loaded (family=kimi)
[K3] prompt: 10 tokens | ngen 32 | temp 0.00
[K3] prefill 10/10 (40.4s)
[K3] prefill done in 40.4s (0.25 tok/s)
 What  [tok 1: 8.3s/tok, hit 1%, 178.5 GB read]
 is  [tok 2: 6.5s/tok, hit 6%, 194.9 GB read]
 the  [tok 3: 5.6s/tok, hit 8%, 214.2 GB read]
 difference  [tok 4: 5.8s/tok, hit 9%, 235.8 GB read]
 between  [tok 5: 5.4s/tok, hit 10%, 254.6 GB read]
 NV  [tok 6: 5.4s/tok, hit 11%, 274.8 GB read]
Me  [tok 7: 5.4s/tok, hit 13%, 291.8 GB read]
 and  [tok 8: 5.2s/tok, hit 14%, 309.6 GB read]
 M  [tok 9: 5.2s/tok, hit 15%, 329.5 GB read]
.  [tok 10: 5.0s/tok, hit 16%, 346.3 GB read]
2  [tok 11: 4.9s/tok, hit 17%, 365.1 GB read]
?

  [tok 12: 5.0s/tok, hit 17%, 384.2 GB read]
NV  [tok 13: 4.9s/tok, hit 18%, 403.7 GB read]
Me  [tok 14: 4.9s/tok, hit 18%, 423.7 GB read]
 is  [tok 15: 4.9s/tok, hit 18%, 442.1 GB read]
 a  [tok 16: 4.9s/tok, hit 19%, 461.8 GB read]
 protocol  [tok 17: 4.9s/tok, hit 18%, 484.3 GB read]
 that  [tok 18: 4.8s/tok, hit 19%, 501.5 GB read]
 is  [tok 19: 4.7s/tok, hit 19%, 520.1 GB read]
 used  [tok 20: 4.7s/tok, hit 19%, 541.4 GB read]
 to  [tok 21: 4.7s/tok, hit 20%, 559.1 GB read]
 access  [tok 22: 4.7s/tok, hit 20%, 578.0 GB read]
 the  [tok 23: 4.7s/tok, hit 20%, 599.7 GB read]
 flash  [tok 24: 4.7s/tok, hit 20%, 622.3 GB read]
 storage  [tok 25: 4.7s/tok, hit 20%, 639.0 GB read]
.  [tok 26: 4.7s/tok, hit 20%, 659.9 GB read]
 It  [tok 27: 4.6s/tok, hit 20%, 680.9 GB read]
 is  [tok 28: 4.6s/tok, hit 20%, 702.4 GB read]
 a  [tok 29: 4.6s/tok, hit 20%, 723.4 GB read]
 new  [tok 30: 4.6s/tok, hit 20%, 741.2 GB read]
 protocol  [tok 31: 4.5s/tok, hit 21%, 758.3 GB read]
 that
[K3] context full

[K3] decode 32 tokens in 140.2s (0.23 tok/s) | expert hit 20.6% (11239/54455) | 758.3 GB streamed
[K3] time: attn 20.6s moe 159.0s (eload 62.9s) head 0.6s | RSS 54.6 GB

K3_EXPERT_GB=22

user@Cartelion:/media/user/6f1e9b86-52a9-4ef6-a5fd-8965ef1db31f/LLM$ ./install-coli.sh
[K3] loaded layer 1/93 (4.2s, RSS 1.4 GB)
[K3] loaded layer 9/93 (20.4s, RSS 4.2 GB)
[K3] loaded layer 17/93 (36.5s, RSS 7.1 GB)
[K3] loaded layer 25/93 (52.5s, RSS 10.0 GB)
[K3] loaded layer 33/93 (68.6s, RSS 12.9 GB)
[K3] loaded layer 41/93 (84.8s, RSS 15.8 GB)
[K3] loaded layer 49/93 (101.5s, RSS 18.7 GB)
[K3] loaded layer 57/93 (118.2s, RSS 21.6 GB)
[K3] loaded layer 65/93 (135.0s, RSS 24.5 GB)
[K3] loaded layer 73/93 (152.0s, RSS 27.4 GB)
[K3] loaded layer 81/93 (168.9s, RSS 30.3 GB)
[K3] loaded layer 89/93 (185.4s, RSS 33.2 GB)
[K3] init done in 197.2s | 93 layers | expert cache 13/layer (17.5 MB/slot) | RSS 35.7 GB
[K3] tokenizer.json loaded (family=kimi)
[K3] prompt: 10 tokens | ngen 32 | temp 0.00
./install-coli.sh: Zeile 15: 1833545 Getötet                K3_EXPERT_GB=22 ./c/kimi_k3 /media/user/6f1e9b86-52a9-4ef6-a5fd-8965ef1db31f/LLM/Kimi-K3 "Explain what NVMe is in two sentences."

Terminated because of RAM Overload

K3_EXPERT_GB=24

[K3] loaded layer 1/93 (4.1s, RSS 1.4 GB)
[K3] loaded layer 9/93 (20.2s, RSS 4.2 GB)
[K3] loaded layer 17/93 (36.3s, RSS 7.1 GB)
[K3] loaded layer 25/93 (52.3s, RSS 10.0 GB)
[K3] loaded layer 33/93 (68.5s, RSS 12.9 GB)
[K3] loaded layer 41/93 (85.2s, RSS 15.8 GB)
[K3] loaded layer 49/93 (101.8s, RSS 18.7 GB)
[K3] loaded layer 57/93 (118.5s, RSS 21.6 GB)
[K3] loaded layer 65/93 (135.3s, RSS 24.5 GB)
[K3] loaded layer 73/93 (152.3s, RSS 27.4 GB)
[K3] loaded layer 81/93 (169.3s, RSS 30.3 GB)
[K3] loaded layer 89/93 (185.8s, RSS 33.2 GB)
[K3] init done in 197.6s | 93 layers | expert cache 14/layer (17.5 MB/slot) | RSS 35.7 GB
[K3] tokenizer.json loaded (family=kimi)
[K3] prompt: 10 tokens | ngen 32 | temp 0.00
./install-coli.sh: Zeile 15: 1832838 Getötet                K3_EXPERT_GB=24 ./c/kimi_k3 /media/user/6f1e9b86-52a9-4ef6-a5fd-8965ef1db31f/LLM/Kimi-K3 "Explain what NVMe is in two sentences."

Terminated because of RAM Overload

[terrizoaguimor]

Two separate things in that run, and the first one explains why nothing I suggested earlier took
effect.

--ram 50 is being silently discarded

Your log still shows expert cache 4/layer — the same as before you added the flag. That is not
the flag failing to help; it never reached the engine. From c/coli:186:

def env_for_engine(a, arch):
    if arch == "glm":
        return env_for(a)              # GLM gets the full CLI translation
    env = os.environ.copy()
    if a.ngen: env["NGEN"] = str(a.ngen)
    if a.temp is not None: env["COLI_TEMP"] = str(a.temp)
    return env                          # kimi / inkling get only these two

For kimi and inkling, only --ngen and --temp are passed through. --ram, --ctx,
--vram, --cap, --topp, --topk and --repin are accepted by the parser and then dropped
without a word. That is worth a fix on its own, @JustVugg — silently ignoring a flag the CLI
advertises is worse than rejecting it, and this issue is what it costs.

The workaround is that os.environ.copy() does survive, so environment variables reach the engine
even though flags do not:

K3_EXPERT_GB=24 ./coli chat --model /path/to/Kimi-K3

With your 64 GB that should give roughly 14 slots per layer instead of 4. Watch RSS — it lands
near 29 GB + K3_EXPERT_GB.

The exception is a second, unrelated problem

Exception occurred during processing of request from ('127.0.0.1', …) is Python's socketserver
error handler, and the traceback it prints immediately afterwards is the part that identifies the
bug. It is missing from your paste — probably lost in the interleaving with the spinner and the
[api] log lines.

Could you capture it cleanly?

./run-coli.sh 2>/tmp/coli-err.log
# reproduce the failing request, then:
grep -A30 "Traceback" /tmp/coli-err.log | head -60

I cannot reproduce this end myself — I have GLM-5.2 on a CUDA box and it serves fine, but I do not
have the 1.5 TB K3 checkout, and this looks specific to the kimi path through
openai_server.py. The traceback would tell us in one line whether it is the Kimi chat renderer
(render_chat_kimi), the engine wire protocol, or something in request parsing.

One detail worth mentioning while you are there: /health returning 200 while every real request
throws suggests the server process and the engine handshake are both fine and it is failing inside
request handling, which is consistent with a renderer or parsing bug rather than anything to do
with the model loading. Your 252s load and tokenizer.json loaded (family=kimi) both look correct.

[MichaelFomenko]

K3_EXPERT_GB=24:
don't work, because of Ram Overload
K3_EXPERT_GB=22:
don't work, because of Ram Overload
K3_EXPERT_GB=20:
[K3] decode 32 tokens in 140.2s (0.23 tok/s) | expert hit 20.6% (11239/54455) | 758.3 GB streamed
[K3] time: attn 20.6s moe 159.0s (eload 62.9s) head 0.6s | RSS 54.6 GB
K3_EXPERT_GB=18:
[K3] decode 32 tokens in 144.8s (0.22 tok/s) | expert hit 18.8% (10246/54455) | 775.7 GB streamed
[K3] time: attn 19.7s moe 153.1s (eload 65.2s) head 0.6s | RSS 53.2 GB
K3_EXPERT_GB=12:
[K3] decode 32 tokens in 292.6s (0.11 tok/s) | expert hit 11.8% (6415/54455) | 843.0 GB streamed
[K3] time: attn 54.7s moe 290.3s (eload 50.9s) head 1.2s | RSS 47.2 GB

I run this:

./run-coli.sh 2>/tmp/coli-err.log

and got this Result:

user@Cartelion:/media/user/6f1e9b86-52a9-4ef6-a5fd-8965ef1db31f/LLM$ cat /tmp/coli-err.log
[K3] loaded layer 1/93 (4.3s, RSS 1.4 GB)
[K3] loaded layer 9/93 (20.3s, RSS 4.2 GB)
[K3] loaded layer 17/93 (36.4s, RSS 7.1 GB)
[K3] loaded layer 25/93 (52.4s, RSS 10.0 GB)
[K3] loaded layer 33/93 (68.4s, RSS 12.9 GB)
[K3] loaded layer 41/93 (84.6s, RSS 15.8 GB)
[K3] loaded layer 49/93 (101.4s, RSS 18.7 GB)
[K3] loaded layer 57/93 (118.2s, RSS 21.6 GB)
[K3] loaded layer 65/93 (135.1s, RSS 24.5 GB)
[K3] loaded layer 73/93 (152.1s, RSS 27.4 GB)
[K3] loaded layer 81/93 (169.0s, RSS 30.3 GB)
[K3] loaded layer 89/93 (185.5s, RSS 33.2 GB)
[K3] init done in 197.3s | 93 layers | expert cache 12/layer (17.5 MB/slot) | RSS 35.7 GB
[K3] tokenizer.json loaded (family=kimi)
OpenAI-compatible API listening on http://127.0.0.1:8000/v1
[api] 127.0.0.1 - "GET /health HTTP/1.1" 200 -
[api] 127.0.0.1 - "GET /health HTTP/1.1" 200 -
----------------------------------------
----------------------------------------
Exception occurred during processing of request from ('127.0.0.1', 55128)
Exception occurred during processing of request from ('127.0.0.1', 55140)
[api] 127.0.0.1 - "GET /health HTTP/1.1" 200 -
[api] 127.0.0.1 - "GET /health HTTP/1.1" 200 -
[api] 127.0.0.1 - "GET /health HTTP/1.1" 200 -
[api] 127.0.0.1 - "GET /health HTTP/1.1" 200 -
----------------------------------------
----------------------------------------
Exception occurred during processing of request from ('127.0.0.1', 52956)
----------------------------------------
----------------------------------------
Exception occurred during processing of request from ('127.0.0.1', 55164)
Exception occurred during processing of request from ('127.0.0.1', 55154)
Exception occurred during processing of request from ('127.0.0.1', 52972)
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
  File "/usr/lib/python3.13/socketserver.py", line 697, in process_request_thread
    self.finish_request(request, client_address)
    ~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.13/socketserver.py", line 362, in finish_request
    self.RequestHandlerClass(request, client_address, self)
    ~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Traceback (most recent call last):
  File "/usr/lib/python3.13/socketserver.py", line 697, in process_request_thread
    self.finish_request(request, client_address)
    ~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.13/socketserver.py", line 697, in process_request_thread
    self.finish_request(request, client_address)
    ~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.13/socketserver.py", line 362, in finish_request
    self.RequestHandlerClass(request, client_address, self)
    ~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.13/socketserver.py", line 697, in process_request_thread
    self.finish_request(request, client_address)
    ~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.13/socketserver.py", line 362, in finish_request
    self.RequestHandlerClass(request, client_address, self)
    ~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.13/socketserver.py", line 766, in __init__
    self.handle()
    ~~~~~~~~~~~^^
  File "/usr/lib/python3.13/socketserver.py", line 697, in process_request_thread
    self.finish_request(request, client_address)
    ~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.13/socketserver.py", line 697, in process_request_thread
    self.finish_request(request, client_address)
    ~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.13/http/server.py", line 436, in handle
    self.handle_one_request()
    ~~~~~~~~~~~~~~~~~~~~~~~^^
  File "/usr/lib/python3.13/socketserver.py", line 766, in __init__
    self.handle()
    ~~~~~~~~~~~^^
  File "/usr/lib/python3.13/socketserver.py", line 766, in __init__
    self.handle()
    ~~~~~~~~~~~^^
  File "/media/user/6f1e9b86-52a9-4ef6-a5fd-8965ef1db31f/LLM/colibri/c/openai_server.py", line 1448, in handle_one_request
    super().handle_one_request()
    ~~~~~~~~~~~~~~~~~~~~~~~~~~^^
  File "/usr/lib/python3.13/socketserver.py", line 362, in finish_request
    self.RequestHandlerClass(request, client_address, self)
    ~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.13/socketserver.py", line 766, in __init__
    self.handle()
    ~~~~~~~~~~~^^
  File "/usr/lib/python3.13/http/server.py", line 436, in handle
    self.handle_one_request()
    ~~~~~~~~~~~~~~~~~~~~~~~^^
  File "/usr/lib/python3.13/http/server.py", line 424, in handle_one_request
    method()
    ~~~~~~^^
  File "/usr/lib/python3.13/http/server.py", line 436, in handle
    self.handle_one_request()
    ~~~~~~~~~~~~~~~~~~~~~~~^^
  File "/usr/lib/python3.13/socketserver.py", line 362, in finish_request
    self.RequestHandlerClass(request, client_address, self)
    ~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.13/socketserver.py", line 362, in finish_request
    self.RequestHandlerClass(request, client_address, self)
    ~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/media/user/6f1e9b86-52a9-4ef6-a5fd-8965ef1db31f/LLM/colibri/c/openai_server.py", line 1448, in handle_one_request
    super().handle_one_request()
    ~~~~~~~~~~~~~~~~~~~~~~~~~~^^
  File "/usr/lib/python3.13/socketserver.py", line 766, in __init__
    self.handle()
    ~~~~~~~~~~~^^
  File "/usr/lib/python3.13/http/server.py", line 436, in handle
    self.handle_one_request()
    ~~~~~~~~~~~~~~~~~~~~~~~^^
  File "/media/user/6f1e9b86-52a9-4ef6-a5fd-8965ef1db31f/LLM/colibri/c/openai_server.py", line 1632, in do_GET
    self.send_json(200, payload, request_id)
    ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.13/http/server.py", line 424, in handle_one_request
    method()
    ~~~~~~^^
  File "/usr/lib/python3.13/http/server.py", line 436, in handle
    self.handle_one_request()
    ~~~~~~~~~~~~~~~~~~~~~~~^^
  File "/media/user/6f1e9b86-52a9-4ef6-a5fd-8965ef1db31f/LLM/colibri/c/openai_server.py", line 1448, in handle_one_request
    super().handle_one_request()
    ~~~~~~~~~~~~~~~~~~~~~~~~~~^^
  File "/usr/lib/python3.13/socketserver.py", line 766, in __init__
    self.handle()
    ~~~~~~~~~~~^^
  File "/media/user/6f1e9b86-52a9-4ef6-a5fd-8965ef1db31f/LLM/colibri/c/openai_server.py", line 1632, in do_GET
    self.send_json(200, payload, request_id)
    ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/media/user/6f1e9b86-52a9-4ef6-a5fd-8965ef1db31f/LLM/colibri/c/openai_server.py", line 1448, in handle_one_request
    super().handle_one_request()
    ~~~~~~~~~~~~~~~~~~~~~~~~~~^^
  File "/usr/lib/python3.13/http/server.py", line 424, in handle_one_request
    method()
    ~~~~~~^^
  File "/media/user/6f1e9b86-52a9-4ef6-a5fd-8965ef1db31f/LLM/colibri/c/openai_server.py", line 1494, in send_json
    self.wfile.write(data)
    ~~~~~~~~~~~~~~~~^^^^^^
  File "/media/user/6f1e9b86-52a9-4ef6-a5fd-8965ef1db31f/LLM/colibri/c/openai_server.py", line 1494, in send_json
    self.wfile.write(data)
    ~~~~~~~~~~~~~~~~^^^^^^
  File "/usr/lib/python3.13/http/server.py", line 424, in handle_one_request
    method()
    ~~~~~~^^
  File "/media/user/6f1e9b86-52a9-4ef6-a5fd-8965ef1db31f/LLM/colibri/c/openai_server.py", line 1632, in do_GET
    self.send_json(200, payload, request_id)
    ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.13/http/server.py", line 436, in handle
    self.handle_one_request()
    ~~~~~~~~~~~~~~~~~~~~~~~^^
  File "/usr/lib/python3.13/socketserver.py", line 845, in write
    self._sock.sendall(b)
    ~~~~~~~~~~~~~~~~~~^^^
  File "/media/user/6f1e9b86-52a9-4ef6-a5fd-8965ef1db31f/LLM/colibri/c/openai_server.py", line 1632, in do_GET
    self.send_json(200, payload, request_id)
    ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/media/user/6f1e9b86-52a9-4ef6-a5fd-8965ef1db31f/LLM/colibri/c/openai_server.py", line 1494, in send_json
    self.wfile.write(data)
    ~~~~~~~~~~~~~~~~^^^^^^
  File "/usr/lib/python3.13/socketserver.py", line 845, in write
    self._sock.sendall(b)
    ~~~~~~~~~~~~~~~~~~^^^
  File "/media/user/6f1e9b86-52a9-4ef6-a5fd-8965ef1db31f/LLM/colibri/c/openai_server.py", line 1448, in handle_one_request
    super().handle_one_request()
    ~~~~~~~~~~~~~~~~~~~~~~~~~~^^
  File "/media/user/6f1e9b86-52a9-4ef6-a5fd-8965ef1db31f/LLM/colibri/c/openai_server.py", line 1494, in send_json
    self.wfile.write(data)
    ~~~~~~~~~~~~~~~~^^^^^^
BrokenPipeError: [Errno 32] Broken pipe
  File "/media/user/6f1e9b86-52a9-4ef6-a5fd-8965ef1db31f/LLM/colibri/c/openai_server.py", line 1448, in handle_one_request
    super().handle_one_request()
    ~~~~~~~~~~~~~~~~~~~~~~~~~~^^
  File "/usr/lib/python3.13/http/server.py", line 424, in handle_one_request
    method()
    ~~~~~~^^
  File "/usr/lib/python3.13/socketserver.py", line 845, in write
    self._sock.sendall(b)
    ~~~~~~~~~~~~~~~~~~^^^
----------------------------------------
BrokenPipeError: [Errno 32] Broken pipe
  File "/usr/lib/python3.13/socketserver.py", line 845, in write
    self._sock.sendall(b)
    ~~~~~~~~~~~~~~~~~~^^^
BrokenPipeError: [Errno 32] Broken pipe
----------------------------------------
BrokenPipeError: [Errno 32] Broken pipe
  File "/media/user/6f1e9b86-52a9-4ef6-a5fd-8965ef1db31f/LLM/colibri/c/openai_server.py", line 1632, in do_GET
    self.send_json(200, payload, request_id)
    ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.13/http/server.py", line 424, in handle_one_request
    method()
    ~~~~~~^^
----------------------------------------
  File "/media/user/6f1e9b86-52a9-4ef6-a5fd-8965ef1db31f/LLM/colibri/c/openai_server.py", line 1494, in send_json
    self.wfile.write(data)
    ~~~~~~~~~~~~~~~~^^^^^^
  File "/media/user/6f1e9b86-52a9-4ef6-a5fd-8965ef1db31f/LLM/colibri/c/openai_server.py", line 1632, in do_GET
    self.send_json(200, payload, request_id)
    ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.13/socketserver.py", line 845, in write
    self._sock.sendall(b)
    ~~~~~~~~~~~~~~~~~~^^^
  File "/media/user/6f1e9b86-52a9-4ef6-a5fd-8965ef1db31f/LLM/colibri/c/openai_server.py", line 1494, in send_json
    self.wfile.write(data)
    ~~~~~~~~~~~~~~~~^^^^^^
----------------------------------------
  File "/usr/lib/python3.13/socketserver.py", line 845, in write
    self._sock.sendall(b)
    ~~~~~~~~~~~~~~~~~~^^^
BrokenPipeError: [Errno 32] Broken pipe
BrokenPipeError: [Errno 32] Broken pipe
----------------------------------------
----------------------------------------
[api] 127.0.0.1 - "GET /health HTTP/1.1" 200 -
[api] 127.0.0.1 - "GET /v1/models HTTP/1.1" 200 -

[AlphaMo99]

Is it allowed/possible to use the GGUF version from unsloth !? https://huggingface.co/unsloth/Kimi-K3-GGUF

[MichaelFomenko]

Is it allowed/possible to use the GGUF version from unsloth !? https://huggingface.co/unsloth/Kimi-K3-GGUF

not now, but technically it should be possible in the future. GGUF is a different Storage Format for the Weights.

[ZacharyZcR]

The latest logs separate two issues that were mixed together in the original
report.

1. The v1.3 release package did have a bootstrap gap: K3 ships
   tiktoken.model, while the engine expected a generated tokenizer.json.
   Building current dev and running the generator unblocked startup. That
   packaging/docs path is the actionable bug here.
2. The current 0.22-0.23 tok/s result is no longer a launch failure. On a
   64 GB host with two GT 1030s and only 18-20 GB of expert residency, the
   20.6% hit rate forces hundreds of GB of expert streaming per short decode.
   That is the expected low-memory I/O regime, not evidence that the tokenizer
   fix failed.

GGUF is not currently a supported Colibri input format; it cannot be substituted
for the native container without a converter/loader path.

I suggest keeping this issue scoped to making the release self-contained:
either ship the generated tokenizer artifact or make first-run bootstrap
automatic and documented. Any work on improving the 64 GB / GT 1030 performance
case should be tracked separately, because it has a different root cause and
acceptance criterion.

[MichaelFomenko]

why does it not just load the tiktoken.model directly? why the extra step with the extra file?

why do we need synthesizes tokenizer.json from tiktoken.model with c/tools/k3_tokenizer.py ?

[terrizoaguimor]

Because tok.h is one tokenizer shared by all four engines, and it speaks HuggingFace
tokenizer.json. Reading tiktoken.model directly would mean a second on-disk format parser in
the C tokenizer — for one model family.

Worth knowing, though: the conversion is a container change, not a lossy one. From
k3_tokenizer.py's own header, the merges list is left empty on purpose:

Rank-recovered merge lists cannot be made exactly faithful (verified: "newlines" merges
new+l+ines instead of new+lines). tok.h detects the empty list and switches to rank-BPE —
merge the adjacent pair whose concatenation has the lowest vocab id — which IS tiktoken's
algorithm, so encoding is exact by construction.

So you are not getting an approximation of tiktoken; you are getting tiktoken's algorithm with the
vocab repackaged into the format the engine already reads.

That said, your underlying point stands and is the actionable part of this issue: it is a manual
step the release does not ship and does not mention. Whether the fix is shipping the generated
artifact or bootstrapping it on first run, nobody should have to discover k3_tokenizer.py from a
source comment.

[JustVugg]

Fixed on dev, and both halves of it — thank you for the report, and to @terrizoaguimor for the diagnosis that found the part you could not see.

What you hit was a launcher message. What was actually blocking you was that the v1.3.0 archive contained no Kimi K3 engine at all. Nothing you could have typed would have worked, so "maybe I done something wrong" — you didn't.

Two changes, both merged:

#736 — the engines did not compile on Windows. inkling and kimi_k3 polled stdin with fd_set/select, which msys2/UCRT64 does not provide in that form, so they failed to build outright. Measured across the release matrix beforehand: Linux ✅, macOS ✅, Windows ❌. The portable version — carrying two bug fixes from colibri.c, #139 (winsock returns SOCKET_ERROR on pipe handles) and #195 (anonymous pipes are not waitable objects) — now lives once in compat.h.

CI also now builds all four engines on Linux, macOS and Windows. Before this it built colibri + inkling on Linux only and kimi_k3 on nothing, which is exactly why a whole engine could stop compiling on two platforms and nobody noticed for months.

#737 — releases now ship every engine, plus tools/k3_tokenizer.py. Kimi ships tiktoken.model rather than tokenizer.json, and without that script the engine would be in the archive and still undrivable. 863 KB for all four. The release job's verification step was extended to assert each engine is present and executable and that the launcher's own resolver finds it — a packaging mistake is invisible to every other job here, so it is asserted rather than assumed.

From the next release you will be able to download and run Kimi K3 directly. Until then the source build you already got working is the way, and your install-coli.sh was correct.

Two things worth keeping from your run, since they are useful data:

• 0.10 tok/s at 6.7% expert hit, 891 GB streamed for 32 tokens. That is the disk-bound regime working as designed, not a defect — the token is bytes off the 990 Pro, not compute. PIN warm-up and K3_TOPP are the levers that reduce bytes-per-token.
• Your GT 1030s are sm_61 and colibrì's CUDA kernels target sm_80+, so they will not be used. CPU + NVMe streaming is the design point anyway, and your drive is the right part for it.

Closing as fixed. Thank you for the clean report — the full install-coli.sh and the complete log with per-token timings is what made this quick to place.