Hardware environment available

[centralware]

To be honest, I quickly skimmed the read-me, but what I read was intriguing. I have two older consumer+ machines I am willing to assign some work to if it would help in your testing. (Dell Precision T5820, single CPU, 64GB ECC, 512GB SSD and dual Tesla P100-16 and Dell Precision T5610 20C/40T 2xCPU, 64GB ECC, 1TB M.2) both running Ubuntu 24.04 headless. If necessary, RAM can be increased with some forewarning. Our "Dell Land" has an image of 24.04, Python3/PIP, Nvidia R570 if GPU(s) are installed, ollama and open-webui all by default.

If you want testing done on either/both of these machines, please provide the specific instructions or bash script file(s) needed to carry out the mission you're after. If you want logs/data returned, please note contact information (email address) and the generated files you want attached.

Sorry, opening the network these machines are on is not currently an option but may be accommodated if the real need arises. During testing, the machines will be locked into their own network with a door to outbound WAN. If additional needs exist, please feel free to request. An LLM with the specs you've listed sounds very interesting where I'd love to see it grow! (In fact, a model that runs on ~25GB CPU RAM without a fraction of a token per second type of response time and has some real meat on its bones... that in itself is worth a look!)

TJ

[JustVugg]

Thank you — and these two boxes are more useful than you might think, each for a different open question:

1. The T5610 (dual-socket, 20C/40T) is exactly the NUMA test machine we need. GPU seems not helpful at all (0.03 TPS) - (4xA100/80G + 4TB RAM Test) #82 found a massive throughput collapse on a multi-socket server (158/160 GB of the process landed on one node), and we have measured pinning gains (+39.6% on 2 sockets) but no systematic 2-socket A/B. If you run one thing, run this: OMP_PROC_BIND=spread OMP_PLACES=cores ./coli run ... vs the same without the env vars, model on local SSD, and post both numbers with numactl --hardware output. That directly informs the NUMA-aware placement work.

2. The T5820 + 2× Tesla P100 (sm_60) tests the CUDA floor — everything GPU-side so far has been validated on modern cards (5090/A100/GB10). P100 is pre-tensor-core: make glm CUDA=1 CUDA_ARCH=sm_60, and whether the expert tier helps or the CPU path wins on a 2016 card is a datapoint nobody has.

Start with the README quickstart (int4 container, ./coli doctor first); 64 GB RAM is plenty for a good expert cache. Post numbers here and I'll fold them into the table. Welcome aboard.

[centralware]

Message received... work order being drawn up... the machines in question should be online this afternoon and we'll start incorporating your request. Now, someday... if this really proves to be a handsome LLM... my only hope would be to see it network-able... cluster-like model able to operate with strength on a CPU, in theory, would be a superhero model if ganged up on multiple machines (10GbE NICs are dirt-cheap these days, 40s are not horribly priced... keep the thought in the back of your mind?)

I'm still reasonably new to the inner-workings of language models and our little 12-GPU lab is more of a training ground for bigger and better days when it's more plausible to invest in higher end devices (though I've tested K series and a few M series Tesla cards and the P100 was hands-down winner for old hardware being taught new tricks.) 3090/5090/H series/A series... spending tens of thousands to put together a development playground isn't yet justified. :) )

Take care and I'll have results your way as soon as time permits.

~ TJ ~

[JustVugg]

Yes please — and your timing is better than you know.

The single biggest constraint on this project right now is that nobody here can test the platforms it claims to support. Today alone:

• Six Windows PRs were merged this week essentially blind, because the maintainer's box is Linux/WSL.
• feat(vk): opt-in Vulkan iGPU backend for quantized expert matmul #84 (Vulkan iGPU backend, 723 lines) and AMD GPU support via HIP (single-source backend) + CUDA_EXTEND VRAM-only expert tier #112 (AMD/HIP) are open and stalled for exactly one reason: no one can run them. The CI can prove they compile; that's all.
• A contributor spent a night debugging CUDA blue screens (cuda+engine: full fmt=4 (grouped int4 gs=64) support + diagnostic harness #298) with no way to isolate his hardware from his code.
• CI landed today (ci: make check on ubuntu / windows (MSYS2 UCRT64) / macos — and it already caught a Windows test bug (#140) #143) and its first Windows run caught a bug — in a test I'd written and only ever run on Linux.

So "two older consumer+ machines" is not a small offer. What would be most useful, in rough order:

1. Windows (native, MSYS2/UCRT64) — now covered by CI for make check, but nobody runs an actual model there. Reproducing Windows (native) installation #198 and Benchmark + Windows GPU-build fixes: GLM-5.2 on Core Ultra 9 285K + RTX 5080 (sm_120) #306 are both real users hitting real Windows friction.
2. Anything with an AMD GPU or an iGPU — that would unblock AMD GPU support via HIP (single-source backend) + CUDA_EXTEND VRAM-only expert tier #112 and feat(vk): opt-in Vulkan iGPU backend for quantized expert matmul #84, which are otherwise going to be closed unverified.
3. Anything with a NUMA/multi-socket layout — GPU seems not helpful at all (0.03 TPS) - (4xA100/80G + 4TB RAM Test) #82 and glm: COLI_NUMA=1 — interleave resident weights across NUMA nodes (+13% decode on 2-socket, raw mbind, zero deps) (#82) #313 turn on that.

If either machine has an NVMe and ~16-32 GB of RAM, it's squarely the target hardware: the whole thesis is a 744B model streamed from disk on a box that shouldn't be able to run it.

Concretely, the most valuable thing you could do without any commitment: run ./coli doctor and ./coli iobench on both and paste the output, plus what the boxes are. That tells us whether they close one of the gaps above. No pressure beyond that — and thank you for offering.

[xxxajk]

What'd be neat is to run this on what I use on llama.cpp, which is this Frankenstein setup:

Took a lot of knob turning to get it to run 27b model across and all on GPU, but it does work.
Just can't run the bigger ones, streaming would make a lot of sense.

[JustVugg]

Thank you — offers of hardware are genuinely useful here, since a lot of open questions are "does X pay on hardware I don't have." The highest-value runs right now, if you're up for them: the multi-worker PILOT A/B on a drive with a real QD8-32 sweet spot (#441/#480 — the one datapoint that would settle whether deeper prefetch pays), and any enterprise-NVMe / big-RAM tok/s numbers for the tracker. Also COLI_CUDA_MTP=1 at full residency if you have a GPU box (#467/#470 — the crossover where speculation might pay). What's the hardware, and I'll point you at the exact repro that unblocks the most.

[JustVugg]

Closing this as recorded, not as ignored — and the distinction deserves stating, because closing someone's contribution without one reads as dismissal.

Hardware datapoints are contributions of data, not requests for work. They stay valuable permanently: someone with your hardware, searching before they commit to a 400 GB download, wants exactly this. But sitting open in the issue tracker they read as unfinished tasks, and fifteen of them made the project look like it had sixty open problems when it had about twenty. That is a signal-to-noise cost paid by everyone who arrives wanting to help and cannot find where.

So: the numbers stay, the "open task" framing goes.

• The issue and its whole thread remain readable and searchable — nothing is deleted.
• Discussion #208 is the standing home for benchmark datapoints, and where new ones are most useful.
• docs/benchmarks.md is where datapoints that map a regime boundary get written up.

Reopen it without asking if there is a defect or an open question in here that I read past. Several benchmark reports in this tracker turned out to contain real bugs — #718's SMT collapse and #634's two defects are staying open for exactly that reason — and if yours is one of them I would rather be corrected than have it quietly closed.

Thank you for taking the time to measure and report it properly.

[centralware]

Good morning!

We did the 400GB download (with a 1GbE fiber used, it didn't take all "that" long, so that part wasn't such a big deal.)
It's sitting on a 1TB M.2 along with the compiled colibri... but between myself and two others; all engineers in electronics mind you... the verbiage being used is mind-blowing without a degree specific to LLM, so us laymen folks murmur things like "What the...???"

Excerpt from colbiri v1.3.0 readme:
"Preview status, stated plainly: the engine and tokenizer are validated — the multi-turn wire matches Moonshot's official encoding_k3.py at 77/77 token ids — and the regression suite passes. Full-model generation is still gated on a host with ~1.6 TB of storage for the snapshot; nobody has run it end to end yet. If you have that hardware, this one is yours to try."

1. multi-turn wire... this is an old-school wire-wrapping technique that hasn't been practiced in decades.
2. encoding_k3.py -- it's a Python script... okay... and it has 77 out of 77 token IDs... okay, that means something to someone...
3. ...is still "gated" on a host -- terminology for "generating" something and "gating" something clashes in OUR education, but maybe the dictionary has changed so much over the past 30 years that Webster should be thrown away, too. :)
   Different flavors of engineering -- guessing we don't blend well enough!

In short -- we stopped at "Step 3" of quickstart.md after reading numerous contradictions of environmental requirements... how VRAM requirements have to be "this much" yet the original concept was VRAM-less builds on a laptop... so we parked the work order for "rainy day" research. OUR in-house models predicted 0.015 tok/sec based on Gen3 hardware regardless of the speed of our storage drive... but we never had the chance to fire it up and see it for ourselves.

[JustVugg]

Good morning — and thank you. This is the most useful thing anyone has sent this project in a while, and I am reopening the issue because of it. I closed it this morning as a hardware-offer datapoint; your comment turned it into a documentation bug report, which is a different thing and worth staying open.

Three engineers with the hardware, the 400 GB already downloaded, and the engine compiled — stopped by our prose. That is entirely on us.

You are right about all three words, and here is what they meant

"multi-turn wire" — I mean the exact sequence of bytes sent to the model when a conversation has more than one back-and-forth turn. Nothing to do with wiring. "Wire format" is web-protocol jargon that has no business in a user-facing note, and your reading of it is the more sensible one.

"encoding_k3.py at 77/77 token ids" — Moonshot (Kimi K3's authors) publish a Python script that turns text into the numbers the model actually consumes. We wrote our own in C, ran both over 77 test conversations, and got identical numbers every time. It was meant to say "our text handling is not subtly wrong". It says nothing of the sort to anyone who has not been staring at tokenizers all week.

"gated on a host with ~1.6 TB" — "we cannot test this because nobody here owns a 1.6 TB drive". "Gated" is engineer-slang for "blocked by", and you are right that it collides badly with the ordinary meanings of gate and generate.

The contradiction you found is real, and it is our fault for putting two models on one page

That paragraph is about Kimi K3 — a 2.8 trillion-parameter model needing ~1.6 TB. The quickstart you were following is for GLM-5.2: ~380 GB, and no GPU at all.

Both true, about different models, printed near each other with nothing saying so. Reading them together produces exactly the contradiction you describe: "needs 1.6 TB and a serious machine" next to "runs on a laptop without a GPU". Nobody should have to work out that those are two different sentences about two different things.

The prerequisites for what you actually downloaded are:

RAM	~16 GB minimum, 24 GB comfortable
Disk	the ~400 GB you already have
GPU	none needed
Speed	disk-bound; on a good NVMe, fractions of a token per second to a few per second

That is it. You are past the hard part — the download is the expensive step and you have done it.

On your 0.015 tok/s prediction — I think your model was too pessimistic

That number would make it pointless, and I understand parking it on that basis. But reported measurements from comparable machines come out 10 to 30 times better:

• i9-12900K, 64 GB, Samsung 990 Pro: 0.34 tok/s ([Performance]: GLM-5.2 on i9-12900K / 64GB / 990 Pro — 0.34 tok/s, and MTP is a wash when disk-bound (mechanism + numbers) #640)
• Ryzen 9 5950X, 62 GB, Gen4 NVMe: 0.45 tok/s (Zen3 data: SMT collapse (2.3× at 16 threads), ±15% run variance, PIN_GB starves LRU — HIP/gfx1100 + NVMe #718)
• 128 GB CPU-only desktop: ~1.8 tok/s once the cache is warm (Two-step shared-expert router prediction: +3.1% recall over PILOT (experiment, with prior art) #200)

The likely gap is that a straight "bytes per token ÷ drive speed" model assumes every token reads the full expert set from disk. In practice a per-layer cache plus pinned hot experts means most of it never comes off the drive after the first few tokens — measured hit rates run 80% and up on a warm cache. Your Gen3 M.2 would land in the same range as those reports, not two orders below.

Worth twenty minutes to find out rather than trusting either your model or my estimate. If it comes out at 0.015 on your box, that is a genuine finding and I would want the numbers.

What we are going to change

Not asking you to translate for us — this is ours to fix:

1. Model-specific requirements next to each model, not one page mixing a 380 GB GLM with a 1.6 TB Kimi.
2. The jargon goes. "wire" → "the exact bytes sent to the model"; "gated on" → "we cannot test this until someone has"; the 77/77 line either explained in one clause or deleted.
3. A plain-language "what you need" block that survives being read by someone who did not build the thing.

You said it well: "different flavours of engineering — guessing we don't blend well enough". The blending is our job. A person who can download 400 GB, install a toolchain and compile a C engine is not the problem; if that person bounces off the words, the words are wrong.

If you do fire it up, ./coli doctor reports what is missing and how to fix it, and I am happy to answer here in plain terms — including "what does this line mean", which is a perfectly good question and one nobody else has bothered to ask.