[Feature]: Streaming from two or more SSDs

[pbz134]

Problem

Hello!

I have 2 M.2 SSDs in my PC (1TB & 2TB), and I want to know if Colibri is capable of streaming the model from both SSDs at the same time?
Is there any performance gain to this?

I also have a RTX 4070S, so can CUDA and dual SSD streaming be combined?

I'm awaiting your reply, thank you for such a great tool! :)

Proposed solution

Store a part of the GLM model on each SSD and make Colibri use both efficiently, for a theoretical streaming performance increase of ~100%

Alternatives considered

No response

Scope and compatibility

No response

[dnnspaul]

Natively, calibri isn't able to do this. But you can configure the SSDs in a RAID0 to get performance benefits, which IMO is a much more native solution than implementing this into the application itself.

[steve-m]

See #421

[JustVugg]

Yes — and it's actively being built right now. PR #421 (Dual-SSD streaming: read the model from two drives at once, COLI_MODEL_MIRROR) does exactly this: stream the model from two drives at once. On a disk-bound setup like yours the two NVMe drives read in parallel, so the effective bandwidth (which is what caps tokens/second here) adds up. Closing this as a duplicate — follow #421 for progress, and thanks for the question!

[pbz134]

Yes — and it's actively being built right now. PR #421 (Dual-SSD streaming: read the model from two drives at once, COLI_MODEL_MIRROR) does exactly this: stream the model from two drives at once. On a disk-bound setup like yours the two NVMe drives read in parallel, so the effective bandwidth (which is what caps tokens/second here) adds up. Closing this as a duplicate — follow #421 for progress, and thanks for the question!

Thank you for your reply @JustVugg !
I am currently 0.3 t/s on my setup, which is honestly more than I expected on pure CPU inference. Once proper dual SSD support is implemented, I might be getting 0.6 t/s. "I/O Wait" is apparently at 64%.

Are you planning to release precompiled Windows CUDA binaries as well? It seems as if the Windows files from the releases only support CPU for now.

[JustVugg]

Reopening — this is now implemented and merged into dev (#421, dual-SSD streaming).

How to try it:

# put a second copy of the model on a different physical drive, then:
COLI_MODEL=/drive1/glm52_i4 COLI_MODEL_MIRROR=/drive2/glm52_i4 ./coli chat

Reads alternate across the two drives, roughly doubling streaming bandwidth on a disk-bound machine (the primary bottleneck for large-MoE decode). Per-drive split is tracked in the mirror counters (g_mir_bytes/g_mir_nread). Output is bit-identical to single-drive — it's a bandwidth trick, never a precision one.

Please report back here if you run it:

1. tok/s with vs without COLI_MODEL_MIRROR (same prompt, same machine) — does it actually ~2× on a disk-bound box?
2. Per-drive balance from the counters — are both drives sharing evenly, or is one starved?
3. Drive types (NVMe/SATA, matched or mixed) — mixed-speed mirrors are the interesting edge case.
4. A short chat transcript confirming output is unchanged.

Keeping this open as the tracking issue for real-world dual-SSD results. On a slow single disk today, a second budget NVMe is the cheapest real speedup available.

[JustVugg]

This shipped: COLI_MODEL_MIRROR (#421) reads the model from two drives at once (roughly doubling streaming bandwidth), and COLI_MODEL_DIRS (#469) spreads one container across N drives with no duplication (capacity aggregation). Both are in v1.1.0. Closing as implemented — reopen if your setup needs something these two don't cover.