Add DeepSeek V4 target-only CPU inference - #165
Conversation
DrewZt
changed the title
rajpratham1
left a comment
rajpratham1
left a comment
There was a problem hiding this comment.
This is a very impressive contribution and clearly represents a significant amount of engineering work. The implementation covers a complete DeepSeek V4 CPU inference pipeline including runtime, expert streaming, quantization, safetensors loading, CLI tooling, documentation, and an extensive unit test suite.
Because this PR introduces an entirely new inference stack across many core components, I'd prefer additional review before approval.
Some areas that would benefit from closer review include:
- Long-term API stability for the new DeepSeek V4 interfaces.
- Memory ownership and lifetime throughout the expert streaming/runtime pipeline.
- Performance characteristics of the NVMe streaming implementation under sustained inference.
- Cross-platform compatibility (Windows/Linux/macOS) for filesystem and I/O paths.
- Validation against larger real-world models beyond the included unit tests.
Overall the direction looks very promising, but given the size and architectural impact of this change, I think it should receive another maintainer review before merging.
|
Thanks for the thoughtful review. I agree that another maintainer review is appropriate given the size of the change. A few clarifications on the areas you mentioned: API stability: the new interfaces are currently scoped to the DeepSeek V4 engine and should be considered experimental. They are not intended to establish a stable generic model API at this stage. I’m happy to address targeted follow-up findings, add more validation, or split parts of the implementation into staged PRs if the maintainers feel that would make review and long-term maintenance easier. |
|
I pushed a follow-up series through The main changes are:
The latest changes pass: The remaining validation work is focused on sustained NVMe behavior and broader full-model comparison against the official Transformers implementation. Those limitations remain documented in the PR and do not represent unresolved API or resource-lifetime issues. I’m marking the PR ready for review and would appreciate another maintainer look, particularly at the revised API boundary and ownership model. |
|
First full-model validation on Linux/x86-64 — plus a build fix and an AVX2 kernel series, branch ready to pull: steve-m/colibri@v4-avx2-kernels (5 commits on top of this PR's Hardware: Ryzen 9 3900X (Zen 2, 12C/24T, AVX2 no AVX-512), 62 GiB RAM, NVMe ~6 GB/s, Manjaro, gcc 15. Model: Build fix you'll want regardless: two amalgam units ( Kernel series (all behind a
Measured, 48-token free-form decode, OMP_NUM_THREADS=12: 0.53 tok/s (PR defaults) → 0.62 (threads = physical cores; SMT only adds barrier traffic) → 0.78 (FP4 int8) → 0.99 (head) → 1.12 tok/s (FP8). Verify phase −27%, TTFT 20 → 16.7 s. Validation at every step: your oracle tool's Also on the branch, off by default: a dual-SSD mirror ( Happy to split any of this into separate PRs against your |
This is extremely helpful — thank you for doing the first independent full-model Linux/x86-64 validation and for documenting the performance progression in such detail. The Ryzen 3900X result directly addresses one of the main remaining validation gaps for this PR, and the 0.53 → 1.12 tok/s breakdown makes it much easier to see where the current bottlenecks are. I checked the branch history and it looks like Those later commits include the revised public/internal API separation, engine/session lifetime accounting, failure-path cleanup, isolated ownership-test objects, and V4 session tokenizer cleanup. The optimization work is still very valuable, but it should be rebased onto One clarification regarding the pthread build issue: I had already addressed the missing declaration problem in Your source-level includes may still be a cleaner and more localized solution, but the issue itself no longer needs a separate build-fix PR. When rebasing, please either drop the overlapping build-fix commit or call out why replacing the current compiler-level include with explicit includes in the two amalgam units would be preferable. For the remaining work, I suggest splitting it into two focused follow-ups:
I also noticed that the branch diff appears to include generated test binaries such as: Please drop those artifacts when preparing the follow-up PRs. After rebasing, please rerun the full-model correctness and performance validation because the runtime and ownership code has changed since The current benchmark result is already valuable as independent validation of the earlier implementation. Once the rebased kernel series is reviewed, I’d be happy to add the reproducible Linux/x86-64 measurements to the documentation with credit to you and the exact hardware and commands. Thanks again — this is a substantial and very useful contribution. Rebasing and splitting it should let us preserve the recent API and ownership work while giving the kernels and storage experiment the focused review they deserve. |
|
Follow-up for @JustVugg: head af86de1 closes the remaining validation findings. The safetensors index now requires payload bytes to equal dtype width times shape numel; V4 config integers, floats, and compress ratios now have finite/integer/range checks. I also removed the dense-cache borrowed config pointer in favor of the engine-owned canonical config and made the DSpark oracle require exact output lengths on both paths. Fresh validation passed make check (all C tests plus 71 Python tests), x86-64-v3/native builds, ASan+UBSan+LSan, and the 48-shard MEMORY_GB=32 oracle (19/19 teacher forcing, 8/8 greedy, 8/8 continuation self-check, exact DSpark on/off identity). This supersedes my earlier blocker summary; AVX2 and dual-SSD work remain out of scope. Could you please review when convenient? |
|
Pushed the rebased AVX2 kernel series to Two changes from the old
The three commits, all behind the
Validation on the rebased branch — Ryzen 9 3900X (Zen 2, AVX2 no AVX-512), 62 GiB, gcc 16.1.1,
Perf — AVX2 on vs the Kernel-only isolation, DSpark disabled (
For reference, DSpark on (same prompt): decode 0.62 → 0.85 tok/s (+36%), TTFT 43.6 → 30.3 s (−30%) — but that pass had asymmetric acceptance (on 3/10 vs off 1/10 speculative tokens), so the isolated The float-path baseline matches the known physical-cores number, so the kill-switch cleanly isolates the kernel gain. TTFT is stable across the DSpark condition (~31 s on / ~48 s off), as expected — the first token is prefill + first decode, before any speculation. (Absolute tok/s is workload-dependent — this was a heavy ~80 GB-streaming pass; a lighter/warmer pass on this box peaks around 1.12 tok/s. The on/off ratio is the stable claim.) |
This looks excellent — thank you for rebasing the series carefully and for preserving the API and ownership work underneath it. The updated scope is exactly what I was hoping for:
I also appreciate the distinction between the DSpark-enabled result and the The zero-copy resident-head path being exercised by the full-model oracle is also useful confirmation that the new path is covered rather than only compiled. Since the kernel branch is stacked on top of #165, I think the cleanest next step is to keep the branch as-is for now and open a dedicated follow-up PR after #165 is merged. At that point it can be rebased onto the resulting upstream Please preserve the current commit separation and validation details in that PR. In particular, the Thanks again — this is a strong follow-up series, and the careful validation and honest performance attribution make it much easier to review. |
|
Pushed follow-up stability fix
Validation:
|
|
I'm interested in this — DeepSeek V4 Flash on CPU with NVMe expert streaming is squarely what colibrì is for, and I'd like it in. The one condition is that I need to run it on my own machine first. Not as a gate to be difficult: it's the rule I've had to learn the hard way this week. So: as soon as I can get a checkpoint on this box, I'll test it and we'll work on it together. That's not a "no" parked forever — it's the next thing I want to do on this front. Two things that would make it land sooner, and I'd rather ask than have you guess:
I saw the stability fix you pushed ( Keeping this open. Let's keep going. |
Thanks — this gives me a clear integration path. I’ll make the tiny DeepSeek V4 fixture the immediate priority. The goal will be a deterministic, independently generated checkpoint and reference that The initial test contract will cover:
The generator will remain available so the fixture is reproducible rather than an opaque committed artifact, while CI itself will use the checked-in tiny checkpoint and reference. I’ll also continue removing local duplication where that does not change runtime or kernel boundaries. For the deeper shared-I/O work—particularly consolidating the streaming read path with the existing DIRECT/URING infrastructure—I would prefer to do that as a focused follow-up after this base engine and the already prepared AVX2 series land. That keeps the current correctness and performance baselines stable, avoids repeatedly invalidating the AVX2 branch, and lets the tiny token-exact oracle protect the later refactor across both scalar and optimized paths. I’ll keep AVX2 and the dual-SSD experiment out of this PR, and I’ll document the shortest full-checkpoint smoke-test path for your machine alongside the tiny fixture. |
|
Implemented the requested deterministic tiny DeepSeek V4 + DSpark oracle at head
Manual local validation (from the repository root): # Dedicated tiny target + DSpark token-exact oracle
make -C c deepseek-v4-tiny-check ARCH=x86-64-v3
# Or run the complete dependency-free check suite
make -C c checkTo exercise the normal runtime explicitly with drafting disabled: cd c
make deepseek-v4 ARCH=x86-64-v3
./deepseek_v4 deepseek_v4_tiny '<t005><t007><t009>' \
--raw-prompt --draft-model deepseek_v4_tiny/dspark --no-dsparkThe generator and regeneration package versions are documented in |
sorry what?! That's impressive for NVMe. |
The prompt 'What is the capital of France' have some uniqueness, the answer token got 100% dspark acceptance rate, that is the fastest speed boost. |
|
On your own Pc, Vincent, how does it compare to GLM in speed if you don't mind me asking (And what are the specs of yours , and what OS) 🤔From: DrewZt ***@***.***>Sent: Saturday, July 18, 2026 12:07:24 pmTo: JustVugg/colibri ***@***.***>Cc: Maikel Frias Mosquea ***@***.***>; Comment ***@***.***>Subject: Re: [JustVugg/colibri] Add DeepSeek V4 Flash CPU inference with NVMe expert streaming (PR #165)DrewZt left a comment (JustVugg/colibri#165)1.12 tok/s (FP8).sorry what?! That's impressive for NVMe.The prompt 'What is the capital of France' have some uniqueness, the answer token got 100% dspark acceptance rate, that is the fastest speed boost.
For another 0% dspark acceptance rate special case 'hello', decode speed will drop to 0.59tok/s, which is slower than no-dspark situation—Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you commented.
|
My pc is ai max 395+128gb ram+6gb/s ssd, tested glm with config think=0 and mtp=1, got around 0.7 tok/s |
|
Status check: |
Yes, I’m still actively working on this. Please keep the PR open. I reviewed the impact of the current I’m going to treat this as a minimal forward port onto current Work I will complete before merging #165
I had already done additional local work around redundant expert reads and the streamed I/O path, but I intentionally did not push it onto #165. At that point the branch had a stable correctness baseline and Steve’s AVX2 series was stacked on top of it; changing the I/O and cache boundaries again would have invalidated both his branch and the existing measurements. Work planned after #165 merges
So the immediate objective is to restore #165 on current |
|
The forward port is now pushed. PR #165 now points to The main Local The new PR workflows are currently waiting for maintainer approval. Could you approve them so the checks can run? |
|
I said earlier "today I merge it". I am walking that back, and you deserve the reasoning rather than silence, because most of it is our fault rather than yours — the tree moved underneath you while this was open. Two things I got wrong about your PR, corrected firstI had this filed in my head as "duplicates I also described it as "seven What changed underneath you, and why it matters
And here is why that is directly relevant. We measured DeepSeek-V4-Flash's actual shard headers rather than trusting
The experts are declared So most of What we would like, and it is mostly deletionThe shape we have settled on across
And please wire the front end from the startThis is the part we would ask for even if nothing else changed. Users do not run engines directly — they run # c/coli:160 model_arch()
if "deepseek" in model_type: return "deepseek" # config.json says model_type: deepseek_v4
# c/coli:176 engine_for()
name = "inkling" if arch == "inkling" else "kimi_k3" # -> add "deepseek_v4"
DSpark
What we can do to help
No deadline from us, and nothing here is a complaint about the engineering. It is a request to land the same work in the shape the tree grew into while this PR was open. If some of it does not fit the way DeepSeek V4 actually works, say so — you have run this model and we have not. |
|
if you want help i'm here too make this merge faster! |
|
Correcting myself on DSpark — I asked you to hold it back, and the reasoning I gave was worse than the counter-argument. What I said was that 4,057 lines of speculative decoding would keep a working engine from landing for another week. That is true of the review cost, and it is the wrong thing to optimise for. DSpark is DeepSeek's, not an optimisation we invented. If it is part of how the model is meant to run, shipping without it hands people a half DeepSeek and asks them to judge it — and the first thing everyone will want to try is exactly the thing that is missing. Judging a model without the decoding path its authors designed for it is not a fair test of either. So: please keep it, but as the second of two consecutive PRs rather than folded into one.
Nothing gets thrown away, you do not rewrite anything, and neither of us reviews 13,000 lines in one sitting. If the first lands and the second is a day behind it, that is fine — people can run the model while the speculation is reviewed, instead of waiting for both. One thing I would still ask of the DSpark PR, and it is the same standard Take the time you need on both. Nothing here is urgent in a way that should push you into rushing a 9,000-line engine — and I would rather wait than have you strip something the model needs because a maintainer told you to. |
|
Understood — I’ll keep the existing DSpark implementation, but split the work into two consecutive PRs as suggested. For #165, I’ll focus on the target-only DeepSeek V4 engine, migrate it to the current shared I’ll preserve the current full implementation on a separate branch, then prepare the DSpark support as an immediately stacked follow-up on top of the target-engine PR. That follow-up will retain The target engine was developed and validated independently before DSpark was added, so this split should preserve the existing work without requiring the target runtime to be redesigned. |
|
For the target-engine cleanup, the main shared-infrastructure blocker is that upstream fmt=8 does not yet support the UE8M0 block scales used by DeepSeek V4. Could you add the UE8M0 scale variant to the shared quant.h implementation? The required weight format is: raw E4M3FN weight bytes in row-major [O, I] layout; It should coexist with the existing f32-scale fmt=8 path. Once that shared support is available, I’ll validate it against the current V4 implementation and the full DeepSeek-V4-Flash-0731 checkpoint, then remove the private FP8 weight path from #165. I’ll keep the activation QDQ and the CPU routed-expert kernels V4-specific for now, since they are separate from the missing shared weight-format support. |
# Conflicts: # c/olmoe.c
DrewZt
changed the title
|
The piece you asked for is on Your spec was exact and I checked it against the real tensors rather than implementing to the description: 128×128 blocks,
|
Native fp8 checkpoints write the fmt=8 block scale as one UE8M0 byte per block rather than one f32. The geometry is identical -- same shape, same meaning, same multiply -- so the load path now reads the sidecar with st_read_scale_f32, which accepts either encoding and always yields f32. matmul_fp8 is untouched and stays a single implementation with no branch in the hot loop, which is the point of expanding at load rather than decoding per block. Every other format still goes through st_read_f32_cap exactly as before: fmt 0/1/2/4/5/6 are byte-for-byte unchanged. An f32-scaled fmt=8 container behaves identically too, since st_read_scale_f32 dispatches to st_read_f32 for an F32 sidecar -- the same call that ran before. Requested by DrewZt on JustVugg#165, where it was the one shared-infrastructure blocker for moving the DeepSeek V4 engine onto the common quant path. Verified against the real DeepSeek-V4-Flash-0731 checkpoint: the attention sidecars read back as exactly 2^-12 and 2^-11, and the dequantised weights land at |max| 0.094 / |mean| 0.018. All four engines build; make check 288 tests OK. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
Following up on the shared #802 is based on the current
The branch is Once #802 is merged, I will update #165 onto the resulting |
|
@JustVugg, the stacked head advanced to
The earlier |
|
Datapoint on the current head ( Build:
#ifndef _WIN32
#include <sys/resource.h>
#endifWith that in place, on this box (DGX Spark GB10: Grace 10× Cortex-X925 + 10× Cortex-A725, 121 GB LPDDR5x, Ubuntu 24.04, gcc 13.3.0): Tiny oracle, head Full model — the tokens/s I promised: DSpark checkpoint (48 shards) in target-only mode, = 0.51 tok/s decode, cold, TTFT 37.6 s, expert hit 74.3%, NEON rows16 active (805 packed slots). Output is coherent (a clean Rayleigh-scattering paragraph). Against the 0.82 tok/s (DSpark checkpoint, Two minor notes while here:
|
|
@DrewZt Will this work on my system with 32 GB ram and 12 GB vram? This is smaller than GLM 5.2 which runs on my system, this should be even easier and faster to run correct? |
|
@DrewZt is all ready for merge? |
8 participants
Summary
This PR now contains the DeepSeek V4 target-only CPU engine requested in the latest review. DSpark speculative decoding has been removed from #165 and preserved on
DrewZt:pr165-full-dspark-backupfor a separate stacked follow-up.The target engine:
st.hfor safetensors indexing/range I/O;quant.hfor canonical fmt7 MXFP4 matmul;coli run,coli chat,coli serve, andcoli web;--no-dsparkas a compatibility no-op.Production is consolidated in
c/deepseek_v4.c. The standalonec/v4launcher, DSpark runtime/build units, DSpark fixtures/tests, and committed.safetensorsfixtures have been removed.Shared infrastructure status
st.hquant.hTODO(upstream-fmt7-rows16): migrate when shared quant exposes a resident rows16 APITODO(upstream-fmt8-ue8m0): replace when shared fmt8 UE8M0 decode existsThe last two paths remain only to keep the target engine usable and are explicitly marked for migration.
Unified serving
openai_server.pydetectsdeepseek_v4, renders native multi-turn V4 markers, and launches the persistentSUBMIT/DATA/DONEprotocol. Serving is target-only, greedy, one KV slot, and rejects tools/grammar. Requests re-prefill context while the engine, dense tensors, head, and expert cache stay warm.Fixture and CI
The target-only fixture is generated from pinned PyTorch
2.13.0+cpu, Transformers5.14.1, and safetensors0.8.0. Its reference comes from officialDeepseekV4ForCausalLM; there is no C-engine oracle fallback. Generated safetensors are ignored and not committed.The V4 CI job checks teacher-forcing and greedy token identity, compressed/long prompts beyond the 64-token prefill boundary, repeated engine/session lifetime,
--no-dsparkcompatibility, and two requests through one persistent server process.Validation
make -C c checkon Windows UCRT64: all C tests and 283 Python tests passed (21 platform skips)make -C c deepseek-v4 ARCH=x86-64-v3st.hpread/mirror and fmt8 loader tests passedThe capital of France is Paris.target_only=1The real-checkpoint output was also reproduced through
c/coli run.Commit structure
feat(st): support V4 checkpoint metadatarefactor(v4): preserve target-only runtimefeat(cli): route DeepSeek V4 through coli servingtest(v4): generate target-only oracle in CIdocs(v4): describe target-only engine splitFollow-up
The stacked DSpark PR will restore the saved speculative runtime on this target baseline, make
--no-dsparka real switch again, restore deterministic DSpark identity tests, and provide on/off performance and acceptance data across high-, medium-, and low-acceptance prompts.