Feature Request: DeepSeek V3.2-Exp support

[createthis]

Prerequisites

• I am running the latest code. Mention the version if possible as well.
• I carefully followed the README.md.
• I searched using keywords relevant to my issue to make sure that I am creating a new issue that is not already open (or closed).
• I reviewed the Discussions, and have a new and useful enhancement to share.

Feature Description

Support DeepSeek V3.2-Exp: https://huggingface.co/deepseek-ai/DeepSeek-V3.2-Exp

Motivation

Because we can.

Possible Implementation

VLLM code is here for reference: vllm-project/vllm#25896

[lovedheart]

I am not an expert on LLM. I put flashmla_sparse.py to Qwen-Max and it shows

[jukofyork]

It's really just the sparse attention that's going to be a problem:

(Ignore the scale_inv as they are for the quantisation)

[jukofyork]

This is the important code in VLLM:

https://github.com/vllm-project/vllm/blob/88ef733ff5125b1b29e7aaae13063b2a109d5e8f/vllm/model_executor/models/deepseek_v2.py#L768

https://github.com/vllm-project/vllm/blob/88ef733ff5125b1b29e7aaae13063b2a109d5e8f/vllm/model_executor/models/deepseek_v2.py#L584

[jukofyork]

https://github.com/deepseek-ai/DeepSeek-V3.2-Exp/blob/main/DeepSeek_V3_2.pdf

So it looks like it will need an extra cache to store a 128-element k_s value for each token at each layer, but other than this the actual calculation is fairly straightforward (assuming GGML has the needed primitive operation for the sparse multiplication).

[jukofyork]

@slaren is ggml_mul_mat_id() likely to work for this:

• Much larger k (2048) and n (~128k) compared to MoE selection logic.
• It's going to be multiplying a lot of 1xm tensors (specifically lots of 512 or 576 element vectors) instead of nxm tensors compared to the MoE logic.

[jukofyork]

Alternatively, we could just mask out the non-top-k tokens with zeros and do a normal ggml_mul_mat, but this seems a bit crap and doesn't really get any of the gains in speed from the sparsity...

[slaren]

I think it is very likely that at least it will be faster than a masked matrix multiplication, but ultimately it will depend on how well optimized are the backends for this case. You should be able to test the performance by using test-backend-ops in perf mode, and adding the relevant test cases to make_test_cases_perf.

[mattepiu]

Checking https://huggingface.co/deepseek-ai/DeepSeek-V3.2-Exp there's some cpp kernel code which might help more than python:
deepseek-ai/DeepGEMM@7c95b14
deepseek-ai/FlashMLA@c28eca9

[jukofyork]

Quick update ... I woke up early this morning and did my best to figure out the tensor mappings. I've updated my branch: https://github.com/createthis/llama.cpp/pull/9/files

Using this command the GGUF bf16 generates successfully:

python convert_hf_to_gguf.py
--outtype f16
--split-max-size 50G
--outfile /data2/DeepSeek-V3.2-Exp-GGUF/bf16
/data2/DeepSeek-V3.2-Exp-bf16/

Shard (30/30): 100%|█████████████████████████████████████████████████████████████████████████████| 7.52G/7.52G [00:16<00:00, 451Mbyte/s]
Writing: 100%|███████████████████████████████████████████████████████████████████████████████████| 1.34T/1.34T [49:23<00:00, 454Mbyte/s]
INFO:hf-to-gguf:Model successfully exported to /data2/DeepSeek-V3.2-Exp-GGUF/bf16/

This was monkey work, but I'm not 100% sure I named everything correctly. Can someone double check before we move forward with the hard part? Thanks!

I'm downloading it now, but will be a couple of days to 4MB/s.

[jukofyork]

I think at this point we're not using the fancy architectural features for max performance. We're just using the same patterns we've always used. I'm going to use V3.2-Exp to audit it's own inference code at this point, then once I'm satisfied with it, I'll look into implementing the fancier stuff.

It will probably work somewhat (at least at low context lengths) as the paper said it was trained from 3.1-terminus using around 1T tokens to learn the top-k stuff, but it won't work 100% properly as the sum going into the attention calculation will be larger that the model expects (ie: the values at the non-top-k indices should add zero into the sum). I suspect you'll start to see it break down more and more as you increase the context length and these sums become larger and larger.

[ciprianveg]

deepseek-ai/DeepSeek-Math-V2 another reason I am waiting for this implementation:)

[ciprianveg]

deepseek-ai/DeepSeek-V3.2 new model today