deepseek_v4: reuse the KV prefix instead of re-prefilling every turn

[JustVugg]

DeepSeek V4 was the only engine in the tree that re-prefilled the entire context on every request. colibri.c has done same-slot prefix reuse since it had a serve path; inkling.c (#786) and kimi_k3.c (#787) got it through the shared kv_prefix.h. This makes DeepSeek the third user of that header.

The visible symptom is the one people report as "the second message is slower than the first". The engine, dense tensors, head and expert cache all stay warm across requests — but turn ten was still paying to prefill turns one through nine.

Why the reusable case is narrow, and why that is enough

The window attention state cannot be truncated to an arbitrary position. The sliding window is a ring buffer, and the compressor carries recurrent kv_state/score_state rather than per-position rows. That is a real difference from GLM: MLA rows there are position-addressed and self-contained, which is why colibri.c can truncate to any shared prefix and even copy rows between slots under COLI_KV_SHARE.

What this state can do is keep going. So the case handled here is exactly the one a conversation produces:

turn N+1's prompt begins with every id turn N fed — prompt and reply alike — and only the tail is new.

kv_prefix_reuse returns 0 unless the record is a strict prefix of the new prompt, so an identical prompt, a shorter one, or a divergent one all fall back to a full reset and prefill. No new failure mode: the fallback is the current behaviour.

Correctness

Positions stay absolute. The fresh tail is prefilled with start=reuse, so every token sees the position it would have seen in a cold run; only the batch indexing shifts, because the batch now holds the tail alone.

The record tracks what was actually fed, not what was asked for. The last generated token is emitted but never fed back, so tokens are recorded inside the decode loop as each one enters the state — recording in bulk afterwards would claim one token too many and corrupt the following turn. Every failure path taints the record, because a half-updated attention state matches neither the old ids nor the new ones.

kv_prefix_alloc failing is not an error. Per the header's own contract it leaves the record empty, kv_prefix_reuse returns 0, and every request prefills in full.

Tests

tests/test_deepseek_v4_prefix.py speaks the SUBMIT/DATA/DONE protocol directly and runs each second turn twice: once continuing a warm session, once against a freshly started engine. It requires the two to agree token for token.

It also asserts that reuse actually fired. That check matters more than it looks: an optimisation that silently never engages passes every correctness test ever written for it. The first version of this test did exactly that — it built a second turn shorter than the recorded sequence, reuse correctly declined, and the only thing that caught it was asserting on the count.

PASS prefix reuse: 11 tokens reused, output identical to a cold prefill
PASS prefix repeat: identical prompt re-prefills, answer unchanged
PASS prefix reset: divergent prompt re-prefills and matches cold

Wired into make deepseek-v4-tiny-check. The 11 existing checks still pass.

The DONE frame gained a trailing reuse count so the test can measure it rather than scrape stderr; openai_server.py parses len(fields) >= 7, so older readers ignore it. V4_PREFIX_LOG=1 prints the reuse length per request.

What this does not do

No cross-slot sharing — V4 serve is single-slot today. No arbitrary-position truncation; that would need a compressor snapshot at the reuse boundary, which is worth doing only if a workload turns up that needs it. Follow-up to #165 (@DrewZt).