feat(kimi): reuse the attention state a previous turn already built; stop leaking Lc/Rc

[JustVugg]

Stacked on #786, which adds kv_prefix.h. Retarget to dev once that merges.

Same defect: a chat client resends the whole transcript each turn, and kimi_k3.c re-processed turns 1..N-1 from scratch. The cost of a message grew with the conversation, and every replayed position pulled its experts off disk again.

K3 needs the shared record more than the other engines, not less

There is no single KV to inspect. 69 of its layers are KDA — Kimi Delta Attention, a recurrent linear-attention state — and only the 24 MLA layers keep Lc/Rc. So "how much of this prompt does the current state already cover" cannot be read off any buffer; it has to be recorded where the tokens are fed. That is exactly what kv_prefix.h does, and it is why I did not want this logic inlined per engine.

The prefill loop already carried absolute positions:

for(int i=0;i<np;i+=chunk){ ... lo=step_chunk(m,ids+i,i,C); }

so reuse is starting it at reuse instead of 0.

Separately: kv_alloc leaked on every request

Serve calls it per turn, and it callocs over the previous pointers without freeing them:

static void kv_alloc(Model *m, int max_t){
    Cfg *c=&m->c; m->max_t=max_t;
    m->Lc=calloc(c->n_layers,sizeof(float*));   // old Lc leaked
    m->Rc=calloc(c->n_layers,sizeof(float*));   // old Rc leaked

n_layers × max_t × (kv_lora + qk_rope) floats per turn — hundreds of MB over a conversation on the 24 MLA layers at 4k context.

The fix belongs in this PR rather than its own because prefix reuse requires keeping those buffers across turns: growing them discards the positions the record describes. The lifetime question had to be answered either way, and answering it correctly happens to close the leak.

What is NOT verified — please read this part

Kimi K3 is ~1.6 TB and I do not have the checkpoint. So this PR has:

• ❌ no speed number for this engine
• ❌ no end-to-end identity run for this engine

and it has:

• ✅ a clean build
• ✅ tests/test_kv_prefix.c covering the decision logic exhaustively — divergence at the first and at the last recorded token, prompts shorter than and equal to the record, buffer overrun, taint, NULL
• ✅ the same mechanism measured on DeepSeek V4: 5.23× on a second turn reusing 82% of its prompt, output byte-identical to a cold engine

I would rather ship a PR that says what it has not checked than one that lets you assume otherwise.

If you have the weights — @brad-evony, anyone from #748 — the numbers would be genuinely useful. K3_PREFIX_LOG=1 prints what was reused:

[PREFIX] reusing 45 of 55 prompt tokens (82%)

and the thing worth checking is that a multi-turn conversation gives the same replies as one-shot prompts containing the same transcript.

🤖 Generated with Claude Code