feat(inkling): reuse the attention state a previous turn already built

[JustVugg]

A chat client resends the whole transcript every turn. colibri.c has pinned each conversation to a KV slot since #639, so its turn N prefills only the new text. inkling.c, kimi_k3.c and the DeepSeek engine re-process turns 1..N-1 from scratch, so the cost of a message grows with the length of the conversation — and every replayed position pulls its experts off disk again, which on a streaming engine is the dominant cost.

Nobody reported this because it does not present as a bug. It presents as "it gets slower the longer we talk", and on an engine that streams a 469 GB model from disk, that reads as normal.

	KV prefix reuse
colibri.c (GLM)	✅ #639
inkling.c	❌ → this PR
kimi_k3.c	❌ → next PR
DeepSeek engine	❌ → local

It was found by auditing what colibri.c has that the others do not — the same shape as #707 (the Apple OMP exclusion landed in inkling.c and never reached colibri.c) and #718 (physical-core sizing added to kimi_k3.c and olmoe.c and not to colibri.c).

The idea

After a turn the state covers some number of positions. If the next prompt begins with the token sequence that produced them, that state already is the state at that position: skip the reset and prefill only the tail.

No snapshot, no rewind. A prompt that diverges anywhere starts over. Either the reused positions are token-identical, or nothing is reused — the emitted tokens are the same in both cases.

Why a record and not a counter

Deriving the reusable length from the caller's bookkeeping (prompt_count + generated - 1) is tempting and wrong. That invariant differs per engine — whether the last sampled token was fed back, whether a chunked prefill ran to completion, whether generation stopped early — and getting it wrong does not crash. It answers the user from an attention state built out of a different conversation, and the reply looks entirely plausible.

So kv_prefix.h records the ids where they are fed, and that record is the only description of the state anyone consults. The failure mode stops being expressible.

Audio taints the record. Every Inkling audio frame carries the same token id (c->audio_tok) while the mel payload differs, so an id-only comparison would happily match two different clips. A state that consumed audio — or a request that brings its own — is never reused.

kv_prefix.h is shared rather than inlined per engine: one implementation to review, testable without a checkpoint, and ready for the two engines that follow.

Measured

On the DeepSeek V4 engine, where this mechanism landed first (local work, not in this PR):

reuse            45 of 55 prompt tokens (82%)
turn 2 cached    61.2s
turn 2 cold     320.0s
speedup         5.23x
identical       YES

The speedup tracks the ratio of positions still to prefill, so it grows with conversation length.

Full disclosure on this PR specifically: the equivalent end-to-end identity run on the 469 GB Inkling checkpoint is still going on my box — two engine loads at once put the machine into OOM twice, which is a hardware limit here, not a code path. I will post the numbers under this PR when it lands. The correctness argument does not rest on that run: it rests on the structure above and on tests/test_kv_prefix.c.

Tests

tests/test_kv_prefix.c covers every rejection rule, including the ones that look paranoid:

• divergence at the first token
• divergence at the last recorded token — the boundary an off-by-one memcmp length would let through
• a prompt shorter than the record (the state cannot be rewound)
• a prompt equal to the record (nothing left to prefill, so no final hidden state to sample)
• a write past the end of the buffer — drops the record rather than truncating it, because claiming coverage the state does not have is the one failure that answers wrongly
• taint, and a NULL record

It builds without a checkpoint, so CI carries it on every platform. make check picks it up automatically via the derived TEST_BINS (#733).

🤖 Generated with Claude Code

[JustVugg]

Update: the end-to-end identity test now runs in CI, so this PR no longer rests on a measurement I could not take.

The test that matters

tests/test_inkling_prefix_serve.py, wired into the existing Inkling-oracle job:

• two turns on one engine, the second reusing the first's state, must emit the same tokens as that second turn alone on a cold engine
• a diverging prompt must not be contaminated by the turn before it

It runs against the tiny random-init fixture that job already builds, so it needs no checkpoint and no fast disk.

Why it is a fixture and not a benchmark

I tried to take the real number on the 469 GB checkpoint and could not, and the reason is worth recording because it is a fact about the model, not about this change:

16,384 experts x ~28 MB          = no useful cache in 24 GB of RAM
one token, minimal prompt        = 79 GB read in 9 minutes, unfinished
cap=8 (the practical maximum)    TIERS 0 0 16384 0.00 0.00
cap=64 would need                117 GB

I built a pristine origin/dev inkling and it hit exactly the same wall, at the same point, so the behaviour is the hardware and not this PR.

Why this gate has to exist

A wrong reuse length does not crash. It answers the user from an attention state that belongs to a different conversation, and the reply still reads plausibly. Nothing else in the tree would notice — which is also why this defect survived in three engines without a single report.

So the evidence now stands at:

decision logic, every rejection rule	tests/test_kv_prefix.c — no model needed
token-identity, end to end	tests/test_inkling_prefix_serve.py — CI fixture
speed, on hardware that can show it	DeepSeek V4: 5.23x, 82% of prompt reused, output byte-identical