test(inkling): enforce cache capacity invariance - #784
Conversation
920bb8f to
2f8957c
Compare
|
I rebased this for you — the conflict was mine to fix, not yours. #786 landed an hour ago and added a step to the same for cap in 1 2 8; do
SNAP=tiny_inkling ./inkling "$cap" 0 tiny_inkling/ref_inkling.json
done
- name: KV prefix reuse is token-identical (serve, two turns)Also: CI never ran on your branch at all. GitHub holds workflows from a first-time contributor's fork in On the change itselfBoth halves are the right instinct. The capacity sweep. A capacity-1 cache evicts on every routed expert, which is where slot bookkeeping breaks; a capacity-8 run on a tiny fixture may never evict at all and would pass over the bug forever. Testing the boundary rather than the comfortable middle is the harder discipline and the one that catches things. The expert-id guard is the part I would defend hardest if anyone objects to the cost: if (e->eid != idx[(int64_t)s*K + kk]) { ... exit(1); }Serving the wrong expert's weights does not crash — it produces fluent, confident, wrong output, and nothing downstream can tell. That failure class is the same one we spent today on from the other end: a KV prefix reused at the wrong length answers from a different conversation and still reads plausibly. A loud refusal beats junk that looks like text. The per-token cost is O(S·K) integer comparisons against a routed-expert matmul — not measurable. Green CI and this merges. Thank you — and sorry the first thing your PR met was our own silence and then our own conflict. |
Summary
Add regression coverage for the cache-capacity fix at capacities 1, 2, and 8.
Add a runtime expert-ID guard so cache mismatches fail clearly instead of using the wrong expert
weights and producing junk output.
Validation
Compatibility