Gate the Kimi pre-tokenizer, which nothing was watching

[ZacharyZcR]

The gap

o200k has had a gate since it was written — tests/test_tok_o200k.c, 40/40 encode + decode against a 4 KB in-repo fixture, no model download.

The Kimi family has had none. tests/test_tok_kimi.c needs a real tokenizer.json plus a cases.bin generated by tiktoken, so it has no build rule and has never run. c/Makefile lists it among the files that "deliberately have no rule".

That matters because the two families are not independent code. tok.h's o2_letters_masked() and pretok_chunk_o2fam() serve both from the same lines, separated by a flag — a change made for o200k reshapes Kimi silently, and only the o200k half had anything watching.

What this adds

The fixture o200k already had, for Kimi: tests/tok_kimi_tiny.json — the same tiny vocabulary carrying the Kimi pre-tokenizer regex, which is what selects the family (tok.h keys on \p{Han} appearing in the pattern).

It asserts the pre-tokenizer boundary, not ids from the real Kimi vocabulary. Reproducing those needs the actual tokenizer, which is precisely what made the existing test unrunnable. The boundary is a property of the rules, and it is what the shared code decides:

assertion	input	expected
a Han run is its own chunk	中文	1 id (a vocabulary entry)
Han does not absorb adjacent Latin	中文abc	3 ids
a Han codepoint at the range edge still splits	龥a	2 ids

The fixture makes those observable: 中文 is one vocabulary entry, so a correct Han-run split encodes it as one id and any other split falls back to per-byte ids.

Two assertions are differential — the same input must not encode identically through the o200k fixture. Without them the file would pass vacuously the moment the Kimi flag stopped reaching the splitter, which is the failure most worth catching.

Verified both directions

  ok   family detection: kimi=1 / o200k kimi=0
  ok   Han run is one chunk         中文       1 ids
  ok   Han does not eat Latin       中文abc    3 ids
  ok   U+9FA5 splits from Latin     龥a         2 ids
  ok   kimi != o200k on 中文       (1 vs 6 ids)
  ok   kimi != o200k on 中文abc    (3 vs 8 ids)

Routing pretok_chunk_kimi through pretok_chunk_o200k on purpose fails three of them, including a differential one — so the gate is not vacuous.

make test-c picks the new binary up automatically: gates are derived from build rules, so there is no shared list to edit and no line for two PRs to conflict on.

Note on scope

This does not replace tests/test_tok_kimi.c — that one still cross-checks the full BPE against tiktoken on a real vocabulary, which is a stronger claim than this makes and still worth running by hand when a real model is available. This covers the part that can run on every commit, for free.