Skip to content

Render chat templates via hf-chat-template for transformers parity#452

Open
GregoryBolshakov wants to merge 1 commit into
floneum:mainfrom
GregoryBolshakov:chat-template-hf-parity
Open

Render chat templates via hf-chat-template for transformers parity#452
GregoryBolshakov wants to merge 1 commit into
floneum:mainfrom
GregoryBolshakov:chat-template-hf-parity

Conversation

@GregoryBolshakov

Copy link
Copy Markdown
Contributor

Fixes #451.

The chat template renderer drifts from what transformers.apply_chat_template produces, mostly around the system role. MessageType::SystemPrompt serializes to "developer", and that value flows through context!{ role }, so any template that checks role == 'system' never matches the system message. Depending on the template that means a default system prompt gets substituted, the message gets dropped, or the render raises "roles must alternate". On top of that, trim_blocks/lstrip_blocks aren't set and tojson doesn't match Python's json.dumps.

I checked this against 20 real model templates from the Hub (68 cases), comparing byte-for-byte against output generated by Python transformers. Of the 48 cases the current renderer can express, 31 matched and 17 didn't, and 3 of those raise an exception on a plain system/user/assistant chat (Falcon 7B, Gemma 3, Mistral 7B). After this change all 48 match.

This delegates rendering to hf-chat-template, a small crate that does one thing: render HF chat templates byte-identically to transformers (Python string methods, strftime_now, trim_blocks/lstrip_blocks, a transformers-compatible tojson, standalone template files). It brings in minijinja, which is already in the tree, plus serde_json.

The renderer's public shape (create/format) doesn't change, and the minijinja::Error that LlamaModelError and LlamaSourceError carry stays the same, so there's no API break. minijinja is kept just for that error type. The five existing chat_template tests are unchanged. Two smaller knock-on changes: serde_json was optional behind hf-config-json and the renderer needs it now, so it's a normal dependency; and chrono moves to dev-dependencies since only a test still uses it.

If you'd rather not take the dependency, the alternative is reimplementing pycompat, the block trimming, a Python-compatible tojson, and strftime_now inline, which is what the crate already does. Happy to go either way.

One note: I couldn't run the full test suite on my machine (the dev-dependencies pull in openssl-sys and I don't have system OpenSSL here), but cargo check, fmt, and clippy are clean on kalosm-llama, and I checked the rendering against the published crate separately.

kalosm-llama's minijinja renderer diverged from
transformers.apply_chat_template on ordinary conversations. System
messages were emitted with role "developer" (the MessageType serde
rename flows through context!{role}), so templates that branch on
role == 'system' substituted a default system prompt, dropped the
message, or raised "roles must alternate". trim_blocks/lstrip_blocks
were also unset, and tojson did not match Python's json.dumps.

Delegate rendering to hf-chat-template, verified byte-identical to
Python transformers across a golden corpus of real model templates.
The renderer's public shape (create/format) and the minijinja::Error
type carried by this crate's error enums are unchanged; bos/eos are
passed per call via the render input.

Drop the direct minijinja Environment and minijinja-contrib usage
(minijinja is retained only for its Error type); move chrono to
dev-dependencies since only a test uses it now.
@github-actions

Copy link
Copy Markdown

Preview available at https://floneum.github.io/kalosm/pr-preview/pr-452/

@GregoryBolshakov

Copy link
Copy Markdown
Contributor Author

Heads up on CI: the only failing tests are the live-API ones in kalosm-language-model (the claude::chat, openai::chat, and openai::embedding suites), which 401 because fork PRs don't get the API-key secrets. The same suite is green on main. The kalosm-llama template tests this PR actually changes all pass (5/5), and Check, Clippy, Rustfmt, and Docs are green. So nothing here should need a code change; it just needs a maintainer to approve the workflow run with secrets.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

System messages render with role "developer", breaking chat templates

1 participant