Skip to content

feature: Hy3 support - #775

Open
ErikTromp wants to merge 21 commits into
JustVugg:mainfrom
ErikTromp:integrate-upstream-v1.4
Open

feature: Hy3 support #775
ErikTromp wants to merge 21 commits into
JustVugg:mainfrom
ErikTromp:integrate-upstream-v1.4

Conversation

@ErikTromp

Copy link
Copy Markdown

Summary

I had the Hy3 support for Colibri already working for quite a while but now there is more focus on supporting different models from the main repo as well, figure a pull request would make sense. I made sure to rewrite anything off that I did before generalization was in place in Colibri to work in a way that consolidates with the current codebase

Validation

  • make -C c check
  • CUDA changes were tested with make -C c cuda-test
  • Performance claims include hardware, commands, and repeatable measurements

Compatibility

  • The default CPU build remains dependency-free
  • No model files, generated binaries, or benchmark artifacts are included

ErikTromp and others added 20 commits July 13, 2026 12:57
Integrate upstream batch serve protocol, tool calling, streaming keepalive,
and Metal/CUDA build targets while preserving Hy3-specific engine selection,
argv, IDOT defaults, and chat template rendering.
Add PERF=1 instrumentation, AVX2 attention, KV_I8 int8 KV cache,
PIPE=2 io_uring expert loads (4096-ring backpressure), CUDA_ATTN GQA
kernel, and optional TREE_DRAFT speculative decode. Fix int8 KV
reallocation and io_uring SQE exhaustion on large prefills.
Report both total and decode-only tok/s in STAT and coli chat footer.
Document new env vars and build flags in README (IOURING=1, tuning guide).
On Windows, hy3.c in CPU mode compiles but hangs on "waking the giant" — stdin/stdout default to text mode on Windows, which mangles the \x02RESET/\x02MORE binary control sequences and the STAT protocol line. Fix forces binary mode + unbuffered stdout, matching the equivalent fix already present in the upstream GLM fork.
Add AMD ROCm support and KV-aware planning
Implement Hy3 tool calling format and reasoning support in APIHandler
…ine paths.

Bring in upstream CI, docs, web dashboard, and GLM CUDA attention batching;
preserve Hy3 GQA attention, ROCm support, Hy3 resource planning, and
Hy3-specific serve/tool/reasoning handling.
…ping Hy3.

Preserve Hy3 engine paths (hy3 build, single-protocol serve, GQA/IOURING) on top of upstream's glm→colibri rename, HIP backend, grammar/response_format, and /v1/messages. Skip Pages deploy on forks and build hy3 in CI.

Co-authored-by: Cursor <cursoragent@cursor.com>
A merge artifact left two identical if(calls&&getenv("COLI_CUDA_PROFILE")) lines, which gcc rejected when building colibri.exe with CUDA_DLL=1.

Co-authored-by: Cursor <cursoragent@cursor.com>
…eeping Hy3.

Adopt upstream Inkling support, E8/iq3 tooling, deferred SSE streaming, and deep safetensors validation on top of Hy3 engine paths (hy3 build, single-protocol serve, GQA/IOURING). Keep Hy3-focused README and build colibri, hy3, and inkling in CI.

Co-authored-by: Cursor <cursoragent@cursor.com>
Adopt upstream multi-model architecture (model_arch routing, openai_server mux/single
protocol split) while preserving Hy3-specific behavior: GQA engine in hy3.c, int4
quant argv, Hunyuan chat template, convert_hy3 routing, and docs/hy3.md.

Co-authored-by: Cursor <cursoragent@cursor.com>
@JustVugg

JustVugg commented Aug 2, 2026

Copy link
Copy Markdown
Owner

Reviewing properly takes longer than this note, but one thing is worth telling you now rather than after a user reports it.

hy3.c emits the READY sentinel but never calls coli_serve_binary_mode().

On Windows the CRT opens stdout in text mode and rewrites the trailing \n as \r\n, so the engine sends \x01\x01READY\x01\x01\r\n, coli — which matches the bytes exactly — never recognises it, and the session hangs forever with no error at all. That is #748: a user loaded 93 Kimi layers over 42 minutes and then sat watching nothing happen.

It is one line, from compat.h, before the sentinel:

static void serve_loop(...){
    coli_serve_binary_mode();   /* #748: TEXT-mode stdout mangles the READY sentinel */
    ...
    fputs("\x01\x01READY\x01\x01\n", stdout);

colibri.c has had it since #195; inkling.c and kimi_k3.c were written without it and it went unnoticed for months, which is exactly why it now lives in compat.h instead of being pasted per engine. Yours is the fourth engine to be written without it, so this is on our onboarding rather than on you — but you get to have it for free instead of finding out from a bug report.

Credit where it is due: your coli wiring is the most complete of any new engine we have received. model_arch recognises hy_v3, the banner is arch-aware, and the serve protocol is there. #712 (Qwen3.6) has none of that and is blocked on it, so you have already done the part that usually gets missed.

The larger question — whether we take a fifth engine right now — is the maintainer’s and is being discussed separately. Nothing about that reflects on the work; four engines already multiply every maintenance problem across four files, and this week produced two Windows bugs that had to be fixed in three of them at once.

@JustVugg JustVugg added discussion Proposta / discussione aperta, non un task model-support Supporto a nuovi modelli labels Aug 2, 2026
@ErikTromp

Copy link
Copy Markdown
Author

Good catch on the ready emit, will fix it

Port JustVugg#749's shared helper so Hy3 (and inkling/kimi on this merge base)
do not hang forever on Windows when CRT TEXT mode rewrites the sentinel
as CRLF (JustVugg#748).

Co-authored-by: Cursor <cursoragent@cursor.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

discussion Proposta / discussione aperta, non un task model-support Supporto a nuovi modelli

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants