feature: Hy3 support - #775
Conversation
Integrate upstream batch serve protocol, tool calling, streaming keepalive, and Metal/CUDA build targets while preserving Hy3-specific engine selection, argv, IDOT defaults, and chat template rendering.
Add PERF=1 instrumentation, AVX2 attention, KV_I8 int8 KV cache, PIPE=2 io_uring expert loads (4096-ring backpressure), CUDA_ATTN GQA kernel, and optional TREE_DRAFT speculative decode. Fix int8 KV reallocation and io_uring SQE exhaustion on large prefills. Report both total and decode-only tok/s in STAT and coli chat footer. Document new env vars and build flags in README (IOURING=1, tuning guide).
On Windows, hy3.c in CPU mode compiles but hangs on "waking the giant" — stdin/stdout default to text mode on Windows, which mangles the \x02RESET/\x02MORE binary control sequences and the STAT protocol line. Fix forces binary mode + unbuffered stdout, matching the equivalent fix already present in the upstream GLM fork.
Update hy3.c fix windows stdio
Add AMD ROCm support and KV-aware planning
Implement Hy3 tool calling format and reasoning support in APIHandler
…ine paths. Bring in upstream CI, docs, web dashboard, and GLM CUDA attention batching; preserve Hy3 GQA attention, ROCm support, Hy3 resource planning, and Hy3-specific serve/tool/reasoning handling.
…ping Hy3. Preserve Hy3 engine paths (hy3 build, single-protocol serve, GQA/IOURING) on top of upstream's glm→colibri rename, HIP backend, grammar/response_format, and /v1/messages. Skip Pages deploy on forks and build hy3 in CI. Co-authored-by: Cursor <cursoragent@cursor.com>
A merge artifact left two identical if(calls&&getenv("COLI_CUDA_PROFILE")) lines, which gcc rejected when building colibri.exe with CUDA_DLL=1.
Co-authored-by: Cursor <cursoragent@cursor.com>
…eeping Hy3. Adopt upstream Inkling support, E8/iq3 tooling, deferred SSE streaming, and deep safetensors validation on top of Hy3 engine paths (hy3 build, single-protocol serve, GQA/IOURING). Keep Hy3-focused README and build colibri, hy3, and inkling in CI. Co-authored-by: Cursor <cursoragent@cursor.com>
Adopt upstream multi-model architecture (model_arch routing, openai_server mux/single protocol split) while preserving Hy3-specific behavior: GQA engine in hy3.c, int4 quant argv, Hunyuan chat template, convert_hy3 routing, and docs/hy3.md. Co-authored-by: Cursor <cursoragent@cursor.com>
|
Reviewing properly takes longer than this note, but one thing is worth telling you now rather than after a user reports it.
On Windows the CRT opens stdout in text mode and rewrites the trailing It is one line, from static void serve_loop(...){
coli_serve_binary_mode(); /* #748: TEXT-mode stdout mangles the READY sentinel */
...
fputs("\x01\x01READY\x01\x01\n", stdout);
Credit where it is due: your The larger question — whether we take a fifth engine right now — is the maintainer’s and is being discussed separately. Nothing about that reflects on the work; four engines already multiply every maintenance problem across four files, and this week produced two Windows bugs that had to be fixed in three of them at once. |
|
Good catch on the ready emit, will fix it |
Port JustVugg#749's shared helper so Hy3 (and inkling/kimi on this merge base) do not hang forever on Windows when CRT TEXT mode rewrites the sentinel as CRLF (JustVugg#748). Co-authored-by: Cursor <cursoragent@cursor.com>
Summary
I had the Hy3 support for Colibri already working for quite a while but now there is more focus on supporting different models from the main repo as well, figure a pull request would make sense. I made sure to rewrite anything off that I did before generalization was in place in Colibri to work in a way that consolidates with the current codebase
Validation
make -C c checkmake -C c cuda-testCompatibility