[WIP] Vulkan stats - #789
Open
Neppord wants to merge 18 commits into
Open
Conversation
Inkling's audio tower is one embedding table plus one RMSNorm (~10 MB bf16 on Inkling-Small), so there is no reason for the int4 container to drop it. --keep-audio passes model.audio.* through under its original names; the engine loads them directly. Vision and MTP stay skipped. Default off, so existing conversions are byte-identical. make_tiny_inkling_audio.py is the audio counterpart of make_tiny_inkling.py: a random-init InklingForConditionalGeneration with an audio tower, saved in the native TML layout, plus a ref_inkling.json that adds "dmel" (flattened [n_frames, n_mel_bins] levels). The C engine must reproduce tf_pred and the greedy continuation token-for-token. The vision tower is stubbed out: it is irrelevant to the audio oracle and its constructor trips on tensor-typed dims with tiny configs.
input_audio content parts on /v1/chat/completions now work when the engine is Inkling: base64 WAV in, spoken answer out. The gateway decodes the WAV (PCM16 or float32, mono after mixdown, 16 kHz enforced - resampling belongs at the capture edge), runs the PCM through a numpy port of the DMel DSP, renders TMLv0 audio framing, and ships the frames to the engine as a binary extension of the SUBMIT header. The DSP is the part that has to be exactly right, so it is tested against the canonical implementation rather than the paper description: byte-identical output vs tinkernel-audio, which is itself byte-golden against the official tml-renderers 0.1.0 wheel. That covers the 100 ms periodic-Hann window centered on i*hop, zero edge padding, slaney mel with 2/(upper-lower) normalization, magnitude-domain projection, the turn-level RMS boost for quiet audio (rms < 0.01), and round-half-up quantization to 16 levels in log10 [-7, 2]. Note the HF processor skips the RMS boost; tml-renderers is what the model saw in training, so that is what we match. Framing: a multipart user message becomes one TMLv0 message per content run, in part order - text runs under <|content_text|>, each audio clip as <|content_audio_input|> + one <|audio|> placeholder per frame + <|audio_end|>. A TMLv0 message carries a single content type, so this split is the trained distribution, not a workaround. Wire: the optional 7th SUBMIT field was grammar bytes for glm; for inkling it is DMel bytes. The two engines never see each other's extension, and inkling rejects grammars upstream, so the framing cannot desync. numpy is imported lazily and only for audio requests; text-only serving still runs on the stdlib.
Two additions that together put Inkling-Small on a 128 GB Apple Silicon box with audio in: the engine consumes DMel frames at <|audio|> placeholders, and the routed-expert math runs on the Apple GPU. Audio. Inkling's audio tower is a [mel_bins*mel_vocab, D] embedding table plus one RMSNorm: a frame's embedding is rmsnorm(sum of 80 rows, eps 1e-6), and it REPLACES the placeholder's text embedding (embed_norm does not apply), matching masked_scatter in HF modeling_inkling.py. The tensors load from model.audio.* when present - either a --keep-audio container or an audio.safetensors sidecar dropped in the snapshot dir (st_init already indexes every *.safetensors). Absent tensors = text-only engine, unchanged. Entry points: --audio file.dmel on the CLI (implies --chat, emits the TMLv0 audio framing), an optional 7th SUBMIT field carrying raw DMel bytes on the serve protocol, and a "dmel" array in ref_inkling.json for the oracle harness. Validated token-exact against the HF multimodal oracle: 38/38 teacher-forced, 24/24 greedy. Frame counts are checked against placeholder counts up front; a mismatch is an error, not a silent skip. Metal (COLI_METAL=1, opt-in). Reuses colibri's batched coli_metal_moe_block: the container's nibble-packed int4 with per-row scales is bit-identical to the fmt=2 kernel, int8 to fmt=1, and the fused p13 tensor needs no reshaping because resolve() accepts interior pointers (gate = p13, up = p13 + I rows). Expert slots move from per-slot mallocs to one page-aligned slab per sparse layer, carved into slot regions up front and registered with the backend once - eviction reuses the region, so there is no register/unregister churn and unified memory stays zero-copy. Each cache round groups its (token, expert) pairs by expert and submits one command buffer; a failed submit falls back to the CPU loop for that round only. Verified token-identical to the CPU path on an int4 container with the audio oracle active (294 blocks on GPU, 0 fallbacks), and a [metal] counts line makes GPU execution provable rather than assumed. Attention, dense MLP, and shared experts stay on the CPU: at high hit rates ~90% of decode is routed-expert matmul, so that is the lever that pays first. Also: mem_avail_bytes gets a macOS branch (free + inactive + purgeable), so the auto expert-cache cap sizes to the machine instead of falling back to 16 experts/layer on a 128 GB Mac.
At S==1 a MoE round is topk pairs of 6-row kernels; the dispatch+sync latency (~135 ms/block measured) swamps the math. Inkling-Small decode: 0.14 tok/s on GPU vs 0.63 on CPU. Prefill rounds carry up to cap pairs and amortize the launch, so the GPU stays on for S >= 2. INK_METAL_MIN_S=1 restores GPU-always for A/Bs.
Three findings from a 10-config knob sweep on Inkling-Small (M-series 128 GB, frozen probe, 2 reps each, all reproducible): - OMP active-spin tuning off on Apple: the spinning team steals the SoC power budget and loses even for pure-CPU decode (0.50 -> 0.84 tok/s) and prefill (49.7 -> 24.4s). Same mechanism the M5 Max Metal report documented for GLM. - Metal residency set on by default: without it every MoE block pays per-buffer useResource churn, ~135 ms per decode block. With it the same block is ~3 ms: 0.17 -> 1.76 tok/s decode. - GPU decode back on (INK_METAL_MIN_S default 2 -> 1): with the residency set the GPU wins decode 2x over CPU, so the prefill-only gate now costs performance instead of saving it. Net on the frozen probe: 1.75 tok/s decode + 10.3s prefill, from 0.50 and 49.7 at the old defaults. Output tokens unchanged; tiny-oracle validation stays exact on both CPU and GPU paths. The phase profile now puts shared experts and attention (both CPU) above expert matmul.
…ugg#748) kimi_k3 and inkling never called _setmode(..., _O_BINARY). On Windows the CRT opens stdout in TEXT mode and rewrites '\n' as '\r\n', so the engine emits \x01\x01READY\x01\x01\r\n and coli -- which matches the sentinel byte-exactly -- waits for a byte that was mangled on the way out. No error is printed, because nothing failed: one side is waiting, the other already sent something else. @brad-evony hit it in JustVugg#748 on Windows 10: 93 layers loaded over 42 minutes, the tokenizer read, and then silence. Ctrl-C landed in read_engine_turn's stream.read(1), which is exactly where the handshake blocks. colibri.c has done this since JustVugg#195 and its comment describes the symptom word for word ('the READY sentinel never matches and chat hangs'). The two newer engines were written without it and nothing noticed for months. Not a v1.4.0 regression -- the bug predates it. v1.4.0 only made it reachable, because before that the archives contained no kimi_k3.exe at all (JustVugg#720). It goes in compat.h rather than being pasted into each engine: pasting is how it went missing, and this is the third Windows-specific behaviour to be found living in colibri.c alone (after select()/fd_set in JustVugg#736). Adds tests/test_serve_sentinel.c, which writes the handshake through a real stream and asserts the bytes contain no CR. Nothing in the tree looked at those bytes before, which is why this reached a user rather than CI. It can only pass on Linux/macOS -- that is the point of a regression guard, and CI builds every engine on Windows since JustVugg#736, so it runs where it can fail. olmoe.c has no serve mode and is untouched. colibri.c is untouched: it is already correct, and 14 open PRs touch that file. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…indows fix(win): put stdout in binary mode before the serve handshake (closes JustVugg#748)
inkling: audio input, Metal expert MoE, and tuned Apple defaults
…Vugg#191) @centralware and two colleagues downloaded the 400 GB model, compiled the engine, and gave up at step 3 of the quickstart -- not on anything technical, on our prose. Their exact objections, and what each one now says: 'multi-turn wire' read as wire-wrapping, a physical technique. It meant the bytes sent to the model across a several-turn conversation. '77/77 token ids' meant nothing without knowing what encoding_k3.py is. It now says we compared our text handling to Moonshot's own and got identical numbers on 77 test conversations. 'gated on a host' meant 'we cannot test this until someone has 1.6 TB'. That paragraph is gone with the v1.4.0 README rewrite. The contradiction they reported is real and was ours: the Kimi K3 note (1.6 TB, serious machine) sits near the GLM-5.2 quickstart (372 GB, no GPU). Both true, about different models, printed together with nothing saying so. Reading them as one requirement produces exactly 'needs 1.6 TB' next to 'runs on a laptop'. Adds a per-model requirements table to the roster -- disk, RAM, GPU, one row each -- so the numbers are attached to the model they belong to, and says plainly that none of the four needs a GPU. Three electronics engineers with the hardware, the download and a working compiler are not the difficult case. If they bounce off the words, the words are wrong. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
docs: plain language for the words that stopped three engineers (JustVugg#191)
Fourth and final engine for JustVugg#700, after colibri.c (JustVugg#716), kimi_k3.c (JustVugg#719) and inkling.c (JustVugg#742). Every engine that routes experts now writes the same format. m->freq moves from one flat [n_layers * n_experts] block to route_trace.h's row per layer, so the three read sites -- the hot-pin ranking, the bump in moe() and the LFRU eviction guard -- change shape but not meaning. last_access stays flat; it is a separate array route_trace.h does not own. Every olmoe layer routes and there is no MTP layer, so the only row dropped is the spare one rt_init leaves. The history is opt-in behind COLI_USAGE, exactly as kimi_k3 does it: with the variable unset nothing is loaded and nothing is written, so the default path is unchanged. PPL=1 deliberately does not save, so a loss sweep cannot fold its own tokens into the persisted ranking. Verified against the real OLMoE-1B-7B-0125-Instruct converted with tools/convert_olmoe_merged.py. Output is byte-identical to dev on the same run, down to hit=1216 miss=832 -- the eviction path decides exactly as before, which is the property the layout change could have broken. The COLI_USAGE round trip writes the -1/-2 header with the olmoe identity, leaves no row for the dropped layer, and accumulates 2048 -> 4096 across two runs. make check green. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Fourth and final engine for JustVugg#700, after colibri.c (JustVugg#716), kimi_k3.c (JustVugg#719) and inkling.c (JustVugg#742). Every engine that routes experts now writes the same format. m->freq moves from one flat [n_layers * n_experts] block to route_trace.h's row per layer, so the three read sites -- the hot-pin ranking, the bump in moe() and the LFRU eviction guard -- change shape but not meaning. last_access stays flat; it is a separate array route_trace.h does not own. Every olmoe layer routes and there is no MTP layer, so the only row dropped is the spare one rt_init leaves. The history is opt-in behind COLI_USAGE, exactly as kimi_k3 does it: with the variable unset nothing is loaded and nothing is written, so the default path is unchanged. PPL=1 deliberately does not save, so a loss sweep cannot fold its own tokens into the persisted ranking. Verified against the real OLMoE-1B-7B-0125-Instruct converted with tools/convert_olmoe_merged.py. Output is byte-identical to dev on the same run, down to hit=1216 miss=832 -- the eviction path decides exactly as before, which is the property the layout change could have broken. The COLI_USAGE round trip writes the -1/-2 header with the olmoe identity, leaves no row for the dropped layer, and accumulates 2048 -> 4096 across two runs. make check green. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…'t trust native (JustVugg#631) On Linux ARM the Makefile fell through to the x86-64 else: -march=native on a gcc that doesn't know the core (gcc 13 vs Cortex-X925) silently degrades to plain armv8-a, the SDOT/SMMLA int8-int4 kernels compile out, and int4 S=1 decode lands on the f32 fallback (g_i4s=2 without DOTPROD). New branch, default ARCH=native: keep -mcpu=native if it already defines __ARM_FEATURE_MATMUL_INT8; otherwise re-add the features /proc/cpuinfo reports (asimddp -> +dotprod, i8mm -> +i8mm) as -march=armv8-a+..., kept only if the compiler accepts the modifiers (gcc < 10 has no +i8mm), else fall back to -mcpu=native with a warning. Explicit ARCH= respected: armv* -> -march, core names -> -mcpu, same convention as the Darwin branch. GB10 default build: 0 -> 22 smmla, banner idot: neon-i8mm, int4 S=1 decode kernel ~2.5x. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Follow-up from JustVugg#750 review. Documents the DMel audio path (what the audio tower actually is, the two ways to get the tensors, input format, gateway example) and inlines the range-request script: ~10 MB moved instead of re-downloading the 532 GB checkpoint. The script was extracted from this page and executed as written; its output is byte-identical to the sidecar used for the live JustVugg#750 validation runs.
build: Linux/aarch64 branch in the Makefile — probe DOTPROD/I8MM instead of trusting native (JustVugg#631)
docs: inkling audio input and the 10 MB sidecar fetch
…trace olmoe: adopt route_trace.h — the last engine on the shared history
Author
|
Seams like i targeted the wrong branch, and should target dev, sorry. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This is a placeholder PR for implementing the same statistic features that exists for cuda, but for vulkan. More specificaly the vmem and the expert recidents when runing the web ui.
Validation
make -C c checkmake -C c cuda-test(if applicable)Compatibility