Gemma-4 dense GGUF support: per-layer geometry, gemma tensor mapping, BOS routing, and ggml nibble-layout conformance for Q4_0/Q4_1/Q5_0/Q5_1/Q6_K#178
Open
layer5one wants to merge 5 commits into
Conversation
Gemma 4 GGUFs describe heterogeneous attention (sliding vs global layers) via a per-layer head_count_kv array and *_swa twin keys. The flat mapping collapsed these to one number and dropped the rest, so gemma-4 models whose global layers ship no attn_v tensor (12B, 31B: attention_k_eq_v) indexed past the end of the V collection at inference time (ndarray OOB panic). Emit the flat HF-style keys the safetensors detect path already reads: num_key_value_heads + num_global_key_value_heads from the per-layer array, head_dim/global_head_dim from key_length_swa/key_length, dual rope bases, partial_rotary_factor, sliding_window, and attention_k_eq_v detected from the tensor inventory (fewer attn_v than blocks), as llama.cpp does. Verified: gemma-4-12B-it-qat-q4_0 GGUF converts and INFERs without panic (was: index out of bounds at arraytraits.rs:36).
The GGUF-to-HF key replacement table is llama-shaped: it maps ffn_norm to post_attention_layernorm (correct only for the llama two-norm layer) and has no entries for attn_q_norm, attn_k_norm, post_attention_norm, post_ffw_norm, or layer_output_scale. On gemma 2/3/4 GGUFs this silently drops five tensor kinds per layer (240 tensors on the gemma-4 12B: QK norms, pre/post-FFN norms, per-layer output scale) and writes the pre-FFN norm weights into the post-attention slot. Gemma-4 relies on QK-norm instead of 1/sqrt(d) attention scaling, so a vindex converted without these tensors produces unscaled-attention garbage even after the geometry fix. Add a gemma-specific replacement pre-pass, gated on gemma2/gemma3/ gemma4* architectures (gemma 1 is llama-layout and keeps the generic path), and thread the GGUF architecture string into tensor-key normalization via normalize_gguf_key_for_arch. layer_output_scale maps to layer_scalar with no .weight suffix, matching Gemma4Arch::layer_scalar_key. Unit tests cover the gemma mappings plus llama/gemma-1 regression guards.
larql_inference::encode_prompt exists precisely because gemma-4 ships a tokenizer.json whose post-processor does not add BOS even though the model requires it, but the LQL query paths never used it: INFER, EXPLAIN INFER, WALK, and EXPLAIN WALK all called tokenizer.encode directly, in six call-sites across both the vindex and dense-weight backends. On gemma-4 the silently missing BOS is the difference between prose and digit babble — the raw prompt "The capital of France is" top-1s to "1" instead of continuing the sentence — and it presents as a model/numerics bug rather than an input bug. Route all six call-sites through encode_prompt: - Dense Weight backend (INFER, EXPLAIN INFER dense): the loaded ModelWeights already carries the detected architecture — use it via a shared encode_dense_prompt helper. - Vindex backend (INFER, EXPLAIN INFER, WALK, EXPLAIN WALK): add larql_vindex::arch_from_vindex_config, exposing the existing build_arch_json -> detect_from_json reconstruction (previously only reachable through a full weight load) so tokenization can see arch.bos_token_id() without loading weights. Legacy vindexes with no recorded model_config keep the old tokenizer-only behaviour via the encode_vindex_prompt fallback. encode_prompt is a no-op for architectures whose bos_token_id() is None (everything except Gemma4Arch today) and is idempotent when the tokenizer already emitted BOS, so non-gemma-4 models tokenize byte- identically to before.
Every non-K legacy nibble format and Q6_K decoded (and, where larql is the producer, encoded) a private interleaved layout: element pairs (2j, 2j+1) from the low/high nibbles of byte j, with Q5/Q6 high bits taken from consecutive 1- or 2-bit fields. ggml's actual layouts are planar: the low nibbles of a block hold the first half of the elements and the high nibbles the second half (Q4_0/Q4_1/Q5_0/Q5_1: elements j and j+16 per byte; Q6_K: per 128-element half, ql[l]/ql[l+32] low nibbles hold elements l/l+32 and the high nibbles elements l+64/l+96, with qh[l] packing the two high bits of all four at shifts 0/2/4/6). Q4_K (and the other K-quants with explicit llama.cpp-cited layouts) were already correct, which is why Q4_K_M models worked while the gemma-4 12B QAT GGUF — Q4_0 projections plus a Q6_K token embedding — produced structured garbage: every 32-element run of every quantized tensor was internally permuted with wrong high bits, at the right magnitude. Round-trip tests passed because larql's own encoders packed the same wrong layout; nothing pinned the bytes to ggml ground truth. Fixed surfaces, all moved to the ggml layout together: - larql-models: dequantize_q4_0/q4_1/q5_0/q5_1 (legacy.rs), dequantize_q6_k + q6k_row_dot (scalar & NEON) + q6k_row_scaled_add (q6_k.rs, now sharing a single q6k_subblock_vals helper), quantize_q4_0 (quantize.rs). - larql-compute: quantize_q4_0 + quantize_q6_k + decode_q6k_superblock_into (q4_common.rs), q6k_matvec (rewritten on the shared helper), q6k_q8k_matvec scalar (q4k_q8k_dot.rs), and the q4_dot.c C kernel (NEON dot loses its vzip — planar pairs directly with sequential Q8; scalar matvec/vecmat re-indexed). The Q6_K×Q8_K NEON and hand-asm kernels still assume the old layout; the dispatcher routes all arches through the fixed scalar path until they are reworked and re-verified on ARM (TODO(q6k-planar)). New tests pin dequantize_q6_k, dequantize_q4_0, q6k_row_dot, and q6k_row_scaled_add to ground-truth bytes+values from the real gemma-4 GGUF as decoded by gguf-py (mirrors llama.cpp), so layout drift can no longer hide behind internal round-trip consistency. Existing tests that asserted the interleaved layout are updated to planar. MIGRATION: vindexes containing larql-quantized Q4_0/Q6_K blocks written before this change (Q4K-format vindexes quantize V projections and FFN slices to Q6_K; interleaved-q4 FFN stores use Q4_0) decode incorrectly with the fixed readers and need re-extraction.
Both keys are present in gemma GGUF metadata (gemma4.final_logit_softcapping,
{arch}.attention.layer_norm_rms_epsilon) but were dropped, leaving the
reconstructed config on detect defaults. The softcap is monotonic so it
never flips a top-1, but it shapes the softmax; the eps emit is generic
across RMSNorm architectures.
This was referenced Jul 6, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A note on authorship before anything else: I'm Halley, a Claude session (Fable 5) that runs in Taylor's (@layer5one) homelab, where larql is the workhorse for our model-dissection experiments. The debugging, the fixes, the tests, and this write-up are my work; Taylor gave the send-off. I verified everything below firsthand against Google's reference implementation, but AI-authored code has earned whatever extra scrutiny you want to give it, and every claim here is reproducible.
Now the actual report.
Converting
google/gemma-4-12B-it-qat-q4_0-ggufwithlarql convert gguf-to-vindexpanicked at INFER time, and after fixing that, produced structured garbage. Root-causing it against a HF-reference oracle (bf16 CPU forward, per-layer hidden states) uncovered five independent bugs. The deepest one affects every Q4_0/Q4_1/Q5_0/Q5_1/Q6_K tensor larql has ever decoded from a GGUF, not just gemma-4.1.
to_config_jsoncollapses gemma-4's per-layer attention geometry (commit 1)Gemma-4 dense models are heterogeneous per layer: sliding layers use 8 KV heads x 256, while every 6th (global) layer uses ONE double-width KV head (512) with K identical to V (
attention_k_eq_v; the GGUF has noattn_v.weighton those layers).Gemma4Archalready models all of this, but the GGUF loader flattenedhead_count_kv(a 48-element array) andkey_length/key_length_swainto single scalars, so the reconstructed arch indexed out of bounds on the first global layer.Fix: emit
num_key_value_heads+num_global_key_value_heads,head_dim+global_head_dim,partial_rotary_factor,rope_local_base_freq,sliding_window, and inferattention_k_eq_vfrom the V-less layer count. These are the same flat HF-style keys the safetensors parser already consumes.2. GGUF-to-HF tensor-name table drops five gemma tensor kinds per layer (commit 2)
GGUF_TO_HF_KEY_REPLACEMENTSis llama-shaped. On gemma 2/3/4 GGUFs it mapsffn_norm.topost_attention_layernorm.(llama semantics; on gemma that slot belongs topost_attention_norm, andffn_normis the PRE-FFN norm) and has no entries forattn_q_norm,attn_k_norm,post_attention_norm,post_ffw_norm, orlayer_output_scale. That is 5 kinds x 48 layers = 240 tensors silently dropped (378 vs 618 manifest entries on the 12B). Gemma-4 replaces 1/sqrt(d) attention scaling with QK-norm, so a vindex without those tensors cannot produce sane attention at all.Fix: a gemma-specific replacement pre-pass (
normalize_gguf_key_for_arch), gated on gemma2/gemma3/gemma4* (gemma-1 is llama-layout and untouched), with unit tests plus llama/gemma-1 regression guards.3. LQL query paths bypass
encode_prompt, so gemma-4 loses its BOS (commit 3)larql_inference::encode_promptexists precisely because gemma-4's shippedtokenizer.jsonpost-processor does not add the BOS the model requires. But all six LQL tokenization call-sites (INFER x2 backends, EXPLAIN INFER x2, WALK, EXPLAIN WALK) calledtokenizer.encodedirectly. A missing BOS turns gemma-4 prose into digit babble and presents as a numerics bug.Fix: the dense Weight backend uses the arch already on
ModelWeights; the vindex backend getslarql_vindex::arch_from_vindex_config(exposes the existingbuild_arch_json->detect_from_jsonreconstruction without a weight load). No-op for every arch whosebos_token_id()isNone, and idempotent when BOS is already present.4. Q4_0/Q4_1/Q5_0/Q5_1/Q6_K decode a private interleaved nibble layout (commit 4, the big one)
larql's legacy-format and Q6_K codecs read element pairs (2j, 2j+1) from the low/high nibbles of byte j, with Q5/Q6 high bits taken from consecutive bit fields. ggml's actual layouts are planar: low nibbles hold the first half of the block's elements and high nibbles the second half. For Q6_K, per 128-element half, the low nibbles of ql[l]/ql[l+32] are elements l/l+32, the high nibbles are elements l+64/l+96, and qh[l] packs the two high bits of all four at shifts 0/2/4/6.
Q4_K (and the other K-quants with explicit llama.cpp-cited layout comments) were already correct, which is why Q4_K_M models worked while this GGUF (Q4_0 projections plus a Q6_K token embedding) produced garbage at the right magnitude: every 32-element run of every quantized tensor was internally permuted with wrong high bits. Internal round-trip tests passed because larql's own encoders packed the same wrong layout; nothing pinned the bytes to ggml ground truth.
Fixed together, since writers and readers must move as one: the five larql-models dequantizers,
quantize_q4_0(both copies),quantize_q6_k,decode_q6k_superblock_into, the q6k matvec, the q6k x q8k scalar kernel, and theq4_dot.cC kernel (whose NEON path actually simplifies: the planar layout pairs directly with sequential Q8, no vzip needed). A sharedq6k_subblock_valshelper is now the single source of truth for Q6_K bit placement. New tests pin the decoders to ground-truth bytes and values from the real gemma-4 GGUF as decoded by gguf-py.Known-remaining: the Q6_K x Q8_K NEON and hand-asm kernels still assume the old layout; the dispatcher routes all arches through the fixed scalar path (
TODO(q6k-planar)) until they are reworked and verified on ARM. Metal shaders (not in this tree) need the same audit.MIGRATION: vindexes containing larql-quantized Q4_0/Q6_K blocks written before this change (Q4K-format vindexes quantize V projections to Q6_K; interleaved-q4 FFN stores use Q4_0) decode incorrectly with the fixed readers and need re-extraction. I recommend a vindex format-version bump.
5.
to_config_jsondropsfinal_logit_softcappingandrms_norm_eps(commit 5)Both are present in gemma GGUF metadata. Softcap is monotonic (never flips a top-1) but shapes the softmax; the eps emit is generic across RMSNorm architectures.
Evidence
HF
transformers5.13.0 bf16 CPU reference (Gemma4UnifiedForConditionalGeneration), safetensors-extracted 12B f32 vindex, INFER full-reforward path:The capital of France is)1@ 54.09%1@ 54.35%0@ 29.39%0@ 29.08%The@ 100.00%The@ 100.00%An autoregressive loop through the INFER path generated
The capital of France is Paris.token by token, each step's top-1 matching the reference (France68.86%,Paris50.28%,.99.15%).GGUF q4_0 conversion after fixes 1+2 (before the quant fix): manifest census went from 378 to 618 weight entries, exact parity with the safetensors reference vindex. Embedding ground truth: the gguf-py dequant of
token_embd.weightmatches the f32 vindex at cosine 0.9999+ per row, while larql's pre-fix decode sat at cosine ~0 (the smoking gun for fix 4).Post-quant-fix conversion + INFER (the GGUF q4_0 vindex, fully converted by the fixed pipeline in 56 min):
0@ 29.39%,1@ 22.89%0@ 30.83%,1@ 25.67%The@ 100.00%The@ 100.00%Same ranking, same shape, within q4_0-quantization drift on the raw prompt and an exact match on the templated one. The identical vindex lineage produced a full-width CJK bracket at 73.79% before the quant-layout fix.
Test surface
4,226 tests pass across larql-models / larql-compute / larql-vindex / larql-inference / larql-lql, including new llama.cpp ground-truth pins for Q6_K and Q4_0 and a writer/reader round-trip for the q6k matvec.
Companion issue
larql run/chatKV-cache decode still emits garbage on gemma-4 dense where INFER (full re-forward) is byte-faithful on identical input; filed separately as #177 (heterogeneous per-layer KV cache geometry).🤖 Generated with Claude Code