fmt=8: accept UE8M0 block scales alongside the f32 ones - #779
Merged
Conversation
…aders
st_dtype_code knew BF16/F16/F32/U8/I8 and called exit(1) on anything else, so a
native fp8 checkpoint could not be opened at all. Pointed a probe at
DeepSeek-V4-Flash-0731 and it died on the first tensor:
unsupported dtype: I64
That checkpoint holds 2,329 F8_E8M0 tensors (every scale), 25 F8_E4M3 (the dense
weights) and 3 I64 (gate.tid2eid, the frozen token->expert table for the hash-MoE
layers). None of them reachable.
Codes 4, 5 and 6 are new; 0-3 are untouched, so no existing container reads
differently.
THE PART THAT MATTERS MORE THAN THE ADDITION. Both float readers ended in an
that assumes F16 for any dtype that is not 0 or 2. With only 0-3 in
existence that was correct. The moment codes 4/5/6 exist it becomes a trap: an
F8_E4M3 or I64 tensor would be read as half-precision and produce plausible,
wrong numbers in silence. They now refuse by name and say which reader to use.
That makes st.h safer than it was before this commit, not less.
Element size moves into st_dtype_esz(). It was written out three times as
, which was right for four types and would have claimed
2 bytes for an 8-byte I64.
Adds ue8m0_to_f32 and st_read_scale_f32: a block-scale sidecar written as one
UE8M0 byte per block (2^(v-127), 0xff NaN) instead of one f32. Same geometry,
same meaning, different encoding of the number -- so it expands to f32 once at
load and every kernel downstream stays a single implementation with no branch in
the hot loop. Scales are ~1/16384 of the weight bytes (half a megabyte for
DeepSeek-V4's 8.4 GB dense set), so the memory cost is noise. kimi_k3.c already
does exactly this for MXFP4's ue8m0 scales in mx4_scale.
ue8m0_to_f32 uses ldexpf rather than the bit trick : that
trick is exact for v in [1,254] and at v==0 produces 0x00000000, which is EXACT
ZERO and not 2^-127 -- a whole weight block scaled to nothing instead of to
almost nothing. Caught by testing all 256 values against the spec.
Verified: all 256 UE8M0 values correct; st.h indexes 72,317 tensors of the real
DeepSeek-V4-Flash checkpoint; a real scale sidecar reads back as exactly 2^-12
and 2^-11 (UE8M0 can only encode powers of two, so an off-by-anything decode
could not produce them); all four engines build; make check 288 tests OK.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
All 256 UE8M0 values rather than a sample: the first implementation used the bit trick (v << 23), exact for v in [1,254] and silently wrong at v == 0, where the all-zero pattern is exact zero and not 2^-127. Sampling would have missed the one value that was broken. Also pins st_dtype_esz for all seven codes. A wrong element size is an out-of-bounds read rather than a wrong number, and that function replaced three copies of a ternary that would have claimed 2 bytes for an 8-byte I64. Includes the two scales actually observed in DeepSeek-V4's attention tensors (2^-12 and 2^-11) so the test is anchored to a real checkpoint, not only to the spec. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Native fp8 checkpoints write the fmt=8 block scale as one UE8M0 byte per block rather than one f32. The geometry is identical -- same shape, same meaning, same multiply -- so the load path now reads the sidecar with st_read_scale_f32, which accepts either encoding and always yields f32. matmul_fp8 is untouched and stays a single implementation with no branch in the hot loop, which is the point of expanding at load rather than decoding per block. Every other format still goes through st_read_f32_cap exactly as before: fmt 0/1/2/4/5/6 are byte-for-byte unchanged. An f32-scaled fmt=8 container behaves identically too, since st_read_scale_f32 dispatches to st_read_f32 for an F32 sidecar -- the same call that ran before. Requested by DrewZt on #165, where it was the one shared-infrastructure blocker for moving the DeepSeek V4 engine onto the common quant path. Verified against the real DeepSeek-V4-Flash-0731 checkpoint: the attention sidecars read back as exactly 2^-12 and 2^-11, and the dequantised weights land at |max| 0.094 / |mean| 0.018. All four engines build; make check 288 tests OK. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This was referenced Aug 2, 2026
JustVugg
pushed a commit
that referenced
this pull request
Aug 4, 2026
…scale aware fmt=8 tensors (raw e4m3 bytes, one f32 scale per 128x128 block of [O,I], matmul_fp8's semantics) were CPU-only: qt_cuda_upload refused them because row_bytes()/weight_at() had no case and every kernel would have applied per-row scale semantics to a block-scaled container. Backend: - quant_matmul gains an fmt=8 branch decoding through c_e4m3, a 256-float __constant__ LUT published from quant.h's E4M3_LUT via the new coli_cuda_fp8_set_lut (same single-source-of-truth arrangement as the E8 codebook). The block geometry derives from I alone (FP8_BLOCK is a fixed property of the format), so the branch is correct from every call site -- dense, expert MLP, and the async fallback loop included. - grouped_hidden_f8_dual / grouped_down_f8 mirror the g4 expert kernels (silu fused in the dual epilogue); the sync and async expert-group paths dispatch all-fmt=8 groups to them. Mixed groups take the per-expert loop (sync) or refuse (async), like E8. - Uploads of fmt=8 are refused until the LUT is published: a kernel reading the zero-initialized table would compute silent zeros. The engine gates on the same condition (g_cuda_fp8_ready), so an old DLL without the symbol degrades to CPU exactly like fmt=6 does without its codebook. - The absorb-path entries refuse fmt>4 via absorb_fmt_ok: weight_at + absorb_scale know per-row and fmt=4 scales only, so fmt=5/6/8 would be mis-decoded there. Callers keep their CPU attention path; a block-scale absorb kernel is follow-up work. (proj tensors are exempt -- they run through quant_matmul.) Engine: qt_cuda_upload/qt_cuda_update/matmul_qt_ex route fmt=8 weights from q8 and gate on g_cuda_fp8_ready, published in cuda_boot next to the E8 grid. Loader: fp8_set_lut resolved as optional, like e8_set_grid. tests/test_fp8_cuda.cu: oracle vs a reference that replicates matmul_fp8 (LUT decode cross-checked arithmetically, float-in-block/double-across accumulation, block scales at the checkpoint-realistic 2^-12 magnitude from #779's verification), covering partial tail blocks on both axes; API phase checks the LUT gate refuses before publish, dense matmul, sync expert group vs oracle, and async issue/take bit-identical to sync.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #778. Requested by @DrewZt on #165, where this was the one shared-infrastructure blocker for moving the DeepSeek V4 engine onto the common quant path.
What changes
Native fp8 checkpoints write the
fmt=8block scale as one UE8M0 byte per block instead of one f32. The geometry is identical — same shape, same meaning, same multiply — only the encoding of the number differs.The load path now reads the sidecar with
st_read_scale_f32(from #778), which accepts either encoding and always yields f32.matmul_fp8is untouched. Expanding at load rather than decoding per block keeps the kernel a single implementation with no branch in the hot loop — the same choicekimi_k3.calready makes for MXFP4 inmx4_scale. The scales are ~1/16384 of the weight bytes, so the memory cost is noise.Nothing else moves
st_read_scale_f32dispatches tost_read_f32for an F32 sidecar, which is the exact call that ran before.Verified against the real checkpoint
DeepSeek-V4-Flash-0731, all 167 GB of it, downloaded and size-verified:The scales come back as exactly 2^-12 and 2^-11. UE8M0 can only encode powers of two, so a decode wrong by anything at all could not produce them. The weight magnitudes are what attention projections should look like.
All four engines build;
make check288 testsOK (skipped=13).