st.h: index F8_E4M3, F8_E8M0 and I64, and refuse them in the float readers - #778
Merged
Conversation
…aders
st_dtype_code knew BF16/F16/F32/U8/I8 and called exit(1) on anything else, so a
native fp8 checkpoint could not be opened at all. Pointed a probe at
DeepSeek-V4-Flash-0731 and it died on the first tensor:
unsupported dtype: I64
That checkpoint holds 2,329 F8_E8M0 tensors (every scale), 25 F8_E4M3 (the dense
weights) and 3 I64 (gate.tid2eid, the frozen token->expert table for the hash-MoE
layers). None of them reachable.
Codes 4, 5 and 6 are new; 0-3 are untouched, so no existing container reads
differently.
THE PART THAT MATTERS MORE THAN THE ADDITION. Both float readers ended in an
that assumes F16 for any dtype that is not 0 or 2. With only 0-3 in
existence that was correct. The moment codes 4/5/6 exist it becomes a trap: an
F8_E4M3 or I64 tensor would be read as half-precision and produce plausible,
wrong numbers in silence. They now refuse by name and say which reader to use.
That makes st.h safer than it was before this commit, not less.
Element size moves into st_dtype_esz(). It was written out three times as
, which was right for four types and would have claimed
2 bytes for an 8-byte I64.
Adds ue8m0_to_f32 and st_read_scale_f32: a block-scale sidecar written as one
UE8M0 byte per block (2^(v-127), 0xff NaN) instead of one f32. Same geometry,
same meaning, different encoding of the number -- so it expands to f32 once at
load and every kernel downstream stays a single implementation with no branch in
the hot loop. Scales are ~1/16384 of the weight bytes (half a megabyte for
DeepSeek-V4's 8.4 GB dense set), so the memory cost is noise. kimi_k3.c already
does exactly this for MXFP4's ue8m0 scales in mx4_scale.
ue8m0_to_f32 uses ldexpf rather than the bit trick : that
trick is exact for v in [1,254] and at v==0 produces 0x00000000, which is EXACT
ZERO and not 2^-127 -- a whole weight block scaled to nothing instead of to
almost nothing. Caught by testing all 256 values against the spec.
Verified: all 256 UE8M0 values correct; st.h indexes 72,317 tensors of the real
DeepSeek-V4-Flash checkpoint; a real scale sidecar reads back as exactly 2^-12
and 2^-11 (UE8M0 can only encode powers of two, so an off-by-anything decode
could not produce them); all four engines build; make check 288 tests OK.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
All 256 UE8M0 values rather than a sample: the first implementation used the bit trick (v << 23), exact for v in [1,254] and silently wrong at v == 0, where the all-zero pattern is exact zero and not 2^-127. Sampling would have missed the one value that was broken. Also pins st_dtype_esz for all seven codes. A wrong element size is an out-of-bounds read rather than a wrong number, and that function replaced three copies of a ternary that would have claimed 2 bytes for an 8-byte I64. Includes the two scales actually observed in DeepSeek-V4's attention tensors (2^-12 and 2^-11) so the test is anchored to a real checkpoint, not only to the spec. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
6 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Shared-reader groundwork, requested by @DrewZt on #165 and needed before any engine can read a native fp8 checkpoint.
st.hcould not open such a checkpoint at allst_dtype_codehandled BF16/F16/F32/U8/I8 and calledexit(1)on anything else. Pointed a probe at the realDeepSeek-V4-Flash-0731and it died on the first tensor:That checkpoint holds 2,329
F8_E8M0tensors (every scale in the model), 25F8_E4M3(the dense weights) and 3I64(gate.tid2eid, the frozen token→expert table for the hash-MoE layers). None of it reachable.Codes 4, 5 and 6 are new. 0–3 are untouched, so no existing container reads differently.
The part that matters more than the addition
Both float readers ended in an
elsethat assumes F16 for any dtype that is not 0 or 2. With only 0–3 in existence that was correct. The moment codes 4/5/6 exist it becomes a trap: anF8_E4M3orI64tensor would be read as half-precision and produce plausible, wrong numbers in silence.They now refuse by name and say which reader to use. This makes
st.hsafer than it was before this PR, not less.Element size moves into
st_dtype_esz(). It was written out three times as adtype==2 ? 4 : 2ternary — correct for four types, and it would have claimed 2 bytes for an 8-byteI64. A wrong element size is an out-of-bounds read, not a wrong number.ue8m0_to_f32andst_read_scale_f32A block-scale sidecar written as one UE8M0 byte per block (
2^(v-127),0xffNaN) instead of one f32. Same geometry, same meaning, different encoding of the number — so it expands to f32 once at load, and every kernel downstream stays a single implementation with no branch in the hot loop.Scales are ~1/16384 of the weight bytes — half a megabyte for DeepSeek-V4’s 8.4 GB dense set — so the memory cost is noise.
kimi_k3.calready does exactly this for MXFP4’s ue8m0 scales inmx4_scale.ldexpf, not the obvious bit trick.(uint32_t)v << 23is exact forvin [1,254] and atv == 0produces the all-zero pattern, which is exact zero, not 2^-127 — a weight block scaled to nothing instead of to almost nothing. Found by testing all 256 values rather than sampling.Verification
tests/test_ue8m0.c, new).st.hindexes 72,317 tensors of the real DeepSeek-V4-Flash checkpoint. Before:exit(1).make check288 testsOK.Nothing here is DeepSeek-specific: any engine gains the ability to read a native fp8 container. The
fmt=8wiring that consumes it follows in a separate PR.