Skip to content

st.h: index F8_E4M3, F8_E8M0 and I64, and refuse them in the float readers - #778

Merged
JustVugg merged 2 commits into
devfrom
feat/st-fp8-dtypes
Aug 2, 2026
Merged

st.h: index F8_E4M3, F8_E8M0 and I64, and refuse them in the float readers#778
JustVugg merged 2 commits into
devfrom
feat/st-fp8-dtypes

Conversation

@JustVugg

@JustVugg JustVugg commented Aug 2, 2026

Copy link
Copy Markdown
Owner

Shared-reader groundwork, requested by @DrewZt on #165 and needed before any engine can read a native fp8 checkpoint.

st.h could not open such a checkpoint at all

st_dtype_code handled BF16/F16/F32/U8/I8 and called exit(1) on anything else. Pointed a probe at the real DeepSeek-V4-Flash-0731 and it died on the first tensor:

unsupported dtype: I64

That checkpoint holds 2,329 F8_E8M0 tensors (every scale in the model), 25 F8_E4M3 (the dense weights) and 3 I64 (gate.tid2eid, the frozen token→expert table for the hash-MoE layers). None of it reachable.

Codes 4, 5 and 6 are new. 0–3 are untouched, so no existing container reads differently.

The part that matters more than the addition

Both float readers ended in an else that assumes F16 for any dtype that is not 0 or 2. With only 0–3 in existence that was correct. The moment codes 4/5/6 exist it becomes a trap: an F8_E4M3 or I64 tensor would be read as half-precision and produce plausible, wrong numbers in silence.

They now refuse by name and say which reader to use. This makes st.h safer than it was before this PR, not less.

Element size moves into st_dtype_esz(). It was written out three times as a dtype==2 ? 4 : 2 ternary — correct for four types, and it would have claimed 2 bytes for an 8-byte I64. A wrong element size is an out-of-bounds read, not a wrong number.

ue8m0_to_f32 and st_read_scale_f32

A block-scale sidecar written as one UE8M0 byte per block (2^(v-127), 0xff NaN) instead of one f32. Same geometry, same meaning, different encoding of the number — so it expands to f32 once at load, and every kernel downstream stays a single implementation with no branch in the hot loop.

Scales are ~1/16384 of the weight bytes — half a megabyte for DeepSeek-V4’s 8.4 GB dense set — so the memory cost is noise. kimi_k3.c already does exactly this for MXFP4’s ue8m0 scales in mx4_scale.

ldexpf, not the obvious bit trick. (uint32_t)v << 23 is exact for v in [1,254] and at v == 0 produces the all-zero pattern, which is exact zero, not 2^-127 — a weight block scaled to nothing instead of to almost nothing. Found by testing all 256 values rather than sampling.

Verification

  • All 256 UE8M0 values decode exactly (tests/test_ue8m0.c, new).
  • st.h indexes 72,317 tensors of the real DeepSeek-V4-Flash checkpoint. Before: exit(1).
  • A real scale sidecar reads back as exactly 2^-12 and 2^-11. UE8M0 can only encode powers of two, so a decode off by anything could not produce them.
  • Dequantised attention weights land at |max| 0.094, |mean| 0.018 — the right magnitudes.
  • All four engines build; make check 288 tests OK.

Nothing here is DeepSeek-specific: any engine gains the ability to read a native fp8 container. The fmt=8 wiring that consumes it follows in a separate PR.

JustVugg and others added 2 commits August 2, 2026 12:06
…aders

st_dtype_code knew BF16/F16/F32/U8/I8 and called exit(1) on anything else, so a
native fp8 checkpoint could not be opened at all. Pointed a probe at
DeepSeek-V4-Flash-0731 and it died on the first tensor:

    unsupported dtype: I64

That checkpoint holds 2,329 F8_E8M0 tensors (every scale), 25 F8_E4M3 (the dense
weights) and 3 I64 (gate.tid2eid, the frozen token->expert table for the hash-MoE
layers). None of them reachable.

Codes 4, 5 and 6 are new; 0-3 are untouched, so no existing container reads
differently.

THE PART THAT MATTERS MORE THAN THE ADDITION. Both float readers ended in an
 that assumes F16 for any dtype that is not 0 or 2. With only 0-3 in
existence that was correct. The moment codes 4/5/6 exist it becomes a trap: an
F8_E4M3 or I64 tensor would be read as half-precision and produce plausible,
wrong numbers in silence. They now refuse by name and say which reader to use.
That makes st.h safer than it was before this commit, not less.

Element size moves into st_dtype_esz(). It was written out three times as
, which was right for four types and would have claimed
2 bytes for an 8-byte I64.

Adds ue8m0_to_f32 and st_read_scale_f32: a block-scale sidecar written as one
UE8M0 byte per block (2^(v-127), 0xff NaN) instead of one f32. Same geometry,
same meaning, different encoding of the number -- so it expands to f32 once at
load and every kernel downstream stays a single implementation with no branch in
the hot loop. Scales are ~1/16384 of the weight bytes (half a megabyte for
DeepSeek-V4's 8.4 GB dense set), so the memory cost is noise. kimi_k3.c already
does exactly this for MXFP4's ue8m0 scales in mx4_scale.

ue8m0_to_f32 uses ldexpf rather than the bit trick : that
trick is exact for v in [1,254] and at v==0 produces 0x00000000, which is EXACT
ZERO and not 2^-127 -- a whole weight block scaled to nothing instead of to
almost nothing. Caught by testing all 256 values against the spec.

Verified: all 256 UE8M0 values correct; st.h indexes 72,317 tensors of the real
DeepSeek-V4-Flash checkpoint; a real scale sidecar reads back as exactly 2^-12
and 2^-11 (UE8M0 can only encode powers of two, so an off-by-anything decode
could not produce them); all four engines build; make check 288 tests OK.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
All 256 UE8M0 values rather than a sample: the first implementation used the
bit trick (v << 23), exact for v in [1,254] and silently wrong at v == 0, where
the all-zero pattern is exact zero and not 2^-127. Sampling would have missed
the one value that was broken.

Also pins st_dtype_esz for all seven codes. A wrong element size is an
out-of-bounds read rather than a wrong number, and that function replaced three
copies of a ternary that would have claimed 2 bytes for an 8-byte I64.

Includes the two scales actually observed in DeepSeek-V4's attention tensors
(2^-12 and 2^-11) so the test is anchored to a real checkpoint, not only to the
spec.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@JustVugg
JustVugg merged commit 57bf04e into dev Aug 2, 2026
13 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant