docs: register int4-rans256-g0 in FORMATS.md; registry parity for ordinal-less rows - #828
Merged
JustVugg merged 3 commits intoAug 4, 2026
Merged
Conversation
β¦ no public ordinal) Add the registry row and sources bullet for int4-rans256-g0, the entropy-coded int4 expert container that merged via JustVugg#671 (a3a5a75) as codec + offline tools only. The row claims NO public ordinal: no engine decode path exists (qt_resolve_fmt has no branch for it and no compiled fmt constant exists, public or private), so there is nothing to compile a number into. The registry's ID-assignment rule says an ordinal is claimed by the first merge into dev that ships it; for a stamp-identified format whose tools merged without any engine consumer, this row adds its own clarifying gloss on that rule: none is claimed before engine code consuming the format (the ladder's PR 2 loader/CPU-decode stage) first merges into dev. Identity on disk is carried by the format's mandatory __metadata__ stamp, not a number, and the row exists so name-collision scans find the NAME β exactly what this registry is for. The row and its sources bullet also state the engine-interaction hazard precisely rather than claiming inertness: discovery-time stamp ingest (st_fmt_stamp_ingest) and load-time byte-arithmetic inference DO run against repacked shards, and the routed-expert load sites β this format's entire target population β pass stamped_name=NULL, so the MANDATORY stamp is not consulted there today. Pointing the current engine at a repacked directory is unsupported (typical outcome: a named refusal from the byte-arithmetic mismatch); the stamp becomes load-bearing only when PR 2 wires stamp-gated dispatch ahead of inference, as the format spec itself warns. Facts verified against dev 7fb1159: RANS_NSTREAMS=256 (c/rans.h:93), record layout (c/rans.h:17-34, docs/int4-rans256-g0.md), stamp/table keys (c/tools/rans_format.py:40-46), ~0.76 census ratio and byte-exact contract (docs/int4-rans256-g0.md), stamped_name=NULL routed-expert sites (c/colibri.c:2217, 2386, 2580), stamp ingest (c/st.h:317).
The spec's 'Proposed registry row' section predates the registration: it proposed a row with ordinal *(maintainer-assigned)* and status 'proposed, offline tools only'. The row has now landed in FORMATS.md with ordinal *(none)* and status 'merged, offline tools only', so the stub (and the header paragraph's 'whenever a registry (FORMATS.md) lands' framing) would otherwise contradict the tree it lives in. Retitle the section 'Registry row (landed)', mirror the landed cells, and carry the same ID-assignment gloss the FORMATS.md row states: with no engine decode path there is nothing to compile an ordinal into, and none is claimed before engine code consuming the format (PR 2) first merges into dev. FORMATS.md remains the authoritative copy.
test_formats_registry.py's doc parser only matched rows with a numeric ordinal cell, so the new ordinal-less int4-rans256-g0 row (*(none)*) escaped every parity rule β an ordinal-less row could have silently shadowed a numbered row's stamp NAME, the exact collision class this registry and test exist to prevent. Parse ordinal-less rows too (>=5 cells and a backticked name cell, so headers/separators stay excluded) and enforce three rules over them: NAME uniqueness across the whole table (numbered + ordinal-less), no claiming a name that IS in FMT_NAMES (a compiled name has an ordinal and its row must carry it), and an explicit 'no public ordinal' declaration β the ordinal-less analogue of the existing 'no in-tree name string' say-so marker. A plausibility floor on total parsed rows keeps a parser drift from passing as an empty ordinal-less set. Proof of bite: an injected row '| *(pending)* | `fp8-e4m3-b128` | ...' fails all three new tests by name (duplicate NAME, compiled NAME, missing declaration); the real doc passes 5/5.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Authored by Fable 5 in Claude Code, analysis in partnership with @monotophic.
Registers
int4-rans256-g0β the entropy-coded container format that mergedwith #671's offline tools β in
docs/FORMATS.md, and extends the registryparity test so rows like it are inside the safety net.
Scope and audience: this changes no runtime behavior and affects no one's
tok/s. It's coordination infrastructure for format authors β the registry
exists because ordinal collisions happened twice in one month (#465 vs fmt=6;
#705 vs #528 on fmt=7, which cost a 191-reference renumber), and with CUDA
work now building on registered formats (#817), name/ordinal coordination is
load-bearing rather than hypothetical. Two concrete protections ship here:
the collision safety net now actually sees ordinal-less rows (it previously
skipped them β a colliding row passed CI green), and anyone who runs the #671
repack tools gets an explicit statement that pointing today's engine at a
repacked directory is unsupported, with the failure mode named instead of
discovered.
The row claims no public ordinal, deliberately. The format has no engine
decode path today (
qt_resolve_fmthas no branch for it; no compiledfmtconstant exists anywhere, not even a private-block one β its identity is the
mandatory
__metadata__["colibri.fmt"]stamp). Per this registry's own "IDassignment" rule, an ordinal is claimed by the first merge into dev that ships
it; the row reads that as ships engine code consuming it for a
stamp-identified format β stated in the row as its own clarifying gloss, since
there is nothing to compile a number into until the ladder's PR 2 (engine
decode stage) lands. The "next free public ordinal is 9" statement stays true.
Registry parity coverage (
c/tests/test_formats_registry.py): the parserpreviously recognized only numeric-ordinal rows, so an ordinal-less row was
invisible to the collision checks this registry exists for. It now parses both
populations and enforces three rules on ordinal-less rows: name uniqueness
across the whole table, no claiming a compiled
FMT_NAMESname, and anexplicit "no public ordinal" declaration. Verified to bite: an injected
colliding row fails all three tests by name.
Two known constraints, disclosed for the future decode stage (both
pre-existing, neither introduced or resolved here; the row's wording reflects
them):
format's entire target population β which pass
stamped_name=NULL(
c/colibri.c:2217/2386/2580). Stamp-gated dispatch must be wired ahead ofbyte-arithmetic inference when the decode stage lands, exactly as
docs/int4-rans256-g0.md's "THE STAMP IS MANDATORY" section warns.ST_FMT_STAMP_MAX(4096, per this doc's own scan-bound section) is sizedfor hundreds of stamped tensors; a full-corpus routed-expert repack needs
tens of thousands of stamp entries. The decode-stage PR must raise the
bound or restructure the map before full-corpus repacks are loadable.
Shared-namespace scan (per this registry's convention): dev HEAD
7fb1159has no rans row and no fmt=9 claim; all 51 open PRs enumerated 2026-08-04 β
none touch
docs/FORMATS.md, claim a format ordinal, or use rans naming.file:linecitation at dev7fb1159β checkable in-treemake checkgreen: C battery 0 failures, Python 304 tests OKVerification detail
docs/int4-rans256-g0.md's registry stub updated inthe same PR to match the landed row (it previously said ordinal
(maintainer-assigned), status "proposed").
parser drift that loses ordinal-less rows fails loudly rather than
silently shrinking coverage.
git show 7fb1159:<file>with zero discrepancies before this was opened.