Skip to content

docs: register int4-rans256-g0 in FORMATS.md; registry parity for ordinal-less rows - #828

Merged
JustVugg merged 3 commits into
JustVugg:devfrom
monotophic:docs/formats-rans-registration-r1
Aug 4, 2026
Merged

docs: register int4-rans256-g0 in FORMATS.md; registry parity for ordinal-less rows#828
JustVugg merged 3 commits into
JustVugg:devfrom
monotophic:docs/formats-rans-registration-r1

Conversation

@monotophic

Copy link
Copy Markdown
Contributor

Authored by Fable 5 in Claude Code, analysis in partnership with @monotophic.

Registers int4-rans256-g0 β€” the entropy-coded container format that merged
with #671's offline tools β€” in docs/FORMATS.md, and extends the registry
parity test so rows like it are inside the safety net.

Scope and audience: this changes no runtime behavior and affects no one's
tok/s. It's coordination infrastructure for format authors β€” the registry
exists because ordinal collisions happened twice in one month (#465 vs fmt=6;
#705 vs #528 on fmt=7, which cost a 191-reference renumber), and with CUDA
work now building on registered formats (#817), name/ordinal coordination is
load-bearing rather than hypothetical. Two concrete protections ship here:
the collision safety net now actually sees ordinal-less rows (it previously
skipped them β€” a colliding row passed CI green), and anyone who runs the #671
repack tools gets an explicit statement that pointing today's engine at a
repacked directory is unsupported, with the failure mode named instead of
discovered.

The row claims no public ordinal, deliberately. The format has no engine
decode path today (qt_resolve_fmt has no branch for it; no compiled fmt
constant exists anywhere, not even a private-block one β€” its identity is the
mandatory __metadata__["colibri.fmt"] stamp). Per this registry's own "ID
assignment" rule, an ordinal is claimed by the first merge into dev that ships
it; the row reads that as ships engine code consuming it for a
stamp-identified format β€” stated in the row as its own clarifying gloss, since
there is nothing to compile a number into until the ladder's PR 2 (engine
decode stage) lands. The "next free public ordinal is 9" statement stays true.

Registry parity coverage (c/tests/test_formats_registry.py): the parser
previously recognized only numeric-ordinal rows, so an ordinal-less row was
invisible to the collision checks this registry exists for. It now parses both
populations and enforces three rules on ordinal-less rows: name uniqueness
across the whole table, no claiming a compiled FMT_NAMES name, and an
explicit "no public ordinal" declaration. Verified to bite: an injected
colliding row fails all three tests by name.

Two known constraints, disclosed for the future decode stage (both
pre-existing, neither introduced or resolved here; the row's wording reflects
them):

  1. The mandatory stamp is not consulted on the routed-expert load paths β€” the
    format's entire target population β€” which pass stamped_name=NULL
    (c/colibri.c:2217/2386/2580). Stamp-gated dispatch must be wired ahead of
    byte-arithmetic inference when the decode stage lands, exactly as
    docs/int4-rans256-g0.md's "THE STAMP IS MANDATORY" section warns.
  2. ST_FMT_STAMP_MAX (4096, per this doc's own scan-bound section) is sized
    for hundreds of stamped tensors; a full-corpus routed-expert repack needs
    tens of thousands of stamp entries. The decode-stage PR must raise the
    bound or restructure the map before full-corpus repacks are loadable.

Shared-namespace scan (per this registry's convention): dev HEAD 7fb1159
has no rans row and no fmt=9 claim; all 51 open PRs enumerated 2026-08-04 β€”
none touch docs/FORMATS.md, claim a format ordinal, or use rans naming.

Requirement Decisive evidence
Every factual cell true at its stated anchor Each carries a file:line citation at dev 7fb1159 β€” checkable in-tree
Ordinal-less rows now inside the parity net Injected colliding row β†’ 3 named test failures; real doc β†’ 5/5 green
Nothing else regresses make check green: C battery 0 failures, Python 304 tests OK
Verification detail
  • Cross-doc consistency: docs/int4-rans256-g0.md's registry stub updated in
    the same PR to match the landed row (it previously said ordinal
    (maintainer-assigned), status "proposed").
  • Table structure: all rows column-count-consistent in both edited docs.
  • The parity suite's plausibility floor now counts both row populations, so a
    parser drift that loses ordinal-less rows fails loudly rather than
    silently shrinking coverage.
  • Independent blind fact-check re-derived every row cell against
    git show 7fb1159:<file> with zero discrepancies before this was opened.

… no public ordinal)

Add the registry row and sources bullet for int4-rans256-g0, the
entropy-coded int4 expert container that merged via JustVugg#671 (a3a5a75) as
codec + offline tools only. The row claims NO public ordinal: no engine
decode path exists (qt_resolve_fmt has no branch for it and no compiled
fmt constant exists, public or private), so there is nothing to compile
a number into. The registry's ID-assignment rule says an ordinal is
claimed by the first merge into dev that ships it; for a stamp-identified
format whose tools merged without any engine consumer, this row adds its
own clarifying gloss on that rule: none is claimed before engine code
consuming the format (the ladder's PR 2 loader/CPU-decode stage) first
merges into dev. Identity on disk is carried by the format's mandatory
__metadata__ stamp, not a number, and the row exists so name-collision
scans find the NAME β€” exactly what this registry is for.

The row and its sources bullet also state the engine-interaction hazard
precisely rather than claiming inertness: discovery-time stamp ingest
(st_fmt_stamp_ingest) and load-time byte-arithmetic inference DO run
against repacked shards, and the routed-expert load sites β€” this format's
entire target population β€” pass stamped_name=NULL, so the MANDATORY
stamp is not consulted there today. Pointing the current engine at a
repacked directory is unsupported (typical outcome: a named refusal from
the byte-arithmetic mismatch); the stamp becomes load-bearing only when
PR 2 wires stamp-gated dispatch ahead of inference, as the format spec
itself warns.

Facts verified against dev 7fb1159: RANS_NSTREAMS=256 (c/rans.h:93),
record layout (c/rans.h:17-34, docs/int4-rans256-g0.md), stamp/table
keys (c/tools/rans_format.py:40-46), ~0.76 census ratio and byte-exact
contract (docs/int4-rans256-g0.md), stamped_name=NULL routed-expert
sites (c/colibri.c:2217, 2386, 2580), stamp ingest (c/st.h:317).
The spec's 'Proposed registry row' section predates the registration:
it proposed a row with ordinal *(maintainer-assigned)* and status
'proposed, offline tools only'. The row has now landed in FORMATS.md
with ordinal *(none)* and status 'merged, offline tools only', so the
stub (and the header paragraph's 'whenever a registry (FORMATS.md)
lands' framing) would otherwise contradict the tree it lives in.
Retitle the section 'Registry row (landed)', mirror the landed cells,
and carry the same ID-assignment gloss the FORMATS.md row states: with
no engine decode path there is nothing to compile an ordinal into, and
none is claimed before engine code consuming the format (PR 2) first
merges into dev. FORMATS.md remains the authoritative copy.
test_formats_registry.py's doc parser only matched rows with a numeric
ordinal cell, so the new ordinal-less int4-rans256-g0 row (*(none)*)
escaped every parity rule β€” an ordinal-less row could have silently
shadowed a numbered row's stamp NAME, the exact collision class this
registry and test exist to prevent.

Parse ordinal-less rows too (>=5 cells and a backticked name cell, so
headers/separators stay excluded) and enforce three rules over them:
NAME uniqueness across the whole table (numbered + ordinal-less), no
claiming a name that IS in FMT_NAMES (a compiled name has an ordinal
and its row must carry it), and an explicit 'no public ordinal'
declaration β€” the ordinal-less analogue of the existing 'no in-tree
name string' say-so marker. A plausibility floor on total parsed rows
keeps a parser drift from passing as an empty ordinal-less set.

Proof of bite: an injected row '| *(pending)* | `fp8-e4m3-b128` | ...'
fails all three new tests by name (duplicate NAME, compiled NAME,
missing declaration); the real doc passes 5/5.
@JustVugg
JustVugg merged commit 19462f4 into JustVugg:dev Aug 4, 2026
13 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants