Skip to content

The Nemotron revision pin is existence-only: a substituted checkpoint still resolves, and W6's token gate cannot see that #569

Description

@localai-bot

The defect

parity::Nemotron35LightningSnapshot() (tests/parity/hf_snapshot.h:121-126)
gates on the existence of the local_dir's
.cache/huggingface/trees/<revision>.json manifest. That records "this
revision was downloaded here once"
— not "these bytes are that revision".

Demonstrated during the review of #561: a directory whose config.json is a
different model entirely, carrying trees/29f2d174….json and a
download/config.json.metadata naming a different revision, still resolves.
The manifest used in the passing case was an empty touched file, and a decoy
deadbeef….json sitting beside it changed nothing.

Why it matters, and when

This is the resolver W6's token gate will use to find the checkpoint its
committed goldens belong to
(tests/parity/goldens/nemotron_35_lightning_greedy/oracle.json, revision
29f2d1746d8f41e316523194b19018707749b1b1).

A token gate is precisely the instrument that cannot see a substituted
checkpoint: the wrong model produces different tokens and reads as a
correctness regression in whatever was last changed, while the right model at
a different revision can produce matching tokens and silently launder a stale
pin. This project has already lost a full campaign to a fixture that did not
match the path under test.

Not blocking today

Two reasons, both demonstrated:

  1. It is strictly better than the previous state, where neither
    CHECKPOINT_ROOT nor VT_NEMOTRON35_SNAPSHOT carried any pin at all.
  2. The current consumer fails loudly on substitution — Status: FAILURE!,
    exit 1 — rather than silently passing.

The risk is entirely forward-facing.

The fix is cheap and the data is already on disk

.cache/huggingface/download/<file>.metadata line 1 is the per-file
commit_hash
, rewritten in place on every download
(huggingface_hub/_local_folder.py, LocalDownloadFileMetadata.commit_hash).
On the NAS copy it reads 29f2d174…. Gating on that pins content at no extra
cost, and it can be checked for config.json alone or for every file the gate
loads.

Open question

Whether hf download --local-dir deletes a superseded
trees/<rev>.json when re-downloading at a new revision. The installed
huggingface_hub 1.7.2 contains no trees code at all (grep -rn trees
rc=1), so the writer is a different version that could not be read or exercised
during the review. If it does not delete, accumulated manifests are exactly
the scenario above
— an old revision's manifest vouching for new bytes.

Done when

Nemotron35LightningSnapshot() (and any sibling that grows the same shape)
resolves only when the checkpoint's own metadata says the bytes are the pinned
revision; a mismatched directory skips loudly rather than resolving; and a
test pins the mismatch case so this cannot regress.

Found while reviewing #561 (W1 of #517). Must close before W6.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions