The defect
parity::Nemotron35LightningSnapshot() (tests/parity/hf_snapshot.h:121-126)
gates on the existence of the local_dir's
.cache/huggingface/trees/<revision>.json manifest. That records "this
revision was downloaded here once" — not "these bytes are that revision".
Demonstrated during the review of #561: a directory whose config.json is a
different model entirely, carrying trees/29f2d174….json and a
download/config.json.metadata naming a different revision, still resolves.
The manifest used in the passing case was an empty touched file, and a decoy
deadbeef….json sitting beside it changed nothing.
Why it matters, and when
This is the resolver W6's token gate will use to find the checkpoint its
committed goldens belong to
(tests/parity/goldens/nemotron_35_lightning_greedy/oracle.json, revision
29f2d1746d8f41e316523194b19018707749b1b1).
A token gate is precisely the instrument that cannot see a substituted
checkpoint: the wrong model produces different tokens and reads as a
correctness regression in whatever was last changed, while the right model at
a different revision can produce matching tokens and silently launder a stale
pin. This project has already lost a full campaign to a fixture that did not
match the path under test.
Not blocking today
Two reasons, both demonstrated:
- It is strictly better than the previous state, where neither
CHECKPOINT_ROOT nor VT_NEMOTRON35_SNAPSHOT carried any pin at all.
- The current consumer fails loudly on substitution —
Status: FAILURE!,
exit 1 — rather than silently passing.
The risk is entirely forward-facing.
The fix is cheap and the data is already on disk
.cache/huggingface/download/<file>.metadata line 1 is the per-file
commit_hash, rewritten in place on every download
(huggingface_hub/_local_folder.py, LocalDownloadFileMetadata.commit_hash).
On the NAS copy it reads 29f2d174…. Gating on that pins content at no extra
cost, and it can be checked for config.json alone or for every file the gate
loads.
Open question
Whether hf download --local-dir deletes a superseded
trees/<rev>.json when re-downloading at a new revision. The installed
huggingface_hub 1.7.2 contains no trees code at all (grep -rn trees →
rc=1), so the writer is a different version that could not be read or exercised
during the review. If it does not delete, accumulated manifests are exactly
the scenario above — an old revision's manifest vouching for new bytes.
Done when
Nemotron35LightningSnapshot() (and any sibling that grows the same shape)
resolves only when the checkpoint's own metadata says the bytes are the pinned
revision; a mismatched directory skips loudly rather than resolving; and a
test pins the mismatch case so this cannot regress.
Found while reviewing #561 (W1 of #517). Must close before W6.
The defect
parity::Nemotron35LightningSnapshot()(tests/parity/hf_snapshot.h:121-126)gates on the existence of the
local_dir's.cache/huggingface/trees/<revision>.jsonmanifest. That records "thisrevision was downloaded here once" — not "these bytes are that revision".
Demonstrated during the review of #561: a directory whose
config.jsonis adifferent model entirely, carrying
trees/29f2d174….jsonand adownload/config.json.metadatanaming a different revision, still resolves.The manifest used in the passing case was an empty
touched file, and a decoydeadbeef….jsonsitting beside it changed nothing.Why it matters, and when
This is the resolver W6's token gate will use to find the checkpoint its
committed goldens belong to
(
tests/parity/goldens/nemotron_35_lightning_greedy/oracle.json, revision29f2d1746d8f41e316523194b19018707749b1b1).A token gate is precisely the instrument that cannot see a substituted
checkpoint: the wrong model produces different tokens and reads as a
correctness regression in whatever was last changed, while the right model at
a different revision can produce matching tokens and silently launder a stale
pin. This project has already lost a full campaign to a fixture that did not
match the path under test.
Not blocking today
Two reasons, both demonstrated:
CHECKPOINT_ROOTnorVT_NEMOTRON35_SNAPSHOTcarried any pin at all.Status: FAILURE!,exit 1 — rather than silently passing.
The risk is entirely forward-facing.
The fix is cheap and the data is already on disk
.cache/huggingface/download/<file>.metadataline 1 is the per-filecommit_hash, rewritten in place on every download(
huggingface_hub/_local_folder.py,LocalDownloadFileMetadata.commit_hash).On the NAS copy it reads
29f2d174…. Gating on that pins content at no extracost, and it can be checked for
config.jsonalone or for every file the gateloads.
Open question
Whether
hf download --local-dirdeletes a supersededtrees/<rev>.jsonwhen re-downloading at a new revision. The installedhuggingface_hub1.7.2 contains notreescode at all (grep -rn trees→rc=1), so the writer is a different version that could not be read or exercised
during the review. If it does not delete, accumulated manifests are exactly
the scenario above — an old revision's manifest vouching for new bytes.
Done when
Nemotron35LightningSnapshot()(and any sibling that grows the same shape)resolves only when the checkpoint's own metadata says the bytes are the pinned
revision; a mismatched directory skips loudly rather than resolving; and a
test pins the mismatch case so this cannot regress.
Found while reviewing #561 (W1 of #517). Must close before W6.