fix(GATE-SNAPSHOT-CONTENT-PIN): pin the Nemotron checkpoint by CONTENT, not by "this revision was downloaded here once" (#569)

[localai-bot]

Closes #569. Row: MODEL-TEXT-nemotron-h-nemotron-hfor-causal-lm. Spec: .agents/specs/nemotron-h-model.md (W6, #517).

What was wrong

parity::Nemotron35LightningSnapshot() pinned the staged hf download --local-dir checkpoint on the existence of .cache/huggingface/trees/<revision>.json. That records "this revision was downloaded into this directory once", not "these bytes are that revision".

RED, against the header at c6b240ed:

pinned revision   : 29f2d1746d8f41e316523194b19018707749b1b1
sidecar revision  : bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb
staged config.json: {"architectures": ["LlamaForCausalLM"], "hidden_size": 8}
resolver returned : "/tmp/vt_569_red_demo/nemotron-3.5-lightning-30b-nvfp4"
RESOLVED -- the gate would load THIS

A different model, whose every download sidecar named a different revision, resolved — with the pinned manifest an empty touched file and a decoy manifest beside it changing nothing. W6 is a token-exact gate that resolves through this function, and a token gate is exactly the instrument that cannot see a substituted checkpoint.

The trees/ deletion question: ANSWERED — it does not delete

huggingface_hub 1.7.2 (this box) has no trees code at all. The writer is _tree_cache.py, present in 1.23.0 / 1.24.0, which is what is installed on the gate host where the NAS copy was fetched. Its module docstring states the design:

Because a commit hash is immutable, its tree listing never changes and can be cached forever without any invalidation logic.

write_tree_cache only os.replaces <commit>.json. The string trees appears five times in the whole library — all reads and path joins. Nothing removes one.

So manifests accumulate, and the worst case is the real case: after a re-download at a later revision the pinned revision's manifest is still sitting there vouching for bytes that are no longer it. That case is staged and pinned by a test.

The fix

Sweep the per-file sidecars. Line 1 of .cache/huggingface/download/<file>.metadata is that file's commit_hash (_local_folder.py module docstring; read_download_metadata parses it back). Unlike the manifest it does not accumulate: file_download.py calls write_download_metadata(..., commit_hash=...) on all four outcomes — fresh download, copy out of the shared cache, LFS hash matched, and the local_metadata.etag == etag early return that downloads nothing — so it always names the revision the bytes were fetched as. Verified on the NAS copy: 69 sidecars, all 29f2d174…, in exact bijection with the 69 staged files (including .gitattributes).

Scope: every staged file, not just config.json

The goldens depend on the weights. hf download <repo> <shard> --revision B into a tree fetched at A leaves a correct config beside wrong shards — the substitution that matters, and the one a config-only check cannot see. 69 sidecar reads of ~130 bytes against a 20.1 GiB checkpoint the gate is about to load is not a trade-off. The check is symmetric: a file with no sidecar refuses too, or substituting a shard would cost one rm.

The manifest check is replaced, not kept as an extra AND-term: it can no longer refuse anything the sweep does not already refuse, while it can refuse a good tree fetched by a huggingface_hub older than 1.23, which writes sidecars and no manifest.

Preserved unchanged

CHECKPOINT_ROOT as the default path; VT_NEMOTRON35_SNAPSHOT as the explicit override that wins over it and is deliberately not revision-gated (naming ONE directory is the different-checkpoint run the escape exists for); a set-but-wrong override refusing rather than falling back. Each is pinned by a subcase.

Loud skip

The resolver returns the reason it refused through an optional why out-param, and test_modelopt_mixed_precision_checkpoint prints it verbatim in its exit-77 banner, so the skip names which file recorded which revision instead of "not staged". The no-argument spelling is unchanged, so test_nemotron_h_scaffold is untouched.

Sibling accessors: checked, NOT changed

Qwen27NvfP4Snapshot, Qwen27nFp8TowerSnapshot, Qwen36A3bNvfP4Snapshot, Qwen27DFlashDraftSnapshot resolve models--org--name/snapshots/<rev>/, where the revision is the directory name and every entry is a symlink into blobs/<etag>. A re-download at a new revision creates a new snapshot directory rather than replacing bytes under the old one — verified on the gate host, where models--unsloth--Qwen3.6-27B-NVFP4 holds @890bdef7 and @ccdaab7e side by side, each a directory of symlinks. Genuinely a different situation; widening this change to them would be unevidenced.

IMP-MUTATE — 4 mutations, 0 survivors

Mutation	Result
wrong-revision refusal disabled	3 cases RED
missing-sidecar refusal deleted	1 case RED
sweep replaced by config.json-only	3 cases RED
vacuous-tree refusal deleted	1 case RED

The missing-sidecar mutant initially survived: falling through to the wrong-revision branch also refuses ("" never equals the pin), so only the message changed. The message is now asserted, and a config.json that is a directory makes the vacuous-tree branch reachable, which is what catches the fourth. Header restored byte-for-byte afterwards, md5 807c9171c3529fa300e6f7729d6a57d4.

Evidence

agent-preflight.sh does not compile the parity TUs, and this header RED'd main today (#546/#551/#556) on accessor ordering, so the header was compiled in a real build rather than only swept:

Gate	Result
test_hf_snapshot_pinning	11 cases / 42 assertions, Status: SUCCESS!
test_qwen36_paged_engine, test_qwen27_paged_engine, test_qwen36_weights, test_nemotron_h_scaffold, test_modelopt_mixed_precision_checkpoint	all recompiled, exit 0
Full Release -Werror build	rc=0
ctest (full)	403/403 passed, 0 failed (rc=0); test_modelopt_mixed_precision_checkpoint SKIPPED (exit 77) with no CHECKPOINT_ROOT
Same arm vs the LIVE NAS checkpoint	resolves, 12181/12181 assertions
scripts/check-snapshot-pins.py, --self-test, tests/scripts/test_check_snapshot_pins.py	OK / 36 fixtures / 19 tests OK
tests/tools/test_online_gate_server_binary.py	20 tests OK
scripts/agent-preflight.sh --staged	All gates green

df -h / beside every number above: 447G total, 68G avail, 85% used.

Owed, out of this row's authority

Authority for this task excluded docs/ and .agents/. #569 is therefore not yet linked in the .agents/roadmap_v1.md issue table or in .agents/specs/nemotron-h-model.md; preflight does not demand it, and it is left to the operator rather than edited here. No public-doc surface is owed — no row changed lifecycle state.

🤖 Generated with Claude Code