Skip to content

Pin HuggingFace model revisions for reproducible benchmarks #10

Description

@urrme

The Embedder pins a HuggingFace model revision, but the reranker, SPLADE retriever, and grounding verifier call from_pretrained without a pinned revision=. If HF republishes those weights, benchmark numbers can drift silently.

Scope (good first issue):

  1. Find the unpinned loads (start at src/nexusrag/retrieval/reranker.py; also SPLADE + grounding).
  2. Record each model repo's current commit SHA and add revision="<sha>" to the call.
  3. Note the pins in PROVENANCE.md.
  4. Re-run make eval and confirm the committed benchmark numbers still reproduce.

Self-contained; no architecture changes needed.

Metadata

Metadata

Assignees

No one assigned

    Labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions