diff --git a/CLAUDE.md b/CLAUDE.md index 3749536..935e904 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -50,6 +50,15 @@ The package is small and self-documenting, so there's no separate `src/` guide. - **Gates (all required, blocking PR checks):** `ruff check .`, `ruff format --check .`, `mypy` (strict), `pytest --cov` at **100% coverage**. +## Publishing model checkpoints to HuggingFace (any agent session) + +Any checkpoint published to the robocurve HF org **must** follow +[docs/model-cards.md](docs/model-cards.md): fill in +[docs/model-card-template.md](docs/model-card-template.md) top-to-bottom, then pass the +gating checklist (any unchecked REQUIRED box blocks publish). Common traps it prevents: +license inheritance from the base model (never default apache-2.0), missing action/observation +I/O contracts, and unverifiable eval claims. Reference implementations are linked in the doc. + ## Adding a benchmark to the catalog Append a `Benchmark(...)` entry to `src/worldevals/catalog.py` with its name, diff --git a/README.md b/README.md index 4f38a94..f21f009 100644 --- a/README.md +++ b/README.md @@ -85,6 +85,14 @@ validates every entry (unique name, well-formed repo URL, ≥1 task key). See [KitchenBench](https://github.com/robocurve/kitchenbench) as the reference implementation. +## Publishing model checkpoints + +Benchmarks compare *policies* — and a benchmark number is only interpretable when the +checkpoint behind it is documented. Every model checkpoint published to the +[robocurve HuggingFace org](https://huggingface.co/robocurve) must follow the +**[model-card standard](docs/model-cards.md)** (start from the +[template](docs/model-card-template.md); the standard includes a gating checklist). + ## Development > **Dependency changes:** after editing dependencies in `pyproject.toml`, run diff --git a/docs/model-card-template.md b/docs/model-card-template.md index 0ac3f93..2faa323 100644 --- a/docs/model-card-template.md +++ b/docs/model-card-template.md @@ -32,6 +32,11 @@ inference: false # +> **Evaluate this checkpoint** with [Inspect Robots](https://github.com/robocurve/inspect-robots) — +> the Inspect-AI-style eval framework for physical AI — using the +> [ adapters](https://github.com/robocurve/inspect-robots-); benchmarks +> are catalogued in [WorldEvals](https://github.com/robocurve/worldevals). + <≤3 sentences: [base model](link) + method (LoRA/full FT; what's frozen) + robot + [dataset](link). Adapter/merge status and where raw adapters live.> diff --git a/docs/model-cards.md b/docs/model-cards.md index f91b3ea..5dcd453 100644 --- a/docs/model-cards.md +++ b/docs/model-cards.md @@ -93,6 +93,10 @@ model-index: At most three sentences: base model + method (LoRA/full FT; what's frozen) + robot/embodiment + dataset. State whether adapters are merged and where raw adapters live if published. +**Immediately after the H1**, add an eval-path blockquote linking +[Inspect Robots](https://github.com/robocurve/inspect-robots), the embodiment's adapter repo +(e.g. inspect-robots-so101 / inspect-robots-yam), and +[WorldEvals](https://github.com/robocurve/worldevals) — see the template. Link every **named artifact**: base model, each dataset, training-code repo, method libraries, papers. (Ordinary nouns like "camera" need no link.) @@ -201,6 +205,7 @@ Frontmatter Body - [ ] Header ≤3 sentences; every named artifact linked; adapter/merge status stated +- [ ] Eval-path blockquote right after the H1 (Inspect Robots + embodiment adapters + WorldEvals) - [ ] Intended use + out-of-scope + validation status (with e-stop/supervision language) - [ ] Training table: data counts + filter/split rule (level, %, seed) - [ ] Training table: embodiment config + state/action key layout