Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 9 additions & 0 deletions CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -50,6 +50,15 @@ The package is small and self-documenting, so there's no separate `src/` guide.
- **Gates (all required, blocking PR checks):** `ruff check .`,
`ruff format --check .`, `mypy` (strict), `pytest --cov` at **100% coverage**.

## Publishing model checkpoints to HuggingFace (any agent session)

Any checkpoint published to the robocurve HF org **must** follow
[docs/model-cards.md](docs/model-cards.md): fill in
[docs/model-card-template.md](docs/model-card-template.md) top-to-bottom, then pass the
gating checklist (any unchecked REQUIRED box blocks publish). Common traps it prevents:
license inheritance from the base model (never default apache-2.0), missing action/observation
I/O contracts, and unverifiable eval claims. Reference implementations are linked in the doc.

## Adding a benchmark to the catalog

Append a `Benchmark(...)` entry to `src/worldevals/catalog.py` with its name,
Expand Down
8 changes: 8 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -85,6 +85,14 @@ validates every entry (unique name, well-formed repo URL, ≥1 task key). See
[KitchenBench](https://github.com/robocurve/kitchenbench) as the reference
implementation.

## Publishing model checkpoints

Benchmarks compare *policies* — and a benchmark number is only interpretable when the
checkpoint behind it is documented. Every model checkpoint published to the
[robocurve HuggingFace org](https://huggingface.co/robocurve) must follow the
**[model-card standard](docs/model-cards.md)** (start from the
[template](docs/model-card-template.md); the standard includes a gating checklist).

## Development

> **Dependency changes:** after editing dependencies in `pyproject.toml`, run
Expand Down
5 changes: 5 additions & 0 deletions docs/model-card-template.md
Original file line number Diff line number Diff line change
Expand Up @@ -32,6 +32,11 @@ inference: false

# <Model name — robot + method in plain words>

> **Evaluate this checkpoint** with [Inspect Robots](https://github.com/robocurve/inspect-robots) —
> the Inspect-AI-style eval framework for physical AI — using the
> [<embodiment> adapters](https://github.com/robocurve/inspect-robots-<embodiment>); benchmarks
> are catalogued in [WorldEvals](https://github.com/robocurve/worldevals).

<≤3 sentences: [base model](link) + method (LoRA/full FT; what's frozen) + robot +
[dataset](link). Adapter/merge status and where raw adapters live.>

Expand Down
5 changes: 5 additions & 0 deletions docs/model-cards.md
Original file line number Diff line number Diff line change
Expand Up @@ -93,6 +93,10 @@ model-index:

At most three sentences: base model + method (LoRA/full FT; what's frozen) + robot/embodiment
+ dataset. State whether adapters are merged and where raw adapters live if published.
**Immediately after the H1**, add an eval-path blockquote linking
[Inspect Robots](https://github.com/robocurve/inspect-robots), the embodiment's adapter repo
(e.g. inspect-robots-so101 / inspect-robots-yam), and
[WorldEvals](https://github.com/robocurve/worldevals) — see the template.
Link every **named artifact**: base model, each dataset, training-code repo, method libraries,
papers. (Ordinary nouns like "camera" need no link.)

Expand Down Expand Up @@ -201,6 +205,7 @@ Frontmatter

Body
- [ ] Header ≤3 sentences; every named artifact linked; adapter/merge status stated
- [ ] Eval-path blockquote right after the H1 (Inspect Robots + embodiment adapters + WorldEvals)
- [ ] Intended use + out-of-scope + validation status (with e-stop/supervision language)
- [ ] Training table: data counts + filter/split rule (level, %, seed)
- [ ] Training table: embodiment config + state/action key layout
Expand Down
Loading