From 0ecb289f81a5d239a4fa24909b853450381e5094 Mon Sep 17 00:00:00 2001 From: Jay Chooi Date: Sat, 18 Jul 2026 18:21:29 -0700 Subject: [PATCH 1/5] docs(readme): add the Inspect Robots logo above the title Copies the robot-inspecting-a-dot product art from robocurve/website into docs/assets/, adding a cream rounded-tile background so the cream-filled head circle reads on GitHub dark mode and PyPI. The README references it by absolute raw.githubusercontent URL because README.md is also the PyPI long description, where relative paths 404. Co-Authored-By: Claude Fable 5 --- README.md | 2 ++ docs/assets/inspect-robots-logo.svg | 43 +++++++++++++++++++++++++++++ 2 files changed, 45 insertions(+) create mode 100644 docs/assets/inspect-robots-logo.svg diff --git a/README.md b/README.md index 21cae33..a2418d3 100644 --- a/README.md +++ b/README.md @@ -1,5 +1,7 @@
+Inspect Robots logo — a line-art robot inspecting a dot through a magnifying lens + # Inspect Robots ### An open-source evaluation framework for physical AI and VLA (vision-language-action) models diff --git a/docs/assets/inspect-robots-logo.svg b/docs/assets/inspect-robots-logo.svg new file mode 100644 index 0000000..e946589 --- /dev/null +++ b/docs/assets/inspect-robots-logo.svg @@ -0,0 +1,43 @@ + + + + + + + + + + + + + + + + + + + + + From 24221efad69a219e9a19db5739373168032cc28d Mon Sep 17 00:00:00 2001 From: Jay Chooi Date: Sat, 18 Jul 2026 18:30:21 -0700 Subject: [PATCH 2/5] docs(readme): drop the two-swappable-inputs section, salvage fail-fast check MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The two-inputs concept already appears in the tagline, the Inspect AI mapping table, and the Python example, and the docs landing page leads with the same section — so Install moves up under the fold. The one detail stated nowhere else, the pre-rollout (policy, embodiment) compatibility check, moves to a Why Inspect Robots bullet. Co-Authored-By: Claude Fable 5 --- README.md | 18 +++--------------- 1 file changed, 3 insertions(+), 15 deletions(-) diff --git a/README.md b/README.md index a2418d3..69446a0 100644 --- a/README.md +++ b/README.md @@ -33,21 +33,6 @@ If you know [Inspect AI](https://inspect.aisi.org.uk/), this is that for robotic --- -## One framework, two swappable inputs - -LLM evaluations have a single swappable input: the model. Robotics evaluations -have two, and Inspect Robots makes both first-class and orthogonal: - -| | | -|---|---| -| **`Policy`**: the VLA | The "brain". Maps an observation + instruction to an action chunk (a horizon of actions executed open-loop, as π0 / ACT / diffusion policies do). | -| **`Embodiment`**: the robot or sim | The "body + world". Produces observations, executes actions, owns the action/observation spaces and control rate. Real-robot-first; sims are a stricter special case. | - -A **`Task`**, a dataset of `Scene`s (initial conditions, instructions, success -targets) plus scorers, is defined *independently* of both. Before any rollout, -Inspect Robots checks the `(policy, embodiment)` pair is compatible (action/observation -spaces, semantics, control rate, scene realizability) and fails fast if not. - ## Install In a fresh directory (or your existing project), create a virtual environment @@ -249,6 +234,9 @@ print(log.status, log.results.metrics) # success {'success_at_end': 1.0} - **Real-world first.** Interfaces assume real-robot reality: human-in-the-loop reset, no privileged success oracle, wall-clock control rate. Simulators just offer more (seeding, privileged success, rendering) via opt-in capabilities. +- **Compatibility checked up front.** Before any rollout, the + `(policy, embodiment)` pair is validated — action/observation spaces, + semantics, control rate, scene realizability — and fails fast if not. - **Reproducible.** Every run yields an immutable, schema-versioned `EvalLog` with the resolved config, git revision, and package versions. It is re-readable across releases and re-scorable offline. From 71f37e608cfc1bd8d473ada6b3fa888bd4fab041 Mon Sep 17 00:00:00 2001 From: Jay Chooi Date: Sat, 18 Jul 2026 18:34:28 -0700 Subject: [PATCH 3/5] docs(readme): trim over-explaining asides Cuts detail a reader acting on the section doesn't need: the PEP 668 parenthetical, the uv-run re-sync play-by-play, the Rerun viewer memory cap and upgrade note, the .env precedence rules, and the CaP-X adapter spec-sheet (now deferred to the plugin README). Co-Authored-By: Claude Fable 5 --- README.md | 42 +++++++++++++++--------------------------- 1 file changed, 15 insertions(+), 27 deletions(-) diff --git a/README.md b/README.md index 69446a0..3f32c83 100644 --- a/README.md +++ b/README.md @@ -36,8 +36,7 @@ If you know [Inspect AI](https://inspect.aisi.org.uk/), this is that for robotic ## Install In a fresh directory (or your existing project), create a virtual environment -and install (system Pythons on modern distros reject bare `pip install`, -per PEP 668): +and install: ```bash uv venv && uv pip install "inspect-robots[rerun]" @@ -49,19 +48,15 @@ The `rerun` extra powers the live run viewer. For the numpy-only core: uv venv && uv pip install inspect-robots ``` -Any venv workflow works the same way (`python3 -m venv .venv` and that venv's -`pip` and console scripts). Either way, activate the venv once -(`source .venv/bin/activate`; `.venv\Scripts\activate` on Windows) and call -`inspect-robots` directly, as shown below. +Any venv workflow works. Activate it once (`source .venv/bin/activate`; +`.venv\Scripts\activate` on Windows) and call `inspect-robots` directly, +as shown below. > [!NOTE] -> Invoke the CLI as plain `inspect-robots`, not `uv run inspect-robots`. -> Inside a uv project, `uv run` first re-syncs the environment to the -> project's lockfile, downgrading whatever the `uv pip install` commands -> above just added back to the locked versions; the only trace is an -> easy-to-miss "Uninstalled N / Installed N packages" line. To use -> `uv run` anyway, pass `--no-sync`, or declare everything as real -> dependencies with `uv add inspect-robots` plus your plugins. +> Invoke the CLI as plain `inspect-robots`, not `uv run inspect-robots` — +> inside a uv project, `uv run` re-syncs to the lockfile and silently +> uninstalls what `uv pip install` just added. To use `uv run` anyway, +> pass `--no-sync`, or declare the packages with `uv add`. ## Quickstart @@ -102,9 +97,7 @@ inspect-robots "place the fork on the plate" Every run opens a live Rerun viewer streaming the cameras, proprioception, and actions straight from the eval pipeline, so you watch exactly what the -policy sees while the robot moves. The viewer starts with a 2 GiB memory cap -so long sessions stay responsive; after upgrading, kill any already-running -Rerun viewer once so the cap applies. CLI flags override any default +policy sees while the robot moves. CLI flags override any default (`--no-rerun`, `--no-store-frames`, `--max-steps 300`, ...). ### Drive the robot with an LLM @@ -114,10 +107,8 @@ The policy slot is not limited to VLAs. With the drives the same rig through tool calls, one approver-checked motion chunk per call. -Put a `.env` with your API key in the working directory, reusing one you -already have or copying the [.env.example](.env.example) template (the CLI -loads it automatically; real environment variables take precedence over its -values): +Put a `.env` with your API key in the working directory (the CLI loads it +automatically; [.env.example](.env.example) is a template): ```ini ANTHROPIC_API_KEY=sk-ant-... @@ -141,13 +132,12 @@ Read the recorded agent conversation with `inspect-robots view LOG.json`, including the camera frames the model saw (for `--store-frames` runs). -### Generate robot policy code with CaP-X: +### Generate robot policy code with CaP-X The [inspect-robots-capx](plugins/inspect-robots-capx/) plugin evaluates a code-as-policy agent in the same policy slot. The LLM writes Python against SAM3 segmentation, Contact-GraspNet planning, Pyroki IK, and speed-limited -joint-motion helpers. CaP-X model servers run separately, while the persistent -code namespace and action queue run inside the evaluator. +joint-motion helpers. ```bash uv pip install inspect-robots-capx @@ -157,10 +147,8 @@ inspect-robots "place the fork on the plate" --policy capx \ -P model=anthropic/claude-fable-5 -P sam3_url=http://gpu-box:8114 ``` -The v1 adapter requires one `joint_pos` arm with a declared gripper, full -joint-state proprioception, a control rate, and a camera. See the -[plugin README](plugins/inspect-robots-capx/) for model-server bringup, depth -metadata, gripper polarity, and the model-code trust boundary. +See the [plugin README](plugins/inspect-robots-capx/) for embodiment +requirements, model-server bringup, and the model-code trust boundary. ### Run in simulation From 125d9193d0456d34a08bcfb36b4ed21022a50beb Mon Sep 17 00:00:00 2001 From: Jay Chooi Date: Sat, 18 Jul 2026 18:55:46 -0700 Subject: [PATCH 4/5] docs(readme): retitle and tighten copy Subtitle becomes 'evaluating AI and robots in the physical world'; the tagline's italicized any-s go roman. Cuts the uv-run note's remedy sentence, the wizard parentheticals, and restructures the buried --store-frames conditional into the sentence. Co-Authored-By: Claude Fable 5 --- README.md | 17 +++++++---------- 1 file changed, 7 insertions(+), 10 deletions(-) diff --git a/README.md b/README.md index 3f32c83..27b04a8 100644 --- a/README.md +++ b/README.md @@ -4,9 +4,9 @@ # Inspect Robots -### An open-source evaluation framework for physical AI and VLA (vision-language-action) models +### An open-source evaluation framework for evaluating AI and robots in the physical world -Define a robotics benchmark once, then run *any* policy against *any* compatible +Define a robotics benchmark once, then run any policy against any compatible embodiment (a real robot or a simulator) with reproducible logs and first-class [Rerun](https://github.com/rerun-io/rerun) visualization. @@ -55,13 +55,11 @@ as shown below. > [!NOTE] > Invoke the CLI as plain `inspect-robots`, not `uv run inspect-robots` — > inside a uv project, `uv run` re-syncs to the lockfile and silently -> uninstalls what `uv pip install` just added. To use `uv run` anyway, -> pass `--no-sync`, or declare the packages with `uv add`. +> uninstalls what `uv pip install` just added. ## Quickstart -Install the plugin for your rig (the wizard suggests this one's components) -and set your defaults once: +Install the plugin for your rig and set your defaults once: ```bash source .venv/bin/activate @@ -69,8 +67,7 @@ uv pip install inspect-robots-yam # provides the molmoact2 policy + yam_arms r inspect-robots setup ``` -The wizard picks your defaults and finds your cameras (unplug one when asked -and it identifies which is which), then writes +The wizard picks your defaults and finds your cameras, then writes `~/.config/inspect-robots/config.ini`. On a different rig, install its plugin instead and type its component names at the prompts; to write the config file by hand, see [the CLI guide](https://inspectrobots.org/guide/cli/). @@ -129,8 +126,8 @@ inspect-robots "place the fork on the plate" --policy agent \ Read the recorded agent conversation with `inspect-robots inspect LOG.json --transcript`, or open the HTML report with -`inspect-robots view LOG.json`, including the camera frames the model saw (for -`--store-frames` runs). +`inspect-robots view LOG.json` — for `--store-frames` runs it includes the +camera frames the model saw. ### Generate robot policy code with CaP-X From e32ca863071530172d5f466a7d521c9a08cf6413 Mon Sep 17 00:00:00 2001 From: Jay Chooi Date: Sat, 18 Jul 2026 18:57:41 -0700 Subject: [PATCH 5/5] docs(readme): benchmarking in subtitle; real plugin names in Pluggable bullet inspect-robots-maniskill and inspect-robots-openvla were aspirational examples that don't exist; point at inspect-robots-yam and the first-party plugins instead. Subtitle 'evaluating' -> 'benchmarking' to stop echoing 'evaluation framework'. Co-Authored-By: Claude Fable 5 --- README.md | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/README.md b/README.md index 27b04a8..566bbd0 100644 --- a/README.md +++ b/README.md @@ -4,7 +4,7 @@ # Inspect Robots -### An open-source evaluation framework for evaluating AI and robots in the physical world +### An open-source evaluation framework for benchmarking AI and robots in the physical world Define a robotics benchmark once, then run any policy against any compatible embodiment (a real robot or a simulator) with reproducible logs and first-class @@ -234,8 +234,9 @@ print(log.status, log.results.metrics) # success {'success_at_end': 1.0} a slow viewer connection drops camera frames first (whole steps only under sustained stall) instead of delaying the robot control loop, and camera streams are JPEG-compressed by default. -- **Pluggable.** Ship `inspect-robots-maniskill` or `inspect-robots-openvla` as separate - packages. Entry points make them appear in `inspect-robots list` automatically. +- **Pluggable.** Backends ship as separate packages — the first-party plugins + below, and rig plugins like `inspect-robots-yam`. Entry points make them + appear in `inspect-robots list` automatically. - **VLA-native.** Action chunking, open-loop execution, and ACT/ALOHA temporal ensembling are built in, with action *semantics* (control mode, rotation representation, gripper, frame) that make compatibility and ensembling correct.