Inspect Robots Analysis

Auditable p-task-quantile (pTQ) and automation-halving analysis for Inspect Robots evaluation logs.

The package turns binary, per-epoch outcomes into the pooled logistic curves specified by RoboCurve's physical-automation methodology draft. It refuses silent comparisons when benchmark versions, protocols, task sets, instance sets, or time horizons do not align.

What it provides

• strict JSON study manifests with explicit benchmark and protocol provenance;
• validation of EvalLog status, schema, policy/embodiment identity, binary outcomes, reductions, trial counts, and horizon metadata;
• mandatory comparability gates before any fit;
• pair-specific intercepts with one shared job-level logistic slope;
• exact task quantiles plus a disclosed q=0 log-boundary rule;
• pTQ and automation-halving estimates at requested success levels;
• paired hierarchical bootstrap intervals over tasks, instances, and rollouts;
• automation-halving OLS trends and optional target-date projections;
• deterministic strict JSON and self-contained HTML reports.
• SHA-256 provenance for the manifest and every exact input log byte stream.

The only runtime dependency is NumPy. The package reads EvalLog JSON directly; it does not require Inspect Robots to be installed.

Quickstart

uv sync --extra dev
uv run python examples/make_synthetic_study.py /tmp/ira-demo
uv run inspect-robots-analysis validate /tmp/ira-demo/manifest.json
uv run inspect-robots-analysis analyze /tmp/ira-demo/manifest.json \
  --output-dir /tmp/ira-demo/results

Open /tmp/ira-demo/results/report.html. Every generated value is clearly identified as synthetic; the example is a pipeline check, not evidence about real systems.

For real data, replace the generated log paths with immutable logs from inspect-robots eval-set. Keep one log per (pair_id, task_id).

Manifest

{
  "schema_version": 1,
  "study_id": "kitchenbench-yam-2026q3",
  "title": "KitchenBench on YAM",
  "success_metric": "task_success",
  "p_levels": [0.5],
  "forecast_quantiles": [0.5, 0.9],
  "bootstrap": {"replicates": 1000, "confidence": 0.95, "seed": 20260721},
  "runs": [
    {
      "log": "logs/molmoact2-yam-pour.json",
      "job_id": "food-preparation",
      "task_id": "kitchenbench/pour_pasta",
      "pair_id": "molmoact2@yam-v1",
      "model_id": "molmoact2",
      "model_release_date": "2026-05-01",
      "embodiment_id": "yam-v1",
      "benchmark_id": "kitchenbench",
      "benchmark_version": "0.4.0",
      "protocol_id": "physical-automation-draft-2026-05-12",
      "policy_name": "molmoact2",
      "embodiment_name": "yam"
    }
  ]
}

Paths are relative to the manifest. task_id, policy_name, and embodiment_name must exactly match the EvalLog. Human-readable aliases belong in pair_id, model_id, and embodiment_id.

Comparability policy

Analysis is blocked when a job mixes protocol fingerprints, pair identity changes between tasks, any pair lacks a task or instance, or task horizons are inconsistent. Unequal rollout counts and mixed Inspect Robots versions are reported as warnings; grouped-binomial likelihood weights unequal counts correctly, but the underlying precision and software provenance still differ.

Seconds-based tasks are compared by declared max_seconds; their resolved max_steps may differ with embodiment control rate. Step-based tasks must share one max_steps value.

Method and limitations

See DESIGN.md for the equations, bootstrap hierarchy, boundary choice, and failure conditions. The most important caveat is methodological: the source draft defines the easiest task quantile as zero and then applies log2(q). This implementation does not conceal that inconsistency. It reports the exact quantile and uses 0.5/n only as the finite regression coordinate.

Projected crossing dates come from an OLS fit in automation-halving space. They are useful as a reproducible scenario summary, not a causal claim.

Development

uv run ruff format --check .
uv run ruff check .
uv run mypy
uv run pytest --cov --cov-branch