Run inspect_swe coding agents in asta-bench

[jbragg]

Summary

Adds an Inspect solver wrapping the inspect_swe coding agents (claude_code, codex_cli, gemini_cli, mini_swe_agent) so any of them can run any asta-bench task. The agent runs in the sample sandbox; model API calls and task tools are proxied back to Inspect via sandbox_agent_bridge so token usage / transcript / model aliases / MCP tools flow through the standard Inspect path.

Coding agents see two surfaces for asta:

• MCP (default) — the task's MCP tools (snippet_search, get_paper, table_editor, python_session, …) reach the agent as mcp__astabench_*.
• Skills + CLI (-S install_asta_skills=asta|asta-preview) — a more native surface for coding agents: setup.sh clones asta-plugins at a tag matching the asta image into .vendor/, the solver passes the bundled skill dirs through inspect_swe's standard skills= plumbing (each agent's wrapper handles its own discovery path), and paper-search MCP tools that have a 1:1 asta papers CLI subcommand are filtered out of the bridge so the agent has one canonical path. Non-paper MCP tools and MCP tools without a CLI equivalent (get_paper_batch, search_paper_by_title) always bridge. (Needs Bring bundled skill frontmatter into Agent Skills spec compliance asta-plugins#56 — fix/skill-frontmatter-spec-compliance — shipped in a tagged release so the bundled SKILL.md files pass inspect_ai.tool.read_skills validation.)

The solver reads state.metadata["insertion_date"] (populated by allenai/asta-bench#151 — expose-insertion-date-metadata) and seeds ASTA_INSERTED_BEFORE / ASTA_PUBLICATION_DATE_RANGE on the agent's env per sample, so the asta CLI subprocess honors the same cutoff as astabench's MCP wrapper. (ASTA_INSERTED_BEFORE on asta papers snippet-search reads via allenai/asta-plugins#57 — feat/date-broker; ASTA_PUBLICATION_DATE_RANGE reads on asta papers get / search / citations already work against v0.16.0+.)

An inspect_ai GenerateFilter plumbed through the bridge strips model-provider-side web tools (WebSearch/WebFetch and provider variants) from every model API request — they execute on the provider's infrastructure and aren't reachable from inside the sandbox. Tasks can still wire astabench's cutoff-aware native search (make_native_search_tools(inserted_before=...), already used by litqa2 / sqa / paper_finder) into state.tools; those bridge as mcp__astabench_* and survive the filter. Agent subprocess access to external hosts (curl, wget, Python requests) is unrestricted at the sandbox layer in this PR.

Validation

One LitQA2_FullText_validation sample (claude_code + sonnet-4-6) run for each of the three install_asta_skills settings, e.g.:

inspect eval astabench/litqa2_validation \
  --solver agent_baselines/solvers/inspect_swe/agent.py@inspect_swe_solver \
  --model anthropic/claude-sonnet-4-6 \
  --sandbox docker:solvers/inspect-swe/sandbox_compose.yaml \
  -S agent=claude_code -S install_asta_skills=asta \
  --limit 1

All three runs exit with status: success. With the asta-bench prereq dev-installed locally, the eval log also shows Sample.metadata.insertion_date = "2024-10-17" reaching the solver. inspect view shows the surfaces each run actually used. When install_asta_skills is set, inspect_swe installs the bundled skills by writing each SKILL.md into the agent's skill-discovery path inside the sandbox — those appear as write_file events in the log:

• install_asta_skills=None (default, MCP-only path) — no write_file events to /app/.claude/skills/; the agent issued 19 mcp__astabench__* calls (mix of search_papers_by_relevance, snippet_search, get_paper) and received real paper data (paperId, corpusId, title, abstract, …) back. 0 Object of type ContentText is not JSON serializable errors — the bridged-MCP serialization fix in inspect_ai 0.3.220 (this PR pins inspect_ai>=0.3.220 for that reason) makes the MCP path usable; older versions fail every bridged MCP call.
• install_asta_skills="asta" — write_file events drop the bundled asta plugin's SKILL.md files into /app/.claude/skills/<skill>/; the agent invokes Skill(skill="semantic-scholar", …) and then runs Bash asta papers snippet-search "…" --limit 10 (×N, refined queries) and asta papers get CorpusId:58004838 --fields title,abstract,year --format text.
• install_asta_skills="asta-preview" — write_file events drop the asta-preview plugin's larger skill set into /app/.claude/skills/<skill>/ (analyze-data, artifacts, local-paper-index, pdf-extraction, preview, research-step, semantic-scholar, workspace, …); the agent again chose Skill(skill="semantic-scholar", …) and then ran many asta papers snippet-search/search/get Bash calls.

Merge order

The three referenced PRs are independent on GitHub but ordered by what each one ships through:

1. Merge Bring bundled skill frontmatter into Agent Skills spec compliance asta-plugins#56 (fix/skill-frontmatter-spec-compliance) and Add date-cutoff env vars to papers subcommands asta-plugins#57 (feat/date-broker) — parallel; both target asta-plugins:main.
2. Cut a new asta-plugins tag containing both, and build/publish a new ghcr.io/allenai/asta image pinning that tag.
3. Merge Set insertion_date Sample metadata on cutoff-bearing tasks asta-bench#151 (expose-insertion-date-metadata) — independent; can land any time, only gates whether per-sample cutoff metadata is populated.
4. Bump ASTA_IMAGE default in this PR's solvers/inspect-swe/README.md + sandbox_compose*.yaml to the new tag and merge. setup.sh clones asta-plugins at ${ASTA_PLUGINS_REF:-$ASTA_IMAGE_TAG}, so the same bump aligns the host-side skill source.

[gitguardian]

️✅ There are no secrets present in this pull request anymore.

If these secrets were true positive and are still valid, we highly recommend you to revoke them.
While these secrets were previously flagged, we no longer have a reference to the
specific commits where they were detected. Once a secret has been leaked into a git
repository, you should consider it compromised, even if it was deleted immediately.
Find here more information about risks.

---

^{_{🦉 GitGuardian detects secrets in your source code to help developers and security teams secure the modern development process. You are seeing this because you or someone else with access to this repository has authorized GitGuardian to scan your pull request.}}

[jbragg]

GitGuardian flag was a local private key I accidentally staged. Force-pushed to remove and rotated the key.

[rodneykinney]

LGTM

[mdarcy220]

fwiw I've seen this a while back although I don't recall it OOMing at the time. The fundamental issue is that astabench eval uses subprocess.run which SIGKILLs the child on interrupt. Could be worth fixing durably on the agent-eval side if we're at a point where we're writing troubleshooting documentation for it.

[jbragg]

Replaces the host-side git clone asta-plugins in setup.sh with extracting the skill trees directly from the asta image (docker run --rm <img> tar from /opt/asta-plugins, which the asta-plugins Dockerfile already populates via COPY . /opt/asta-plugins). One ref — ASTA_IMAGE — now controls both the CLI runtime in the sandbox and the host-loaded skills the solver passes via install_asta_skills=. No more ASTA_PLUGINS_REF knob to keep in sync, and any image form works (semver tag, :latest, digest).

The original clone setup couldn't keep ASTA_PLUGINS_REF aligned with ASTA_IMAGE for :latest (no matching git ref) or digest pins (the tail isn't a git ref), and even when both refs named the same release, nothing structurally prevented the host clone from drifting from what the image actually has baked in.

A second drift window — operator changes ASTA_IMAGE between setup and eval without re-running setup.sh — is closed by stamping the resolved image ID into .vendor/asta-plugins/.image-id and comparing it against the live ASTA_IMAGE's docker ID at solver construction. Mismatch raises with a "re-run setup.sh" message. This generalizes the existing _check_plugin_image_version_match preflight (semver tags only) to all image forms.

Tests: +5 covering no-stamp / no-env / no-docker / matching-IDs / mismatch. Suite is 19 baseline → 24.