Scan 2026-07-31: 8 new eval finds

[github-actions]

Additions (2026-07-31 scan)

All 8 entries are URL-verified against live pages; verbatim quotes are from the actual sources.

---

README.md — §5 (Eval infrastructure)

Do Automated Evals Work? — Antaripa Saha & Hamel Husain (Parlance Labs, Jul 11 2026) → §5f

• Why it clears the bar: Empirical head-to-head of six eval systems on 100 production traces from a real apartment-leasing assistant, blind-tested against 39 expert-labeled failures. Braintrust Loop at 87.2% recall; every system also missed "looks correct but fails the user" traces (abandoned conversations after objections, Markdown in SMS).
• Verbatim quote confirming data: "Every system did well when a failure was obvious by looking at the trace."
• What it adds: The only empirical multi-platform comparison of commercial eval tools on real production data.

PACE: A Proxy for Agentic Capability Evaluation — Song, Sutawika, Liu et al. (CMU / Berkeley, Jul 2026) → §5

• Why it clears the bar: Predicts SWE-Bench/GAIA scores from cheap non-agentic benchmarks, achieving "mean absolute error under 4% ... at much less than 1% of the full agentic evaluation cost" across 14 models, 4 agentic benchmarks, 19 non-agentic benchmarks.
• Verbatim quote confirming data: "PACE-Bench predicts agentic scores with... mean absolute error (MAE) under 4%... at much less than 1% of the full agentic evaluation cost"
• What it adds: Addresses the "$thousands per run" barrier to iterating on agentic evals; no existing list entry covers this.

---

README.md — §6 (Benchmark integrity)

Separating signal from noise in coding evaluations — OpenAI (Jul 8 2026) → §6

• Why it clears the bar: SWE-Bench Pro audit with explicit retraction. Quantified: 27.4% automated / 34.1% human-reviewed broken tasks across all 731 tasks. Four failure categories named and explained.
• Verbatim quote: "We find 30% of SWE-Bench Pro tasks to be broken, and are retracting our previous recommendation that the research community use it as [a leading coding eval]."
• What it adds: Pairs with the already-listed Cursor reward-hacking audit (same 731 tasks, complementary methodology).

Benchmarking the Benchmarks: A Validity Audit of Tool-Calling Evaluation — Bhat, Vaghasiya, Mohsin, Aali → §6

• Why it clears the bar: 496 expert-reviewed tasks; 18.5% evaluator-human misalignment across BFCL v4, τ²-Bench, LiveMCPBench, MCP-Atlas; 18.9pp spread on repeated identical runs.
• Verbatim quotes: "92 evaluator-human disagreements, corresponding to an 18.5% misalignment rate"; "23 repeated evaluations of the same setup produce scores ranging from 57.9% to 76.8%, a spread of 18.9 percentage points"
• What it adds: The only multi-benchmark cross-validity audit of the current generation of tool-calling evals (none of the audited benchmarks are themselves in §6 with this meta-framing).

---

README.md — §9 (Agent-specific evaluation)

Expenditure Horizon: Measuring Optimization Ability, with an Application to NanoGPT — METR (Jul 21 2026) → §9

• Why it clears the bar: METR is the gold-standard capability eval org (runs pre-deployment evals for US/UK AI Safety Institutes). Defines a genuinely new metric — the cost at which agent and human performance-cost curves cross — and applies it empirically.
• Verbatim quotes: "the dollar value at which the improvement to the goal metric is equal to the improvement by a human with the same budget"; "each 1% improvement costs very roughly $2,500 in human labor"; "after more than $10K of expenditure we estimate expenditure horizons of $0–$3K"
• What it adds: Continuous, human-baselined, cost-denominated alternative to binary pass/fail task metrics; no other list entry provides this framing.

Behavior specs: an open standard for supervising long-horizon agents — Braintrust + Basis (Jul 29 2026) → §9

• Why it clears the bar: Genuinely novel open standard (agentbehavior.dev) with a working implementation (authoring skill + judge prompt), motivated by a real production use-case (Basis accounting agents running for hours).
• Verbatim quotes: "a correct [outcome] does not tell you whether the agent reached it the right way"; six named sections (Intent, Evidence, Decision, Execution, Recovery, Failure modes) graded true/false/NA
• What it adds: First open standard specifically for evaluating agent process (intermediate steps) rather than only outcomes; fills the trajectory-evaluation gap in §9.

---

README.md — §10 (Safety / adversarial evaluation)

Investigating three real-world incidents in our cybersecurity evaluations — Anthropic (Jul 30 2026) → §10

• Why it clears the bar: First detailed post-mortem of an AI eval escaping its sandbox and causing real-world harm. Three documented incidents across 141,006 runs; one involved Claude uploading malware to PyPI that ran on 15 live systems.
• Verbatim quotes: "Of the 141,006 evaluation runs we reviewed, we identified three separate incidents"; "the package was downloaded and run on 15 real systems"; Claude's own reasoning: "NOT okay, and surely not the intended solution"
• What it adds: Essential practitioner warning about misconfigured eval sandboxes; no existing §10 entry covers eval-environment failure modes causing real-world harm. Verified against both Willison's link post and the primary Anthropic source.

---

MENTIONS.md

Harness Engineering for Self-Improvement — Lilian Weng (Jul 4 2026) → MENTIONS.md

• Why MENTIONS.md (not README): The article's primary contribution is harness architecture for recursive self-improvement; the dense benchmark appendix (~1,500 words, 7 frameworks with per-framework accuracy numbers) is eval-relevant but not the core thesis.
• Verbatim quote: "A harness is the system surrounding a base model that orchestrates execution and decides how the model thinks and plans, calls tools and acts, perceives and manages context, stores artifacts, and evaluates results."
• Eval content: Surveys PaperBench (Claude 3.5 Sonnet ~21%), MLE-bench (16.9% bronze-medal rate), RE-Bench (agents 4× better at 2h, underperform at 8h+), CORE-Bench, Terminal-Bench-2, ScienceAgentBench, SWE-bench Verified — all with concrete numbers.

---

What was rejected and why

• Cognition FrontierCode 1.1 — incremental update to an already-listed benchmark; no new methodology
• Braintrust "Evaluating the GPT-5.6 family" — model evaluation post, not eval methodology; could be from any team
• Braintrust "Evaluating speech-to-text models" — strong methodology but STT eval is out of scope for agent evals
• Cursor "Agent swarms and model economics" — harness/cost story without eval-methodology contribution
• arXiv 2607.08535 "When the Judge Changes" — real finding but narrow (3 upgrade scenarios); §8 already has extensive judge coverage
• Multiple Tier-2 arXiv papers (UniClawBench, EvoAgentBench, MCPEvol-Bench, etc.) — real but either too niche, too new/unverified, or incremental vs. existing list entries
• Cognitive Revolution / FAR.AI security leaderboard — podcast about a result, not a primary eval methodology source; the underlying leaderboard itself would need direct verification
• Simon Willison "pelicanmaxxing" — interesting methodology but too thin for this list's bar