From c07599bd1f3e119e1a54cd9b763d51772bc02593 Mon Sep 17 00:00:00 2001
From: macanderson <mac@oxagen.sh>
Date: Sat, 18 Jul 2026 19:34:53 -0700
Subject: [PATCH 1/4] fix: verification integrity, metric honesty, and CLI
 hardening

Highest-severity first:
- Timeouts kill the agent's whole process tree (POSIX process groups);
  trials can no longer hang on pipes held by orphaned grandchildren,
  and survivors can no longer tamper with held-out verification.
- Zero-token (unparseable-envelope) trials are excluded from token/cost
  medians and deltas instead of dragging them toward zero past the gate.
- claude-code preserves full model ids (no floating "sonnet" alias), so
  version pinning, matchedModels, and pricing lookups hold.
- Gemini normalization counts thoughts (reasoning) and tool tokens.
- Pretty-printed JSON envelopes parse (TS + Harbor Python mirror).
- Per-trial error containment + incremental results.json; wall clock
  measures spawn-to-exit only; git-diff failure is "unknown", never
  "empty"; report table shares perAgentSummary with the gate (no NaN).
- CLI: -o short flag (as documented), strict numeric flags, unknown
  verify ids error, duplicate agent specs rejected, typo'd gate
  thresholds error instead of silently disabling the check.
- Cost with unpriced cache writes is null, never guessed.
- Harbor: float-tolerant ARENA_TIMEOUT (never silently unbounded),
  loud empty-{budget} failure, metrics.kind validated at load.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---
 CHANGELOG.md                   | 40 +++++++++++++++++
 harbor/arena_harbor/base.py    | 14 +++++-
 harbor/arena_harbor/command.py |  7 +++
 harbor/arena_harbor/metrics.py | 77 +++++++++++++++++++++++++++++++--
 harbor/arena_harbor/spec.py    |  7 ++-
 harbor/pyproject.toml          |  3 +-
 src/adapters/base.ts           | 57 +++++++++++++++++++++---
 src/adapters/claude-code.ts    | 12 ++++--
 src/adapters/gemini.ts         | 16 +++++--
 src/cli.ts                     | 79 ++++++++++++++++++++++++++++------
 src/orchestrator.ts            | 74 ++++++++++++++++++++++++++++---
 src/parse.ts                   | 73 +++++++++++++++++++++++++++++--
 src/pricing.ts                 |  7 ++-
 src/report.ts                  | 37 ++++++++++------
 src/summary.ts                 | 13 +++++-
 src/workspace.ts               | 17 ++++++--
 test/adapters.test.ts          | 26 +++++++----
 test/parse.test.ts             | 33 ++++++++++++++
 18 files changed, 516 insertions(+), 76 deletions(-)

diff --git a/CHANGELOG.md b/CHANGELOG.md
index 81d2219..c80e17f 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -11,6 +11,46 @@ minor bumps may include breaking changes).
 - Open-source launch artifacts: `CONTRIBUTING.md`, `CODE_OF_CONDUCT.md`, issue
   and PR templates, and this changelog.
 - `.stella/` agent caches are now gitignored.
+- **Site** (`site/`) — arena.oxagen.sh: landing page, the research paper
+  *The State of Agent Benchmarking*, and the draft **Agent Benchmark Protocol**
+  spec (`docs/agent-benchmark-protocol.md`,
+  `docs/agent-engine-benchmarks-2026.md`).
+- Pretty-printed (multi-line) JSON envelopes are now parsed — both in the TS
+  harness (`parseJsonEnvelope`) and the Harbor adapter (`last_json_object`).
+- `arena run -o <dir>` short flag (the form the README documents).
+
+### Fixed
+- **Timeouts kill the agent's whole process tree** (POSIX process groups), and
+  a trial can no longer hang on stdio pipes held open by orphaned
+  grandchildren — which could previously also outlive the agent and tamper
+  with verification.
+- **Zero-token trials no longer poison medians or the gate**: scored trials
+  whose envelope had no parseable usage are excluded from token/cost medians
+  and deltas (reported separately), instead of dragging them toward zero.
+- **claude-code model pinning**: `anthropic/claude-sonnet-5` now resolves to
+  `claude-sonnet-5` instead of the floating `sonnet` alias, preserving version
+  pinning, `matchedModels`, and pricing lookups.
+- **Gemini token normalization** counts `thoughts` (reasoning) and `tool`
+  tokens; both were previously dropped, understating Gemini usage.
+- Per-trial error containment: a harness-side failure (git/FS error) scores
+  that one trial `agent-error` instead of aborting the run; `results.json` is
+  rewritten after every trial so a crash never loses completed trials.
+- Wall clock now measures spawn-to-exit only (workspace seeding excluded),
+  matching METHODOLOGY.md.
+- `arena verify <unknown-id>` errors instead of vacuously passing; numeric
+  CLI flags are validated (`--timeout 10m` errors instead of parsing as 10);
+  a typo'd gate threshold errors instead of silently disabling the check;
+  duplicate `--agents` specs are rejected (their trial ids would collide).
+- `git diff` failures are now distinguished from an empty diff, so a trial
+  with real changes can no longer be misclassified `agent-error`.
+- Cache-write tokens with no `cacheWritePerM` price now yield a null cost
+  (never guessed with the input rate).
+- Report per-agent table shares `perAgentSummary` with the baseline/gate (no
+  more NaN rows when every trial of an agent errored).
+- Harbor adapter: non-integer `ARENA_TIMEOUT` falls back to 1800s with a
+  warning instead of silently disabling the container timeout; an empty
+  `{budget}` in a run template fails loudly; unknown `metrics.kind` values are
+  rejected at spec load.
 
 ## [0.1.0] — initial release
 
diff --git a/harbor/arena_harbor/base.py b/harbor/arena_harbor/base.py
index bdf9c39..282201f 100644
--- a/harbor/arena_harbor/base.py
+++ b/harbor/arena_harbor/base.py
@@ -132,6 +132,18 @@ async def run(
         )
         env = forwarded_env(spec)
 
+        # Accept "600", "600.5", etc. A non-numeric ARENA_TIMEOUT must not
+        # silently disable the container timeout.
+        try:
+            timeout_sec: int | None = int(float(timeout))
+        except ValueError:
+            timeout_sec = 1800
+            self.logger.warning(
+                "arena-harbor: ARENA_TIMEOUT=%r is not numeric; using %ss",
+                timeout,
+                timeout_sec,
+            )
+
         # Run directly (not exec_as_agent) so a non-zero agent exit does NOT
         # abort the trial before Harbor's verifier gets to judge the workspace.
         # The verifier — never the CLI's exit code — decides pass/fail.
@@ -139,7 +151,7 @@ async def run(
             result = await environment.exec(
                 command=command,
                 env=env,
-                timeout_sec=int(timeout) if timeout.isdigit() else None,
+                timeout_sec=timeout_sec,
             )
             self._agent_output = "\n".join(
                 part
diff --git a/harbor/arena_harbor/command.py b/harbor/arena_harbor/command.py
index d94fb73..6f39e33 100644
--- a/harbor/arena_harbor/command.py
+++ b/harbor/arena_harbor/command.py
@@ -55,6 +55,13 @@ def build_command(
         "timeout": timeout or "1800",
         "instruction": shlex.quote(instruction),
     }
+    if "{budget}" in spec.run_template and not mapping["budget"]:
+        # An empty {budget} would render e.g. "--max-usd --json", making the
+        # flag silently eat the next token. Fail loudly instead.
+        raise ValueError(
+            f"agent '{spec.name}': run_template references {{budget}} but no "
+            "budget is set (set ARENA_BUDGET or default_budget in the spec)."
+        )
     return render_template(spec.run_template, mapping)
 
 
diff --git a/harbor/arena_harbor/metrics.py b/harbor/arena_harbor/metrics.py
index faea373..22268cf 100644
--- a/harbor/arena_harbor/metrics.py
+++ b/harbor/arena_harbor/metrics.py
@@ -55,9 +55,10 @@ def is_empty(self) -> bool:
 
 
 def last_json_object(text: str) -> dict[str, Any] | None:
-    """Return the last ``{"type":"result"}`` JSON object in ``text`` (JSONL or a
-    single object), else the last parseable object. Non-JSON lines are ignored.
-    Mirrors the TS ``parseJsonEnvelope``.
+    """Return the last ``{"type":"result"}`` JSON object in ``text`` (JSONL, a
+    single object, or a pretty-printed multi-line object), else the last
+    parseable object. Non-JSON lines are ignored. Mirrors the TS
+    ``parseJsonEnvelope``.
     """
     if not text:
         return None
@@ -73,7 +74,75 @@ def last_json_object(text: str) -> dict[str, Any] | None:
             fallback = obj
         if obj.get("type") == "result":
             return obj
-    return fallback
+
+    # No single-line result envelope: pretty-printed envelopes span lines, so
+    # fall back to a string-aware brace scan over the whole output.
+    scanned = _scan_json_objects(text)
+    for obj in reversed(scanned):
+        if obj.get("type") == "result":
+            return obj
+    if fallback is not None:
+        return fallback
+    return scanned[-1] if scanned else None
+
+
+def _scan_json_objects(text: str) -> list[dict[str, Any]]:
+    """Find top-level JSON objects in text that may contain non-JSON noise.
+
+    Candidates are anchored at lines that START with ``{`` (a pretty-printed
+    envelope's opening line), so a stray brace mid-way through a log line can
+    never derail the scan. From each anchor, string-aware brace matching finds
+    the balanced end; spans that fail to parse are skipped.
+    """
+    found: list[dict[str, Any]] = []
+    offset = 0
+    consumed_up_to = -1
+    for line in text.split("\n"):
+        line_start = offset
+        offset += len(line) + 1
+        if line_start < consumed_up_to:
+            continue  # inside an already-parsed object
+        stripped = line.lstrip()
+        if not stripped.startswith("{"):
+            continue
+        start = line_start + (len(line) - len(stripped))
+        end = _balanced_object_end(text, start)
+        if end == -1:
+            continue
+        try:
+            obj = json.loads(text[start : end + 1])
+        except json.JSONDecodeError:
+            continue  # balanced braces but not JSON (shell/awk) — skip
+        if isinstance(obj, dict):
+            found.append(obj)
+            consumed_up_to = end + 1
+    return found
+
+
+def _balanced_object_end(text: str, start: int) -> int:
+    """Index of the ``}`` closing the object opened at ``start``, or -1."""
+    depth = 0
+    in_string = False
+    escaped = False
+    for i in range(start, len(text)):
+        ch = text[i]
+        if in_string:
+            if escaped:
+                escaped = False
+            elif ch == "\\":
+                escaped = True
+            elif ch == '"':
+                in_string = False
+            continue
+        if ch == '"':
+            in_string = True
+        elif ch == "{":
+            depth += 1
+        elif ch == "}":
+            depth -= 1
+            if depth == 0:
+                return i
+    return -1
 
 
 def dotted_get(obj: Any, path: str | None) -> Any:
diff --git a/harbor/arena_harbor/spec.py b/harbor/arena_harbor/spec.py
index 5981d79..dc40a97 100644
--- a/harbor/arena_harbor/spec.py
+++ b/harbor/arena_harbor/spec.py
@@ -136,8 +136,13 @@ def spec_from_dict(data: dict[str, Any]) -> AgentSpec:
     if metrics_raw is not None:
         if not isinstance(metrics_raw, dict):
             raise ValueError("'metrics' must be a table/object")
+        metrics_kind = metrics_raw.get("kind", "json_tail")
+        if metrics_kind not in ("json_tail", "regex"):
+            raise ValueError(
+                f"metrics.kind must be 'json_tail' or 'regex', got '{metrics_kind}'"
+            )
         metrics = MetricsSpec(
-            kind=metrics_raw.get("kind", "json_tail"),
+            kind=metrics_kind,
             input_path=metrics_raw.get("input_path"),
             output_path=metrics_raw.get("output_path"),
             cache_read_path=metrics_raw.get("cache_read_path"),
diff --git a/harbor/pyproject.toml b/harbor/pyproject.toml
index 2b942c4..03bcdd7 100644
--- a/harbor/pyproject.toml
+++ b/harbor/pyproject.toml
@@ -20,7 +20,8 @@ classifiers = [
   "Programming Language :: Python :: 3.12",
   "Programming Language :: Python :: 3.13",
 ]
-# Pinned to the 0.6.x line the adapter is built and tested against.
+# Built and tested against 0.6.x; the range admits later 0.x releases the
+# spec-driven surface has stayed compatible with. Tighten if Harbor breaks API.
 dependencies = [
   "harbor>=0.6,<0.20",
 ]
diff --git a/src/adapters/base.ts b/src/adapters/base.ts
index c055d81..502c71b 100644
--- a/src/adapters/base.ts
+++ b/src/adapters/base.ts
@@ -87,29 +87,62 @@ export abstract class Adapter {
   /**
    * Run the agent. Overridable (the mock adapter runs in-process). argv arrays
    * only — nothing is ever interpolated through a shell.
+   *
+   * On POSIX the child gets its own process group so a timeout (or exit) can
+   * kill the agent's whole process tree, not just the direct child: coding
+   * agents routinely spawn shells, watchers, and servers, and a surviving
+   * grandchild could otherwise hold the stdio pipes open forever (hanging the
+   * run past its timeout) or keep mutating the workspace while the held-out
+   * verifier runs.
    */
   execute(args: AdapterRunArgs): Promise<ExecOutcome> {
     return new Promise((resolve) => {
+      const posix = process.platform !== "win32";
       const child = spawn(this.bin(), this.buildArgs(args), {
         cwd: args.workDir,
         env: { ...process.env, ...this.env(args) },
         stdio: ["ignore", "pipe", "pipe"],
+        detached: posix,
       });
 
       let stdout = "";
       let stderr = "";
       let timedOut = false;
+      let settled = false;
+      let drainTimer: NodeJS.Timeout | undefined;
+
+      const killTree = (): void => {
+        if (posix && child.pid !== undefined) {
+          try {
+            process.kill(-child.pid, "SIGKILL");
+            return;
+          } catch {
+            // Process group already gone — fall through to the direct kill.
+          }
+        }
+        child.kill("SIGKILL");
+      };
+
+      const settle = (outcome: ExecOutcome): void => {
+        if (settled) return;
+        settled = true;
+        clearTimeout(timer);
+        if (drainTimer !== undefined) clearTimeout(drainTimer);
+        resolve(outcome);
+      };
+
       const timer = setTimeout(() => {
         timedOut = true;
-        child.kill("SIGKILL");
+        killTree();
       }, args.timeoutSeconds * 1000);
 
-      child.stdout.on("data", (chunk: Buffer) => (stdout += chunk.toString()));
-      child.stderr.on("data", (chunk: Buffer) => (stderr += chunk.toString()));
+      child.stdout.setEncoding("utf8");
+      child.stderr.setEncoding("utf8");
+      child.stdout.on("data", (chunk: string) => (stdout += chunk));
+      child.stderr.on("data", (chunk: string) => (stderr += chunk));
 
       child.on("error", (err) => {
-        clearTimeout(timer);
-        resolve({
+        settle({
           stdout,
           stderr,
           exitCode: null,
@@ -118,9 +151,19 @@ export abstract class Adapter {
         });
       });
 
-      child.on("close", (code) => {
+      // "close" waits for the stdio pipes to drain, which an orphaned
+      // grandchild that inherited them can prevent forever. "exit" is the
+      // authoritative signal: reap anything left in the process group, then
+      // give the pipes a short grace period to flush before settling.
+      child.on("exit", (code) => {
         clearTimeout(timer);
-        resolve({ stdout, stderr, exitCode: code, timedOut });
+        killTree();
+        drainTimer = setTimeout(() => {
+          settle({ stdout, stderr, exitCode: code, timedOut });
+        }, 2000);
+      });
+      child.on("close", (code) => {
+        settle({ stdout, stderr, exitCode: code, timedOut });
       });
     });
   }
diff --git a/src/adapters/claude-code.ts b/src/adapters/claude-code.ts
index 96e2b7e..845edf8 100644
--- a/src/adapters/claude-code.ts
+++ b/src/adapters/claude-code.ts
@@ -21,11 +21,15 @@ export class ClaudeCodeAdapter extends Adapter {
   readonly name = "claude-code";
   protected readonly defaultBinary = "claude";
 
-  /** `anthropic/claude-sonnet-5` → `sonnet`; unrecognized slugs pass through. */
+  /**
+   * `anthropic/claude-sonnet-5` → `claude-sonnet-5`; bare ids and aliases
+   * pass through. The full id is preserved (never collapsed to a family
+   * alias like `sonnet`): an alias floats to whatever the installed CLI's
+   * default is, which silently breaks version pinning, `matchedModels`, and
+   * pricing-table lookups.
+   */
   override resolveModel(model: string): string {
-    const bare = model.replace(/^anthropic\//, "");
-    const family = bare.match(/^claude-(opus|sonnet|haiku|fable)/)?.[1];
-    return family ?? bare;
+    return model.replace(/^anthropic\//, "");
   }
 
   buildArgs(args: AdapterRunArgs): string[] {
diff --git a/src/adapters/gemini.ts b/src/adapters/gemini.ts
index 2ae1f5f..390e7b4 100644
--- a/src/adapters/gemini.ts
+++ b/src/adapters/gemini.ts
@@ -8,9 +8,13 @@
  * `--permission-mode acceptEdits` (edits auto-approved, arbitrary shell not).
  *
  * JSON output shape (gemini-cli ≥ 0.x): { response, stats: { models: {
- *   "<model>": { tokens: { prompt, candidates, cached, total, ... } } } } }
+ *   "<model>": { tokens: { prompt, candidates, thoughts, tool, cached,
+ *   total, ... } } } } }
  * Multiple model entries are summed. Gemini's `prompt` count INCLUDES cached
- * tokens, so normalized input = prompt − cached.
+ * tokens, so normalized input = prompt + tool − cached. `thoughts` (reasoning)
+ * tokens are billed as output, so normalized output = candidates + thoughts —
+ * dropping them would systematically understate Gemini vs. agents whose
+ * output counts already include reasoning.
  *
  * NOTE: gemini-cli flags/envelope evolve quickly; `arena doctor` surfaces the
  * installed version, and this adapter is covered by unit tests over a captured
@@ -52,6 +56,8 @@ export class GeminiAdapter extends Adapter {
 
     let prompt = 0;
     let candidates = 0;
+    let thoughts = 0;
+    let tool = 0;
     let cached = 0;
     let toolCalls: number | null = null;
     for (const entry of Object.values(models)) {
@@ -59,6 +65,8 @@ export class GeminiAdapter extends Adapter {
       const tokens = isRecord(entry["tokens"]) ? entry["tokens"] : {};
       prompt += num(tokens["prompt"]);
       candidates += num(tokens["candidates"]);
+      thoughts += num(tokens["thoughts"]);
+      tool += num(tokens["tool"]);
       cached += num(tokens["cached"]);
     }
     const tools = isRecord(stats["tools"]) ? stats["tools"] : {};
@@ -70,8 +78,8 @@ export class GeminiAdapter extends Adapter {
     const cacheRead = Math.min(cached, prompt);
     return {
       tokens: totalize({
-        input: prompt - cacheRead,
-        output: candidates,
+        input: prompt + tool - cacheRead,
+        output: candidates + thoughts,
         cacheRead,
         cacheWrite: 0,
       }),
diff --git a/src/cli.ts b/src/cli.ts
index aa01a57..27ce140 100644
--- a/src/cli.ts
+++ b/src/cli.ts
@@ -137,7 +137,7 @@ async function cmdRun(argv: string[]): Promise<void> {
       budget: { type: "string" },
       timeout: { type: "string" },
       seed: { type: "string" },
-      out: { type: "string" },
+      out: { type: "string", short: "o" },
     },
   });
 
@@ -157,6 +157,18 @@ async function cmdRun(argv: string[]): Promise<void> {
     return { adapter: adapter.trim(), model: resolved.trim() };
   });
 
+  // Duplicate (adapter, model) specs would produce colliding trial ids and
+  // silently overwrite each other's receipts.
+  const specKeys = agents.map((a) => `${a.adapter}=${a.model}`);
+  const dupes = [...new Set(specKeys.filter((k, i) => specKeys.indexOf(k) !== i))];
+  if (dupes.length > 0) {
+    console.error(
+      `Duplicate agent spec(s): ${dupes.join(", ")}. ` +
+        "Trial ids would collide — run A/A comparisons as two separate runs.",
+    );
+    process.exit(1);
+  }
+
   const allTasks = loadTasks(TASK_ROOT);
   let tasks: LoadedTask[] = allTasks;
   if (values.tasks && values.tasks !== "all") {
@@ -172,10 +184,12 @@ async function cmdRun(argv: string[]): Promise<void> {
   const config = {
     agents,
     tasks,
-    trials: values.trials ? parseInt(values.trials, 10) : 3,
-    ...(values.budget !== undefined ? { budgetUsd: parseFloat(values.budget) } : {}),
-    timeoutSeconds: values.timeout ? parseInt(values.timeout, 10) : 600,
-    seed: values.seed ? parseInt(values.seed, 10) : 42,
+    trials: numericFlag("trials", values.trials, 3, { integer: true, min: 1 }),
+    ...(values.budget !== undefined
+      ? { budgetUsd: numericFlag("budget", values.budget, 0, { min: 0 }) }
+      : {}),
+    timeoutSeconds: numericFlag("timeout", values.timeout, 600, { integer: true, min: 1 }),
+    seed: numericFlag("seed", values.seed, 42, { integer: true, min: 0 }),
     outDir: resolve(values.out ?? "results"),
   };
 
@@ -185,6 +199,31 @@ async function cmdRun(argv: string[]): Promise<void> {
   console.log(`\nRun complete. Report: ${join(runDir, "report.md")}`);
 }
 
+/**
+ * Parse a numeric flag strictly. `parseInt`-style prefix parsing is rejected
+ * ("10m" is an error, not 10) and NaN never leaks into the run config, where
+ * it would silently produce instant timeouts or an empty "successful" run.
+ */
+function numericFlag(
+  flag: string,
+  raw: string | undefined,
+  fallback: number,
+  opts: { integer?: boolean; min?: number } = {},
+): number {
+  if (raw === undefined) return fallback;
+  const n = Number(raw);
+  const wholeOk = opts.integer !== true || Number.isInteger(n);
+  const minOk = opts.min === undefined || n >= opts.min;
+  if (!Number.isFinite(n) || !wholeOk || !minOk) {
+    console.error(
+      `--${flag} must be ${opts.integer ? "an integer" : "a number"}` +
+        `${opts.min !== undefined ? ` >= ${String(opts.min)}` : ""}, got "${raw}"`,
+    );
+    process.exit(1);
+  }
+  return n;
+}
+
 function cmdReport(argv: string[]): void {
   const runDir = argv[0];
   if (!runDir) {
@@ -203,6 +242,15 @@ function cmdReport(argv: string[]): void {
  */
 function cmdVerify(argv: string[]): void {
   const all = loadTasks(TASK_ROOT);
+  if (argv.length > 0) {
+    // An id that matches nothing must fail loudly: CI configured with a
+    // renamed task would otherwise "audit" zero tasks and pass forever.
+    const missing = argv.filter((id) => !all.some((t) => t.id === id));
+    if (missing.length > 0) {
+      console.error(`Unknown task id(s): ${missing.join(", ")}. Run \`arena list\`.`);
+      process.exit(1);
+    }
+  }
   const tasks = argv.length > 0 ? all.filter((t) => argv.includes(t.id)) : all;
   let failures = 0;
 
@@ -315,20 +363,25 @@ function cmdGate(argv: string[]): void {
   const configPath =
     values.config ?? (existsSync(resolve("arena-gate.json")) ? "arena-gate.json" : undefined);
   const thresholds: GateThresholds = loadGateConfig(configPath ? resolve(configPath) : undefined);
-  // A finite number, or null (a non-numeric flag value disables the check).
-  const num = (v: string | undefined): number | null => {
+  // Strict: a typo'd threshold ("--tokens-increase ten") must error, not
+  // silently disable the guard it was meant to set.
+  const num = (flag: string, v: string | undefined): number | null => {
     if (v === undefined) return null;
-    const n = parseFloat(v);
-    return Number.isFinite(n) ? n : null;
+    const n = Number(v);
+    if (!Number.isFinite(n)) {
+      console.error(`--${flag} must be a number, got "${v}"`);
+      process.exit(1);
+    }
+    return n;
   };
   if (values["accuracy-drop"] !== undefined)
-    thresholds.accuracyMaxDropPoints = num(values["accuracy-drop"]) ?? 0;
+    thresholds.accuracyMaxDropPoints = num("accuracy-drop", values["accuracy-drop"]) ?? 0;
   if (values["tokens-increase"] !== undefined)
-    thresholds.tokensMaxIncreasePct = num(values["tokens-increase"]);
+    thresholds.tokensMaxIncreasePct = num("tokens-increase", values["tokens-increase"]);
   if (values["cost-increase"] !== undefined)
-    thresholds.costMaxIncreasePct = num(values["cost-increase"]);
+    thresholds.costMaxIncreasePct = num("cost-increase", values["cost-increase"]);
   if (values["speed-increase"] !== undefined)
-    thresholds.speedMaxIncreasePct = num(values["speed-increase"]);
+    thresholds.speedMaxIncreasePct = num("speed-increase", values["speed-increase"]);
   if (values["require-significant"]) thresholds.accuracyRequireSignificant = true;
   if (values["allow-task-mismatch"]) thresholds.allowTaskMismatch = true;
 
diff --git a/src/orchestrator.ts b/src/orchestrator.ts
index ce675f4..bb5cd60 100644
--- a/src/orchestrator.ts
+++ b/src/orchestrator.ts
@@ -41,7 +41,6 @@ export async function executeRun(
   config: RunConfig,
   log: RunProgress = () => {},
 ): Promise<{ runDir: string; manifest: RunManifest; results: TrialResult[] }> {
-  const _pricing = loadPricing();
   const runId = `run-${new Date().toISOString().replace(/[:.]/g, "-").slice(0, 19)}-${randomUUID().slice(0, 8)}`;
   const runDir = join(config.outDir, runId);
   mkdirSync(join(runDir, "trials"), { recursive: true });
@@ -99,6 +98,9 @@ export async function executeRun(
         const result = await runOne(task, spec, adapter, trial, config, runId, runDir);
         results.push(result);
         writeFileSync(join(runDir, "trials", `${result.id}.json`), JSON.stringify(result, null, 2));
+        // Rewritten after every trial so a mid-run crash still leaves an
+        // aggregatable results.json behind.
+        writeFileSync(join(runDir, "results.json"), JSON.stringify({ manifest, results }, null, 2));
         const mark =
           result.outcome === "passed" ? "✅" : result.outcome === "agent-error" ? "⚠️" : "❌";
         log(
@@ -109,8 +111,6 @@ export async function executeRun(
     }
   }
 
-  writeFileSync(join(runDir, "results.json"), JSON.stringify({ manifest, results }, null, 2));
-
   return { runDir, manifest, results };
 }
 
@@ -124,13 +124,15 @@ async function runOne(
   runDir: string,
 ): Promise<TrialResult> {
   const workDir = join(tmpdir(), `arena-${sanitizeSegment(task.id)}-${randomUUID()}`);
-  const startedAt = new Date();
   const id = `${task.id}-${spec.adapter}-${sanitizeSegment(spec.model)}-t${trial}`;
+  let startedAt = new Date();
 
   try {
     seedWorkspace(task, workDir);
     MockAdapter.currentTaskDir = task.dir;
 
+    // Started only now: wall clock measures spawn-to-exit, not harness seeding.
+    startedAt = new Date();
     const exec = await adapter.execute({
       prompt: buildPrompt(task),
       model: spec.model,
@@ -142,14 +144,21 @@ async function runOne(
     const finishedAt = new Date();
     const wallClockSeconds = (finishedAt.getTime() - startedAt.getTime()) / 1000;
 
-    const diff = collectDiff(workDir);
+    const diffRaw = collectDiff(workDir);
+    const diff = diffRaw ?? "";
     const envelope = adapter.parseEnvelope(exec.stdout);
 
     // Invocation-level failure: the agent never actually ran (bad flags,
-    // missing binary, immediate non-zero exit with no work product).
+    // missing binary, immediate non-zero exit with no work product). A null
+    // diff means git failed — the diff is unknown, not empty, so it can never
+    // count as evidence that the agent did nothing.
     const invocationFailure =
       exec.spawnError !== undefined ||
-      (exec.exitCode !== 0 && !exec.timedOut && diff.length === 0 && envelope.tokens.total === 0);
+      (exec.exitCode !== 0 &&
+        !exec.timedOut &&
+        diffRaw !== null &&
+        diff.length === 0 &&
+        envelope.tokens.total === 0);
 
     let outcome: Outcome;
     let verify = { passed: false, output: "" };
@@ -227,6 +236,57 @@ async function runOne(
       diffPath,
       ...(errorText !== undefined ? { error: errorText } : {}),
     };
+  } catch (error) {
+    // Per-trial containment: a harness-side failure (git error, FS hiccup,
+    // missing fixture) scores this one trial `agent-error` — consistent with
+    // the project rule that harness failures never read as the agent losing —
+    // instead of aborting the whole run and losing every completed trial.
+    const finishedAt = new Date();
+    const message = error instanceof Error ? error.message : String(error);
+    const transcriptPath = join("transcripts", `${id}.txt`);
+    const diffPath = join("diffs", `${id}.patch`);
+    try {
+      writeFileSync(join(runDir, transcriptPath), `# ${id}\n# harness error\n\n${message}\n`);
+      writeFileSync(join(runDir, diffPath), "");
+    } catch {
+      // Receipts are best-effort here; the error itself is preserved below.
+    }
+    return {
+      id,
+      taskId: task.id,
+      trial,
+      agent: {
+        adapter: spec.adapter,
+        model: spec.model,
+        resolvedModel: adapter.resolveModel(spec.model),
+        version: adapter.version(),
+        bin: adapter.bin(),
+      },
+      outcome: "agent-error",
+      verify: { passed: false, output: "" },
+      timing: {
+        wallClockSeconds: (finishedAt.getTime() - startedAt.getTime()) / 1000,
+        agentReportedSeconds: null,
+      },
+      tokens: { input: 0, output: 0, cacheRead: 0, cacheWrite: 0, total: 0 },
+      cost: { computedUsd: null, agentReportedUsd: null, pricingModel: null },
+      activity: {
+        toolCalls: null,
+        iterations: null,
+        filesTouched: 0,
+        linesAdded: 0,
+        linesRemoved: 0,
+        diffBytes: 0,
+      },
+      provenance: {
+        runId,
+        startedAt: startedAt.toISOString(),
+        finishedAt: finishedAt.toISOString(),
+      },
+      transcriptPath,
+      diffPath,
+      error: `harness error: ${message}`,
+    };
   } finally {
     MockAdapter.currentTaskDir = null;
     rmSync(workDir, { recursive: true, force: true });
diff --git a/src/parse.ts b/src/parse.ts
index 1827144..10cbb10 100644
--- a/src/parse.ts
+++ b/src/parse.ts
@@ -22,9 +22,10 @@ export function isRecord(value: unknown): value is Record<string, unknown> {
 /**
  * Extract the machine-readable JSON envelope from an agent's stdout.
  *
- * Headless agent CLIs print either a single JSON object or JSONL (one event
- * per line). We want the last `type: "result"` object, falling back to the
- * last parseable object. Banners and log lines are ignored.
+ * Headless agent CLIs print either a single JSON object, JSONL (one event per
+ * line), or a pretty-printed (multi-line) object. We want the last
+ * `type: "result"` object, falling back to the last parseable object. Banners
+ * and log lines are ignored.
  */
 export function parseJsonEnvelope(stdout: string): Record<string, unknown> | null {
   if (!stdout) return null;
@@ -47,7 +48,71 @@ export function parseJsonEnvelope(stdout: string): Record<string, unknown> | nul
     }
   }
 
-  return fallback;
+  // No single-line result envelope. Pretty-printed envelopes span lines, so
+  // fall back to a string-aware brace scan over the whole output.
+  const scanned = scanJsonObjects(stdout);
+  for (let i = scanned.length - 1; i >= 0; i--) {
+    const obj = scanned[i] as Record<string, unknown>;
+    if (obj["type"] === "result") return obj;
+  }
+
+  return fallback ?? scanned[scanned.length - 1] ?? null;
+}
+
+/**
+ * Find every top-level JSON object in a text that may also contain non-JSON
+ * noise. Candidates are anchored at lines that START with `{` (a
+ * pretty-printed envelope's opening line), so a stray brace mid-way through a
+ * log line can never derail the scan. From each anchor, string-aware brace
+ * matching finds the balanced end; spans that fail `JSON.parse` are skipped.
+ */
+function scanJsonObjects(text: string): Record<string, unknown>[] {
+  const found: Record<string, unknown>[] = [];
+  let offset = 0;
+  let consumedUpTo = -1;
+  for (const line of text.split("\n")) {
+    const lineStart = offset;
+    offset += line.length + 1;
+    if (lineStart < consumedUpTo) continue; // inside an already-parsed object
+    const firstNonSpace = line.search(/\S/);
+    if (firstNonSpace === -1 || line[firstNonSpace] !== "{") continue;
+    const start = lineStart + firstNonSpace;
+    const end = balancedObjectEnd(text, start);
+    if (end === -1) continue;
+    try {
+      const obj: unknown = JSON.parse(text.slice(start, end + 1));
+      if (isRecord(obj)) {
+        found.push(obj);
+        consumedUpTo = end + 1;
+      }
+    } catch {
+      // Balanced braces but not JSON — skip this anchor.
+    }
+  }
+  return found;
+}
+
+/** Index of the `}` closing the object opened at `start`, or -1. */
+function balancedObjectEnd(text: string, start: number): number {
+  let depth = 0;
+  let inString = false;
+  let escaped = false;
+  for (let i = start; i < text.length; i++) {
+    const ch = text[i];
+    if (inString) {
+      if (escaped) escaped = false;
+      else if (ch === "\\") escaped = true;
+      else if (ch === '"') inString = false;
+      continue;
+    }
+    if (ch === '"') inString = true;
+    else if (ch === "{") depth++;
+    else if (ch === "}") {
+      depth--;
+      if (depth === 0) return i;
+    }
+  }
+  return -1;
 }
 
 /** Make a string safe as a single path segment (model slugs contain "/"). */
diff --git a/src/pricing.ts b/src/pricing.ts
index b351de2..ed756b3 100644
--- a/src/pricing.ts
+++ b/src/pricing.ts
@@ -44,11 +44,14 @@ export function computeCost(
 ): number | null {
   const rate = pricing[model];
   if (!rate) return null;
-  const cacheWriteRate = rate.cacheWritePerM ?? rate.inputPerM;
+  // Cache-write rates differ per vendor. If the trial wrote cache tokens and
+  // the table has no cacheWritePerM, the true cost is unknown — report null
+  // rather than guessing with the input rate ("cost is never guessed").
+  if (tokens.cacheWrite > 0 && rate.cacheWritePerM === undefined) return null;
   return (
     (tokens.input / 1e6) * rate.inputPerM +
     (tokens.output / 1e6) * rate.outputPerM +
     (tokens.cacheRead / 1e6) * rate.cacheReadPerM +
-    (tokens.cacheWrite / 1e6) * cacheWriteRate
+    (tokens.cacheWrite / 1e6) * (rate.cacheWritePerM ?? 0)
   );
 }
diff --git a/src/report.ts b/src/report.ts
index ab17ae6..bad8c98 100644
--- a/src/report.ts
+++ b/src/report.ts
@@ -15,7 +15,8 @@
 import { readFileSync } from "node:fs";
 import { join } from "node:path";
 import { isRecord } from "./parse.js";
-import { mcnemarExact, median, pairedBootstrapDelta, wilsonInterval } from "./stats.js";
+import { mcnemarExact, median, pairedBootstrapDelta } from "./stats.js";
+import { perAgentSummary } from "./summary.js";
 import type { RunManifest, TrialResult } from "./types.js";
 
 export function loadRun(runDir: string): {
@@ -97,22 +98,21 @@ export function generateReport(runDir: string): string {
   }
 
   // ── Per-agent summary ──
+  // perAgentSummary is the single source of truth shared with the baseline
+  // and the gate, so the report can never disagree with what CI enforces.
+  const summaries = perAgentSummary(results);
   lines.push("## Results by agent", "");
   lines.push(
     "| Agent | Scored trials | Passed | Success rate (95% CI) | Median wall clock | Median tokens (billed) | Median computed cost | Self-reported cost |",
     "|---|---|---|---|---|---|---|---|",
   );
-  for (const key of keys) {
-    const all = byAgent.get(key) ?? [];
-    const scored = all.filter((r) => r.outcome !== "agent-error");
-    const passed = scored.filter((r) => r.outcome === "passed").length;
-    const [lo, hi] = wilsonInterval(passed, scored.length);
-    const wall = median(scored.map((r) => r.timing.wallClockSeconds));
-    const toks = median(scored.map((r) => r.tokens.total));
-    const costs = scored.map((r) => r.cost.computedUsd).filter((c): c is number => c !== null);
-    const self = scored.map((r) => r.cost.agentReportedUsd).filter((c): c is number => c !== null);
+  for (const s of summaries) {
+    const wall =
+      s.medianWallClockSeconds === null ? "—" : `${s.medianWallClockSeconds.toFixed(1)}s`;
+    const toks =
+      s.medianTotalTokens === null ? "—" : Math.round(s.medianTotalTokens).toLocaleString();
     lines.push(
-      `| ${key} | ${String(scored.length)} | ${String(passed)} | ${pct(scored.length ? passed / scored.length : 0)} (${pct(lo)}–${pct(hi)}) | ${wall.toFixed(1)}s | ${Math.round(toks).toLocaleString()} | ${fmtUsd(costs.length ? median(costs) : null)} | ${fmtUsd(self.length ? median(self) : null)} |`,
+      `| ${s.key} | ${String(s.scoredTrials)} | ${String(s.passed)} | ${pct(s.successRate)} (${pct(s.successCI[0])}–${pct(s.successCI[1])}) | ${wall} | ${toks} | ${fmtUsd(s.medianComputedCost)} | ${fmtUsd(s.medianAgentReportedCost)} |`,
     );
   }
   lines.push("");
@@ -120,6 +120,13 @@ export function generateReport(runDir: string): string {
     "Token counts are normalized (input excludes cache reads; cache reads and writes tracked separately). Computed cost applies one shared pricing table to every agent; “—” means the model has no pricing entry (cost is never guessed).",
     "",
   );
+  const unmetered = summaries.reduce((n, s) => n + s.unmeteredTrials, 0);
+  if (unmetered > 0) {
+    lines.push(
+      `> ℹ️ ${String(unmetered)} scored trial(s) reported no parseable token usage (envelope missing or truncated). They count toward success rates but are excluded from token/cost medians and deltas.`,
+      "",
+    );
+  }
 
   // ── Pairwise comparisons ──
   if (keys.length >= 2) {
@@ -228,11 +235,13 @@ function pairwiseSection(
     "",
   );
 
+  // Zero-token trials mean "usage unknown" (unparseable envelope), so token
+  // and cost metrics treat them as missing rather than as literal zeros.
   const metrics: { label: string; get: (r: TrialResult) => number | null }[] = [
     { label: "Wall clock (s)", get: (r) => r.timing.wallClockSeconds },
-    { label: "Total tokens", get: (r) => r.tokens.total },
-    { label: "Output tokens", get: (r) => r.tokens.output },
-    { label: "Computed cost (USD)", get: (r) => r.cost.computedUsd },
+    { label: "Total tokens", get: (r) => (r.tokens.total > 0 ? r.tokens.total : null) },
+    { label: "Output tokens", get: (r) => (r.tokens.total > 0 ? r.tokens.output : null) },
+    { label: "Computed cost (USD)", get: (r) => (r.tokens.total > 0 ? r.cost.computedUsd : null) },
   ];
   lines.push(`| Metric | ${aKey} (median) | ${bKey} (median) | Δ relative to ${aKey} (95% CI) |`);
   lines.push("|---|---|---|---|");
diff --git a/src/summary.ts b/src/summary.ts
index 6c8508c..cba0611 100644
--- a/src/summary.ts
+++ b/src/summary.ts
@@ -19,6 +19,8 @@ export interface AgentSummary {
   /** Trials excluding `agent-error`. */
   scoredTrials: number;
   errorTrials: number;
+  /** Scored trials whose envelope had no parseable token usage (total = 0). */
+  unmeteredTrials: number;
   passed: number;
   /** passed / scoredTrials (0 when nothing scored). */
   successRate: number;
@@ -58,7 +60,13 @@ export function perAgentSummary(results: TrialResult[]): AgentSummary[] {
     const scored = all.filter((r) => r.outcome !== "agent-error");
     const passed = scored.filter((r) => r.outcome === "passed").length;
     const first = all[0] as TrialResult;
-    const costs = scored.map((r) => r.cost.computedUsd).filter((c): c is number => c !== null);
+    // A scored trial with zero total tokens means the envelope was
+    // unparseable (timeout kill mid-print, envelope drift), not that the
+    // agent used zero tokens. Excluding them keeps token/cost medians — and
+    // the drift gate built on them — from being dragged toward zero by parse
+    // failures.
+    const metered = scored.filter((r) => r.tokens.total > 0);
+    const costs = metered.map((r) => r.cost.computedUsd).filter((c): c is number => c !== null);
     const selfCosts = scored
       .map((r) => r.cost.agentReportedUsd)
       .filter((c): c is number => c !== null);
@@ -68,11 +76,12 @@ export function perAgentSummary(results: TrialResult[]): AgentSummary[] {
       model: first.agent.model,
       scoredTrials: scored.length,
       errorTrials: all.length - scored.length,
+      unmeteredTrials: scored.length - metered.length,
       passed,
       successRate: scored.length ? passed / scored.length : 0,
       successCI: wilsonInterval(passed, scored.length),
       medianWallClockSeconds: medianOrNull(scored.map((r) => r.timing.wallClockSeconds)),
-      medianTotalTokens: medianOrNull(scored.map((r) => r.tokens.total)),
+      medianTotalTokens: medianOrNull(metered.map((r) => r.tokens.total)),
       medianComputedCost: medianOrNull(costs),
       medianAgentReportedCost: medianOrNull(selfCosts),
     };
diff --git a/src/workspace.ts b/src/workspace.ts
index e5062bd..d11d5b7 100644
--- a/src/workspace.ts
+++ b/src/workspace.ts
@@ -32,7 +32,12 @@ export function loadTasks(taskRoot: string): LoadedTask[] {
     const dir = join(taskRoot, entry.name);
     const defPath = join(dir, "task.json");
     if (!existsSync(defPath)) continue;
-    const def: unknown = JSON.parse(readFileSync(defPath, "utf8"));
+    let def: unknown;
+    try {
+      def = JSON.parse(readFileSync(defPath, "utf8"));
+    } catch (error) {
+      throw new Error(`Invalid JSON in ${defPath}: ${(error as Error).message}`);
+    }
     if (!isRecord(def) || typeof def["id"] !== "string") {
       throw new Error(`Malformed task.json in ${dir}`);
     }
@@ -82,8 +87,12 @@ export function seedWorkspace(task: LoadedTask, workDir: string): void {
   git(["commit", "-m", "arena: seed workspace", "--no-verify"]);
 }
 
-/** Diff of the agent's changes (tracked + new files) against the seed commit. */
-export function collectDiff(workDir: string): string {
+/**
+ * Diff of the agent's changes (tracked + new files) against the seed commit.
+ * Returns null when git itself failed (oversized diff, corrupted repo) — an
+ * unknown diff, which callers must not conflate with "no changes".
+ */
+export function collectDiff(workDir: string): string | null {
   try {
     execFileSync("git", ["add", "-A"], { cwd: workDir, stdio: "ignore" });
     return execFileSync("git", ["diff", "--cached", "HEAD"], {
@@ -92,7 +101,7 @@ export function collectDiff(workDir: string): string {
       maxBuffer: 64 * 1024 * 1024,
     });
   } catch {
-    return "";
+    return null;
   }
 }
 
diff --git a/test/adapters.test.ts b/test/adapters.test.ts
index 88c834f..71a9f6e 100644
--- a/test/adapters.test.ts
+++ b/test/adapters.test.ts
@@ -26,9 +26,9 @@ describe("registry", () => {
 describe("claude-code", () => {
   const adapter = new ClaudeCodeAdapter();
 
-  it("maps slugs to family aliases", () => {
-    expect(adapter.resolveModel("anthropic/claude-sonnet-5")).toBe("sonnet");
-    expect(adapter.resolveModel("anthropic/claude-opus-4.8")).toBe("opus");
+  it("preserves the full model id (version pinning), stripping only the provider prefix", () => {
+    expect(adapter.resolveModel("anthropic/claude-sonnet-5")).toBe("claude-sonnet-5");
+    expect(adapter.resolveModel("anthropic/claude-opus-4.8")).toBe("claude-opus-4.8");
     expect(adapter.resolveModel("weird-model")).toBe("weird-model");
   });
 
@@ -38,7 +38,7 @@ describe("claude-code", () => {
       "--output-format",
       "json",
       "--model",
-      "sonnet",
+      "claude-sonnet-5",
       "--permission-mode",
       "acceptEdits",
       "--max-budget-usd",
@@ -147,13 +147,20 @@ describe("gemini", () => {
     expect(argv).toContain("auto_edit");
   });
 
-  it("sums per-model token stats and subtracts cached from prompt", () => {
+  it("sums per-model token stats, subtracts cached, and counts thoughts + tool tokens", () => {
     const stdout = JSON.stringify({
       response: "done",
       stats: {
         models: {
           "gemini-2.5-pro": {
-            tokens: { prompt: 5000, candidates: 700, cached: 2000, total: 5700 },
+            tokens: {
+              prompt: 5000,
+              candidates: 700,
+              thoughts: 300,
+              tool: 50,
+              cached: 2000,
+              total: 6050,
+            },
           },
           "gemini-2.5-flash": { tokens: { prompt: 100, candidates: 20, cached: 0, total: 120 } },
         },
@@ -161,9 +168,12 @@ describe("gemini", () => {
       },
     });
     const parsed = adapter.parseEnvelope(stdout);
-    expect(parsed.tokens.input).toBe(5100 - 2000);
+    // input = prompt (5100) + tool (50) − cached (2000)
+    expect(parsed.tokens.input).toBe(5100 + 50 - 2000);
     expect(parsed.tokens.cacheRead).toBe(2000);
-    expect(parsed.tokens.output).toBe(720);
+    // output = candidates (720) + thoughts (300): reasoning tokens are billed
+    // as output and must not be dropped from the head-to-head comparison.
+    expect(parsed.tokens.output).toBe(720 + 300);
     expect(parsed.toolCalls).toBe(9);
   });
 });
diff --git a/test/parse.test.ts b/test/parse.test.ts
index 88129ba..39f29ef 100644
--- a/test/parse.test.ts
+++ b/test/parse.test.ts
@@ -26,6 +26,39 @@ describe("parseJsonEnvelope", () => {
     const stdout = ["INFO starting", '{"type":"result","ok":true}', "INFO done"].join("\n");
     expect(parseJsonEnvelope(stdout)).toEqual({ type: "result", ok: true });
   });
+
+  it("parses a pretty-printed (multi-line) envelope", () => {
+    const stdout = [
+      "INFO starting",
+      "{",
+      '  "type": "result",',
+      '  "usage": {',
+      '    "input_tokens": 1200',
+      "  }",
+      "}",
+      "INFO done",
+    ].join("\n");
+    expect(parseJsonEnvelope(stdout)).toEqual({
+      type: "result",
+      usage: { input_tokens: 1200 },
+    });
+  });
+
+  it("prefers a multi-line result envelope and survives braces inside strings", () => {
+    const stdout = [
+      'log with a stray { brace and "quote',
+      "{",
+      '  "type": "result",',
+      '  "note": "contains } and { inside a string",',
+      '  "n": 9',
+      "}",
+    ].join("\n");
+    expect(parseJsonEnvelope(stdout)).toEqual({
+      type: "result",
+      note: "contains } and { inside a string",
+      n: 9,
+    });
+  });
 });
 
 describe("num / countOf", () => {

From 3d7c5f51bf56ef91ffcd3b90a83a98627483e0c4 Mon Sep 17 00:00:00 2001
From: macanderson <mac@oxagen.sh>
Date: Sat, 18 Jul 2026 19:34:59 -0700
Subject: [PATCH 2/4] feat(site): arena.oxagen.sh landing page + Agent
 Benchmark Protocol draft

Static site (site/) for arena.oxagen.sh: landing page with the
matched-model scoreboard, and the ABP design spec. Spec source lives at
docs/agent-benchmark-protocol.md: four interchange schemas (task, run,
trace, verdict), the live GitHub-issue task pipeline with golden traces,
smallest-normalized-diff scoring, conformance levels, and roadmap.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---
 docs/agent-benchmark-protocol.md | 227 ++++++++++
 site/.gitignore                  |   2 +
 site/index.html                  | 194 +++++++++
 site/spec/index.html             | 351 ++++++++++++++++
 site/style.css                   | 691 +++++++++++++++++++++++++++++++
 site/vercel.json                 |  14 +
 6 files changed, 1479 insertions(+)
 create mode 100644 docs/agent-benchmark-protocol.md
 create mode 100644 site/.gitignore
 create mode 100644 site/index.html
 create mode 100644 site/spec/index.html
 create mode 100644 site/style.css
 create mode 100644 site/vercel.json

diff --git a/docs/agent-benchmark-protocol.md b/docs/agent-benchmark-protocol.md
new file mode 100644
index 0000000..e285851
--- /dev/null
+++ b/docs/agent-benchmark-protocol.md
@@ -0,0 +1,227 @@
+# Agent Benchmark Protocol (ABP)
+
+**Draft 0.1 · July 2026 · Oxagen**
+
+An open standard for benchmarking agent engines: the harness, prompts, tools, and loop an agent ships with, measured separately from the model it happens to run on.
+
+Status: draft for public comment. The reference implementation is [Arena](https://github.com/oxageninc/arena). Nothing here is final, including the name.
+
+---
+
+## 1. The problem this standard exists to solve
+
+Teams that ship agents change their engine every week: a new system prompt, a new tool, a reworked planning loop, a different context strategy. None of those changes show up on a model leaderboard, because model leaderboards hold the harness fixed and vary the model. The people who build agent engines have the opposite need: hold the model fixed and vary the engine.
+
+Today that comparison is nearly impossible to run honestly:
+
+1. Every harness invokes agents differently, so "same task" rarely means same prompt, same budget, same timeout, or same autonomy level.
+2. Every harness logs differently, so traces cannot be compared across engines.
+3. Every harness grades differently, and most let the agent's own output influence the grade.
+4. Almost no benchmark measures efficiency. Two agents that both fix a bug are scored as equals even when one shipped a 4-line patch in 90 seconds and the other shipped a 400-line rewrite in 20 minutes.
+5. Almost no benchmark runs in CI. A benchmark you run once for a launch tweet cannot catch the Tuesday your agent quietly got worse.
+
+ABP standardizes the four interfaces that make honest engine comparison possible: how a task is packaged, how a run is configured, how a trace is recorded, and how a verdict is reported. Any engine that speaks these formats can be benchmarked by any conforming harness, and any harness can be audited by anyone.
+
+## 2. Why "protocol"
+
+We considered "standard", "format", "spec", and "benchmark". Protocol is the right word for the same reason it was right for the Model Context Protocol: the deliverable is a set of interchange contracts between independent parties (task authors, engines, harnesses, graders, and auditors), not a single dataset or a single leaderboard. Datasets and leaderboards are built on top of ABP; they are products, and ABP is the plumbing. It also keeps the family resemblance with the Open Context Protocol, which Oxagen authored and built [Stella](https://github.com/oxageninc/stella) on.
+
+One naming caution, stated openly: "protocol" sets an expectation of wire-level rigor. This draft earns that word only in Sections 5 through 8, where the schemas live. If the community reads those sections and concludes the word oversells, the fallback name is the Agent Benchmark Format (ABF). The acronym to protect is ABP, one syllable per letter, easy to say in CI logs: `abp gate failed`.
+
+## 3. Design principles
+
+These are inherited from Arena's methodology and are non-negotiable for conformance:
+
+1. **The engine is the unit under test.** The model, provider, budget, and timeout are pinned inputs. Runs that vary the model are stamped as such and cannot back engine-vs-engine claims.
+2. **The agent never grades itself.** Verification material must be absent from the workspace while the agent runs, injected only after it exits, and immune to anything the agent planted.
+3. **Every number carries its uncertainty.** Success rates get Wilson intervals. Paired comparisons get exact McNemar. Continuous metrics get seeded bootstrap CIs. No blended scores, ever.
+4. **Harness failures are not agent failures.** An engine that could not be invoked is excluded from comparison, not counted as a loss.
+5. **Receipts or it did not happen.** A conforming result ships its manifest, transcripts, diffs, and a one-line reproduce command.
+6. **Efficiency is a first-class metric.** Resolution alone is a tie. Diff size, tokens, wall clock, and cost break it.
+
+## 4. The live task pipeline
+
+This is the part of ABP that no existing benchmark provides end to end.
+
+### 4.1 Sourcing
+
+A curated registry of the top 500 open-source repositories on GitHub, filtered for: OSI license, active maintainership (merged PR in the last 30 days), a runnable test suite, and issue hygiene (reproducible reports, labeled backlog). The registry is versioned and published; entries rotate quarterly so the pool tracks the real ecosystem.
+
+From the registry, tasks are drawn by seeded random selection: pick a repo, pick an open issue from its backlog that passes eligibility (reproducible, in-scope for a code change, not a question or a feature debate). The seed is published with every draw, so the selection itself is auditable.
+
+### 4.2 Solving and promotion
+
+A frontier engine (or a human, or both) builds a fix for the issue against the repo's contribution rules and submits it upstream. The fix is **verified** when the project's own CI passes and a maintainer merges it into the production branch. Maintainer merge is the ground truth no synthetic benchmark has: a real reviewer accepted this change into a real codebase.
+
+Two artifacts fall out of every verified fix:
+
+- **Training data.** The (issue, repo state, verified diff) triple.
+- **A golden trace.** The full ABP trace (Section 6) the solving engine printed along the way: every tool call, every file read, every edit, every token count, timestamped.
+
+### 4.3 Replay and grading
+
+Once a task is verified, it freezes: repo pinned at the pre-fix commit, issue text captured, and a verification bundle built from the repo's own test suite plus the tests the verified fix added (the FAIL_TO_PASS set). Other engines then attempt the same frozen task under matched config. They are graded on:
+
+1. **Resolution.** The held-out verification bundle passes.
+2. **Efficiency, in strict order:** smaller normalized git diff wins, then fewer tokens, then less wall clock, then lower cost.
+
+The golden trace is published as evidence and as a study aid, not as the rubric. Version 1 deliberately does not grade trace similarity: an engine that reaches a passing, smaller fix by a different route beats the golden trace's author. Judging architectural quality and trajectory quality is a stated version 2 direction, and it needs its own adversarial review before it can be a metric.
+
+### 4.4 Contamination control
+
+Live sourcing is also the contamination story. Issues are selected from the current backlog, after every candidate model's training cutoff, and each task carries a `sourcedAt` timestamp plus the model cutoffs it was verified against. Tasks age out of the scored set once their fix has been public for 180 days; they remain available as a practice set. Frozen-set benchmarks decay into training data. A pipeline that keeps drawing from the living backlog does not.
+
+### 4.5 Known objections, answered in the design
+
+- **"Smallest diff is gameable."** Diff size is measured on a normalized form: formatting-only changes are canonicalized before counting, generated files and lockfiles are excluded, and test files are counted separately (deleting or weakening tests disqualifies the run, it does not shrink the diff). Resolution is still the gate; diff size only ranks engines that already passed. And the metric is honest about what it is: a proxy for surgical precision, not for design quality. Arena publishes the raw diffs, so anyone can audit whether small meant surgical or small meant degenerate.
+- **"Maintainers did not consent to being a benchmark."** Sourcing follows each repo's contribution guidelines, fixes are submitted as normal PRs with disclosure, registry entries can opt out, and replay runs happen on frozen clones, never against the live repo.
+- **"Issue selection will favor easy tasks."** Selection is seeded-random within eligibility, the seed is published, and difficulty is stratified in reporting (task metadata records diff size and files touched of the verified fix).
+
+## 5. Task manifest (`abp-task/v1`)
+
+```jsonc
+{
+  "schema": "abp-task/v1",
+  "id": "gh-vercel-next.js-81234",
+  "source": {
+    "kind": "github-issue",              // or "fixture" for synthetic tasks
+    "repo": "vercel/next.js",
+    "issue": 81234,
+    "baseCommit": "9f2c41d…",            // repo frozen here (pre-fix)
+    "sourcedAt": "2026-07-02T14:11:09Z",
+    "registryVersion": "abp-registry/2026Q3",
+    "drawSeed": 424242
+  },
+  "prompt": "…full task statement given to every engine…",
+  "environment": {
+    "image": "ghcr.io/oxageninc/abp-node:22",   // container digest-pinned
+    "setup": ["pnpm install --frozen-lockfile"]
+  },
+  "verification": {
+    "kind": "commands",
+    "failToPass": ["pnpm test -- test/router/edge-case.test.ts"],
+    "passToPass": ["pnpm test -- test/router/"],
+    "timeoutSeconds": 900,
+    "heldOut": true                       // must not be on disk during the run
+  },
+  "golden": {
+    "diffBytesNormalized": 1843,
+    "filesTouched": 2,
+    "trace": "traces/gh-vercel-next.js-81234.abp-trace.jsonl",
+    "mergedPr": "https://github.com/vercel/next.js/pull/81301"
+  },
+  "cutoffsVerified": ["claude-sonnet-5:2026-01", "gpt-5.2:2026-03"]
+}
+```
+
+Synthetic fixtures (like Arena's built-in tasks) use the same schema with `source.kind: "fixture"` and no `golden` block. A harness must treat both identically at run time.
+
+## 6. Trace format (`abp-trace/v1`)
+
+One JSONL file per (task, engine, trial). Every line is an event:
+
+```jsonc
+{"t":"2026-07-02T14:12:01.402Z","seq":1,"kind":"run.start","engine":"oxagen/1.4.2","model":"anthropic/claude-sonnet-5","task":"gh-vercel-next.js-81234","config":{"budgetUsd":5,"timeoutSeconds":1200}}
+{"t":"…","seq":2,"kind":"model.call","tokens":{"input":1204,"output":388,"cacheRead":9120,"cacheWrite":0}}
+{"t":"…","seq":3,"kind":"tool.call","tool":"read_file","args":{"path":"src/router/match.ts"},"durationMs":12}
+{"t":"…","seq":4,"kind":"tool.call","tool":"edit_file","args":{"path":"src/router/match.ts"},"diffBytes":412}
+{"t":"…","seq":5,"kind":"subagent.start","id":"sa-1","purpose":"run tests"}
+{"t":"…","seq":6,"kind":"run.end","outcome":"resolved","wallClockMs":184201,"tokens":{"input":48210,"output":9114,"cacheRead":301200,"cacheWrite":8100},"diffBytesNormalized":1610,"filesTouched":2}
+```
+
+Rules:
+
+- `seq` is a strictly increasing integer; `t` is RFC 3339 UTC. Together they make traces diffable and mergeable.
+- `tool.call` events record the tool name and a redacted argument summary, never raw secrets. A published redaction list is part of conformance.
+- Multi-agent engines emit `subagent.start` / `subagent.end` with nested attribution, so a fan-out engine's token bill is visible per branch. Multi-model engines record `model` per `model.call`. This is how ABP stays meaningful for orchestrators, not just single-loop CLIs.
+- Token counts follow Arena's normalization: `input` never includes cache reads; cache reads and writes are tracked separately.
+- An engine that cannot emit ABP traces natively can ship a sidecar translator; the harness records which path produced the trace.
+
+## 7. Run configuration (`abp-run/v1`)
+
+The manifest a harness must write before the first trial:
+
+```jsonc
+{
+  "schema": "abp-run/v1",
+  "engines": [{"name": "oxagen", "version": "1.4.2"}, {"name": "claude-code", "version": "2.1.0"}],
+  "model": "anthropic/claude-sonnet-5",
+  "matchedModels": true,
+  "budgetUsd": 5,
+  "timeoutSeconds": 1200,
+  "trials": 3,
+  "ordering": "abba",
+  "seed": 20260718,
+  "tasks": ["gh-vercel-next.js-81234", "…"],
+  "host": {"os": "linux", "arch": "arm64", "containerized": true},
+  "reproduce": "abp run --config run.json"
+}
+```
+
+`matchedModels: false` runs are legal (you may want to test your engine across models) but a conforming report must lead with the warning and must not present engine-vs-engine conclusions from them.
+
+## 8. Verdict format (`abp-verdict/v1`)
+
+```jsonc
+{
+  "schema": "abp-verdict/v1",
+  "run": "run-20260718-a41c",
+  "perEngine": [{
+    "engine": "oxagen",
+    "scoredTrials": 72,
+    "excludedEngineErrors": 1,
+    "resolveRate": {"point": 0.84, "wilson95": [0.71, 0.92]},
+    "medianDiffBytesNormalized": 1650,
+    "medianTokensTotal": 361000,
+    "medianWallClockMs": 190334,
+    "medianComputedUsd": 1.42
+  }],
+  "pairwise": [{
+    "a": "oxagen", "b": "claude-code",
+    "mcnemar": {"discordant": [14, 5], "pExact": 0.041},
+    "diffDelta": {"relMedian": -0.22, "bootstrap95": [-0.31, -0.09], "seed": 20260718}
+  }],
+  "receipts": {"trials": "trials/", "transcripts": "transcripts/", "diffs": "diffs/", "traces": "traces/"}
+}
+```
+
+## 9. Conformance levels
+
+- **ABP-core.** The harness consumes `abp-task/v1`, enforces held-out verification and matched config, and emits `abp-run/v1` + `abp-verdict/v1` with the required statistics. Arena today is one file-format migration away from ABP-core.
+- **ABP-traces.** Everything in core, plus `abp-trace/v1` emission for every trial, with subagent and multi-model attribution.
+- **ABP-live.** Everything in traces, plus participation in the live task pipeline: consuming registry draws, contributing verified fixes, and publishing golden traces.
+
+Conformance is claimed by shipping a passing run of the public conformance suite (a set of adversarial fixtures: a task that tries to plant verification files, an envelope that lies about tokens, a trial that must be scored engine-error) and linking the receipts.
+
+## 10. CI integration
+
+The reason ABP should win: it is the only benchmark designed to be run on every pull request, not once per launch.
+
+```yaml
+# .github/workflows/agent-quality.yml
+- run: abp run --config abp-run.json -o results
+- run: abp gate results/run-latest --baseline abp-baseline.json --require-significant
+```
+
+The gate semantics are Arena's, generalized: fail on a statistically significant resolve-rate drop, or on median token, cost, diff-size, or wall-clock growth past committed thresholds; refuse to compare across mismatched task sets. The baseline is a committed JSON artifact, so the PR that degrades the agent goes red the same day, with receipts attached.
+
+## 11. Roadmap
+
+| Phase | Deliverable | Exit criterion |
+|---|---|---|
+| 0 (now) | Arena as reference harness; this draft public at arena.oxagen.sh | Draft survives public review |
+| 1 | Schemas frozen at v1; Arena emits and consumes all four formats; conformance suite published | Two non-Oxagen engines emit valid traces |
+| 2 | Live pipeline pilot: 25-repo registry subset, 100 verified tasks, golden traces published | External team reproduces a published verdict from receipts alone |
+| 3 | Registry at 500 repos; hosted leaderboard with matched-model brackets; GitHub Action in the marketplace | ABP gate running in 100 external repos' CI |
+| 4 | Version 2 exploration: trajectory quality, architectural review, multi-agent orchestration scoring | Community RFC process |
+
+## 12. Open questions
+
+1. Trace redaction: how much tool-argument detail can be published from runs on private code without leaking it? Current answer: conformance requires the redaction list, and private runs may withhold traces while still claiming ABP-core.
+2. Who signs verified tasks? A task's freeze bundle should be content-addressed and signed by the registry, or a poisoned mirror can grade dishonestly.
+3. Diff normalization is specified per-language (formatter canonicalization). The v1 scope is the languages with deterministic formatters; the escape hatch is byte counts on `git diff --numstat` with published exclusions.
+4. Should golden traces ever become the rubric? Version 1 says no. If v2 explores it, trace-similarity grading must never punish a better route to a passing fix.
+
+---
+
+*Feedback: open an issue at [github.com/oxageninc/arena](https://github.com/oxageninc/arena/issues) with the `abp` label.*
diff --git a/site/.gitignore b/site/.gitignore
new file mode 100644
index 0000000..245259b
--- /dev/null
+++ b/site/.gitignore
@@ -0,0 +1,2 @@
+.vercel
+.env*
diff --git a/site/index.html b/site/index.html
new file mode 100644
index 0000000..29df89b
--- /dev/null
+++ b/site/index.html
@@ -0,0 +1,194 @@
+<!DOCTYPE html>
+<html lang="en">
+<head>
+  <meta charset="utf-8" />
+  <meta name="viewport" content="width=device-width, initial-scale=1" />
+  <title>Arena: benchmark the agent engine, not the model</title>
+  <meta name="description" content="Arena runs coding agents head-to-head on the same model, budget, and timeout, grades them with held-out tests, and gates your CI so agent quality never slips silently." />
+  <link rel="stylesheet" href="/style.css" />
+  <link rel="icon" href="data:image/svg+xml,%3Csvg xmlns='http://www.w3.org/2000/svg' viewBox='0 0 100 100'%3E%3Ctext y='.9em' font-size='90'%3E%E2%8F%B1%3C/text%3E%3C/svg%3E" />
+  <meta property="og:title" content="Arena: benchmark the agent engine, not the model" />
+  <meta property="og:description" content="Head-to-head benchmarks for agentic coding CLIs. Held-out verification, matched configs, real statistics, full receipts, and a CI regression gate." />
+  <meta property="og:url" content="https://arena.oxagen.sh" />
+</head>
+<body>
+  <div class="wrap">
+    <header class="site-head">
+      <a class="brand" href="/">arena<span>.oxagen.sh</span></a>
+      <nav>
+        <a href="/paper/">Paper</a>
+        <a href="/spec/">Spec</a>
+        <a href="https://github.com/oxageninc/arena/blob/main/METHODOLOGY.md">Methodology</a>
+        <a href="https://github.com/oxageninc/arena">GitHub</a>
+      </nav>
+    </header>
+  </div>
+
+  <main>
+    <div class="wrap">
+      <div class="hero">
+        <h1>Your agent got worse last Tuesday. Nobody noticed.</h1>
+        <p class="sub">
+          Every leaderboard measures the model. Nobody measures <strong>your agent engine</strong>:
+          the harness, the prompts, the tools, the loop you actually ship. Arena runs coding agents
+          head-to-head on the <strong>same model, same budget, same timeout</strong>, grades them with
+          <strong>held-out tests</strong> they can never see, and reports each metric separately with
+          real statistics and full receipts.
+        </p>
+        <div class="cta-row">
+          <a class="btn solid" href="https://github.com/oxageninc/arena">git clone oxageninc/arena</a>
+          <a class="btn ghost" href="/paper/">Read the paper</a>
+          <a class="btn ghost" href="/spec/">Read the spec</a>
+        </div>
+
+        <div class="scoreboard" role="img" aria-label="Example Arena report: two agents compared on the same model with confidence intervals and a McNemar p-value">
+          <div class="bar">
+            <span class="cmd">arena run --agents oxagen,claude-code --model anthropic/claude-sonnet-5 --trials 3</span>
+            <span>matchedModels: true</span>
+          </div>
+          <div class="sb-body">
+            <div class="sb-row">
+              <span class="name">oxagen</span>
+              <span class="ci"><span class="band" style="left:71%;width:21%"></span><span class="pt" style="left:84%"></span></span>
+              <span class="rate">84% <small>Wilson 95% [71, 92]</small></span>
+            </div>
+            <div class="sb-row">
+              <span class="name">claude-code</span>
+              <span class="ci"><span class="band" style="left:64%;width:24%"></span><span class="pt" style="left:78%"></span></span>
+              <span class="rate">78% <small>Wilson 95% [64, 88]</small></span>
+            </div>
+          </div>
+          <div class="sb-foot">
+            <span class="p">exact McNemar p = 0.041 on paired trials</span>
+            <span>receipts: manifest · transcripts · diffs · reproduce command</span>
+          </div>
+        </div>
+        <p class="sb-note">Report format shown with example data. Every real report carries its own intervals, p-values, and raw run directory.</p>
+      </div>
+
+      <section id="why">
+        <p class="eyebrow">The problem</p>
+        <h2>Model benchmarks answer a question you are not asking</h2>
+        <div class="prose">
+          <p>
+            When your coding agent gets better or worse, the model is usually not what changed.
+            You changed a system prompt, added a tool, reworked the planning loop, or bumped a
+            dependency. Model leaderboards cannot see any of that. They hold the harness fixed
+            and vary the model. Your job is the opposite: the model is a config value, and the
+            engine is the product.
+          </p>
+          <p>
+            Arena isolates the engine. Two or more agents, one model, one budget, one timeout,
+            the same tasks. If the numbers differ, the harness is the reason. That is the
+            comparison that tells you whether your last sprint helped, and the one no public
+            leaderboard runs for you.
+          </p>
+        </div>
+      </section>
+
+      <section id="scrutiny">
+        <p class="eyebrow">Trust</p>
+        <h2>Built to survive skeptical reading</h2>
+        <div class="defense">
+          <div class="row">
+            <div class="attack">The agent graded itself</div>
+            <div class="answer"><strong>Held-out verification.</strong> Tests live outside the workspace, are copied in only after the agent exits, and anything planted at the verify path is deleted first. CI proves every task fails pristine and passes the reference solution.</div>
+          </div>
+          <div class="row">
+            <div class="attack">Different models or budgets</div>
+            <div class="answer"><strong>One config drives every agent.</strong> Unmatched runs are stamped <code>matchedModels: false</code> and the report leads with a warning banner.</div>
+          </div>
+          <div class="row">
+            <div class="attack">Your harness broke the competitor</div>
+            <div class="answer"><strong>Harness failures are excluded.</strong> A CLI that cannot be invoked scores <code>agent-error</code> and is removed from every comparison. It never counts as the agent losing.</div>
+          </div>
+          <div class="row">
+            <div class="attack">n = 1, no stats</div>
+            <div class="answer"><strong>Real statistics.</strong> Wilson 95% intervals, exact McNemar on paired outcomes, seeded paired-bootstrap CIs on speed, tokens, and cost. Same data plus same seed gives the same report, bit for bit.</div>
+          </div>
+          <div class="row">
+            <div class="attack">Where is the raw data</div>
+            <div class="answer"><strong>Full receipts.</strong> Every trial writes its transcript, workspace diff, and a manifest with CLI versions, models, host, seed, and a one-line reproduce command.</div>
+          </div>
+        </div>
+      </section>
+
+      <section id="metrics">
+        <p class="eyebrow">Measurement</p>
+        <h2>Four metrics, never blended</h2>
+        <p>
+          A single blended score hides the tradeoff you care about. Arena reports each metric
+          separately, with its own interval, and refuses to average them.
+        </p>
+        <div class="metrics">
+          <div class="metric">
+            <div class="k">accuracy</div>
+            <div class="v">Held-out tests pass. Wilson 95% interval on every rate.</div>
+          </div>
+          <div class="metric">
+            <div class="k">speed</div>
+            <div class="v">Wall-clock, harness-measured. Medians with bootstrap CIs.</div>
+          </div>
+          <div class="metric">
+            <div class="k">tokens</div>
+            <div class="v">Normalized per adapter. Cache reads never inflate input.</div>
+          </div>
+          <div class="metric">
+            <div class="k">cost</div>
+            <div class="v">One shared price table, or no number at all. Never a guess.</div>
+          </div>
+        </div>
+      </section>
+
+      <section id="gate">
+        <p class="eyebrow">CI integration</p>
+        <h2>The regression gate: a tripwire for agent quality</h2>
+        <div class="prose">
+          <p>
+            Benchmarks are not only for bragging. Snapshot a good run as a baseline, commit it,
+            and fail CI the day a change makes your agent less accurate, slower, or more
+            expensive. With <code>--require-significant</code>, a drop only fails once it clears
+            95% interval noise, so small samples do not cry wolf.
+          </p>
+        </div>
+        <pre class="shell"><span class="c"># establish a baseline once, commit it</span>
+pnpm arena run --agents mine --model anthropic/claude-sonnet-5 --trials 5
+pnpm arena baseline save results/run-…        <span class="c"># writes arena-baseline.json</span>
+
+<span class="c"># in CI: rerun and gate. non-zero exit = regression</span>
+pnpm arena run --agents mine --model anthropic/claude-sonnet-5 --trials 5
+pnpm arena gate results/run-… <span class="y">--require-significant</span></pre>
+      </section>
+
+      <section id="read">
+        <p class="eyebrow">Go deeper</p>
+        <h2>The paper, the spec, and the methodology</h2>
+        <div class="docs">
+          <a class="doc-card" href="/paper/">
+            <div class="t">The State of Agent Benchmarking</div>
+            <div class="d">A survey of the 2026 landscape: what exists, what it measures, and why almost none of it evaluates the engine you ship.</div>
+            <div class="m">research paper</div>
+          </a>
+          <a class="doc-card" href="/spec/">
+            <div class="t">Agent Benchmark Protocol</div>
+            <div class="d">A draft open standard: live tasks from real repos, verified fixes, golden traces, and grading any engine can plug into.</div>
+            <div class="m">design spec · draft</div>
+          </a>
+          <a class="doc-card" href="https://github.com/oxageninc/arena/blob/main/METHODOLOGY.md">
+            <div class="t">Methodology</div>
+            <div class="d">The contract behind any number Arena produces, including what it takes to publish an A-beats-B claim.</div>
+            <div class="m">github.com/oxageninc/arena</div>
+          </a>
+        </div>
+      </section>
+    </div>
+  </main>
+
+  <div class="wrap">
+    <footer class="site-foot">
+      <span>MIT © Mac Anderson · <a href="https://oxagen.sh">Oxagen</a></span>
+      <span><a href="https://github.com/oxageninc/arena">github.com/oxageninc/arena</a></span>
+    </footer>
+  </div>
+</body>
+</html>
diff --git a/site/spec/index.html b/site/spec/index.html
new file mode 100644
index 0000000..b371bd3
--- /dev/null
+++ b/site/spec/index.html
@@ -0,0 +1,351 @@
+<!DOCTYPE html>
+<html lang="en">
+<head>
+  <meta charset="utf-8" />
+  <meta name="viewport" content="width=device-width, initial-scale=1" />
+  <title>Agent Benchmark Protocol (ABP) · Draft 0.1</title>
+  <meta name="description" content="A draft open standard for benchmarking agent engines: task manifests, golden traces, verdicts, and a CI gate any engine can plug into." />
+  <link rel="stylesheet" href="/style.css" />
+  <link rel="icon" href="data:image/svg+xml,%3Csvg xmlns='http://www.w3.org/2000/svg' viewBox='0 0 100 100'%3E%3Ctext y='.9em' font-size='90'%3E%E2%8F%B1%3C/text%3E%3C/svg%3E" />
+</head>
+<body>
+  <div class="wrap">
+    <header class="site-head">
+      <a class="brand" href="/">arena<span>.oxagen.sh</span></a>
+      <nav>
+        <a href="/paper/">Paper</a>
+        <a href="/spec/" aria-current="page">Spec</a>
+        <a href="https://github.com/oxageninc/arena/blob/main/METHODOLOGY.md">Methodology</a>
+        <a href="https://github.com/oxageninc/arena">GitHub</a>
+      </nav>
+    </header>
+
+    <article class="article">
+      <p class="kicker">Design spec · Draft 0.1 · July 2026</p>
+      <h1>Agent Benchmark Protocol</h1>
+      <p class="byline">Oxagen · reference implementation: <a href="https://github.com/oxageninc/arena">Arena</a> · status: draft for public comment</p>
+
+      <div class="abstract">
+        <strong>An open standard for benchmarking agent engines:</strong> the harness, prompts,
+        tools, and loop an agent ships with, measured separately from the model it happens to run
+        on. ABP standardizes four interchange contracts (task manifest, run config, trace, and
+        verdict) plus a live task pipeline in which verified fixes to real GitHub issues become
+        golden traces. Nothing here is final, including the name.
+      </div>
+
+      <nav class="toc" aria-label="Contents">
+        <div class="t">Contents</div>
+        <ol>
+          <li><a href="#problem">The problem this standard exists to solve</a></li>
+          <li><a href="#why-protocol">Why "protocol"</a></li>
+          <li><a href="#principles">Design principles</a></li>
+          <li><a href="#pipeline">The live task pipeline</a></li>
+          <li><a href="#task">Task manifest (abp-task/v1)</a></li>
+          <li><a href="#trace">Trace format (abp-trace/v1)</a></li>
+          <li><a href="#run">Run configuration (abp-run/v1)</a></li>
+          <li><a href="#verdict">Verdict format (abp-verdict/v1)</a></li>
+          <li><a href="#conformance">Conformance levels</a></li>
+          <li><a href="#ci">CI integration</a></li>
+          <li><a href="#roadmap">Roadmap</a></li>
+          <li><a href="#open">Open questions</a></li>
+        </ol>
+      </nav>
+
+      <div class="article-body">
+        <h2 id="problem">1. The problem this standard exists to solve</h2>
+        <p>
+          Teams that ship agents change their engine every week: a new system prompt, a new tool, a
+          reworked planning loop, a different context strategy. None of those changes show up on a
+          model leaderboard, because model leaderboards hold the harness fixed and vary the model.
+          The people who build agent engines have the opposite need: hold the model fixed and vary
+          the engine.
+        </p>
+        <p>Today that comparison is nearly impossible to run honestly:</p>
+        <ol>
+          <li>Every harness invokes agents differently, so "same task" rarely means same prompt, same budget, same timeout, or same autonomy level.</li>
+          <li>Every harness logs differently, so traces cannot be compared across engines.</li>
+          <li>Every harness grades differently, and most let the agent's own output influence the grade.</li>
+          <li>Almost no benchmark measures efficiency. Two agents that both fix a bug are scored as equals even when one shipped a 4-line patch in 90 seconds and the other shipped a 400-line rewrite in 20 minutes.</li>
+          <li>Almost no benchmark runs in CI. A benchmark you run once for a launch tweet cannot catch the Tuesday your agent quietly got worse.</li>
+        </ol>
+        <p>
+          ABP standardizes the four interfaces that make honest engine comparison possible: how a
+          task is packaged, how a run is configured, how a trace is recorded, and how a verdict is
+          reported. Any engine that speaks these formats can be benchmarked by any conforming
+          harness, and any harness can be audited by anyone.
+        </p>
+
+        <h2 id="why-protocol">2. Why "protocol"</h2>
+        <p>
+          We considered "standard", "format", "spec", and "benchmark". Protocol is the right word
+          for the same reason it was right for the Model Context Protocol: the deliverable is a set
+          of interchange contracts between independent parties (task authors, engines, harnesses,
+          graders, and auditors), not a single dataset or a single leaderboard. Datasets and
+          leaderboards are built on top of ABP; they are products, and ABP is the plumbing. It also
+          keeps the family resemblance with the Open Context Protocol, which Oxagen authored and
+          built <a href="https://github.com/oxageninc/stella">Stella</a> on.
+        </p>
+        <p>
+          One naming caution, stated openly: "protocol" sets an expectation of wire-level rigor.
+          This draft earns that word only in Sections 5 through 8, where the schemas live. If the
+          community reads those sections and concludes the word oversells, the fallback name is the
+          Agent Benchmark Format (ABF). The acronym to protect is ABP, one syllable per letter, easy
+          to say in CI logs: <code>abp gate failed</code>.
+        </p>
+
+        <h2 id="principles">3. Design principles</h2>
+        <p>These are inherited from Arena's methodology and are non-negotiable for conformance:</p>
+        <ol>
+          <li><strong>The engine is the unit under test.</strong> The model, provider, budget, and timeout are pinned inputs. Runs that vary the model are stamped as such and cannot back engine-vs-engine claims.</li>
+          <li><strong>The agent never grades itself.</strong> Verification material must be absent from the workspace while the agent runs, injected only after it exits, and immune to anything the agent planted.</li>
+          <li><strong>Every number carries its uncertainty.</strong> Success rates get Wilson intervals. Paired comparisons get exact McNemar. Continuous metrics get seeded bootstrap CIs. No blended scores, ever.</li>
+          <li><strong>Harness failures are not agent failures.</strong> An engine that could not be invoked is excluded from comparison, not counted as a loss.</li>
+          <li><strong>Receipts or it did not happen.</strong> A conforming result ships its manifest, transcripts, diffs, and a one-line reproduce command.</li>
+          <li><strong>Efficiency is a first-class metric.</strong> Resolution alone is a tie. Diff size, tokens, wall clock, and cost break it.</li>
+        </ol>
+
+        <h2 id="pipeline">4. The live task pipeline</h2>
+        <p>This is the part of ABP that no existing benchmark provides end to end.</p>
+
+        <h3>4.1 Sourcing</h3>
+        <p>
+          A curated registry of the top 500 open-source repositories on GitHub, filtered for: OSI
+          license, active maintainership (merged PR in the last 30 days), a runnable test suite, and
+          issue hygiene (reproducible reports, labeled backlog). The registry is versioned and
+          published; entries rotate quarterly so the pool tracks the real ecosystem.
+        </p>
+        <p>
+          From the registry, tasks are drawn by seeded random selection: pick a repo, pick an open
+          issue from its backlog that passes eligibility (reproducible, in-scope for a code change,
+          not a question or a feature debate). The seed is published with every draw, so the
+          selection itself is auditable.
+        </p>
+
+        <h3>4.2 Solving and promotion</h3>
+        <p>
+          A frontier engine (or a human, or both) builds a fix for the issue against the repo's
+          contribution rules and submits it upstream. The fix is <strong>verified</strong> when the
+          project's own CI passes and a maintainer merges it into the production branch. Maintainer
+          merge is the ground truth no synthetic benchmark has: a real reviewer accepted this change
+          into a real codebase.
+        </p>
+        <p>Two artifacts fall out of every verified fix:</p>
+        <ul>
+          <li><strong>Training data.</strong> The (issue, repo state, verified diff) triple.</li>
+          <li><strong>A golden trace.</strong> The full ABP trace (Section 6) the solving engine printed along the way: every tool call, every file read, every edit, every token count, timestamped.</li>
+        </ul>
+
+        <h3>4.3 Replay and grading</h3>
+        <p>
+          Once a task is verified, it freezes: repo pinned at the pre-fix commit, issue text
+          captured, and a verification bundle built from the repo's own test suite plus the tests
+          the verified fix added (the FAIL_TO_PASS set). Other engines then attempt the same frozen
+          task under matched config. They are graded on:
+        </p>
+        <ol>
+          <li><strong>Resolution.</strong> The held-out verification bundle passes.</li>
+          <li><strong>Efficiency, in strict order:</strong> smaller normalized git diff wins, then fewer tokens, then less wall clock, then lower cost.</li>
+        </ol>
+        <p>
+          The golden trace is published as evidence and as a study aid, not as the rubric. Version 1
+          deliberately does not grade trace similarity: an engine that reaches a passing, smaller
+          fix by a different route beats the golden trace's author. Judging architectural quality
+          and trajectory quality is a stated version 2 direction, and it needs its own adversarial
+          review before it can be a metric.
+        </p>
+
+        <h3>4.4 Contamination control</h3>
+        <p>
+          Live sourcing is also the contamination story. Issues are selected from the current
+          backlog, after every candidate model's training cutoff, and each task carries a
+          <code>sourcedAt</code> timestamp plus the model cutoffs it was verified against. Tasks age
+          out of the scored set once their fix has been public for 180 days; they remain available
+          as a practice set. Frozen-set benchmarks decay into training data. A pipeline that keeps
+          drawing from the living backlog does not.
+        </p>
+
+        <h3>4.5 Known objections, answered in the design</h3>
+        <ul>
+          <li><strong>"Smallest diff is gameable."</strong> Diff size is measured on a normalized form: formatting-only changes are canonicalized before counting, generated files and lockfiles are excluded, and test files are counted separately (deleting or weakening tests disqualifies the run, it does not shrink the diff). Resolution is still the gate; diff size only ranks engines that already passed. And the metric is honest about what it is: a proxy for surgical precision, not for design quality. Arena publishes the raw diffs, so anyone can audit whether small meant surgical or small meant degenerate.</li>
+          <li><strong>"Maintainers did not consent to being a benchmark."</strong> Sourcing follows each repo's contribution guidelines, fixes are submitted as normal PRs with disclosure, registry entries can opt out, and replay runs happen on frozen clones, never against the live repo.</li>
+          <li><strong>"Issue selection will favor easy tasks."</strong> Selection is seeded-random within eligibility, the seed is published, and difficulty is stratified in reporting (task metadata records diff size and files touched of the verified fix).</li>
+        </ul>
+
+        <h2 id="task">5. Task manifest (abp-task/v1)</h2>
+        <pre><code>{
+  "schema": "abp-task/v1",
+  "id": "gh-vercel-next.js-81234",
+  "source": {
+    "kind": "github-issue",              // or "fixture" for synthetic tasks
+    "repo": "vercel/next.js",
+    "issue": 81234,
+    "baseCommit": "9f2c41d…",            // repo frozen here (pre-fix)
+    "sourcedAt": "2026-07-02T14:11:09Z",
+    "registryVersion": "abp-registry/2026Q3",
+    "drawSeed": 424242
+  },
+  "prompt": "…full task statement given to every engine…",
+  "environment": {
+    "image": "ghcr.io/oxageninc/abp-node:22",
+    "setup": ["pnpm install --frozen-lockfile"]
+  },
+  "verification": {
+    "kind": "commands",
+    "failToPass": ["pnpm test -- test/router/edge-case.test.ts"],
+    "passToPass": ["pnpm test -- test/router/"],
+    "timeoutSeconds": 900,
+    "heldOut": true
+  },
+  "golden": {
+    "diffBytesNormalized": 1843,
+    "filesTouched": 2,
+    "trace": "traces/gh-vercel-next.js-81234.abp-trace.jsonl",
+    "mergedPr": "https://github.com/vercel/next.js/pull/81301"
+  },
+  "cutoffsVerified": ["claude-sonnet-5:2026-01", "gpt-5.2:2026-03"]
+}</code></pre>
+        <p>
+          Synthetic fixtures (like Arena's built-in tasks) use the same schema with
+          <code>source.kind: "fixture"</code> and no <code>golden</code> block. A harness must treat
+          both identically at run time.
+        </p>
+
+        <h2 id="trace">6. Trace format (abp-trace/v1)</h2>
+        <p>One JSONL file per (task, engine, trial). Every line is an event:</p>
+        <pre><code>{"t":"2026-07-02T14:12:01.402Z","seq":1,"kind":"run.start",
+ "engine":"oxagen/1.4.2","model":"anthropic/claude-sonnet-5",
+ "task":"gh-vercel-next.js-81234","config":{"budgetUsd":5,"timeoutSeconds":1200}}
+{"t":"…","seq":2,"kind":"model.call",
+ "tokens":{"input":1204,"output":388,"cacheRead":9120,"cacheWrite":0}}
+{"t":"…","seq":3,"kind":"tool.call","tool":"read_file",
+ "args":{"path":"src/router/match.ts"},"durationMs":12}
+{"t":"…","seq":4,"kind":"tool.call","tool":"edit_file",
+ "args":{"path":"src/router/match.ts"},"diffBytes":412}
+{"t":"…","seq":5,"kind":"subagent.start","id":"sa-1","purpose":"run tests"}
+{"t":"…","seq":6,"kind":"run.end","outcome":"resolved","wallClockMs":184201,
+ "tokens":{"input":48210,"output":9114,"cacheRead":301200,"cacheWrite":8100},
+ "diffBytesNormalized":1610,"filesTouched":2}</code></pre>
+        <p>Rules:</p>
+        <ul>
+          <li><code>seq</code> is a strictly increasing integer; <code>t</code> is RFC 3339 UTC. Together they make traces diffable and mergeable.</li>
+          <li><code>tool.call</code> events record the tool name and a redacted argument summary, never raw secrets. A published redaction list is part of conformance.</li>
+          <li>Multi-agent engines emit <code>subagent.start</code> / <code>subagent.end</code> with nested attribution, so a fan-out engine's token bill is visible per branch. Multi-model engines record <code>model</code> per <code>model.call</code>. This is how ABP stays meaningful for orchestrators, not just single-loop CLIs.</li>
+          <li>Token counts follow Arena's normalization: <code>input</code> never includes cache reads; cache reads and writes are tracked separately.</li>
+          <li>An engine that cannot emit ABP traces natively can ship a sidecar translator; the harness records which path produced the trace.</li>
+        </ul>
+
+        <h2 id="run">7. Run configuration (abp-run/v1)</h2>
+        <p>The manifest a harness must write before the first trial:</p>
+        <pre><code>{
+  "schema": "abp-run/v1",
+  "engines": [{"name": "oxagen", "version": "1.4.2"},
+              {"name": "claude-code", "version": "2.1.0"}],
+  "model": "anthropic/claude-sonnet-5",
+  "matchedModels": true,
+  "budgetUsd": 5,
+  "timeoutSeconds": 1200,
+  "trials": 3,
+  "ordering": "abba",
+  "seed": 20260718,
+  "tasks": ["gh-vercel-next.js-81234", "…"],
+  "host": {"os": "linux", "arch": "arm64", "containerized": true},
+  "reproduce": "abp run --config run.json"
+}</code></pre>
+        <p>
+          <code>matchedModels: false</code> runs are legal (you may want to test your engine across
+          models) but a conforming report must lead with the warning and must not present
+          engine-vs-engine conclusions from them.
+        </p>
+
+        <h2 id="verdict">8. Verdict format (abp-verdict/v1)</h2>
+        <pre><code>{
+  "schema": "abp-verdict/v1",
+  "run": "run-20260718-a41c",
+  "perEngine": [{
+    "engine": "oxagen",
+    "scoredTrials": 72,
+    "excludedEngineErrors": 1,
+    "resolveRate": {"point": 0.84, "wilson95": [0.71, 0.92]},
+    "medianDiffBytesNormalized": 1650,
+    "medianTokensTotal": 361000,
+    "medianWallClockMs": 190334,
+    "medianComputedUsd": 1.42
+  }],
+  "pairwise": [{
+    "a": "oxagen", "b": "claude-code",
+    "mcnemar": {"discordant": [14, 5], "pExact": 0.041},
+    "diffDelta": {"relMedian": -0.22, "bootstrap95": [-0.31, -0.09],
+                  "seed": 20260718}
+  }],
+  "receipts": {"trials": "trials/", "transcripts": "transcripts/",
+               "diffs": "diffs/", "traces": "traces/"}
+}</code></pre>
+
+        <h2 id="conformance">9. Conformance levels</h2>
+        <ul>
+          <li><strong>ABP-core.</strong> The harness consumes <code>abp-task/v1</code>, enforces held-out verification and matched config, and emits <code>abp-run/v1</code> + <code>abp-verdict/v1</code> with the required statistics. Arena today is one file-format migration away from ABP-core.</li>
+          <li><strong>ABP-traces.</strong> Everything in core, plus <code>abp-trace/v1</code> emission for every trial, with subagent and multi-model attribution.</li>
+          <li><strong>ABP-live.</strong> Everything in traces, plus participation in the live task pipeline: consuming registry draws, contributing verified fixes, and publishing golden traces.</li>
+        </ul>
+        <p>
+          Conformance is claimed by shipping a passing run of the public conformance suite (a set of
+          adversarial fixtures: a task that tries to plant verification files, an envelope that lies
+          about tokens, a trial that must be scored engine-error) and linking the receipts.
+        </p>
+
+        <h2 id="ci">10. CI integration</h2>
+        <p>
+          The reason ABP should win: it is the only benchmark designed to be run on every pull
+          request, not once per launch.
+        </p>
+        <pre><code># .github/workflows/agent-quality.yml
+- run: abp run --config abp-run.json -o results
+- run: abp gate results/run-latest --baseline abp-baseline.json --require-significant</code></pre>
+        <p>
+          The gate semantics are Arena's, generalized: fail on a statistically significant
+          resolve-rate drop, or on median token, cost, diff-size, or wall-clock growth past
+          committed thresholds; refuse to compare across mismatched task sets. The baseline is a
+          committed JSON artifact, so the PR that degrades the agent goes red the same day, with
+          receipts attached.
+        </p>
+
+        <h2 id="roadmap">11. Roadmap</h2>
+        <div class="tablewrap">
+        <table>
+          <thead><tr><th>Phase</th><th>Deliverable</th><th>Exit criterion</th></tr></thead>
+          <tbody>
+            <tr><td>0 (now)</td><td>Arena as reference harness; this draft public at arena.oxagen.sh</td><td>Draft survives public review</td></tr>
+            <tr><td>1</td><td>Schemas frozen at v1; Arena emits and consumes all four formats; conformance suite published</td><td>Two non-Oxagen engines emit valid traces</td></tr>
+            <tr><td>2</td><td>Live pipeline pilot: 25-repo registry subset, 100 verified tasks, golden traces published</td><td>External team reproduces a published verdict from receipts alone</td></tr>
+            <tr><td>3</td><td>Registry at 500 repos; hosted leaderboard with matched-model brackets; GitHub Action in the marketplace</td><td>ABP gate running in 100 external repos' CI</td></tr>
+            <tr><td>4</td><td>Version 2 exploration: trajectory quality, architectural review, multi-agent orchestration scoring</td><td>Community RFC process</td></tr>
+          </tbody>
+        </table>
+        </div>
+
+        <h2 id="open">12. Open questions</h2>
+        <ol>
+          <li>Trace redaction: how much tool-argument detail can be published from runs on private code without leaking it? Current answer: conformance requires the redaction list, and private runs may withhold traces while still claiming ABP-core.</li>
+          <li>Who signs verified tasks? A task's freeze bundle should be content-addressed and signed by the registry, or a poisoned mirror can grade dishonestly.</li>
+          <li>Diff normalization is specified per-language (formatter canonicalization). The v1 scope is the languages with deterministic formatters; the escape hatch is byte counts on <code>git diff --numstat</code> with published exclusions.</li>
+          <li>Should golden traces ever become the rubric? Version 1 says no. If v2 explores it, trace-similarity grading must never punish a better route to a passing fix.</li>
+        </ol>
+
+        <p class="muted">
+          Feedback: open an issue at
+          <a href="https://github.com/oxageninc/arena/issues">github.com/oxageninc/arena</a> with the
+          <code>abp</code> label. The markdown source of this spec lives in the repo at
+          <code>docs/agent-benchmark-protocol.md</code>.
+        </p>
+      </div>
+    </article>
+  </div>
+
+  <div class="wrap">
+    <footer class="site-foot">
+      <span>MIT © Mac Anderson · <a href="https://oxagen.sh">Oxagen</a></span>
+      <span><a href="https://github.com/oxageninc/arena">github.com/oxageninc/arena</a></span>
+    </footer>
+  </div>
+</body>
+</html>
diff --git a/site/style.css b/site/style.css
new file mode 100644
index 0000000..565dc55
--- /dev/null
+++ b/site/style.css
@@ -0,0 +1,691 @@
+/* Arena — arena.oxagen.sh
+   Light "evidence" page, one dark scoreboard signature element.
+   Mono display (transcripts, diffs, receipts are the brand material). */
+
+:root {
+  --paper: #f7f8f7;
+  --ink: #14171a;
+  --muted: #5c6670;
+  --line: #dde1de;
+  --panel: #101317;
+  --panel-line: #262c33;
+  --amber: #ffb020;
+  --amber-dim: #b98217;
+  --pass: #0b7a3e;
+  --fail: #c6362c;
+  --link: #0b5aa2;
+  --mono: ui-monospace, "SF Mono", SFMono-Regular, Menlo, Consolas, monospace;
+  --sans: -apple-system, BlinkMacSystemFont, "Segoe UI", Roboto, Helvetica,
+    Arial, sans-serif;
+  --serif: Charter, "Iowan Old Style", Georgia, Cambria, serif;
+}
+
+@media (prefers-color-scheme: dark) {
+  :root {
+    --paper: #0e1114;
+    --ink: #e8eaec;
+    --muted: #98a2ab;
+    --line: #232930;
+    --panel: #090b0d;
+    --panel-line: #232930;
+    --link: #6db3f2;
+  }
+}
+
+* {
+  margin: 0;
+  padding: 0;
+  box-sizing: border-box;
+}
+
+html {
+  scroll-behavior: smooth;
+}
+
+@media (prefers-reduced-motion: reduce) {
+  html {
+    scroll-behavior: auto;
+  }
+  * {
+    animation: none !important;
+    transition: none !important;
+  }
+}
+
+body {
+  background: var(--paper);
+  color: var(--ink);
+  font-family: var(--sans);
+  font-size: 17px;
+  line-height: 1.6;
+  -webkit-font-smoothing: antialiased;
+}
+
+a {
+  color: var(--link);
+  text-decoration-thickness: 1px;
+  text-underline-offset: 3px;
+}
+
+a:focus-visible,
+button:focus-visible {
+  outline: 2px solid var(--link);
+  outline-offset: 3px;
+  border-radius: 2px;
+}
+
+.wrap {
+  max-width: 880px;
+  margin: 0 auto;
+  padding: 0 24px;
+}
+
+/* ---------- header ---------- */
+
+.site-head {
+  display: flex;
+  align-items: baseline;
+  justify-content: space-between;
+  gap: 16px;
+  padding: 28px 0 0;
+  font-family: var(--mono);
+  font-size: 14px;
+}
+
+.site-head .brand {
+  color: var(--ink);
+  text-decoration: none;
+  font-weight: 700;
+  letter-spacing: 0.02em;
+}
+
+.site-head .brand span {
+  color: var(--muted);
+  font-weight: 400;
+}
+
+.site-head nav {
+  display: flex;
+  gap: 20px;
+  flex-wrap: wrap;
+}
+
+.site-head nav a {
+  color: var(--muted);
+  text-decoration: none;
+}
+
+.site-head nav a:hover,
+.site-head nav a[aria-current="page"] {
+  color: var(--ink);
+}
+
+/* ---------- hero ---------- */
+
+.hero {
+  padding: 88px 0 40px;
+}
+
+.hero h1 {
+  font-family: var(--mono);
+  font-weight: 700;
+  font-size: clamp(30px, 5.4vw, 52px);
+  line-height: 1.12;
+  letter-spacing: -0.015em;
+  max-width: 21ch;
+}
+
+.hero .sub {
+  margin-top: 24px;
+  max-width: 58ch;
+  font-size: 19px;
+  color: var(--muted);
+}
+
+.hero .sub strong {
+  color: var(--ink);
+  font-weight: 600;
+}
+
+.cta-row {
+  display: flex;
+  gap: 14px;
+  flex-wrap: wrap;
+  margin-top: 32px;
+  font-family: var(--mono);
+  font-size: 14px;
+}
+
+.btn {
+  display: inline-block;
+  padding: 11px 18px;
+  border-radius: 6px;
+  text-decoration: none;
+  border: 1px solid var(--ink);
+}
+
+.btn.solid {
+  background: var(--ink);
+  color: var(--paper);
+}
+
+.btn.ghost {
+  color: var(--ink);
+}
+
+.btn:hover {
+  transform: translateY(-1px);
+}
+
+/* ---------- scoreboard (signature) ---------- */
+
+.scoreboard {
+  margin: 48px 0 8px;
+  background: var(--panel);
+  border: 1px solid var(--panel-line);
+  border-radius: 10px;
+  overflow: hidden;
+  font-family: var(--mono);
+  color: #cfd6dd;
+  box-shadow: 0 24px 60px -32px rgb(0 0 0 / 0.5);
+}
+
+.scoreboard .bar {
+  display: flex;
+  justify-content: space-between;
+  gap: 12px;
+  padding: 12px 18px;
+  border-bottom: 1px solid var(--panel-line);
+  font-size: 12.5px;
+  color: #8a949e;
+  flex-wrap: wrap;
+}
+
+.scoreboard .bar .cmd {
+  color: #cfd6dd;
+  overflow-wrap: anywhere;
+}
+
+.scoreboard .bar .cmd::before {
+  content: "$ ";
+  color: var(--amber);
+}
+
+.sb-body {
+  padding: 20px 18px 8px;
+  overflow-x: auto;
+}
+
+.sb-row {
+  display: grid;
+  grid-template-columns: 130px 1fr 150px;
+  gap: 14px;
+  align-items: center;
+  padding: 10px 0;
+  min-width: 520px;
+}
+
+.sb-row .name {
+  font-size: 14px;
+  color: #e8eaec;
+}
+
+.sb-row .rate {
+  font-size: 14px;
+  color: var(--amber);
+  text-align: right;
+  white-space: nowrap;
+}
+
+.sb-row .rate small {
+  color: #8a949e;
+  font-size: 11.5px;
+  display: block;
+}
+
+.ci {
+  position: relative;
+  height: 14px;
+  background: #1b2129;
+  border-radius: 7px;
+}
+
+.ci .band {
+  position: absolute;
+  top: 3px;
+  bottom: 3px;
+  background: rgb(255 176 32 / 0.28);
+  border-radius: 4px;
+}
+
+.ci .pt {
+  position: absolute;
+  top: 0;
+  bottom: 0;
+  width: 3px;
+  background: var(--amber);
+  border-radius: 2px;
+}
+
+.sb-foot {
+  display: flex;
+  justify-content: space-between;
+  gap: 12px;
+  flex-wrap: wrap;
+  padding: 12px 18px 14px;
+  border-top: 1px solid var(--panel-line);
+  font-size: 12.5px;
+  color: #8a949e;
+}
+
+.sb-foot .p {
+  color: #cfd6dd;
+}
+
+.sb-note {
+  font-family: var(--mono);
+  font-size: 12.5px;
+  color: var(--muted);
+  margin-bottom: 56px;
+}
+
+/* ---------- sections ---------- */
+
+section {
+  padding: 56px 0 8px;
+}
+
+.eyebrow {
+  font-family: var(--mono);
+  font-size: 12.5px;
+  letter-spacing: 0.08em;
+  text-transform: uppercase;
+  color: var(--muted);
+  margin-bottom: 10px;
+}
+
+section h2 {
+  font-family: var(--mono);
+  font-size: clamp(22px, 3vw, 30px);
+  font-weight: 700;
+  letter-spacing: -0.01em;
+  line-height: 1.2;
+  max-width: 30ch;
+}
+
+section > .wrap > p,
+section .prose p {
+  margin-top: 18px;
+  max-width: 66ch;
+}
+
+section .prose p + p {
+  margin-top: 14px;
+}
+
+.muted {
+  color: var(--muted);
+}
+
+/* defense table */
+
+.defense {
+  margin-top: 28px;
+  border-top: 1px solid var(--line);
+}
+
+.defense .row {
+  display: grid;
+  grid-template-columns: minmax(180px, 2fr) 3fr;
+  gap: 8px 28px;
+  padding: 16px 0;
+  border-bottom: 1px solid var(--line);
+}
+
+.defense .attack {
+  font-family: var(--mono);
+  font-size: 14px;
+  font-weight: 600;
+}
+
+.defense .attack::before {
+  content: "“";
+  color: var(--muted);
+}
+
+.defense .attack::after {
+  content: "”";
+  color: var(--muted);
+}
+
+.defense .answer {
+  font-size: 15.5px;
+  color: var(--muted);
+}
+
+.defense .answer strong {
+  color: var(--ink);
+  font-weight: 600;
+}
+
+@media (max-width: 640px) {
+  .defense .row {
+    grid-template-columns: 1fr;
+  }
+}
+
+/* metric cards */
+
+.metrics {
+  display: grid;
+  grid-template-columns: repeat(auto-fit, minmax(190px, 1fr));
+  gap: 14px;
+  margin-top: 28px;
+}
+
+.metric {
+  border: 1px solid var(--line);
+  border-radius: 8px;
+  padding: 16px;
+}
+
+.metric .k {
+  font-family: var(--mono);
+  font-size: 13px;
+  font-weight: 700;
+}
+
+.metric .v {
+  margin-top: 6px;
+  font-size: 14px;
+  color: var(--muted);
+}
+
+/* code block */
+
+pre.shell {
+  margin-top: 24px;
+  background: var(--panel);
+  color: #cfd6dd;
+  border: 1px solid var(--panel-line);
+  border-radius: 10px;
+  padding: 18px;
+  font-family: var(--mono);
+  font-size: 13.5px;
+  line-height: 1.7;
+  overflow-x: auto;
+}
+
+pre.shell .c {
+  color: #6f7a84;
+}
+
+pre.shell .y {
+  color: var(--amber);
+}
+
+/* doc cards */
+
+.docs {
+  display: grid;
+  grid-template-columns: repeat(auto-fit, minmax(240px, 1fr));
+  gap: 14px;
+  margin-top: 28px;
+}
+
+.doc-card {
+  display: block;
+  border: 1px solid var(--line);
+  border-radius: 8px;
+  padding: 18px;
+  text-decoration: none;
+  color: var(--ink);
+}
+
+.doc-card:hover {
+  border-color: var(--muted);
+}
+
+.doc-card .t {
+  font-family: var(--mono);
+  font-size: 14.5px;
+  font-weight: 700;
+}
+
+.doc-card .d {
+  margin-top: 8px;
+  font-size: 14.5px;
+  color: var(--muted);
+}
+
+.doc-card .m {
+  margin-top: 12px;
+  font-family: var(--mono);
+  font-size: 12px;
+  color: var(--muted);
+}
+
+/* ---------- footer ---------- */
+
+.site-foot {
+  margin-top: 88px;
+  border-top: 1px solid var(--line);
+  padding: 28px 0 48px;
+  font-family: var(--mono);
+  font-size: 13px;
+  color: var(--muted);
+  display: flex;
+  justify-content: space-between;
+  gap: 16px;
+  flex-wrap: wrap;
+}
+
+.site-foot a {
+  color: var(--muted);
+}
+
+/* ---------- article pages (paper, spec) ---------- */
+
+.article {
+  padding: 64px 0 24px;
+}
+
+.article .kicker {
+  font-family: var(--mono);
+  font-size: 12.5px;
+  letter-spacing: 0.08em;
+  text-transform: uppercase;
+  color: var(--muted);
+}
+
+.article h1 {
+  margin-top: 14px;
+  font-family: var(--mono);
+  font-weight: 700;
+  font-size: clamp(26px, 4vw, 40px);
+  line-height: 1.15;
+  letter-spacing: -0.015em;
+  max-width: 26ch;
+}
+
+.article .byline {
+  margin-top: 18px;
+  font-family: var(--mono);
+  font-size: 13px;
+  color: var(--muted);
+}
+
+.article .abstract {
+  margin-top: 28px;
+  padding: 20px 22px;
+  border: 1px solid var(--line);
+  border-radius: 8px;
+  font-size: 16px;
+  color: var(--muted);
+  max-width: 72ch;
+}
+
+.article .abstract strong {
+  color: var(--ink);
+}
+
+.article-body {
+  font-family: var(--serif);
+  font-size: 18px;
+  line-height: 1.72;
+  max-width: 72ch;
+  padding-bottom: 40px;
+}
+
+.article-body h2 {
+  font-family: var(--mono);
+  font-size: 22px;
+  font-weight: 700;
+  letter-spacing: -0.01em;
+  margin: 52px 0 4px;
+  line-height: 1.3;
+}
+
+.article-body h3 {
+  font-family: var(--mono);
+  font-size: 16.5px;
+  font-weight: 700;
+  margin: 36px 0 0;
+}
+
+.article-body p {
+  margin-top: 16px;
+}
+
+.article-body ul,
+.article-body ol {
+  margin: 16px 0 0 26px;
+}
+
+.article-body li {
+  margin-top: 8px;
+}
+
+.article-body li::marker {
+  color: var(--muted);
+  font-family: var(--mono);
+  font-size: 14px;
+}
+
+.article-body code {
+  font-family: var(--mono);
+  font-size: 0.82em;
+  background: rgb(127 127 127 / 0.12);
+  padding: 2px 5px;
+  border-radius: 4px;
+}
+
+.article-body pre {
+  margin-top: 18px;
+  background: var(--panel);
+  color: #cfd6dd;
+  border: 1px solid var(--panel-line);
+  border-radius: 10px;
+  padding: 18px;
+  font-family: var(--mono);
+  font-size: 13px;
+  line-height: 1.65;
+  overflow-x: auto;
+}
+
+.article-body pre code {
+  background: none;
+  padding: 0;
+  font-size: 1em;
+}
+
+.article-body blockquote {
+  margin-top: 18px;
+  padding-left: 18px;
+  border-left: 3px solid var(--line);
+  color: var(--muted);
+}
+
+.article-body table {
+  margin-top: 20px;
+  border-collapse: collapse;
+  width: 100%;
+  font-family: var(--sans);
+  font-size: 14.5px;
+}
+
+.article-body .tablewrap {
+  overflow-x: auto;
+  margin-top: 20px;
+}
+
+.article-body .tablewrap table {
+  margin-top: 0;
+  min-width: 640px;
+}
+
+.article-body th {
+  text-align: left;
+  font-family: var(--mono);
+  font-size: 12.5px;
+  text-transform: uppercase;
+  letter-spacing: 0.05em;
+  color: var(--muted);
+  border-bottom: 1px solid var(--ink);
+  padding: 8px 12px 8px 0;
+  vertical-align: bottom;
+}
+
+.article-body td {
+  border-bottom: 1px solid var(--line);
+  padding: 10px 12px 10px 0;
+  vertical-align: top;
+}
+
+.article-body .refs {
+  font-family: var(--sans);
+  font-size: 14.5px;
+  color: var(--muted);
+}
+
+.article-body .refs li {
+  margin-top: 10px;
+  overflow-wrap: anywhere;
+}
+
+.article-body sup a {
+  text-decoration: none;
+  font-family: var(--mono);
+  font-size: 0.72em;
+}
+
+.toc {
+  margin-top: 28px;
+  padding: 18px 22px;
+  border: 1px solid var(--line);
+  border-radius: 8px;
+  font-family: var(--mono);
+  font-size: 13.5px;
+  max-width: 72ch;
+}
+
+.toc .t {
+  font-weight: 700;
+  margin-bottom: 10px;
+}
+
+.toc ol {
+  margin-left: 20px;
+}
+
+.toc li {
+  margin-top: 6px;
+}
+
+.toc a {
+  text-decoration: none;
+}
+
+.toc a:hover {
+  text-decoration: underline;
+}
diff --git a/site/vercel.json b/site/vercel.json
new file mode 100644
index 0000000..530b6a0
--- /dev/null
+++ b/site/vercel.json
@@ -0,0 +1,14 @@
+{
+  "$schema": "https://openapi.vercel.sh/vercel.json",
+  "cleanUrls": true,
+  "trailingSlash": true,
+  "headers": [
+    {
+      "source": "/(.*)",
+      "headers": [
+        { "key": "X-Content-Type-Options", "value": "nosniff" },
+        { "key": "Referrer-Policy", "value": "strict-origin-when-cross-origin" }
+      ]
+    }
+  ]
+}

From 79caa418f7ae1c527f6a1f7ff168a489db5a81a1 Mon Sep 17 00:00:00 2001
From: macanderson <mac@oxagen.sh>
Date: Sat, 18 Jul 2026 19:39:15 -0700
Subject: [PATCH 3/4] docs: The State of Agent Benchmarking (paper) + site
 rendering

Research paper surveying the July 2026 agent-benchmark landscape through
the engine-vs-model lens: 28 cited references, gap analysis of four
capabilities (matched-model isolation, efficiency grading, statistical
CI gating, live tasks with golden traces), COI disclosed. Markdown
source in docs/, rendered at arena.oxagen.sh/paper/. README links the
site, paper, and spec.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---
 README.md                            |   2 +
 docs/agent-engine-benchmarks-2026.md | 114 +++++++++
 site/paper/index.html                | 335 +++++++++++++++++++++++++++
 3 files changed, 451 insertions(+)
 create mode 100644 docs/agent-engine-benchmarks-2026.md
 create mode 100644 site/paper/index.html

diff --git a/README.md b/README.md
index 35369ab..4a20c90 100644
--- a/README.md
+++ b/README.md
@@ -6,6 +6,8 @@
 
 **Head-to-head benchmarks for agentic coding CLIs — built to survive scrutiny.**
 
+Website: [arena.oxagen.sh](https://arena.oxagen.sh) · Paper: [The State of Agent Benchmarking](docs/agent-engine-benchmarks-2026.md) · Spec: [Agent Benchmark Protocol (draft)](docs/agent-benchmark-protocol.md)
+
 Arena runs two or more coding agents (Claude Code, Gemini CLI, [Oxagen](https://oxagen.sh), [Stella](https://github.com/oxageninc/stella), or your own) on the **same tasks, same model, same budget, same timeout**, grades them with **held-out tests the agent can never see or author**, and reports each metric separately with real statistics and full receipts.
 
 ```bash
diff --git a/docs/agent-engine-benchmarks-2026.md b/docs/agent-engine-benchmarks-2026.md
new file mode 100644
index 0000000..515ecd2
--- /dev/null
+++ b/docs/agent-engine-benchmarks-2026.md
@@ -0,0 +1,114 @@
+# The State of Agent Benchmarking
+
+## Everyone measures the model. Almost nobody measures the engine you ship.
+
+**Mac Anderson (Oxagen) · July 2026 · v1.0**
+
+*Conflict of interest, stated up front: the author builds Arena, an open-source agent-engine benchmark harness, and the Oxagen and Stella agents that appear in its examples. This paper argues a position. Every load-bearing claim carries a citation, and the gap analysis in Section 6 names the systems that already do parts of what Arena does.*
+
+---
+
+## Abstract
+
+Teams that ship AI agents change their engine every week: prompts, tools, planning loops, context strategies, and orchestration. The model changes a few times a year. Yet nearly every public benchmark ranks models with the harness held fixed, which is the exact inverse of the question a working team needs answered: did my last change make my agent better or worse, on the model I already run? Recent controlled evidence says this inversion is not cosmetic. In a 3x3 factorial study on SWE-bench Verified, harness-induced variance exceeded model-induced variance by 7.8x, and model rankings reversed in six of nine harness pairings [3]. Two 2026 surveys independently name model-vs-harness conflation as an open methodological gap [1, 2]. This paper surveys the benchmark landscape as of July 2026 through that lens: what each system actually measures, whether it can isolate the engine from the model, whether it grades efficiency, whether it can run in CI as a regression gate, and whether its tasks stay fresh. We find that no existing system combines matched-model engine isolation, efficiency grading (diff size, tokens, wall clock, cost), statistically honest CI gating, and live task sourcing with verified fixes as reference traces. We close by describing that missing system and an interchange standard, the Agent Benchmark Protocol, that would let any harness provide it.
+
+## 1. The question benchmarks answer, and the question teams ask
+
+A model leaderboard answers: given a fixed scaffold, which model scores highest? That is the right question for a lab choosing a base model, and the wrong question for everyone downstream. Downstream, the model is a config value. The product is the engine: the system prompt, the tool set, the retrieval and context policy, the planning loop, the subagent topology, and the guardrails. One survey puts the distinction plainly: evaluating the LLM alone is like examining an engine on a stand, while agent evaluation must assess the whole car under driving conditions [2].
+
+The scale of what the scaffold contributes has been measured repeatedly. SWE-agent showed in 2024 that the agent-computer interface alone moves resolve rates at a fixed model [24]. Agentless showed the same year that a deliberately non-agentic pipeline could outperform elaborate agents at a fraction of the cost [25]. And in 2026 a controlled factorial study quantified it: across three models and three harness configurations on a 100-task SWE-bench Verified subset, the harness contributed 18.48 pp² of score variance against the model's 2.37 pp², a ratio of 7.8x, with six model-ranking reversals across nine harness pairings [3]. The authors' conclusion is the thesis of this paper: the execution harness, the infrastructure layer that governs context construction, tool interaction, orchestration, and verification, is often a stronger determinant of agent performance than the model it wraps [3].
+
+The benchmark ecosystem has not caught up. The most comprehensive survey of agent evaluation to date lists "Decoupling LLM & Harness Evaluation" as future work: most current benchmarks conflate the two targets, and no established benchmark provides controlled protocols that vary each factor independently [1]. The same survey finds cost and efficiency metrics broadly neglected, echoing the argument of "AI Agents That Matter" that accuracy-only leaderboards drive needlessly costly agents [1, 23].
+
+## 2. Survey: coding-agent benchmarks
+
+**SWE-bench** (Princeton/Stanford, 2023) established the template: 2,294 real GitHub issues from 12 Python repos, graded by the repo's own FAIL_TO_PASS and PASS_TO_PASS tests [4]. **SWE-bench Verified** (OpenAI, 2024) human-filtered 500 instances to remove impossible or underspecified tasks [5]. **SWE-bench Multimodal** adds visual, JavaScript-centric issues; **SWE-bench Pro** (Scale AI, 2025) raises difficulty with long-horizon, enterprise-style tasks and a held-out commercial subset [6]. **SWE-bench Live** (Microsoft Research and community, 2025) attacks contamination directly with a monthly-refreshed feed of new issues [7]. All of these grade one thing: did the repo's tests pass. None grades diff size, token spend, or wall clock as a ranked metric, and their leaderboards mix (model, scaffold) pairs, so a score is attributable to neither alone. They are, however, the raw material engine benchmarking needs: real repos, real issues, executable verification.
+
+**Terminal-Bench** (Stanford and the Laude Institute, 2025) measures terminal-native task completion in containers; its 2.0 release ships on **Harbor** [8, 9]. Harbor deserves its own entry: it is a meta-harness from the Terminal-Bench creators for "evaluating and optimizing agents and language models", wrapping third-party benchmarks (Terminal-Bench 2.0, SWE-bench, Aider Polyglot) behind one containerized runner, with the agent scaffold (Claude Code, OpenHands, Codex CLI, and more) selected independently of the `--model` flag [9]. That structure enables matched-model harness comparison, and a 2026 survey names Harbor (with Exgentic) as the first frameworks pushing a unified protocol for general agent assessment [1]. What Harbor does not do: rank efficiency, gate CI on regressions with statistics, or source fresh tasks with reference traces. **Harness-Bench** (2026) is the academic complement: a 6-harness by 8-model factorial over 106 sandboxed tasks (5,194 trajectories) that fixes task, sandbox, budget, timeout, and evaluator while letting each harness keep its native behavior [10]. It is a study rather than an ongoing service, but it is the cleanest matched-model prior art to date, and this paper leans on its design.
+
+**Aider Polyglot** (225 hard Exercism exercises) is honest about being a model benchmark: one fixed harness (Aider), many models, with cost per run displayed [11]. **Commit0** (build a library from scratch against a spec and tests) and **RepoBench** (repo-level completion) sit closer to the model end. **SWE-Gym** and **R2E** are environment generators, aimed at training rather than refereeing [12].
+
+## 3. Survey: general agent benchmarks
+
+**AgentBench** (2023) spans eight environments from OS shells to web shopping [13]. **GAIA** (Meta/HF, 2023) asks 466 tool-requiring questions with unambiguous answers [14]. **WebArena** and **VisualWebArena** grade functional task completion on self-hosted websites [15]; **OSWorld** does the same for 369 tasks on real desktop operating systems [16]. **tau-bench** and **tau2-bench** (Sierra) simulate customer-service conversations with an LLM user and domain policies, and contribute the ecosystem's best reliability statistic: pass^k, the probability that an agent succeeds on all k independent attempts, which exposes flakiness that a single-run pass rate hides [17]. **TheAgentCompany** (CMU, 2024) drops agents into a simulated software company (GitLab, RocketChat, ownCloud) and grades checkpointed long-horizon work with partial credit [18]. **BFCL** (Berkeley) is the standard for function-call correctness, including multi-turn tool use [19]; **ToolBench** covers 16,000+ real APIs [20]. **MLE-bench** (OpenAI) uses 75 Kaggle competitions; **CORE-bench** (Princeton) tests computational reproducibility of real papers [21].
+
+Two observations. First, nearly all of these publish leaderboards keyed by model, with the benchmark's own reference scaffold underneath: they are model benchmarks with agentic tasks. Second, the exceptions prove the demand: **HAL**, the Holistic Agent Leaderboard (Princeton), re-runs (model, scaffold) pairs across many of the benchmarks above and plots accuracy against cost, an explicit two-axis Pareto view [22]. HAL is the closest thing to a public engine-aware leaderboard. It still is not a tool a team can point at its own agent in its own CI, and cost is its only efficiency axis.
+
+## 4. Survey: trace-based evaluation and eval platforms
+
+Trajectory grading has real prior art. **AgentBoard**'s progress rate compares an agent's actual trajectory against an expected one for per-step progress [2, 26]. **T-Eval** decomposes tool use into step-wise next-call alignment [26]. LLM-as-judge trajectory scoring is now a stock feature of eval platforms. What does not exist is what we will call a golden-trace corpus: reference traces from verified, production-merged fixes, published as auditable evidence alongside outcome grades.
+
+The observability and eval platforms are where CI actually happens today. **LangSmith** (LangChain), **Langfuse** (open source), **Braintrust**, **Arize Phoenix** (open source, OpenTelemetry-based), and **W&B Weave** all offer tracing plus dataset-driven evals, and most can fail a CI job on a metric drop [27]. **DeepEval** runs evals as pytest tests; **promptfoo** runs config-driven evals and red-team suites in CI; **OpenAI Evals** seeded the genre [27]. On standards: OpenTelemetry's GenAI semantic conventions are emerging as the de facto wire format for agent traces (spans for model calls and tool executions), and the Model Context Protocol showed that a small interchange spec can reorganize an ecosystem within a year [28]. There is no equivalent interchange standard for benchmark tasks, runs, or verdicts. A 2026 factorial study proposes "Harness Cards", a structured disclosure taxonomy (execution, tool, context, scheduling, observability, verification, governance) for exactly this reason [3].
+
+The platforms' limitation is the mirror image of the benchmarks'. They grade your agent on your dataset with judge-model rubrics: perfect for product regression, but with no held-out verification (the eval data lives in your repo, next to the agent that will be graded on it), no cross-engine comparability, and no shared task corpus, so a score means nothing outside your org.
+
+## 5. Best practices: what a credible agent benchmark must do in 2026
+
+The literature has converged on a checklist, even if no system implements all of it.
+
+1. **Separate the engine from the model, by construction.** Hold one fixed while varying the other; stamp the run; refuse cross-attribution [1, 3, 10].
+2. **Held-out verification.** The grader must be absent from the workspace while the agent runs, and immune to anything the agent planted. Benchmarks with repo-native tests (SWE-bench) get this half right; eval platforms mostly do not attempt it [4].
+3. **Cost and efficiency as ranked metrics, not footnotes.** Accuracy-cost Pareto curves (AI Agents That Matter, HAL), plus tokens, wall clock, and, we argue, diff size: no surveyed system ranks the size of the change an agent makes, though every reviewer knows a 4-line fix and a 400-line rewrite are not equal [22, 23, 2].
+4. **Statistics that respect small n.** Confidence intervals on rates; paired tests for head-to-heads; reliability metrics like pass^k; seeded, reproducible resampling [17, 23].
+5. **Contamination control.** Frozen sets decay into training data; live sourcing (SWE-bench Live) with recorded cutoffs is the credible answer [7].
+6. **Receipts.** Full transcripts, diffs, manifests, and a reproduce command; disclosure of the harness configuration (Harness Cards) [3].
+7. **Run in CI.** A benchmark you run at launch is marketing; a benchmark that fails a pull request is engineering.
+
+## 6. Gap analysis: the four capabilities, and who has them
+
+**(a) Matched-model engine isolation.** Harbor enables it structurally; Harness-Bench and the factorial study execute it as research; HAL reports (model, scaffold) pairs [9, 10, 3, 22]. None operationalizes it as a stamped, enforced property of every published run.
+
+**(b) Efficiency grading beyond cost.** HAL plots cost; Aider Polyglot displays it; tau-bench measures reliability [22, 11, 17]. No surveyed system ranks normalized git-diff size, and only ad hoc reporting covers tokens and wall clock together [2].
+
+**(c) CI regression gating with statistics.** Eval platforms gate CI on judge-scored private datasets; no benchmark harness offers a significance-aware gate over held-out, executable verification [27].
+
+**(d) Live tasks with verified fixes as golden traces.** SWE-bench Live sources fresh issues; nobody promotes fixes upstream, treats maintainer merge as ground truth, and publishes the solving trace as a reference artifact [7].
+
+No system does all four. That is the hole in the landscape, stated with the evidence above.
+
+## 7. Arena, and the Agent Benchmark Protocol
+
+**Arena** (github.com/oxageninc/arena, MIT) is our attempt at the first three capabilities, today: two or more coding agents on the same tasks, same model, same budget, same timeout; held-out verification injected only after the agent exits; Wilson intervals, exact McNemar on paired outcomes, and seeded paired-bootstrap CIs on wall-clock, token, and cost deltas; no blended score; full receipts with a one-line reproduce command; and a CI gate (`arena baseline save` / `arena gate --require-significant`) that fails a pull request only when a regression clears interval noise. A Harbor adapter scales the same discipline to SWE-bench Verified with the official containerized verifier [9].
+
+The fourth capability needs a standard more than a product. The **Agent Benchmark Protocol** (draft spec published alongside this paper) defines four interchange schemas: a task manifest, a run configuration, a trace format with per-subagent and per-model attribution, and a verdict format with required statistics. On top of them it specifies a live pipeline: a versioned registry of the top 500 open-source GitHub repositories; seeded random draws of real backlog issues; fixes built and submitted upstream under each repo's contribution rules; maintainer merge as verification; and the solving engine's full trace published as a golden trace beside the frozen task. Replays are graded on resolution first, then efficiency in strict order: smaller normalized diff, then fewer tokens, then less wall clock, then lower cost. Golden traces are evidence and study material, deliberately not the rubric, because trajectory-similarity grading punishes better routes to a passing fix; AgentBoard-style progress metrics are a version 2 question [26]. Contamination is handled by construction: tasks are drawn after candidate model cutoffs, stamped, and retired from the scored set 180 days after their fix goes public.
+
+Why a protocol rather than another leaderboard: the measurement problem is an interoperability problem. Tasks, runs, traces, and verdicts need to move between task authors, engines, harnesses, and auditors, exactly as context moves between hosts and tools under MCP [28]. Arena is the reference implementation, not the standard; the standard is the four schemas and the conformance suite.
+
+## 8. Limitations
+
+This survey characterizes fast-moving systems from their public documentation as of July 2026; version specifics will drift. The 7.8x variance ratio comes from one study on one benchmark family and one task size; it needs replication, though its direction agrees with SWE-agent, Agentless, and Harness-Bench [3, 24, 25, 10]. Diff-size ranking is a proxy for surgical precision, not design quality, and is honest only with the normalization and disqualification rules the spec defines. And the author's conflict of interest is real: the reader should treat Section 7 as a proposal to be attacked, with the receipts to attack it.
+
+## References
+
+1. Yehudai et al., *Survey on Evaluation of LLM-based Agents*, v2 (2026). arxiv.org/abs/2503.16416
+2. *A Survey of AI Agent Evaluation* (2025). arxiv.org/abs/2507.21504
+3. *Harness variance factorial study: Harness Cards and ETCSOVG taxonomy* (2026). arxiv.org/abs/2605.23950
+4. Jimenez et al., *SWE-bench: Can Language Models Resolve Real-World GitHub Issues?* (2023). arxiv.org/abs/2310.06770
+5. OpenAI, *Introducing SWE-bench Verified* (2024). openai.com/index/introducing-swe-bench-verified
+6. Scale AI, *SWE-bench Pro* (2025). github.com/scaleapi/SWE-bench_Pro-os
+7. *SWE-bench Live* (2025). swe-bench-live.github.io
+8. Terminal-Bench (2025). tbench.ai
+9. Harbor Framework (2026). github.com/harbor-framework/harbor
+10. *Harness-Bench* (2026). arxiv.org/abs/2605.27922
+11. Aider Polyglot leaderboard. aider.chat/docs/leaderboards
+12. SWE-Gym: github.com/SWE-Gym · R2E: r2e.dev · Commit0: github.com/commit-0/commit0 · RepoBench (2023)
+13. Liu et al., *AgentBench* (2023). arxiv.org/abs/2308.03688
+14. Mialon et al., *GAIA* (2023). arxiv.org/abs/2311.12983
+15. Zhou et al., *WebArena* (2023). arxiv.org/abs/2307.13854
+16. Xie et al., *OSWorld* (2024). arxiv.org/abs/2404.07972
+17. Yao et al., *tau-bench* (2024). arxiv.org/abs/2406.12045 · tau2-bench: github.com/sierra-research/tau2-bench
+18. Xu et al., *TheAgentCompany* (2024). arxiv.org/abs/2412.14161
+19. Berkeley Function-Calling Leaderboard. gorilla.cs.berkeley.edu/leaderboard.html
+20. Qin et al., *ToolLLM/ToolBench* (2023). arxiv.org/abs/2307.16789
+21. OpenAI, *MLE-bench* (2024). arxiv.org/abs/2410.07095 · CORE-bench (Princeton, 2024)
+22. HAL: Holistic Agent Leaderboard (Princeton). hal.cs.princeton.edu
+23. Kapoor et al., *AI Agents That Matter* (2024). arxiv.org/abs/2407.01502
+24. Yang et al., *SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering* (2024). arxiv.org/abs/2405.15793
+25. Xia et al., *Agentless* (2024). arxiv.org/abs/2407.01489
+26. Ma et al., *AgentBoard* (2024). arxiv.org/abs/2401.13178 · T-Eval (2023). arxiv.org/abs/2312.14033
+27. LangSmith: smith.langchain.com · Langfuse: langfuse.com · Braintrust: braintrust.dev · Arize Phoenix: phoenix.arize.com · W&B Weave: wandb.ai · DeepEval: github.com/confident-ai/deepeval · promptfoo: promptfoo.dev · OpenAI Evals: github.com/openai/evals
+28. Model Context Protocol. modelcontextprotocol.io · OpenTelemetry GenAI semantic conventions. opentelemetry.io
+
+---
+
+*Companion documents: the [Agent Benchmark Protocol draft spec](agent-benchmark-protocol.md) and [Arena's methodology](../METHODOLOGY.md). Corrections: open an issue with the `paper` label.*
diff --git a/site/paper/index.html b/site/paper/index.html
new file mode 100644
index 0000000..8907cf4
--- /dev/null
+++ b/site/paper/index.html
@@ -0,0 +1,335 @@
+<!DOCTYPE html>
+<html lang="en">
+<head>
+  <meta charset="utf-8" />
+  <meta name="viewport" content="width=device-width, initial-scale=1" />
+  <title>The State of Agent Benchmarking · Arena</title>
+  <meta name="description" content="A July 2026 survey of agent benchmarks: everyone measures the model, almost nobody measures the engine you ship. Evidence, gap analysis, and the case for an Agent Benchmark Protocol." />
+  <link rel="stylesheet" href="/style.css" />
+  <link rel="icon" href="data:image/svg+xml,%3Csvg xmlns='http://www.w3.org/2000/svg' viewBox='0 0 100 100'%3E%3Ctext y='.9em' font-size='90'%3E%E2%8F%B1%3C/text%3E%3C/svg%3E" />
+</head>
+<body>
+  <div class="wrap">
+    <header class="site-head">
+      <a class="brand" href="/">arena<span>.oxagen.sh</span></a>
+      <nav>
+        <a href="/paper/" aria-current="page">Paper</a>
+        <a href="/spec/">Spec</a>
+        <a href="https://github.com/oxageninc/arena/blob/main/METHODOLOGY.md">Methodology</a>
+        <a href="https://github.com/oxageninc/arena">GitHub</a>
+      </nav>
+    </header>
+
+    <article class="article">
+      <p class="kicker">Research paper · July 2026 · v1.0</p>
+      <h1>The State of Agent Benchmarking</h1>
+      <p class="byline">Everyone measures the model. Almost nobody measures the engine you ship. · Mac Anderson, Oxagen</p>
+
+      <div class="abstract">
+        <strong>Abstract.</strong> Teams that ship AI agents change their engine every week: prompts,
+        tools, planning loops, context strategies, and orchestration. The model changes a few times a
+        year. Yet nearly every public benchmark ranks models with the harness held fixed, the exact
+        inverse of the question a working team needs answered: did my last change make my agent
+        better or worse, on the model I already run? Recent controlled evidence says this inversion
+        is not cosmetic. In a 3x3 factorial study on SWE-bench Verified, harness-induced variance
+        exceeded model-induced variance by 7.8x, and model rankings reversed in six of nine harness
+        pairings [3]. This paper surveys the landscape as of July 2026 through that lens and finds
+        that no existing system combines matched-model engine isolation, efficiency grading,
+        statistically honest CI gating, and live task sourcing with verified fixes as reference
+        traces. We close by describing that missing system, and an interchange standard that would
+        let any harness provide it.
+      </div>
+
+      <div class="abstract" style="margin-top:14px">
+        <strong>Conflict of interest, stated up front:</strong> the author builds
+        <a href="https://github.com/oxageninc/arena">Arena</a>, an open-source agent-engine benchmark
+        harness, and the Oxagen and Stella agents that appear in its examples. This paper argues a
+        position. Every load-bearing claim carries a citation, and Section 6 names the systems that
+        already do parts of what Arena does.
+      </div>
+
+      <nav class="toc" aria-label="Contents">
+        <div class="t">Contents</div>
+        <ol>
+          <li><a href="#question">The question benchmarks answer, and the question teams ask</a></li>
+          <li><a href="#coding">Survey: coding-agent benchmarks</a></li>
+          <li><a href="#general">Survey: general agent benchmarks</a></li>
+          <li><a href="#traces">Survey: trace-based evaluation and eval platforms</a></li>
+          <li><a href="#practices">Best practices for a credible agent benchmark</a></li>
+          <li><a href="#gap">Gap analysis: four capabilities, and who has them</a></li>
+          <li><a href="#arena">Arena, and the Agent Benchmark Protocol</a></li>
+          <li><a href="#limits">Limitations</a></li>
+          <li><a href="#refs">References</a></li>
+        </ol>
+      </nav>
+
+      <div class="article-body">
+        <h2 id="question">1. The question benchmarks answer, and the question teams ask</h2>
+        <p>
+          A model leaderboard answers: given a fixed scaffold, which model scores highest? That is
+          the right question for a lab choosing a base model, and the wrong question for everyone
+          downstream. Downstream, the model is a config value. The product is the engine: the system
+          prompt, the tool set, the retrieval and context policy, the planning loop, the subagent
+          topology, and the guardrails. One survey puts the distinction plainly: evaluating the LLM
+          alone is like examining an engine on a stand, while agent evaluation must assess the whole
+          car under driving conditions [2].
+        </p>
+        <p>
+          The scale of what the scaffold contributes has been measured repeatedly. SWE-agent showed
+          in 2024 that the agent-computer interface alone moves resolve rates at a fixed model [24].
+          Agentless showed the same year that a deliberately non-agentic pipeline could outperform
+          elaborate agents at a fraction of the cost [25]. And in 2026 a controlled factorial study
+          quantified it: across three models and three harness configurations on a 100-task
+          SWE-bench Verified subset, the harness contributed 18.48 pp² of score variance against the
+          model's 2.37 pp², a ratio of 7.8x, with six model-ranking reversals across nine harness
+          pairings [3]. The authors' conclusion is the thesis of this paper: the execution harness,
+          the infrastructure layer that governs context construction, tool interaction,
+          orchestration, and verification, is often a stronger determinant of agent performance than
+          the model it wraps [3].
+        </p>
+        <p>
+          The benchmark ecosystem has not caught up. The most comprehensive survey of agent
+          evaluation to date lists "Decoupling LLM &amp; Harness Evaluation" as future work: most
+          current benchmarks conflate the two targets, and no established benchmark provides
+          controlled protocols that vary each factor independently [1]. The same survey finds cost
+          and efficiency metrics broadly neglected, echoing "AI Agents That Matter": accuracy-only
+          leaderboards drive needlessly costly agents [1, 23].
+        </p>
+
+        <h2 id="coding">2. Survey: coding-agent benchmarks</h2>
+        <p>
+          <strong>SWE-bench</strong> (Princeton/Stanford, 2023) established the template: 2,294 real
+          GitHub issues from 12 Python repos, graded by the repo's own FAIL_TO_PASS and PASS_TO_PASS
+          tests [4]. <strong>SWE-bench Verified</strong> (OpenAI, 2024) human-filtered 500 instances
+          to remove impossible or underspecified tasks [5]. <strong>SWE-bench Multimodal</strong>
+          adds visual, JavaScript-centric issues; <strong>SWE-bench Pro</strong> (Scale AI, 2025)
+          raises difficulty with long-horizon, enterprise-style tasks and a held-out commercial
+          subset [6]. <strong>SWE-bench Live</strong> (Microsoft Research and community, 2025)
+          attacks contamination directly with a monthly-refreshed feed of new issues [7]. All of
+          these grade one thing: did the repo's tests pass. None grades diff size, token spend, or
+          wall clock as a ranked metric, and their leaderboards mix (model, scaffold) pairs, so a
+          score is attributable to neither alone. They are, however, the raw material engine
+          benchmarking needs: real repos, real issues, executable verification.
+        </p>
+        <p>
+          <strong>Terminal-Bench</strong> (Stanford and the Laude Institute, 2025) measures
+          terminal-native task completion in containers; its 2.0 release ships on
+          <strong>Harbor</strong> [8, 9]. Harbor deserves its own entry: it is a meta-harness from
+          the Terminal-Bench creators for "evaluating and optimizing agents and language models",
+          wrapping third-party benchmarks (Terminal-Bench 2.0, SWE-bench, Aider Polyglot) behind one
+          containerized runner, with the agent scaffold (Claude Code, OpenHands, Codex CLI, and
+          more) selected independently of the <code>--model</code> flag [9]. That structure enables
+          matched-model harness comparison, and a 2026 survey names Harbor (with Exgentic) as the
+          first frameworks pushing a unified protocol for general agent assessment [1]. What Harbor
+          does not do: rank efficiency, gate CI on regressions with statistics, or source fresh
+          tasks with reference traces. <strong>Harness-Bench</strong> (2026) is the academic
+          complement: a 6-harness by 8-model factorial over 106 sandboxed tasks (5,194 trajectories)
+          that fixes task, sandbox, budget, timeout, and evaluator while letting each harness keep
+          its native behavior [10]. It is a study rather than an ongoing service, but it is the
+          cleanest matched-model prior art to date.
+        </p>
+        <p>
+          <strong>Aider Polyglot</strong> (225 hard Exercism exercises) is honest about being a
+          model benchmark: one fixed harness, many models, with cost per run displayed [11].
+          <strong>Commit0</strong> (build a library from scratch against a spec and tests) and
+          <strong>RepoBench</strong> (repo-level completion) sit closer to the model end.
+          <strong>SWE-Gym</strong> and <strong>R2E</strong> are environment generators, aimed at
+          training rather than refereeing [12].
+        </p>
+
+        <h2 id="general">3. Survey: general agent benchmarks</h2>
+        <p>
+          <strong>AgentBench</strong> (2023) spans eight environments from OS shells to web shopping
+          [13]. <strong>GAIA</strong> (Meta/HF, 2023) asks 466 tool-requiring questions with
+          unambiguous answers [14]. <strong>WebArena</strong> and <strong>VisualWebArena</strong>
+          grade functional task completion on self-hosted websites [15]; <strong>OSWorld</strong>
+          does the same for 369 tasks on real desktop operating systems [16].
+          <strong>tau-bench</strong> and <strong>tau2-bench</strong> (Sierra) simulate
+          customer-service conversations with an LLM user and domain policies, and contribute the
+          ecosystem's best reliability statistic: pass^k, the probability that an agent succeeds on
+          all k independent attempts, which exposes flakiness a single-run pass rate hides [17].
+          <strong>TheAgentCompany</strong> (CMU, 2024) drops agents into a simulated software
+          company (GitLab, RocketChat, ownCloud) and grades checkpointed long-horizon work with
+          partial credit [18]. <strong>BFCL</strong> (Berkeley) is the standard for function-call
+          correctness, including multi-turn tool use [19]; <strong>ToolBench</strong> covers
+          16,000+ real APIs [20]. <strong>MLE-bench</strong> (OpenAI) uses 75 Kaggle competitions;
+          <strong>CORE-bench</strong> (Princeton) tests computational reproducibility of real
+          papers [21].
+        </p>
+        <p>
+          Two observations. First, nearly all of these publish leaderboards keyed by model, with the
+          benchmark's own reference scaffold underneath: they are model benchmarks with agentic
+          tasks. Second, the exceptions prove the demand: <strong>HAL</strong>, the Holistic Agent
+          Leaderboard (Princeton), re-runs (model, scaffold) pairs across many of the benchmarks
+          above and plots accuracy against cost, an explicit two-axis Pareto view [22]. HAL is the
+          closest thing to a public engine-aware leaderboard. It still is not a tool a team can
+          point at its own agent in its own CI, and cost is its only efficiency axis.
+        </p>
+
+        <h2 id="traces">4. Survey: trace-based evaluation and eval platforms</h2>
+        <p>
+          Trajectory grading has real prior art. <strong>AgentBoard</strong>'s progress rate
+          compares an agent's actual trajectory against an expected one for per-step progress
+          [2, 26]. <strong>T-Eval</strong> decomposes tool use into step-wise next-call alignment
+          [26]. LLM-as-judge trajectory scoring is now a stock feature of eval platforms. What does
+          not exist is what we will call a golden-trace corpus: reference traces from verified,
+          production-merged fixes, published as auditable evidence alongside outcome grades.
+        </p>
+        <p>
+          The observability and eval platforms are where CI actually happens today.
+          <strong>LangSmith</strong>, <strong>Langfuse</strong>, <strong>Braintrust</strong>,
+          <strong>Arize Phoenix</strong>, and <strong>W&amp;B Weave</strong> all offer tracing plus
+          dataset-driven evals, and most can fail a CI job on a metric drop [27].
+          <strong>DeepEval</strong> runs evals as pytest tests; <strong>promptfoo</strong> runs
+          config-driven evals and red-team suites in CI; <strong>OpenAI Evals</strong> seeded the
+          genre [27]. On standards: OpenTelemetry's GenAI semantic conventions are emerging as the
+          de facto wire format for agent traces, and the Model Context Protocol showed that a small
+          interchange spec can reorganize an ecosystem within a year [28]. There is no equivalent
+          interchange standard for benchmark tasks, runs, or verdicts. A 2026 factorial study
+          proposes "Harness Cards", a structured disclosure taxonomy for exactly this reason [3].
+        </p>
+        <p>
+          The platforms' limitation is the mirror image of the benchmarks'. They grade your agent on
+          your dataset with judge-model rubrics: perfect for product regression, but with no
+          held-out verification (the eval data lives in your repo, next to the agent that will be
+          graded on it), no cross-engine comparability, and no shared task corpus, so a score means
+          nothing outside your org.
+        </p>
+
+        <h2 id="practices">5. Best practices: what a credible agent benchmark must do in 2026</h2>
+        <ol>
+          <li><strong>Separate the engine from the model, by construction.</strong> Hold one fixed while varying the other; stamp the run; refuse cross-attribution [1, 3, 10].</li>
+          <li><strong>Held-out verification.</strong> The grader must be absent from the workspace while the agent runs, and immune to anything the agent planted [4].</li>
+          <li><strong>Cost and efficiency as ranked metrics, not footnotes.</strong> Accuracy-cost Pareto curves, plus tokens, wall clock, and, we argue, diff size: no surveyed system ranks the size of the change an agent makes, though every reviewer knows a 4-line fix and a 400-line rewrite are not equal [22, 23, 2].</li>
+          <li><strong>Statistics that respect small n.</strong> Confidence intervals on rates; paired tests for head-to-heads; reliability metrics like pass^k; seeded, reproducible resampling [17, 23].</li>
+          <li><strong>Contamination control.</strong> Frozen sets decay into training data; live sourcing with recorded cutoffs is the credible answer [7].</li>
+          <li><strong>Receipts.</strong> Full transcripts, diffs, manifests, a reproduce command, and disclosure of the harness configuration [3].</li>
+          <li><strong>Run in CI.</strong> A benchmark you run at launch is marketing; a benchmark that fails a pull request is engineering.</li>
+        </ol>
+
+        <h2 id="gap">6. Gap analysis: the four capabilities, and who has them</h2>
+        <div class="tablewrap">
+        <table>
+          <thead><tr><th>Capability</th><th>Closest today</th><th>What's missing</th></tr></thead>
+          <tbody>
+            <tr>
+              <td>(a) Matched-model engine isolation</td>
+              <td>Harbor (structural) [9]; Harness-Bench, factorial study (research) [10, 3]; HAL (reporting) [22]</td>
+              <td>An enforced, stamped property of every published run</td>
+            </tr>
+            <tr>
+              <td>(b) Efficiency grading beyond cost</td>
+              <td>HAL and Aider show cost [22, 11]; tau-bench measures reliability [17]</td>
+              <td>Nobody ranks normalized git-diff size; tokens + wall clock rarely reported together</td>
+            </tr>
+            <tr>
+              <td>(c) CI regression gating with statistics</td>
+              <td>Eval platforms gate CI on judge-scored private datasets [27]</td>
+              <td>No significance-aware gate over held-out, executable verification</td>
+            </tr>
+            <tr>
+              <td>(d) Live tasks, verified fixes, golden traces</td>
+              <td>SWE-bench Live sources fresh issues [7]</td>
+              <td>Nobody promotes fixes upstream, uses maintainer merge as ground truth, or publishes the solving trace</td>
+            </tr>
+          </tbody>
+        </table>
+        </div>
+        <p>No system does all four. That is the hole in the landscape.</p>
+
+        <h2 id="arena">7. Arena, and the Agent Benchmark Protocol</h2>
+        <p>
+          <strong>Arena</strong> (<a href="https://github.com/oxageninc/arena">github.com/oxageninc/arena</a>,
+          MIT) is our attempt at the first three capabilities, today: two or more coding agents on
+          the same tasks, same model, same budget, same timeout; held-out verification injected only
+          after the agent exits; Wilson intervals, exact McNemar on paired outcomes, and seeded
+          paired-bootstrap CIs on wall-clock, token, and cost deltas; no blended score; full
+          receipts with a one-line reproduce command; and a CI gate that fails a pull request only
+          when a regression clears interval noise. A Harbor adapter scales the same discipline to
+          SWE-bench Verified with the official containerized verifier [9].
+        </p>
+        <p>
+          The fourth capability needs a standard more than a product. The
+          <a href="/spec/">Agent Benchmark Protocol</a> defines four interchange schemas: a task
+          manifest, a run configuration, a trace format with per-subagent and per-model attribution,
+          and a verdict format with required statistics. On top of them it specifies a live
+          pipeline: a versioned registry of the top 500 open-source GitHub repositories; seeded
+          random draws of real backlog issues; fixes built and submitted upstream under each repo's
+          contribution rules; maintainer merge as verification; and the solving engine's full trace
+          published as a golden trace beside the frozen task. Replays are graded on resolution
+          first, then efficiency in strict order: smaller normalized diff, then fewer tokens, then
+          less wall clock, then lower cost. Golden traces are evidence and study material,
+          deliberately not the rubric, because trajectory-similarity grading punishes better routes
+          to a passing fix [26]. Contamination is handled by construction: tasks are drawn after
+          candidate model cutoffs, stamped, and retired from the scored set 180 days after their fix
+          goes public.
+        </p>
+        <p>
+          Why a protocol rather than another leaderboard: the measurement problem is an
+          interoperability problem. Tasks, runs, traces, and verdicts need to move between task
+          authors, engines, harnesses, and auditors, exactly as context moves between hosts and
+          tools under MCP [28]. Arena is the reference implementation, not the standard; the
+          standard is the four schemas and the conformance suite.
+        </p>
+
+        <h2 id="limits">8. Limitations</h2>
+        <p>
+          This survey characterizes fast-moving systems from their public documentation as of July
+          2026; version specifics will drift. The 7.8x variance ratio comes from one study on one
+          benchmark family and one task size; it needs replication, though its direction agrees with
+          SWE-agent, Agentless, and Harness-Bench [3, 24, 25, 10]. Diff-size ranking is a proxy for
+          surgical precision, not design quality, and is honest only with the normalization and
+          disqualification rules the spec defines. And the author's conflict of interest is real:
+          the reader should treat Section 7 as a proposal to be attacked, with the receipts to
+          attack it.
+        </p>
+
+        <h2 id="refs">References</h2>
+        <ol class="refs">
+          <li>Yehudai et al., <em>Survey on Evaluation of LLM-based Agents</em>, v2 (2026). arxiv.org/abs/2503.16416</li>
+          <li><em>A Survey of AI Agent Evaluation</em> (2025). arxiv.org/abs/2507.21504</li>
+          <li><em>Harness variance factorial study: Harness Cards and the ETCSOVG taxonomy</em> (2026). arxiv.org/abs/2605.23950</li>
+          <li>Jimenez et al., <em>SWE-bench</em> (2023). arxiv.org/abs/2310.06770</li>
+          <li>OpenAI, <em>Introducing SWE-bench Verified</em> (2024). openai.com/index/introducing-swe-bench-verified</li>
+          <li>Scale AI, <em>SWE-bench Pro</em> (2025). github.com/scaleapi/SWE-bench_Pro-os</li>
+          <li><em>SWE-bench Live</em> (2025). swe-bench-live.github.io</li>
+          <li>Terminal-Bench (2025). tbench.ai</li>
+          <li>Harbor Framework (2026). github.com/harbor-framework/harbor</li>
+          <li><em>Harness-Bench</em> (2026). arxiv.org/abs/2605.27922</li>
+          <li>Aider Polyglot leaderboard. aider.chat/docs/leaderboards</li>
+          <li>SWE-Gym · R2E (r2e.dev) · Commit0 (github.com/commit-0/commit0) · RepoBench (2023)</li>
+          <li>Liu et al., <em>AgentBench</em> (2023). arxiv.org/abs/2308.03688</li>
+          <li>Mialon et al., <em>GAIA</em> (2023). arxiv.org/abs/2311.12983</li>
+          <li>Zhou et al., <em>WebArena</em> (2023). arxiv.org/abs/2307.13854</li>
+          <li>Xie et al., <em>OSWorld</em> (2024). arxiv.org/abs/2404.07972</li>
+          <li>Yao et al., <em>tau-bench</em> (2024). arxiv.org/abs/2406.12045 · tau2-bench: github.com/sierra-research/tau2-bench</li>
+          <li>Xu et al., <em>TheAgentCompany</em> (2024). arxiv.org/abs/2412.14161</li>
+          <li>Berkeley Function-Calling Leaderboard. gorilla.cs.berkeley.edu/leaderboard.html</li>
+          <li>Qin et al., <em>ToolLLM/ToolBench</em> (2023). arxiv.org/abs/2307.16789</li>
+          <li>OpenAI, <em>MLE-bench</em> (2024). arxiv.org/abs/2410.07095 · CORE-bench (Princeton, 2024)</li>
+          <li>HAL: Holistic Agent Leaderboard (Princeton). hal.cs.princeton.edu</li>
+          <li>Kapoor et al., <em>AI Agents That Matter</em> (2024). arxiv.org/abs/2407.01502</li>
+          <li>Yang et al., <em>SWE-agent</em> (2024). arxiv.org/abs/2405.15793</li>
+          <li>Xia et al., <em>Agentless</em> (2024). arxiv.org/abs/2407.01489</li>
+          <li>Ma et al., <em>AgentBoard</em> (2024). arxiv.org/abs/2401.13178 · T-Eval (2023). arxiv.org/abs/2312.14033</li>
+          <li>LangSmith · Langfuse · Braintrust · Arize Phoenix · W&amp;B Weave · DeepEval · promptfoo · OpenAI Evals</li>
+          <li>Model Context Protocol (modelcontextprotocol.io) · OpenTelemetry GenAI semantic conventions</li>
+        </ol>
+
+        <p class="muted">
+          The markdown source of this paper lives in the repo at
+          <code>docs/agent-engine-benchmarks-2026.md</code>. Corrections: open an issue with the
+          <code>paper</code> label.
+        </p>
+      </div>
+    </article>
+  </div>
+
+  <div class="wrap">
+    <footer class="site-foot">
+      <span>MIT © Mac Anderson · <a href="https://oxagen.sh">Oxagen</a></span>
+      <span><a href="https://github.com/oxageninc/arena">github.com/oxageninc/arena</a></span>
+    </footer>
+  </div>
+</body>
+</html>

From 467c3e2fbb8631d198281f38e2c5480e03b1b76d Mon Sep 17 00:00:00 2001
From: macanderson <mac@oxagen.sh>
Date: Sat, 18 Jul 2026 19:40:21 -0700
Subject: [PATCH 4/4] docs: architecture map (module graph, trial data flow,
 contracts)

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---
 docs/ARCHITECTURE.md | 62 ++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 62 insertions(+)
 create mode 100644 docs/ARCHITECTURE.md

diff --git a/docs/ARCHITECTURE.md b/docs/ARCHITECTURE.md
new file mode 100644
index 0000000..65f4fe3
--- /dev/null
+++ b/docs/ARCHITECTURE.md
@@ -0,0 +1,62 @@
+# Arena architecture
+
+Two subsystems share one repo: a TypeScript CLI harness (`src/`) and a Python Harbor adapter (`harbor/`). They share no code, only a token-normalization convention: `input` never includes cache reads.
+
+```mermaid
+graph TD
+  subgraph CLI["TypeScript harness"]
+    cli["cli.ts (entry / dispatch)"]
+    cli --> c_run["run"] & c_verify["verify"] & c_gate["gate / baseline"] & c_report["report"] & c_list["list / doctor"]
+
+    subgraph CORE["core pipeline"]
+      orch["orchestrator.ts executeRun/runOne"]
+      ws["workspace.ts seed/diff/verify"]
+      adp["adapters/ claude-code · gemini · oxagen · stella · mock"]
+      parse["parse.ts envelope/diff parsing"]
+      pricing["pricing.ts + pricing.json"]
+    end
+
+    stats["stats.ts wilson/mcnemar/bootstrap"]
+    summary["summary.ts perAgentSummary"]
+    report["report.ts"]
+    baseline["baseline.ts gate/baseline"]
+    tasksdir["tasks/<id>/ task.json + workspace + verify + solution"]
+
+    c_run --> orch
+    c_verify --> ws
+    c_report --> report
+    c_gate --> baseline
+    orch --> ws & adp & parse & pricing
+    ws --> tasksdir
+    report --> stats & summary
+    baseline --> summary
+    summary --> stats
+  end
+
+  subgraph HARBOR["harbor/ Python adapter (separate process)"]
+    h_agents["agents.py Byo/Oxagen/StellaAgent"]
+    h_base["base.py ArenaInstalledAgent"]
+    h_spec["spec.py AgentSpec (TOML/JSON)"]
+    h_cmd["command.py build_command"]
+    h_metrics["metrics.py extract_metrics"]
+    HarborFW["Harbor framework (Docker verifier)"]
+    h_agents --> h_base --> h_cmd & h_metrics & h_spec
+    h_base --> HarborFW
+  end
+
+  CLI -. "token normalization convention only" .- HARBOR
+```
+
+## One trial, end to end
+
+1. `cli.ts:cmdRun` parses argv into a `RunConfig`, loads tasks.
+2. `orchestrator.ts:executeRun` creates the run dir, checks each adapter's availability, writes `manifest.json`, then loops trials × tasks × agents with ABBA order flipping.
+3. `runOne`: `seedWorkspace` copies the fixture into a temp dir and git-commits the seed → `adapter.execute` spawns the CLI (own process group, SIGKILL tree on timeout) → `collectDiff` captures exactly what the agent changed → `parseEnvelope` normalizes tokens → held-out `runVerification` (wipe `.arena-verify/`, copy tests in, `node --test`, wipe again) → `TrialResult` written; `results.json` rewritten every trial.
+4. `report.ts` renders per-agent summaries (via `summary.ts`, the same aggregation the gate uses), pairwise McNemar and bootstrap deltas, the per-task matrix, and receipts.
+5. `baseline.ts` snapshots a run (`baseline save`) and gates later runs (`gate`), refusing task-set mismatches and, with `--require-significant`, only failing accuracy drops that clear the 95% CIs.
+
+## Contracts
+
+- **Adapter** (`src/adapters/base.ts`): implement `name`, `defaultBinary`, `buildArgs(args)` (argv array, never shell), and `parseEnvelope(stdout)` returning normalized tokens (`input` excludes cache reads; use `totalize`/`emptyEnvelope`). Optional overrides: `resolveModel`, `env`, `execute`, `isAvailable`/`version`. Register in `src/adapters/index.ts`.
+- **Task fixture** (`tasks/<id>/`): `task.json` (`id` must equal the dir name), `workspace/` (what the agent sees), `verify/` (held-out `node:test` suite, never on disk during the run), `solution/` (reference proving solvability). `arena verify` enforces: pristine fails, solution passes.
+- **Harbor spec** (`harbor/arena_harbor/spec.py`): a TOML/JSON file with `name`, `binary`, `run_template` (`{bin} {model} {budget} {timeout} {instruction}` placeholders; instruction shell-quoted for you), plus install and metrics blocks. Point `ARENA_AGENT_SPEC` at it; no Python needed.