Verification integrity fixes, arena.oxagen.sh site, ABP spec + research paper

CHANGELOG.md

@@ -11,6 +11,46 @@ minor bumps may include breaking changes).
 - Open-source launch artifacts: `CONTRIBUTING.md`, `CODE_OF_CONDUCT.md`, issue
   and PR templates, and this changelog.
 - `.stella/` agent caches are now gitignored.
+- **Site** (`site/`) — arena.oxagen.sh: landing page, the research paper
+  *The State of Agent Benchmarking*, and the draft **Agent Benchmark Protocol**
+  spec (`docs/agent-benchmark-protocol.md`,
+  `docs/agent-engine-benchmarks-2026.md`).
+- Pretty-printed (multi-line) JSON envelopes are now parsed — both in the TS
+  harness (`parseJsonEnvelope`) and the Harbor adapter (`last_json_object`).
+- `arena run -o <dir>` short flag (the form the README documents).
+
+### Fixed
+- **Timeouts kill the agent's whole process tree** (POSIX process groups), and
+  a trial can no longer hang on stdio pipes held open by orphaned
+  grandchildren — which could previously also outlive the agent and tamper
+  with verification.
+- **Zero-token trials no longer poison medians or the gate**: scored trials
+  whose envelope had no parseable usage are excluded from token/cost medians
+  and deltas (reported separately), instead of dragging them toward zero.
+- **claude-code model pinning**: `anthropic/claude-sonnet-5` now resolves to
+  `claude-sonnet-5` instead of the floating `sonnet` alias, preserving version
+  pinning, `matchedModels`, and pricing lookups.
+- **Gemini token normalization** counts `thoughts` (reasoning) and `tool`
+  tokens; both were previously dropped, understating Gemini usage.
+- Per-trial error containment: a harness-side failure (git/FS error) scores
+  that one trial `agent-error` instead of aborting the run; `results.json` is
+  rewritten after every trial so a crash never loses completed trials.
+- Wall clock now measures spawn-to-exit only (workspace seeding excluded),
+  matching METHODOLOGY.md.
+- `arena verify <unknown-id>` errors instead of vacuously passing; numeric
+  CLI flags are validated (`--timeout 10m` errors instead of parsing as 10);
+  a typo'd gate threshold errors instead of silently disabling the check;
+  duplicate `--agents` specs are rejected (their trial ids would collide).
+- `git diff` failures are now distinguished from an empty diff, so a trial
+  with real changes can no longer be misclassified `agent-error`.
+- Cache-write tokens with no `cacheWritePerM` price now yield a null cost
+  (never guessed with the input rate).
+- Report per-agent table shares `perAgentSummary` with the baseline/gate (no
+  more NaN rows when every trial of an agent errored).
+- Harbor adapter: non-integer `ARENA_TIMEOUT` falls back to 1800s with a
+  warning instead of silently disabling the container timeout; an empty
+  `{budget}` in a run template fails loudly; unknown `metrics.kind` values are
+  rejected at spec load.
 
 ## [0.1.0] — initial release
 

README.md

@@ -11,6 +11,8 @@
 
 **Head-to-head benchmarks for agentic coding CLIs — built to survive scrutiny.** · [arena.oxagen.sh](https://arena.oxagen.sh)
 
+Paper: [The State of Agent Benchmarking](docs/agent-engine-benchmarks-2026.md) · Spec: [Agent Benchmark Protocol (draft)](docs/agent-benchmark-protocol.md)
+
 Arena runs two or more coding agents (Claude Code, Gemini CLI, [Oxagen](https://oxagen.sh), [Stella](https://github.com/macanderson/stella), or your own) on the **same tasks, same model, same budget, same timeout**, grades them with **held-out tests the agent can never see or author**, and reports each metric separately with real statistics and full receipts.
 
 ```bash

docs/ARCHITECTURE.md

@@ -0,0 +1,62 @@
+# Arena architecture
+
+Two subsystems share one repo: a TypeScript CLI harness (`src/`) and a Python Harbor adapter (`harbor/`). They share no code, only a token-normalization convention: `input` never includes cache reads.
+
+```mermaid
+graph TD
+  subgraph CLI["TypeScript harness"]
+    cli["cli.ts (entry / dispatch)"]
+    cli --> c_run["run"] & c_verify["verify"] & c_gate["gate / baseline"] & c_report["report"] & c_list["list / doctor"]
+
+    subgraph CORE["core pipeline"]
+      orch["orchestrator.ts executeRun/runOne"]
+      ws["workspace.ts seed/diff/verify"]
+      adp["adapters/ claude-code · gemini · oxagen · stella · mock"]
+      parse["parse.ts envelope/diff parsing"]
+      pricing["pricing.ts + pricing.json"]
+    end
+
+    stats["stats.ts wilson/mcnemar/bootstrap"]
+    summary["summary.ts perAgentSummary"]
+    report["report.ts"]
+    baseline["baseline.ts gate/baseline"]
+    tasksdir["tasks/<id>/ task.json + workspace + verify + solution"]
+
+    c_run --> orch
+    c_verify --> ws
+    c_report --> report
+    c_gate --> baseline
+    orch --> ws & adp & parse & pricing
+    ws --> tasksdir
+    report --> stats & summary
+    baseline --> summary
+    summary --> stats
+  end
+
+  subgraph HARBOR["harbor/ Python adapter (separate process)"]
+    h_agents["agents.py Byo/Oxagen/StellaAgent"]
+    h_base["base.py ArenaInstalledAgent"]
+    h_spec["spec.py AgentSpec (TOML/JSON)"]
+    h_cmd["command.py build_command"]
+    h_metrics["metrics.py extract_metrics"]
+    HarborFW["Harbor framework (Docker verifier)"]
+    h_agents --> h_base --> h_cmd & h_metrics & h_spec
+    h_base --> HarborFW
+  end
+
+  CLI -. "token normalization convention only" .- HARBOR
+```
+
+## One trial, end to end
+
+1. `cli.ts:cmdRun` parses argv into a `RunConfig`, loads tasks.
+2. `orchestrator.ts:executeRun` creates the run dir, checks each adapter's availability, writes `manifest.json`, then loops trials × tasks × agents with ABBA order flipping.
+3. `runOne`: `seedWorkspace` copies the fixture into a temp dir and git-commits the seed → `adapter.execute` spawns the CLI (own process group, SIGKILL tree on timeout) → `collectDiff` captures exactly what the agent changed → `parseEnvelope` normalizes tokens → held-out `runVerification` (wipe `.arena-verify/`, copy tests in, `node --test`, wipe again) → `TrialResult` written; `results.json` rewritten every trial.
+4. `report.ts` renders per-agent summaries (via `summary.ts`, the same aggregation the gate uses), pairwise McNemar and bootstrap deltas, the per-task matrix, and receipts.
+5. `baseline.ts` snapshots a run (`baseline save`) and gates later runs (`gate`), refusing task-set mismatches and, with `--require-significant`, only failing accuracy drops that clear the 95% CIs.
+
+## Contracts
+
+- **Adapter** (`src/adapters/base.ts`): implement `name`, `defaultBinary`, `buildArgs(args)` (argv array, never shell), and `parseEnvelope(stdout)` returning normalized tokens (`input` excludes cache reads; use `totalize`/`emptyEnvelope`). Optional overrides: `resolveModel`, `env`, `execute`, `isAvailable`/`version`. Register in `src/adapters/index.ts`.
+- **Task fixture** (`tasks/<id>/`): `task.json` (`id` must equal the dir name), `workspace/` (what the agent sees), `verify/` (held-out `node:test` suite, never on disk during the run), `solution/` (reference proving solvability). `arena verify` enforces: pristine fails, solution passes.
+- **Harbor spec** (`harbor/arena_harbor/spec.py`): a TOML/JSON file with `name`, `binary`, `run_template` (`{bin} {model} {budget} {timeout} {instruction}` placeholders; instruction shell-quoted for you), plus install and metrics blocks. Point `ARENA_AGENT_SPEC` at it; no Python needed.