Verification integrity fixes, arena.oxagen.sh site, ABP spec + research paper#22
Merged
Merged
Conversation
Highest-severity first:
- Timeouts kill the agent's whole process tree (POSIX process groups);
trials can no longer hang on pipes held by orphaned grandchildren,
and survivors can no longer tamper with held-out verification.
- Zero-token (unparseable-envelope) trials are excluded from token/cost
medians and deltas instead of dragging them toward zero past the gate.
- claude-code preserves full model ids (no floating "sonnet" alias), so
version pinning, matchedModels, and pricing lookups hold.
- Gemini normalization counts thoughts (reasoning) and tool tokens.
- Pretty-printed JSON envelopes parse (TS + Harbor Python mirror).
- Per-trial error containment + incremental results.json; wall clock
measures spawn-to-exit only; git-diff failure is "unknown", never
"empty"; report table shares perAgentSummary with the gate (no NaN).
- CLI: -o short flag (as documented), strict numeric flags, unknown
verify ids error, duplicate agent specs rejected, typo'd gate
thresholds error instead of silently disabling the check.
- Cost with unpriced cache writes is null, never guessed.
- Harbor: float-tolerant ARENA_TIMEOUT (never silently unbounded),
loud empty-{budget} failure, metrics.kind validated at load.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…raft Static site (site/) for arena.oxagen.sh: landing page with the matched-model scoreboard, and the ABP design spec. Spec source lives at docs/agent-benchmark-protocol.md: four interchange schemas (task, run, trace, verdict), the live GitHub-issue task pipeline with golden traces, smallest-normalized-diff scoring, conformance levels, and roadmap. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
![sourcery-ai[bot]](https://avatars.githubusercontent.com/in/48477?s=60&v=4){kind=small}
There was a problem hiding this comment.
Sorry @macanderson, you have reached your weekly rate limit of 500000 diff characters.
Please try again later or upgrade to continue using Sourcery
Research paper surveying the July 2026 agent-benchmark landscape through the engine-vs-model lens: 28 cited references, gap analysis of four capabilities (matched-model isolation, efficiency grading, statistical CI gating, live tasks with golden traces), COI disclosed. Markdown source in docs/, rendered at arena.oxagen.sh/paper/. README links the site, paper, and spec. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
macanderson
marked this pull request as ready for review
July 19, 2026 04:48
There was a problem hiding this comment.
Sorry @macanderson, you have reached your weekly rate limit of 500000 diff characters.
Please try again later or upgrade to continue using Sourcery
macanderson
enabled auto-merge (squash)
July 19, 2026 04:49
…ntend Main's #20 independently implemented several of the same fixes. Kept main's architecture (async workspace, taskDir threading, pricing param, intFlag/numFlag, report perAgentSummary) and layered this branch's unique fixes on top: - anchor-based JSON block scan (survives stray braces in log lines; main's char-scan fails that test) - drain-timer settle in adapter execute (grace timer not unref'd, so the promise always settles) + utf8 stream encoding - collectDiff returns null on git failure (unknown never equals empty) - bounded 1800s fallback for non-numeric ARENA_TIMEOUT (main's helper silently disabled the container timeout) - kept both sides' parse tests (4 pretty-printed envelope cases) Also applied #21's oxageninc->macanderson rename to site/ and docs/. Note: repo now has two frontends, web/ (from #20) and site/ (deployed at arena.oxagen.sh); consolidation is a follow-up decision. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Both heads merged origin/main with different conflict resolutions. This keeps the locally verified resolution (55/55 tests): both parse test suites, the anchor-based JSON scan, drain-timer settle, and the bounded ARENA_TIMEOUT fallback. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Three workstreams in one branch:
1. Bug fixes (src/, harbor/) — from a full-repo review, ranked by severity:
sonnetfamily alias, restoring version pinning,matchedModels, and pricing integrity.thoughts+tooltokens (previously dropped, understating Gemini in head-to-heads).agent-errorinstead of aborting the run), incrementalresults.json, spawn-to-exit wall clock, diff-unknown vs diff-empty distinction, strict numeric CLI flags,-oshort flag, loud failures for unknown verify ids / duplicate agent specs / typo'd gate thresholds, null cost for unpriced cache writes, NaN-free report rows.ARENA_TIMEOUT, loud empty-{budget}failure,metrics.kindvalidation.2. Site (site/) — static site for arena.oxagen.sh: landing page, research paper, ABP spec. Deployed via the Vercel project
arena(oxagen team).3. Docs (docs/) — the Agent Benchmark Protocol draft spec and the research paper The State of Agent Benchmarking (sources in markdown, rendered on the site).
Verification
pnpm typecheckclean,pnpm test53/53,pnpm lintclean,pnpm arena verifyinvariants covered by the pipeline test.🤖 Generated with Claude Code