benchmark: harden discover-to-call v2 and publish corrected baseline

[linfangw]

What changed

• adds isolated configured-model and curated-reference adapters plus immutable task sets through tasks/v4.jsonl
• publishes sanitized benchmark artifacts while keeping raw operational records private
• removes operational IDs, ordered catalog rows, raw parameter values, and inspected parameter names from public artifacts
• records and validates task, model, adapter, toolkit, runtime, API, and catalog provenance
• validates task files, execution controls, run/session IDs, task digest, comparison lane, output paths, and clean toolkit revision before any external request
• derives fail-closed parameter schemas and validates required, enum, nested, unknown, and one_of_required values before billed execution
• sends only compact routing cards and a least-privilege inspected-tool projection to model adapters
• strips every case variant of QVERIS_* environment variables at adapter process boundaries
• fixes structural result scoring to inspect result.data and explicitly requests respond_with: "full"
• avoids retrying ambiguous Execute failures, preventing duplicate side effects or charges
• introduces corrected methodology discover-call-v2; summaries cannot mix historical v1 and corrected v2 records
• publishes public runs and summary files as one application-level transaction with rollback if either replacement fails
• requires every checked-in public runs file and summary file to have a matching counterpart
• assigns every observed strict failure to its earliest failed gate
• retains earlier v3/v4 data as diagnostic history and publishes a fresh corrected-v2 configured-model baseline

Corrected v2 baseline

The 2026-07-24 run is the first official configured-model baseline collected with corrected discover-call-v2, immutable tasks/v4.jsonl, 18 tasks, three trials per task, Top-10 discovery, real calls, and complete failed-trial retention.

Metric	Curated reference route	gpt-5.6-sol configured model
Runs	54	54
Completed and executed	51/54	52/54
Selection grounded	94.44%	100%
Inspection grounded	94.44%	100%
Required-parameter accuracy	100%	100%
Constraint accuracy	94.44%	88.89%
Call success among attempted calls	100% (51/51)	100% (52/52)
Result non-empty among successful calls	100% (51/51)	100% (52/52)
Strict workflow success	94.44% (51/54)	88.89% (48/54)
Task-cluster bootstrap 95% interval	83.33%–100%	72.22%–100%

The strict benchmark gap is 3/54 = 5.56 percentage points. It is not presented as a pure routing effect: both lanes observed API revision 2026-07-23.2, but the API reported no catalog revision and the sequential lanes have different catalog-observation digests. The intervals overlap, so this 18-task result is not a statistical-significance claim.

Reference failures are three timezone-tokyo catalog-coverage misses. The configured lane has three Tokyo constraint mismatches, two domain-intel tool_use_rejected adapter failures, and one domain-intel constraint mismatch. Every attempted call in both lanes succeeded and returned a non-empty result.

The configured lane used gpt-5.6-sol, medium reasoning, CLI 0.144.1, and toolkit revision a7f2aa60ef143dbbb35eaf9006ed8123d778fb13. Its provider model revision is unreported, so this is an official configured-model baseline, not a pinned-model snapshot.

Validation

• benchmark tests: 81/81 passing
• task/fixture/artifact validation: v1/v2/v3/v4, 6 fixture records, 7 public result files
• synthetic fixture scoring passes
• new public artifacts pass forbidden-field scans
• full monorepo tests pass locally: CLI 132, MCP 179, JS SDK 64, OpenClaw plugin 78, Python SDK 148 passed/1 skipped, release coordinator 8
• formatting and git diff --check pass

Advances #163. The remaining benchmark work is per-release repetition and a future pinned provider snapshot when one is available.

[ax2]

@codex review

[ax2]

@codex review

[ax2]

@codex review

[ax2]

@codex review

[ax2]

@codex review

[ax2]

@codex review

[ax2]

@codex review

[ax2]

@codex review

[ax2]

@codex review

[ax2]

@codex review

[ax2]

@codex review