Context. METHODOLOGY says headline claims should run on SWE-bench Verified via the Harbor adapter with a pre-registered instance list. Make that reproducible, and guard the adapter with a real container smoke.
Scope
- A script selecting a deterministic instance sample (published seed, n ≥ 100) that runs the
harbor/ adapters plus a Harbor built-in competitor (e.g. claude-code).
- An opt-in Docker E2E smoke that runs one instance end-to-end in CI to catch adapter breakage.
- Publish the seed + resolved instance list.
Acceptance. A single command reproduces the sampled run; the 1-instance smoke passes on demand.
Context. METHODOLOGY says headline claims should run on SWE-bench Verified via the Harbor adapter with a pre-registered instance list. Make that reproducible, and guard the adapter with a real container smoke.
Scope
harbor/adapters plus a Harbor built-in competitor (e.g.claude-code).Acceptance. A single command reproduces the sampled run; the 1-instance smoke passes on demand.