Skip to content

fix: enforce fair benchmark budgets - #16

Merged
wq19901103wq merged 1 commit into
mainfrom
codex/fair-direct-budget
Jul 17, 2026
Merged

fix: enforce fair benchmark budgets#16
wq19901103wq merged 1 commit into
mainfrom
codex/fair-direct-budget

Conversation

@wq19901103wq

Copy link
Copy Markdown
Owner

Root cause

The official DeepSeek comparison showed direct mode stopping with empty patches. Its logs proved both runs ended at the inherited 100k cumulative-token cap, while SWE-agent used roughly 400k input tokens on the same task. This made the comparison invalid.

Fix

  • explicitly set direct mode to 100 steps and a 1M cumulative-token budget
  • restore user config after every task and record limits in report metadata
  • abort instead of scoring budget exhaustion or runner timeouts as accuracy failures
  • use fully isolated per-task venvs so editable installs cannot overwrite the main Python environment

Validation

  • ruff and mypy pass
  • 25 benchmark-related tests pass
  • regression test verifies benchmark limits and config restoration

@wq19901103wq
wq19901103wq merged commit 28d0e43 into main Jul 17, 2026
@wq19901103wq
wq19901103wq deleted the codex/fair-direct-budget branch July 17, 2026 23:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant