Add test bench for speed + input-token cost#2
Merged
Conversation
Two-mode bench for watching bully's speed and input-token cost over time: fixture suite for regression trends (history.jsonl) and per-config cost analysis for users evaluating rule packs. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Ten-task TDD plan implementing the test bench spec: phase-timer hook, fixture loader, token counter, single-fixture runner, Mode A (history), --compare, Mode B (config cost), 8 fixtures, and README. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Bench harness needs to measure per-phase timing without duplicating the pipeline loop. Default is a zero-cost no-op so normal hook invocations are unaffected. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
… test Ruff UP037 flagged the self-referential return-type quotes on _NoopPhaseTimer (the file doesn't use `from __future__ import annotations`, so the annotations aren't actually needed on a private no-op class). Also tighten the phase-timer test to assert the exact set of 7 phase names, so a future refactor that drops one will fail the test. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
`bully bench` now dispatches to pipeline.bench. Flags parsed but not yet implemented -- subsequent tasks fill in fixture loading, token counting, and both modes. Also adds module-level __getattr__ to pipeline.py so `from pipeline import bench` works in the flat-module test context. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Task 2's test attempted `from pipeline import bench`, which required a module-level __getattr__ in pipeline.py to work with the test's sys.path setup. That was the wrong fix -- the existing pattern for sibling-module imports (see test_analyzer.py) is `import bench`, treating bench.py as a top-level module once pipeline/ is on sys.path. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Fixtures live in bench/fixtures/<name>/{config.yml,fixture.json}.
load_fixture validates both files exist and the metadata has required
fields; discover_fixtures returns all subdirectories sorted by name.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
count_tokens uses messages/count_tokens when an API key and the SDK are available; otherwise returns len(json.dumps(payload)) + len(system) and tags the result 'proxy'. System prompt is loaded from the real bully-evaluator agent file so counts match production. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
PhaseTimer implements the pipeline hook protocol and records per-phase elapsed time. run_fixture runs one fixture N times in-process, samples cold-start once via subprocess, and builds the real semantic payload to count tokens. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Previously, run_fixture called count_tokens even on fixtures with no semantic rules -- producing a misleading non-zero token count for a dispatch that would never happen. Fix to report 0 tokens with method "n/a-no-semantic-rules" so history entries accurately reflect what real bully invocations would send. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
\`bully bench\` now enumerates bench/fixtures/*, runs each fixture, and appends one JSONL line to bench/history.jsonl with per-fixture phase timings + aggregate totals. Human-readable summary to stdout by default; --json emits the raw record. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Per-fixture + aggregate deltas for wall time and input tokens. Fails clearly when history has fewer than two runs. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
\`bully bench --config <path>\` parses a .bully.yml and reports floor tokens per dispatch, per-rule marginal cost (sorted), diff scaling at 1/10/100/1000 added lines, and a scope-grouped breakdown. Script and ast rules are listed separately as zero model-token cost. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Fixtures exercise: single-engine smoke (01-03), mixed-engine path (04), multi-level extends chain (05), payload size scaling (06, 07), and the auto-generated skip short-circuit (08). Integration test runs all of them against run_mode_a. Also fix bench.py import to work in both package context (pipeline.bench) and direct sys.path context (used by pytest), and fix pre-existing ruff isort violations in test_bench.py. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Covers Mode A (fixture suite + history.jsonl), Mode B (config cost analysis), and the ANTHROPIC_API_KEY / optional anthropic SDK requirement for real token counts. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Previously run_fixture set BULLY_TRUST_ALL=1 permanently. Fine for the CLI path (process exits) but leaks state if bench is consumed as a library. Save the prior value and restore in a finally block. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Previously returned {returncode, report}; the dict wrapper was unique to
Mode B. Align with the simpler "None = failure, dict = success" shape
used elsewhere.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Previously `--config ... --compare` silently dropped --compare (config branch won the if/elif). Use argparse's mutually_exclusive_group so users get an explicit error. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Fixture targets (.py files under bench/fixtures/*) intentionally model user code with unused vars, incomplete functions, and style violations -- they are data, not source. Exclude them from ruff so CI stops flagging them as real violations. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Bully can catch style violations at edit time before CI does. The --force-exclude flag makes ruff respect pyproject's extend-exclude list even when a single file is passed, so bench fixtures are skipped. After pulling, run `bully trust --refresh` to re-approve the modified config (trust gate hashes the .bully.yml contents). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
CI's `ruff format --check` step flagged formatting differences on both bench files. Apply the formatter output directly; no semantic changes. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds a two-mode local bench for watching bully's own speed and input-token cost over time. Runs entirely on the user's machine; no CI integration.
Mode A — fixture suite (
bully bench): runs 8 hand-authored fixtures underbench/fixtures/throughrun_pipeline, records per-phase timings + cold-start subprocess sample + Anthropic-tokenized payload sizes, appends one JSONL record tobench/history.jsonlper run.--comparediffs the last two runs.Mode B — config cost analysis (
bully bench --config <path>): given any.bully.yml, reports floor tokens per dispatch, per-rule marginal cost (sorted), diff scaling at 1/10/100/1000 added lines, and a scope-grouped breakdown. No history append.Pipeline stays stdlib-only.
anthropicis an optional dep (pip install -e ".[bench]"); without it, both modes fall back to alen(json.dumps(payload))proxy and tag outputmethod: proxy. The bench never dispatches real model calls — onlymessages/count_tokens, which is free.What's in this PR
pipeline/pipeline.py: optionalphase_timerkeyword arg onrun_pipeline(zero-cost no-op by default, wraps 7 named phases); subcommand dispatch forbully benchat top ofmain()pipeline/bench.py(new, ~650 LoC): CLI, fixture loader, token counter (real + proxy fallback),PhaseTimer,run_fixture,run_mode_a,run_mode_b,run_comparebench/fixtures/01-08/: 8 paired-file fixtures covering script/ast/semantic/mixed engines, extends chain, 20-rule payload, 500-line diff, auto-generated-skip short-circuitpipeline/tests/test_bench.py: 24 testspyproject.toml: new[bench]optional dependencies groupREADME.md: "Test Bench" sectiondocs/superpowers/specs/2026-04-17-test-bench-design.md+docs/superpowers/plans/2026-04-17-test-bench.md: spec and planTest Plan
pytest pipeline/tests -q— 267/267 passruff check .+ruff format --check .— cleanpython3 -m pipeline.pipeline bench --no-tokensruns all 8 fixtures and appends a line tobench/history.jsonlpython3 -m pipeline.pipeline bench --config .bully.yml --no-tokensprints a per-rule cost table for the repo's own configpython3 -m pipeline.pipeline bench --comparediffs the last two history entriesANTHROPIC_API_KEYset +pip install -e ".[bench]": confirmmethod: count_tokensin the output (requires real key; defer to local verification)🤖 Generated with Claude Code