Add test bench for speed + input-token cost#2

Merged

christopherarter merged 21 commits into

mainfrom

p1-review-epic

Apr 17, 2026

Commits on Apr 17, 2026

Add test bench design spec

Chris Arter
and
claude
committed
Add test bench implementation plan

Chris Arter
and
claude
committed
Add optional phase_timer hook to run_pipeline

Chris Arter
and
claude
committed
Address Task 1 review: drop redundant type annotations, tighten phase test

Chris Arter
and
claude
committed
Add bench module skeleton and CLI entry point

Chris Arter
and
claude
committed
Drop __getattr__ import hack; match existing sibling-module test pattern

Chris Arter
and
claude
committed
Add bench fixture loader

Chris Arter
and
claude
committed
Add token counter with Anthropic SDK + proxy fallback

Chris Arter
and
claude
committed
Add PhaseTimer and single-fixture runner

Chris Arter
and
claude
committed
Short-circuit token counting when no semantic rules match

Chris Arter
and
claude
committed
Add Mode A: fixture suite runner + history writer

Chris Arter
and
claude
committed
Add bench --compare: diff last two runs in history

Chris Arter
and
claude
committed
Add Mode B: config cost analysis

Chris Arter
and
claude
committed
Add 8 bench fixtures covering script/ast/semantic engines

Chris Arter
and
claude
committed
Document the bench in README

Chris Arter
and
claude
committed
Save and restore BULLY_TRUST_ALL across run_fixture

Chris Arter
and
claude
committed
Simplify run_mode_b return type to dict | None

Chris Arter
and
claude
committed
Make --config and --compare mutually exclusive in bully bench

Chris Arter
and
claude
committed
Exclude bench/fixtures from ruff

Chris Arter
and
claude
committed
Dogfood ruff: add ruff-clean rule to .bully.yml

Chris Arter
and
claude
committed
Apply ruff format to pipeline/bench.py and test_bench.py

Chris Arter
and
claude
committed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Add test bench for speed + input-token cost#2

Add test bench for speed + input-token cost#2
christopherarter merged 21 commits into
mainfrom
p1-review-epic

Commits on Apr 17, 2026

Add test bench design spec

Add test bench implementation plan

Add optional phase_timer hook to run_pipeline

Address Task 1 review: drop redundant type annotations, tighten phase test

Add bench module skeleton and CLI entry point

Drop getattr import hack; match existing sibling-module test pattern

Add bench fixture loader

Add token counter with Anthropic SDK + proxy fallback

Add PhaseTimer and single-fixture runner

Short-circuit token counting when no semantic rules match

Add Mode A: fixture suite runner + history writer

Add bench --compare: diff last two runs in history

Add Mode B: config cost analysis

Add 8 bench fixtures covering script/ast/semantic engines

Document the bench in README

Save and restore BULLY_TRUST_ALL across run_fixture

Simplify run_mode_b return type to dict | None

Make --config and --compare mutually exclusive in bully bench

Exclude bench/fixtures from ruff

Dogfood ruff: add ruff-clean rule to .bully.yml

Apply ruff format to pipeline/bench.py and test_bench.py