v1.1.1

First experiment results

Experiment 001 (CLAUDE.md effect) ran to completion - 5 runs per condition on claude-haiku-4-5 against the typescript-eslint codebase. Results are now tracked in the repo alongside experiment configs.

Headline numbers: hand-curated CLAUDE.md improved pass@1 from 60% to 100%, cut duration by 15%, and reduced token usage by 13%. Effect sizes are large (Cohen's d around -0.85 to -1.0), though N=5 means confidence intervals still overlap. Full report in experiments/001-claude-md-effect/results/report.md.

Results tracking changed: .gitignore no longer blanket-ignores results/ directories. Metrics, diffs, reports, and summaries are now committed. Only session.json files (400-800KB agent logs) stay ignored.

---

Fixes

• Report median was wrong: the Duration (median) row displayed the average for every condition except the first, due to a broken lambda that only resolved against conditions[0]. Median is now computed directly per condition, matching how the CI row already worked.
• Dead code in row(): removed a conditional where both branches did the same thing (avg if "avg" not in label else avg) and an unused bold parameter that wrapped labels in ** nobody consumed.

---

Housekeeping

• Added a task backlog (.tasks/) with 8 items tracking planned framework improvements, using YAML frontmatter for metadata
• Fixed step numbering in the experiment creation skill - rounds now number from 1 instead of continuing a global counter across all rounds
• Applied ruff formatting across report_generator.py
• Cleaned up project documentation

v1.1.0

What's new

Worktree dependency setup

Worktrees created by git worktree add only contain tracked files — dependencies like node_modules or venv are absent. Experiments can now declare a project.setup_command in experiment.yaml that runs in each worktree before the agent starts:

project:
  setup_command: "pnpm install --frozen-lockfile"

An optional project.setup_timeout (defaults to verification timeout) prevents hung installs from blocking the experiment.

Quick iteration with --runs

No more editing experiment.yaml to do a smoke test. Override runs per condition from the CLI:

bash framework/orchestrator.sh experiments/001-claude-md-effect/ --runs 1

The report now shows actual vs configured count (e.g., Runs/condition | 2 (config: 5)) so partial runs are transparent.

Cost estimation

Reports now include estimated USD cost per run and per condition, computed from official Claude API pricing. Token breakdown covers input, output, cache read, and cache creation separately. Pricing lives in framework/pricing.json for easy updates.

---

Fixes

• Agent SIGTTIN hang — the timeout command creates a new process group, causing claude to lose terminal access. Any stdin read during init stopped the process permanently with zero output. Fixed by redirecting stdin from /dev/null.
• Report per_condition mismatch — the report displayed the YAML config value even when --runs overrode it. Now derived from actual metrics data.
• Setup errors silent — project setup previously swallowed stderr. Errors are now visible for debugging.

---

Internal improvements

• Report generator extracted from a 390-line bash heredoc into standalone report_generator.py with pricing.json
• Agent and verification commands wrapped in configurable timeout (defaults: agent 3600s, verification 600s)
• Orchestrator uses proper option parsing instead of positional args
• Creating-experiment skill updated with setup_command interview question and template field
• CLAUDE.md artifact for typescript-eslint experiment simplified and reorganized

v1.0.0

Multi-hypothesis benchmark framework with git worktree isolation, interleaved scheduling, and statistical rigor.