11.3.1 — Analysis and Balance Reporting (M11.3)

[SorraTheOrc]

Description

Build analysis tooling that consumes aggregated sweep data and generates actionable balance reports identifying overpowered/underpowered mechanics, dominant strategies, unused content, and parameter sensitivity. Extend existing analyze_ai_games.py functionality with statistical rigor and trend detection.

Acceptance Criteria

• Script scripts/analyze_balance.py processes aggregated sweep results and produces HTML or Markdown balance reports
• Reports include sections for:
  • Dominant strategies (win rate deltas >10%)
  • Underperforming mechanics (actions/policies rarely chosen)
  • Unused story seeds
  • Parameter sensitivity analysis (impact of difficulty/config changes)
• Statistical analysis includes confidence intervals, significance testing (e.g., t-tests for win rate differences), and trend detection across historical runs
• Visual outputs (charts/graphs) showing win rate distributions, metric trends over time, and parameter correlations
• Report highlights regressions (new sweeps showing significant deviations from baseline)
• At least 12 tests covering report generation, statistical calculations, and edge cases (empty data, single run)

Priority

High

Dependencies

• Task 11.2.1 (Result Aggregation and Storage) - ✅ COMPLETED
• Task 9.4.1 (AI Tournament Analysis Script) - ✅ COMPLETED

Risks & Mitigations

• Risk: Statistical tests produce false positives
  • Mitigation: Use appropriate significance thresholds and multiple comparison corrections
• Risk: Reports become too verbose
  • Mitigation: Summary-first design with detailed breakdowns in appendices

Next Steps

1. Define report structure and key metrics to surface
2. Implement statistical analysis functions (win rate deltas, significance tests, trend detection)
3. Add visualization generation (matplotlib/plotly for charts)
4. Create test suite with synthetic sweep data

Reference

See .pm/tracker.md task 11.3.1 for full details.

[SorraTheOrc]

FIXED by PR #64: Analysis and Balance Reporting tool and tests merged to main.