Keep sequential confidence intervals finite and warn on negative variance estimates

[ondraz]

Problem

Two related issues that surface as NaN/inf confidence intervals and NaN p-values when evaluating experiments:

1. Infinite confidence intervals early in a sequential test. Statistics.obf_alpha_spending_function rounds the adjusted confidence level to 4 decimal places. Early in an experiment the O'Brien-Fleming adjustment produces values like 0.9999984 which round to exactly 1.0. The t-quantile of confidence level 1.0 is infinite, so every confidence interval evaluates to inf or NaN (rel_se * inf), accompanied by a cryptic RuntimeWarning: invalid value encountered in multiply from ttest_evaluation. For example, on day 3 of an 18-day test the adjusted alpha is ~1.6e-6, which rounds to a confidence level of exactly 1.0.

2. Silent NaN results on inconsistent pre-aggregated data. evaluate_agg requires sum_sqr_value to be a sum of squared per-unit goal totals. When a metric matches multiple pre-aggregated rows per unit — typically a dimensional goal summed across several dimension values per unit, with squares computed per dimension value — the variance estimate (sum_sqr_value - sum_value^2/count) / (count - 1) goes negative, and std, p-value and confidence interval all turn NaN with no explanation. This cost us a lengthy investigation in production; the failure mode deserves a loud, named warning.

Changes

• Cap the OBF-adjusted confidence level at MAX_SEQUENTIAL_CONFIDENCE_LEVEL = 0.9999. Confidence intervals early in a test are now finite and appropriately wide instead of infinite. Test expectations and the test-sequential-v1 golden data updated accordingly.
• Emit an explicit UserWarning from metric evaluation when any metric has a negative variance estimate, naming the affected metrics, the likely cause and the remedies (pre-aggregate sum_sqr_value from per-unit totals or use evaluate_by_unit).
• Add a regression test pinning that evaluate_by_unit computes sum_sqr_value from per-unit totals when a unit's goal is split into multiple input rows (the pivot in evaluate_by_unit collapses them before squaring — this behavior is correct today and now guarded).

Testing

make check passes: ruff clean, all 152 tests pass including the new ones (test_obf_alpha_spending_function_keeps_confidence_intervals_finite, test_agg_warns_on_negative_variance, test_by_unit_sums_unit_rows_before_squaring).

🤖 Generated with Claude Code