Cap outliers by percentile winsorization

[ondraz]

Summary

Adds optional percentile winsorization to metrics so that by-unit experiment evaluation stays robust when heavy-tailed metrics (revenue, time spent, actions per user, …) contain extreme outliers. A handful of extreme units can dominate a variant's mean and variance; when those outliers are concentrated in one variant they distort the comparison and cost statistical power.

Approach

Per-unit nominator values are capped (winsorized) to a percentile threshold before aggregation:

• The threshold is computed once from the pooled per-unit values across all variants and applied identically to every variant. Using the same absolute cap in every arm makes winsorization a single fixed transformation of the metric — it cannot, on its own, introduce a difference between variants. Computing the threshold per variant would cut each arm's tail at a different absolute value and could bias (or even create) the measured effect, so this is deliberately avoided.
• Units are capped, not removed. Unlike trimming, the denominator / sample size is preserved and no information is discarded.
• Upper tail by default, lower tail opt-in. Heavy-tailed metrics typically only have upper-tail outliers; the lower tail is usually a pile of legitimate zeros.

Configuration

Two optional parameters on Metric / SimpleMetric (and the HTTP API metric definition), each a percentage 0–50 of the tail to cap:

• outlier_upper_percentile — cap the upper tail. 1 caps every per-unit value above the pooled 99th percentile down to the 99th-percentile value.
• outlier_lower_percentile — cap the lower tail (analogous, off by default).

Metric(
    1,
    'Revenue per User',
    'value(test_unit_type.unit.revenue)',
    'count(test_unit_type.unit.exposure)',
    outlier_upper_percentile=1,  # cap the most extreme 1% of users
)

Scope & caveats

• Applies only to evaluate_by_unit (it needs the individual per-unit values); the pre-aggregated evaluate_agg path is unaffected.
• Winsorization shifts the estimand slightly — you estimate the mean of the capped metric. This is the intended robustness/bias trade-off; keep the capped fraction small (typically ≤ 1%).
• The standard deviation reported after winsorization is the ordinary capped sample SD; at small capped fractions the difference from a dedicated winsorized-variance estimator is negligible.

Changes

• parser.py — pooled winsorization in Parser.evaluate_by_unit (no-op when disabled; the original code path is unchanged).
• metric.py — outlier_upper_percentile / outlier_lower_percentile config + validation to [0, 50).
• server/req.py — exposed both fields on the API metric definition and wired them through to_metric.
• tests/epstats/toolkit/test_outlier_winsorization.py — caps preserve count (vs. trimming), pooled threshold applied identically across variants, lower-tail flooring, disabled == 0-percentiles, and validation.
• docs/stats/outliers.md (+ Statistics nav) — methodology page; Metric docstrings updated.

🤖 Generated with Claude Code