Cap outliers by percentile winsorization

docs/stats/outliers.md

@@ -0,0 +1,74 @@
+# Outlier Winsorization
+
+Some metrics are heavy-tailed: revenue, time spent, number of actions per user, etc. A handful of
+extreme units (e.g. a single user who spends 1000× the typical amount) can dominate the per-variant
+mean and, even more so, the variance. When such outliers are concentrated in one variant, the
+[by-unit evaluation][epstats.toolkit.experiment.Experiment.evaluate_by_unit] of that metric becomes
+unreliable — the test loses power and the estimated effect is distorted.
+
+Ep-Stats can make the evaluation robust to these outliers by **winsorizing** the per-unit values
+before the metric is aggregated.
+
+## What winsorization does
+
+Winsorization **caps** (clips) extreme per-unit values to a percentile threshold instead of letting
+them through unchanged. For example, capping the upper `1%` replaces every per-unit value above the
+99th percentile with the 99th-percentile value.
+
+Two design choices make this safe for experiment evaluation:
+
+1. **The threshold is computed once from the pooled data across all variants and applied
+   identically to every variant.** Because the very same absolute cap is used in every arm,
+   winsorization is a single fixed transformation of the metric — it cannot, on its own, introduce a
+   difference between variants. Computing the threshold *separately per variant* would cut the tail
+   of each arm at a different absolute value and could bias (or even create) the measured effect, so
+   Ep-Stats deliberately does **not** do that.
+
+2. **Units are capped, not removed.** Unlike trimming, the units stay in the sample, so the
+   denominator and the sample size are preserved. This keeps more information and avoids removing a
+   different number of units from each arm.
+
+## Upper vs. lower tail
+
+Heavy-tailed metrics typically only have outliers in the **upper** tail; the lower tail is usually a
+pile of legitimate zeros (most users do not convert). Winsorizing the lower tail of such a metric
+would floor those real zeros and bias the mean upward. For that reason upper-tail winsorization is
+the common case and lower-tail winsorization is opt-in and off by default.
+
+## Configuration
+
+Winsorization is configured per [`Metric`][epstats.toolkit.metric.Metric] through two optional
+parameters, each expressed as a percentage (`0`–`50`) of the tail to cap:
+
+- `outlier_upper_percentile` — cap the upper tail. `1` caps everything above the 99th pooled
+  percentile down to the 99th-percentile value.
+- `outlier_lower_percentile` — cap the lower tail (analogous, off by default).
+
+```python
+from epstats.toolkit import Metric
+
+Metric(
+    1,
+    'Revenue per User',
+    'value(test_unit_type.unit.revenue)',
+    'count(test_unit_type.unit.exposure)',
+    # cap the most extreme 1% of users to the pooled 99th percentile
+    outlier_upper_percentile=1,
+)
+```
+
+Over the HTTP API the same parameters are available on the metric definition as
+`outlier_upper_percentile` and `outlier_lower_percentile`.
+
+## Scope and caveats
+
+- Winsorization only applies to [`evaluate_by_unit`][epstats.toolkit.experiment.Experiment.evaluate_by_unit],
+  because it needs the individual per-unit values. The pre-aggregated
+  [`evaluate_agg`][epstats.toolkit.experiment.Experiment.evaluate_agg] path has no per-unit data and
+  is unaffected.
+- Winsorization changes the estimand slightly: you are estimating the mean of the *capped* metric
+  rather than of the raw metric. This is the intended trade-off — a small, controlled bias in
+  exchange for robustness to extreme values. Keep the capped fraction small (typically `≤ 1%`).
+- The standard deviation reported after winsorization is the ordinary (capped) sample standard
+  deviation. With large samples and a small capped fraction the difference from a dedicated
+  winsorized-variance estimator is negligible.

mkdocs.yml

@@ -28,6 +28,7 @@ nav:
       - CTR Metric Redefined: 'stats/ctr.md'
       - Multiple Comparison Correction: 'stats/multiple.md'
       - Compound Metrics: 'stats/compound_metrics.md'
+      - Outlier Winsorization: 'stats/outliers.md'
   - API:
     - Metric: 'api/metric.md'
     - Experiment: 'api/experiment.md'

src/epstats/server/req.py

@@ -49,6 +49,27 @@ class Metric(BaseModel):
         description=f"""The minimum effect of interest is the smallest relative difference that is meaningful to detect,
         defining it allows us to estimate the size of the sample data required to reach {DEFAULT_POWER:.0%} power.""",
     )
+    outlier_upper_percentile: Optional[float] = Field(
+        None,
+        ge=0,
+        lt=50,
+        title="Upper-tail winsorization percentile",
+        description="""Percentage (`0`-`50`) of the upper tail to winsorize (cap) before the metric is evaluated.
+        E.g. `1` caps every per-unit value above the 99th percentile down to the 99th percentile value. The threshold
+        is computed once from the pooled per-unit values across all variants and applied identically to every variant,
+        so it makes the evaluation robust to extreme outliers without biasing the comparison between variants. Units
+        are kept (only their extreme values are capped). Applies only to by-unit evaluation. Defaults to no
+        winsorization.""",
+    )
+    outlier_lower_percentile: Optional[float] = Field(
+        None,
+        ge=0,
+        lt=50,
+        title="Lower-tail winsorization percentile",
+        description="""Percentage (`0`-`50`) of the lower tail to winsorize (floor), analogous to
+        `outlier_upper_percentile`. Usually left empty because heavy-tailed metrics only have outliers in the upper
+        tail. Applies only to by-unit evaluation. Defaults to no winsorization.""",
+    )
 
     @model_validator(mode="after")
     def check_nominator_denominator(self):
@@ -60,10 +81,14 @@ def check_nominator_denominator(self):
         try:
             parser = Parser(nominator, denominator)
             if not parser.get_goals():
-                raise ValueError("We expect the metric to have at least one goal in nominator and denominator")
+                raise ValueError(
+                    "We expect the metric to have at least one goal in nominator and denominator"
+                )
             return self
         except ParseException as e:
-            raise ValueError(f"Cannot parse nominator '{nominator}' or '{denominator}' because of '{e}'")
+            raise ValueError(
+                f"Cannot parse nominator '{nominator}' or '{denominator}' because of '{e}'"
+            )
 
     def to_metric(self):
         return EvMetric(
@@ -72,6 +97,8 @@ def to_metric(self):
             nominator=self.nominator,
             denominator=self.denominator,
             minimum_effect=self.minimum_effect,
+            outlier_upper_percentile=self.outlier_upper_percentile,
+            outlier_lower_percentile=self.outlier_lower_percentile,
         )
 
 
@@ -125,7 +152,9 @@ def _validate_nominator_or_denominator(value, which):
         try:
             parser = Parser(value, value)
             if not parser.get_goals():
-                raise ValueError(f"We expect the check to have at least one goal in {which}")
+                raise ValueError(
+                    f"We expect the check to have at least one goal in {which}"
+                )
             return value
         except ParseException as e:
             raise ValueError(f"Cannot parse '{value}' because of '{e}'")
@@ -163,7 +192,9 @@ class Filter(BaseModel):
 
     value: Optional[List[Any]] = Field(None, title="List of possible values")
 
-    goal: Optional[str] = Field(None, title="Specify goals if filter scope is `trigger`")
+    goal: Optional[str] = Field(
+        None, title="Specify goals if filter scope is `trigger`"
+    )
 
     @model_validator(mode="after")
     def check_trigger(self):
@@ -197,7 +228,9 @@ class Experiment(BaseModel):
     )
 
     variants: Optional[List[str]] = Field(
-        None, title="Variants", description="""List of experiment variants to evaluate for."""
+        None,
+        title="Variants",
+        description="""List of experiment variants to evaluate for.""",
     )
 
     unit_type: str = Field(
@@ -305,8 +338,12 @@ def date_to_must_be_date(cls, value):
 
     @model_validator(mode="after")
     def check_date_from_to(self):
-        if self.date_for is not None and (self.date_from is None or self.date_to is None):
-            raise ValueError("date_for requires date_from and date_to to be present as well")
+        if self.date_for is not None and (
+            self.date_from is None or self.date_to is None
+        ):
+            raise ValueError(
+                "date_for requires date_from and date_to to be present as well"
+            )
         if self.date_from is not None and self.date_to is not None:
             try:
                 df = datetime.strptime(self.date_from, "%Y-%m-%d")  # noqa: PD901
@@ -319,11 +356,17 @@ def check_date_from_to(self):
                 except ValueError:
                     raise ValueError("cannot parse date_for")
                 if dfor < df:
-                    raise ValueError("we expect date_for to be greater or equal to date_from")
+                    raise ValueError(
+                        "we expect date_for to be greater or equal to date_from"
+                    )
                 if dfor > dt:
-                    raise ValueError("we expect date_for to be less or equal to date_to")
+                    raise ValueError(
+                        "we expect date_for to be less or equal to date_to"
+                    )
             if dt < df:
-                raise ValueError("we expect date_to to be greater or equal to date_from")
+                raise ValueError(
+                    "we expect date_to to be greater or equal to date_from"
+                )
 
         return self
 

src/epstats/toolkit/metric.py

@@ -20,6 +20,8 @@ def __init__(
         metric_format: str = "{:.2%}",
         metric_value_multiplier: int = 1,
         minimum_effect: Optional[float] = None,
+        outlier_upper_percentile: Optional[float] = None,
+        outlier_lower_percentile: Optional[float] = None,
     ):
         """
         Constructor of the general metric definition.
@@ -37,6 +39,19 @@ def __init__(
             denominator: definition of denominator
             metric_format: specify format of the metric, e.g. '${:,.1f}' for RPM
             metric_value_multiplier: specify multiplier, e.g. 1000 for RPM
+            minimum_effect: minimum (relative) effect of interest used to estimate the required sample size
+            outlier_upper_percentile: percentage (`0`-`50`) of the upper tail to winsorize (cap) before the
+                metric is evaluated. E.g. `1` caps every per-unit value above the 99th percentile down to the
+                99th percentile value. The threshold is computed once from the pooled per-unit values across
+                all variants and applied identically to every variant, so it makes the evaluation robust to
+                extreme outliers without biasing the comparison between variants. Unlike trimming, units are
+                kept (only their extreme values are capped), so the denominator / sample size is preserved.
+                Applies only to [`evaluate_by_unit`][epstats.toolkit.experiment.Experiment.evaluate_by_unit].
+                `None` (default) disables upper-tail winsorization.
+            outlier_lower_percentile: percentage (`0`-`50`) of the lower tail to winsorize (floor), analogous
+                to `outlier_upper_percentile`. Usually left as `None` because heavy-tailed metrics (revenue,
+                time spent, ...) typically only have outliers in the upper tail while the lower tail is made of
+                legitimate zeros.
 
         Usage:
 
@@ -48,6 +63,12 @@ def __init__(
             'count(test_unit_type.global.exposure)')
         ```
         """
+        self._validate_outlier_percentile(
+            "outlier_upper_percentile", outlier_upper_percentile
+        )
+        self._validate_outlier_percentile(
+            "outlier_lower_percentile", outlier_lower_percentile
+        )
 
         self.id = id
         self.name = name
@@ -58,6 +79,15 @@ def __init__(
         self.metric_format = metric_format
         self.metric_value_multiplier = metric_value_multiplier
         self.minimum_effect = minimum_effect
+        self.outlier_upper_percentile = outlier_upper_percentile
+        self.outlier_lower_percentile = outlier_lower_percentile
+
+    @staticmethod
+    def _validate_outlier_percentile(name: str, percentile: Optional[float]) -> None:
+        if percentile is not None and not 0 <= percentile < 50:
+            raise ValueError(
+                f"`{name}` must be in `[0, 50)` but `{percentile}` received."
+            )
 
     def get_goals(self) -> Set:
         """
@@ -97,7 +127,18 @@ def get_evaluate_columns_by_unit(self, goals: pd.DataFrame) -> np.array:
             numpy array of shape (variants, metrics) where metrics are in order of
             (count, sum_value, sum_sqr_value)
         """
-        return self._parser.evaluate_by_unit(goals)
+        lower_quantile, upper_quantile = self._get_outlier_quantiles()
+        return self._parser.evaluate_by_unit(goals, lower_quantile, upper_quantile)
+
+    def _get_outlier_quantiles(self) -> (float, float):
+        """
+        Translate the upper/lower outlier percentiles (percentage winsorized from each tail) into
+        the `(lower_quantile, upper_quantile)` pair used to cap per-unit values. Returns `(0.0, 1.0)`
+        (no winsorization) when outlier handling is disabled.
+        """
+        lower_quantile = (self.outlier_lower_percentile or 0) / 100
+        upper_quantile = 1 - (self.outlier_upper_percentile or 0) / 100
+        return lower_quantile, upper_quantile
 
 
 class SimpleMetric(Metric):
@@ -115,6 +156,8 @@ def __init__(
         metric_format: str = "{:.2%}",
         metric_value_multiplier: int = 1,
         minimum_effect: Optional[float] = None,
+        outlier_upper_percentile: Optional[float] = None,
+        outlier_lower_percentile: Optional[float] = None,
     ):
         """
         Constructor of the simplified metric definition.
@@ -149,4 +192,14 @@ def __init__(
         num = "value" + "(" + unit_type + "." + agg_type + "." + numerator + ")"
         den = "value" + "(" + unit_type + "." + agg_type + "." + denominator + ")"
 
-        super().__init__(id, name, num, den, metric_format, metric_value_multiplier, minimum_effect)
+        super().__init__(
+            id,
+            name,
+            num,
+            den,
+            metric_format,
+            metric_value_multiplier,
+            minimum_effect,
+            outlier_upper_percentile,
+            outlier_lower_percentile,
+        )

src/epstats/toolkit/parser.py

@@ -112,13 +112,22 @@ def evaluate_agg(self, goals: pd.DataFrame):
         count = self._denominator_expr.evaluate_agg(goals)
         return count, sum_value, sum_sqr_value
 
-    def evaluate_by_unit(self, goals: pd.DataFrame):
+    def evaluate_by_unit(
+        self,
+        goals: pd.DataFrame,
+        lower_quantile: float = 0.0,
+        upper_quantile: float = 1.0,
+    ):
         """
         Get `count`, `sum_value`, `sum_value_sqr` numpy array of shape (variants, metrics) after
         evaluating nominator and denominator expressions from goals aggregated by unit.
 
         Arguments:
             goals: one row per experiment variant
+            lower_quantile: per-unit nominator values below this pooled quantile (in `[0, 1]`) are capped
+                (winsorized) to it. `0.0` caps nothing.
+            upper_quantile: per-unit nominator values above this pooled quantile (in `[0, 1]`) are capped
+                (winsorized) to it. `1.0` caps nothing.
 
         See [`Experiment.evaluate_agg`][epstats.toolkit.experiment.Experiment.evaluate_by_unit] for details
         on `goals` at input.
@@ -130,6 +139,10 @@ def evaluate_by_unit(self, goals: pd.DataFrame):
         value_variants, value = self._nominator_expr.evaluate_by_unit(
             goals, "sum_value"
         )
+
+        if lower_quantile > 0.0 or upper_quantile < 1.0:
+            value = self._winsorize(value, lower_quantile, upper_quantile)
+
         value_df = (
             pd.DataFrame(
                 {
@@ -152,6 +165,25 @@ def evaluate_by_unit(self, goals: pd.DataFrame):
         metrics_df = value_df.join(count_df)
         return metrics_df["count"], metrics_df["sum_value"], metrics_df["sum_sqr_value"]
 
+    @staticmethod
+    def _winsorize(
+        value: pd.Series,
+        lower_quantile: float,
+        upper_quantile: float,
+    ) -> pd.Series:
+        """
+        Cap per-unit `value`s to the `[lower_quantile, upper_quantile]` range (winsorization).
+
+        The capping thresholds are computed once from the pooled per-unit values across all variants
+        and applied identically to every variant. Because the same absolute threshold is used for all
+        variants, winsorization makes the evaluation robust to extreme outliers without biasing the
+        comparison between variants. Units are kept (only their extreme values are capped), so the
+        denominator / sample size is preserved.
+        """
+        lower = value.quantile(lower_quantile) if lower_quantile > 0.0 else None
+        upper = value.quantile(upper_quantile) if upper_quantile < 1.0 else None
+        return value.clip(lower=lower, upper=upper)
+
     def get_goals(self) -> Set:
         """
         Get set of goals that appear in `nominator` and `denominator` expressions as `EpGoal` instances.