Skip to content

BUG/API: ewm with equally spaced times disagrees with no-times for adjust=False, ignore_na=False on NaN data #66523

Description

@jbrockmendel

Pandas version checks

  • I have checked that this issue has not already been reported.
  • I have confirmed this bug exists on the latest version of pandas.
  • I have confirmed this bug exists on the main branch of pandas.

Reproducible Example

import numpy as np
import pandas as pd

ser = pd.Series([1.0, np.nan, 5.0])
times = pd.date_range("2000", freq="D", periods=3)  # equally spaced

ser.ewm(com=1, adjust=False, ignore_na=False).mean()
# 2    3.666667

ser.ewm(halflife="1D", times=times, adjust=False, ignore_na=False).mean()
# 2    4.000000

Issue Description

times that are equally spaced should be equivalent to not passing times at all —
that invariant is asserted directly by test_ewma_with_times_equal_spacing. It holds
for three of the four adjust × ignore_na combinations, but breaks for
adjust=False, ignore_na=False once the data contains NaN:

data = np.arange(10.0)
data[::2] = np.nan
df = pd.DataFrame({"A": data})
times = pd.date_range("2000", freq="D", periods=10)

for adjust in (True, False):
    for ignore_na in (True, False):
        lhs = df.ewm(halflife="1D", times=times, adjust=adjust, ignore_na=ignore_na).mean()
        rhs = df.ewm(halflife=1.0, adjust=adjust, ignore_na=ignore_na).mean()
        print(adjust, ignore_na, np.allclose(lhs, rhs, equal_nan=True))

# True  True   True
# True  False  True
# False True   True
# False False  False   <-- diverges

The two paths generalize adjust=False to a gap of Δ differently. They agree at
Δ = 1 and diverge otherwise:

  • no times — normalized, matching the recursion documented for ignore_na:
    y_t = ((1-α)^Δ · y_u + α · x_t) / ((1-α)^Δ + α)
  • times — convex, added in ENH: Allow adjust=False when times is provided #59142 to support irregular intervals:
    y_t = (1-α)^Δ · y_u + (1 - (1-α)^Δ) · x_t

With ignore_na=False a NaN gap pushes Δ above 1, so the two disagree.

test_ewma_with_times_equal_spacing does not catch this because it is not
parametrized over adjust, and GH-59142's own test_ewma_with_times_variable_spacing
uses np.arange(3) with no NaN, so the interaction with NaN was never covered.

Worth noting: this divergence is newly visible as of GH-66521. Before that fix both
spellings agreed — but they agreed on the wrong value, because the com == 1 special
case intended for the times path was leaking into the no-times path (GH-31178).
GH-66521 deliberately fixed only the no-times path and left times exactly as 3.0
shipped it, which is what surfaces the inconsistency.

Expected Behavior

Undecided, and that is really the question here — the two candidate generalizations
are both defensible, so this needs a call on intended semantics:

  1. Make times normalized across NaN gaps, restoring the equal-spacing
    invariant and making ewm NaN handling is wrong : simple example #31178's example return 3.666667 in both spellings. Note
    this cannot be done by simply switching the times path to the normalized form:
    test_ewma_with_times_variable_spacing value-pins the convex form for
    adjust=False (0.2376251864...; the normalized form gives 0.39608). It would
    require distinguishing decay from elapsed interval from decay accumulated across
    skipped NaN rows.
  2. Declare the convex form correct for times and document that
    ignore_na=False weights NaN gaps differently once times is supplied,
    accepting that equally spaced times is not equivalent to no times in this one
    combination.

Related: GH-54328 (the request GH-59142 implemented) and GH-40098.

Installed Versions

Details

Reproduces on main (3.1.0.dev0). The times side is unchanged since 3.0.0; the
divergence between the two spellings appears with GH-66521.

Metadata

Metadata

Assignees

No one assigned

    Labels

    API - ConsistencyInternal Consistency of API/BehaviorBugMissing-datanp.nan, pd.NaT, pd.NA, dropna, isnull, interpolateNeeds DiscussionRequires discussion from core team before further actionWindowrolling, ewma, expanding

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions