Pandas version checks
Reproducible Example
import pandas as pd
df = pd.DataFrame({"A": [1.0, 2.0, 3.0, 4.0]})
df.ewm(com=1).online().mean()
# A
# 0 1.0
# 1 1.5
# 2 2.0
# 3 2.5
df.ewm(com=1).mean()
# A
# 0 1.000000
# 1 1.666667
# 2 2.428571
# 3 3.266667Issue Description
For a single-column DataFrame, .online().mean() drops the decay entirely and
returns the cumulative unweighted mean instead of the exponentially weighted mean.
Two separate off-by-a-dimension mistakes combine to cause it:
-
ExponentialMovingWindow.online builds the deltas array from the column count
rather than the row count:
# pandas/core/window/ewm.py
update_deltas = np.ones(
max(self._selected_obj.shape[-1] - 1, 0), dtype=np.float64
)
-
online_ewma then indexes that array with the column index j, taken from
numba.prange(len(cur)), rather than the row index i — despite the comment
immediately above it stating the intended contract:
# pandas/core/window/online.py
for j in numba.prange(len(cur)):
...
# note that len(deltas) = len(vals) - 1 and deltas[i] is to be
# used in conjunction with vals[i+1]
old_wt[j] *= old_wt_factor ** deltas[j - 1]
For a 1-column frame, (1) makes update_deltas a length-0 array, so deltas[j - 1]
with j = 0 is an out-of-bounds read on an empty array. Under numba's default
(bounds checking off) this yields 0.0, so old_wt_factor ** 0 == 1 and old_wt
never decays.
This is why it has gone unnoticed: for a Series and for frames with 2 or more
columns, every in-range index and the wrapped -1 index all happen to hold 1.0,
so the result is accidentally correct. pandas/tests/window/test_online.py only ever
uses 2-column frames and Series, so nothing covers the broken shape.
The update= path is affected too:
oe = df.ewm(com=1).online()
oe.mean() # [1.0, 1.5, 2.0, 2.5]
oe.mean(update=pd.DataFrame({"A": [5.0]}, index=[4])) # [3.0]
pd.DataFrame({"A": [1.0, 2.0, 3.0, 4.0, 5.0]}).ewm(com=1).mean()
# # [1.0, 1.666667, 2.428571, 3.266667, 4.16129]Both lines date to GH-41888, which added the online EWM operations, so this reproduces
back to 1.3.0. (The axis deprecation in GH-57186 later rewrote
shape[self.axis - 1] to shape[-1], but for the default axis=0 those are the same
expression, so it did not change the behavior.)
Expected Behavior
.online().mean() should agree with .mean() regardless of the number of columns:
df.ewm(com=1).online().mean()
# A
# 0 1.000000
# 1 1.666667
# 2 2.428571
# 3 3.266667
Installed Versions
Details
Reproduces on main (3.1.0.dev0) and on every release back to 1.3.0.
Pandas version checks
Reproducible Example
Issue Description
For a single-column
DataFrame,.online().mean()drops the decay entirely andreturns the cumulative unweighted mean instead of the exponentially weighted mean.
Two separate off-by-a-dimension mistakes combine to cause it:
ExponentialMovingWindow.onlinebuilds the deltas array from the column countrather than the row count:
online_ewmathen indexes that array with the column indexj, taken fromnumba.prange(len(cur)), rather than the row indexi— despite the commentimmediately above it stating the intended contract:
For a 1-column frame, (1) makes
update_deltasa length-0 array, sodeltas[j - 1]with
j = 0is an out-of-bounds read on an empty array. Under numba's default(bounds checking off) this yields
0.0, soold_wt_factor ** 0 == 1andold_wtnever decays.
This is why it has gone unnoticed: for a
Seriesand for frames with 2 or morecolumns, every in-range index and the wrapped
-1index all happen to hold1.0,so the result is accidentally correct.
pandas/tests/window/test_online.pyonly everuses 2-column frames and
Series, so nothing covers the broken shape.The
update=path is affected too:Both lines date to GH-41888, which added the online EWM operations, so this reproduces
back to 1.3.0. (The
axisdeprecation in GH-57186 later rewroteshape[self.axis - 1]toshape[-1], but for the defaultaxis=0those are the sameexpression, so it did not change the behavior.)
Expected Behavior
.online().mean()should agree with.mean()regardless of the number of columns:Installed Versions
Details
Reproduces on main (3.1.0.dev0) and on every release back to 1.3.0.