REF: Use moments accumulator from libalgos to compute variance by Alvaro-Kothe · Pull Request #66302 · pandas-dev/pandas

Alvaro-Kothe · 2026-07-12T13:16:33Z

xref REF: Reductions #53261 (Replace xxxx with the GitHub issue number)
xref Discussion: SIMD strategy for pandas C/C++ code #64884
closes BUG: nansem with mask + skipna + axis={0,1} returns scalar np.nan instead of array of np.nan #65373
Tests added and passed if fixing a bug or adding a new feature
All code checks passed.
Added type annotations to new arguments/methods/functions.
Added an entry in the latest doc/source/whatsnew/vX.X.X.rst file if fixing a bug or adding a new feature.
I have reviewed and followed all the contribution guidelines
If I used AI to develop this pull request, I prompted it to follow AGENTS.md.

This is a continuation of #64848. It wasn't possible to reopen that, so I am opening this new PR instead.

Use the moments accumulator functions introduced in #64366 and #65801 to compute variance, standard error and standard error of the mean.

The performance increase for var and std only applies if bottleneck isn't installed (or is disabled). Otherwise, it prefers the bottleneck implementation.

Benchmarks

without bottleneck

Details

Change	Before [`7986b42`] <refactor/combine-var-impl~7>	After [`0240c2e`] <refactor/combine-var-impl>	Ratio	Benchmark (Parameter)
+	2.38±0.02ms	3.32±0.3ms	1.39	stat_ops.FrameOps.time_op('var', 'int', 1)
+	72.5±0.3μs	91.8±2μs	1.27	groupby.GroupByMethods.time_dtype_as_field('int', 'var', 'direct', 1, 'cython')
+	4.47±0.03ms	5.28±0.03ms	1.18	stat_ops.FrameOps.time_op('std', 'Int64', 1)
+	4.38±0.02ms	5.15±0.03ms	1.17	stat_ops.FrameOps.time_op('var', 'Int64', 1)
+	3.76±0.04ms	4.36±0.03ms	1.16	rolling.Methods.time_method('DataFrame', ('expanding', {}), 'float', 'sem')
+	1.58±0.01ms	1.81±0.01ms	1.15	rolling.Methods.time_method('DataFrame', ('expanding', {}), 'float', 'std')
+	3.77±0.02ms	4.26±0.1ms	1.13	rolling.Methods.time_method('DataFrame', ('expanding', {}), 'int', 'sem')
+	4.96±0.04ms	5.57±0.05ms	1.12	rolling.Methods.time_method('Series', ('rolling', {'window': 10}), 'int', 'sem')
+	73.6±2μs	81.8±1μs	1.11	groupby.GroupByMethods.time_dtype_as_field('int16', 'var', 'direct', 1, 'cython')
+	79.9±0.7μs	88.2±0.6μs	1.11	groupby.GroupByMethods.time_dtype_as_group('int', 'std', 'direct', 1, 'cython')
+	2.54±0.02ms	2.80±0.02ms	1.1	rolling.Methods.time_method('Series', ('rolling', {'window': 10}), 'float', 'std')
-	5.32±0.05ms	4.68±0.03ms	0.88	stat_ops.FrameOps.time_op('sem', 'Int64', 1)
-	4.31±0.2ms	3.44±0.4ms	0.8	stat_ops.FrameOps.time_op('std', 'float', 1)
-	298±5μs	220±20μs	0.74	stat_ops.SeriesOps.time_op('var', 'int')
-	1.72±0.01ms	1.15±0.08ms	0.67	stat_ops.FrameOps.time_op('var', 'int', None)
-	328±10μs	218±4μs	0.66	stat_ops.SeriesOps.time_op('std', 'int')
-	1.73±0.01ms	1.13±0.07ms	0.65	stat_ops.FrameOps.time_op('std', 'int', None)
-	1.89±0.1ms	1.18±0.1ms	0.63	stat_ops.FrameOps.time_op('std', 'int', 0)
-	48.8±1μs	30.4±0.4μs	0.62	series_methods.NanOps.time_func('std', 1000, 'float64')
-	1.82±0.02ms	1.12±0.07ms	0.62	stat_ops.FrameOps.time_op('var', 'int', 0)
-	4.13±0.1ms	2.54±0.01ms	0.61	stat_ops.FrameOps.time_op('var', 'float', 1)
-	4.97±1ms	2.79±0.02ms	0.56	series_methods.NanOps.time_func('var', `1000000`, 'int64')
-	9.02±0.5ms	4.68±0.02ms	0.52	stat_ops.FrameOps.time_op('sem', 'float', 1)
-	5.68±1ms	2.88±0.08ms	0.51	series_methods.NanOps.time_func('std', `1000000`, 'int64')
-	38.5±1μs	18.0±0.2μs	0.47	series_methods.NanOps.time_func('std', 1000, 'int64')
-	38.2±1μs	17.5±0.3μs	0.46	series_methods.NanOps.time_func('std', 1000, 'int32')
-	32.6±0.5μs	15.0±0.5μs	0.46	series_methods.NanOps.time_func('var', 1000, 'int32')
-	39.8±0.2μs	17.8±0.3μs	0.45	series_methods.NanOps.time_func('std', 1000, 'int8')
-	33.8±1μs	15.3±0.7μs	0.45	series_methods.NanOps.time_func('var', 1000, 'int64')
-	7.23±0.3ms	3.23±0.02ms	0.45	stat_ops.FrameOps.time_op('sem', 'int', 1)
-	81.7±1μs	36.0±1μs	0.44	series_methods.NanOps.time_func('sem', 1000, 'float64')
-	17.2±0.02ms	7.62±0.01ms	0.44	stat_ops.FrameMixedDtypesOps.time_op('std', 0)
-	36.3±3μs	15.1±0.2μs	0.42	series_methods.NanOps.time_func('var', 1000, 'int8')
-	102±3μs	41.5±0.7μs	0.41	series_methods.NanOps.time_func('sem', 1000, 'Int64')
-	88.3±2μs	35.4±1μs	0.4	series_methods.NanOps.time_func('sem', 1000, 'int32')
-	101±5μs	38.7±0.7μs	0.39	series_methods.NanOps.time_func('sem', 1000, 'boolean')
-	89.1±2μs	34.9±1μs	0.39	series_methods.NanOps.time_func('sem', 1000, 'int64')
-	6.92±0.2ms	2.67±0.09ms	0.39	series_methods.NanOps.time_func('std', `1000000`, 'int32')
-	6.52±0.3ms	2.52±0.08ms	0.39	series_methods.NanOps.time_func('std', `1000000`, 'int8')
-	6.87±0.2ms	2.58±0.01ms	0.38	series_methods.NanOps.time_func('var', `1000000`, 'int32')
-	6.42±0.3ms	2.44±0.01ms	0.38	series_methods.NanOps.time_func('var', `1000000`, 'int8')
-	90.4±0.8μs	33.2±0.6μs	0.37	series_methods.NanOps.time_func('sem', 1000, 'int8')
-	41.7±0.8μs	13.5±0.6μs	0.32	series_methods.NanOps.time_func('var', 1000, 'float64')
-	3.05±0.02ms	906±60μs	0.3	stat_ops.FrameMultiIndexOps.time_op('std')
-	2.94±0.1ms	887±50μs	0.3	stat_ops.FrameOps.time_op('std', 'float', 0)
-	569±30μs	166±8μs	0.29	stat_ops.SeriesOps.time_op('var', 'float')
-	641±30μs	179±10μs	0.28	stat_ops.SeriesMultiIndexOps.time_op('std')
-	600±5μs	167±4μs	0.28	stat_ops.SeriesMultiIndexOps.time_op('var')
-	3.01±0.01ms	800±6μs	0.27	stat_ops.FrameMultiIndexOps.time_op('var')
-	637±20μs	160±3μs	0.25	stat_ops.SeriesOps.time_op('std', 'float')
-	8.64±0.3ms	2.00±0.04ms	0.23	series_methods.NanOps.time_func('std', `1000000`, 'float64')
-	1.15±0.08ms	263±20μs	0.23	stat_ops.SeriesOps.time_op('sem', 'int')
-	5.05±0.1ms	1.09±0.01ms	0.22	stat_ops.FrameOps.time_op('sem', 'int', 0)
-	2.89±0.1ms	643±10μs	0.22	stat_ops.FrameOps.time_op('var', 'float', 0)
-	5.82±0.4ms	1.24±0.01ms	0.21	stat_ops.FrameOps.time_op('sem', 'Int64', 0)
-	8.86±0.3ms	1.77±0ms	0.2	series_methods.NanOps.time_func('var', `1000000`, 'float64')
-	5.99±0.2ms	1.20±0.01ms	0.2	stat_ops.FrameOps.time_op('sem', 'float', 0)
-	15.8±1ms	2.98±0.02ms	0.19	series_methods.NanOps.time_func('sem', `1000000`, 'int64')
-	4.34±0.01ms	830±50μs	0.19	stat_ops.FrameOps.time_op('std', 'float', None)
-	17.5±2ms	3.09±0.03ms	0.18	series_methods.NanOps.time_func('sem', `1000000`, 'Int64')
-	14.7±1ms	2.60±0.02ms	0.18	series_methods.NanOps.time_func('sem', `1000000`, 'boolean')
-	6.20±0.02ms	1.14±0.08ms	0.18	stat_ops.FrameMultiIndexOps.time_op('sem')
-	1.27±0.02ms	233±8μs	0.18	stat_ops.SeriesMultiIndexOps.time_op('sem')
-	1.27±0.05ms	233±9μs	0.18	stat_ops.SeriesOps.time_op('sem', 'float')
-	17.5±0.7ms	2.79±0.01ms	0.16	series_methods.NanOps.time_func('sem', `1000000`, 'int32')
-	17.8±0.6ms	2.60±0.02ms	0.15	series_methods.NanOps.time_func('sem', `1000000`, 'float64')
-	17.6±0.8ms	2.63±0.01ms	0.15	series_methods.NanOps.time_func('sem', `1000000`, 'int8')
-	11.9±1ms	1.82±0.08ms	0.15	stat_ops.FrameOps.time_op('sem', 'Int64', None)
-	6.84±0.4ms	1.04±0.01ms	0.15	stat_ops.FrameOps.time_op('sem', 'int', None)
-	4.71±0.4ms	605±6μs	0.13	stat_ops.FrameOps.time_op('var', 'float', None)
-	9.58±0.7ms	1.13±0.01ms	0.12	stat_ops.FrameOps.time_op('sem', 'float', None)

with bottleneck

Details

Change	Before [`f135efd`]	After [a73c47dc] <refactor/combine-var-impl>	Ratio	Benchmark (Parameter)
+	9.19±0.2ms	12.0±0.2ms	1.3	stat_ops.FrameOps.time_op('var', 'int', 1)
+	1.10±0.01ms	1.33±0.01ms	1.21	stat_ops.FrameOps.time_op('var', 'int', None)
+	14.0±0.1μs	16.2±0.3μs	1.15	series_methods.NanOps.time_func('std', 1000, 'float64')
+	110±0.9ms	125±2ms	1.14	groupby.GroupByCythonAggEaDtypes.time_frame_agg('Int64', 'var')
+	74.3±1μs	83.8±0.8μs	1.13	groupby.GroupByMethods.time_dtype_as_field('int', 'var', 'direct', 1, 'cython')
+	1.58±0.02ms	1.78±0.02ms	1.13	rolling.Methods.time_method('Series', ('expanding', {}), 'float', 'std')
+	75.0±0.5μs	83.8±0.4μs	1.12	groupby.GroupByMethods.time_dtype_as_field('float', 'sem', 'direct', 1, 'cython')
+	14.4±0.4μs	16.1±0.4μs	1.12	series_methods.NanOps.time_func('std', 1000, 'int64')
+	2.90±0.03ms	3.19±0.02ms	1.1	rolling.VariableWindowMethods.time_method('Series', '1h', 'float', 'std')
+	14.6±0.3μs	16.2±0.4μs	1.1	series_methods.NanOps.time_func('std', 1000, 'int32')
+	1.64±0.02ms	1.79±0.05ms	1.09	rolling.EWMMethods.time_ewm('DataFrame', ({'halflife': 1000}, 'std'), 'float')
+	3.04±0.02ms	3.31±0.02ms	1.09	rolling.VariableWindowMethods.time_method('Series', '1h', 'int', 'std')
+	5.96±0.04ms	6.38±0.03ms	1.07	rolling.VariableWindowMethods.time_method('Series', '1h', 'int', 'sem')
+	2.84±0.02ms	3.02±0.1ms	1.06	rolling.VariableWindowMethods.time_method('DataFrame', '1d', 'float', 'std')
+	3.00±0.02ms	3.17±0.1ms	1.06	rolling.VariableWindowMethods.time_method('DataFrame', '1d', 'int', 'std')
-	1.42±0.01ms	1.17±0.01ms	0.82	stat_ops.FrameOps.time_op('var', 'float', 0)
-	12.8±0.05ms	10.4±0.1ms	0.81	stat_ops.FrameMixedDtypesOps.time_op('std', 0)
-	50.7±0.5μs	39.6±0.8μs	0.78	series_methods.NanOps.time_func('sem', 1000, 'int32')
-	52.7±3μs	39.9±0.3μs	0.76	series_methods.NanOps.time_func('sem', 1000, 'int64')
-	20.0±0.7ms	14.9±0.3ms	0.74	stat_ops.FrameOps.time_op('sem', 'float', 1)
-	2.70±0.02ms	1.72±0.1ms	0.64	stat_ops.FrameOps.time_op('sem', 'int', 0)
-	7.14±0.2ms	4.21±0.02ms	0.59	series_methods.NanOps.time_func('sem', `1000000`, 'int32')
-	7.42±0.2ms	4.41±0.1ms	0.59	series_methods.NanOps.time_func('sem', `1000000`, 'int64')
-	7.20±0.2ms	4.09±0.02ms	0.57	series_methods.NanOps.time_func('sem', `1000000`, 'int8')
-	3.14±0.2ms	1.72±0.1ms	0.55	stat_ops.FrameOps.time_op('sem', 'int', None)
-	706±30μs	376±4μs	0.53	stat_ops.SeriesOps.time_op('sem', 'int')
-	73.7±5μs	35.4±0.5μs	0.48	series_methods.NanOps.time_func('sem', 1000, 'float64')
-	95.8±2μs	39.9±1μs	0.42	series_methods.NanOps.time_func('sem', 1000, 'int8')
-	113±5μs	40.9±1μs	0.36	series_methods.NanOps.time_func('sem', 1000, 'Int64')
-	113±4μs	38.9±1μs	0.34	series_methods.NanOps.time_func('sem', 1000, 'boolean')
-	4.04±0.04ms	1.26±0.03ms	0.31	stat_ops.FrameOps.time_op('sem', 'float', 0)
-	4.08±0.08ms	1.17±0.09ms	0.29	stat_ops.FrameMultiIndexOps.time_op('sem')
-	991±60μs	248±4μs	0.25	stat_ops.SeriesMultiIndexOps.time_op('sem')
-	5.99±0.4ms	1.43±0.1ms	0.24	stat_ops.FrameOps.time_op('sem', 'Int64', 0)
-	1.04±0.04ms	219±4μs	0.21	stat_ops.SeriesOps.time_op('sem', 'float')
-	5.71±0.04ms	1.11±0.05ms	0.2	stat_ops.FrameOps.time_op('sem', 'float', None)
-	13.9±0.4ms	2.67±0.06ms	0.19	series_methods.NanOps.time_func('sem', `1000000`, 'float64')
-	19.3±2ms	3.29±0.09ms	0.17	series_methods.NanOps.time_func('sem', `1000000`, 'Int64')
-	18.2±0.4ms	2.88±0.2ms	0.16	series_methods.NanOps.time_func('sem', `1000000`, 'boolean')
-	12.4±2ms	1.84±0.01ms	0.15	stat_ops.FrameOps.time_op('sem', 'Int64', None)

jbrockmendel · 2026-07-21T01:22:15Z

 - APIs that accept an ``engine="numba"`` parameter with ``engine_kwargs`` will no longer pass through a ``nopython`` argument to ``numba.jit``. This argument has had no effect since numba 0.59.0 (:issue:`64483`).
 - Removed the ``freq`` and ``freqstr`` attributes from :class:`.DatetimeArray` and :class:`.TimedeltaArray`. Frequency is now stored only on :class:`DatetimeIndex` and :class:`TimedeltaIndex`; access ``Series.dt.freq`` or wrap the array in an Index to retrieve a frequency. The ``check_freq`` keyword on :func:`testing.assert_extension_array_equal` for these array types has also been removed (:issue:`24566`).
-
+- :meth:`DataFrame.var`, :meth:`DataFrame.std`, :meth:`DataFrame.sem`, :meth:`DataFrame.skew` and :meth:`DataFrame.kurt`  with ``numeric_only=False`` now converts numeric strings to their float representation for its intermediate computations (:issue:`64366`, :issue:`66302`)


is this API change central to the refactor here? i dont think its a change we want to make

There is a requirement that the array must be of floats. So, somewhere we must convert the array datatype to np.float64.

There are some ways to avoid the behavior change:

Keep and use the old implementation for object dtype.

Use _ensure_numeric to emit an error if it finds a string in the array before trying to convert to float64. The main problem in using this function is the error message that prints the whole array and due to error message mismatch, several tests fail.

In the current implementation, it converts numeric strings when computing avg , but fails when computing the central differences and this is the error message asserted in test_numeric_only_flag.

Whatever we decide here must also apply to nankurt and nanskew.

i lean toward a consistent _ensure_numeric call across the board. Though i agree printing the whole array doesn't make sense

Updated the code and the benchmark for the case without bottleneck. LMK if the performance regression reported in there is acceptable. Otherwise, we can just patch the skewness and kurtosis case - I opened a PR for them in #66495.

Alvaro-Kothe force-pushed the refactor/combine-var-impl branch from a70538c to 55b85c0 Compare July 12, 2026 20:06

Alvaro-Kothe marked this pull request as ready for review July 12, 2026 20:28

jbrockmendel reviewed Jul 13, 2026

View reviewed changes

Comment thread pandas/core/nanops.py

Alvaro-Kothe force-pushed the refactor/combine-var-impl branch 2 times, most recently from e3cec07 to dd294b1 Compare July 18, 2026 01:31

jbrockmendel reviewed Jul 21, 2026

View reviewed changes

Comment thread pandas/core/nanops.py Outdated

Alvaro-Kothe mentioned this pull request Jul 21, 2026

BUG: fix mask order mismatch with values for nanskew and nankurt #66408

Merged

7 tasks

Alvaro-Kothe added 7 commits July 27, 2026 18:54

perf(moments-axis): use SIMD scalar reduction in contiguous assess

af371bb

refactor(var): use moments accumulator to compute variance

d922e27

fix: mismatched mask order with values

bd85fdb

fix: fix mypy error for order variable

4bbf829

fix(nanvar): don't implicitly convert numeric strings

a1ae0b6

fix: fix failing tests

3ef08fd

docs: remove entry about behavior change

0240c2e

Alvaro-Kothe force-pushed the refactor/combine-var-impl branch from d9bb1fc to 0240c2e Compare July 27, 2026 22:40

xirafa mentioned this pull request Jul 27, 2026

REF: Use moments accumulator from libalgos to compute variance autodag-dev/pandas#4

Open

Alvaro-Kothe mentioned this pull request Jul 27, 2026

BUG: don't implicitly convert numeric strings in nanskew and nankurt #66495

Draft

7 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

REF: Use moments accumulator from libalgos to compute variance - #66302

REF: Use moments accumulator from libalgos to compute variance#66302
Alvaro-Kothe wants to merge 7 commits into
pandas-dev:mainfrom
Alvaro-Kothe:refactor/combine-var-impl

Alvaro-Kothe commented Jul 12, 2026 •

edited

Loading

Uh oh!

Uh oh!

jbrockmendel Jul 21, 2026

Uh oh!

Alvaro-Kothe Jul 21, 2026

Uh oh!

jbrockmendel Jul 22, 2026

Uh oh!

Alvaro-Kothe Jul 28, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Uh oh!

Conversation

Alvaro-Kothe commented Jul 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Benchmarks

without bottleneck

with bottleneck

Uh oh!

Uh oh!

jbrockmendel Jul 21, 2026

Choose a reason for hiding this comment

Uh oh!

Alvaro-Kothe Jul 21, 2026

Choose a reason for hiding this comment

Uh oh!

jbrockmendel Jul 22, 2026

Choose a reason for hiding this comment

Uh oh!

Alvaro-Kothe Jul 28, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Alvaro-Kothe commented Jul 12, 2026 •

edited

Loading