Skip to content

BUG: object and extension-dtype reductions honour skipna=False (GH-4147) - #66508

Open
jbrockmendel wants to merge 4 commits into
pandas-dev:mainfrom
jbrockmendel:bug-18588-skipna
Open

BUG: object and extension-dtype reductions honour skipna=False (GH-4147)#66508
jbrockmendel wants to merge 4 commits into
pandas-dev:mainfrom
jbrockmendel:bug-18588-skipna

Conversation

@jbrockmendel

Copy link
Copy Markdown
Member

Stacked on GH-66507 (test-only, closes the five duplicate reports) — please review that
one first; this PR's diff assumes it. Supersedes GH-66471, which mixed the two and has
been closed.

GH-66507 covers the skipna=True half of object-dtype min/max, already fixed on
main by GH-65526. Writing those tests turned up four further defects in the same code
path, all of which are fixed here. References GH-4147 and GH-18588 for context; the
closing keywords live on GH-66507.

1. skipna=False on object dtype

_nanminmax gated the NA fill on dtype == object and skipna. With skipna=False the
_get_values fallback never builds a mask for object, so NAs went into the comparison
as-is. A float NaN propagates through a comparison on its own; an object NA does not:

>>> pd.Series(["a", None, "d"], dtype=object).min(skipna=False)
TypeError: '<=' not supported between instances of 'str' and 'float'
>>> pd.Series([1.0, np.nan, 3.0], dtype=object).max(skipna=False)   # wrong
3.0

Both now return a missing value. A reduction that covers the whole array short-circuits
to NA; a per-axis 2-D reduction blanks the NA-holding slices and nulls them through
_maybe_null_out. Values in a nulled slice are never compared, so they need not even be
comparable with each other — Series([1, "a", None], dtype=object).min(skipna=False) is
NA rather than a TypeError, while the NA-free Series([1, "a"]) still raises.

2. idxmin/idxmax on object dtype

nanargmin/nanargmax kept the ±inf fill, so they still raised on exactly the data
min/max now handles:

>>> pd.Series([date(2018, 1, 1), None, date(2018, 1, 3)]).idxmax()
TypeError: '>' not supported between instances of 'float' and 'datetime.date'

They now share _nanminmax's same-slice fill. _maybe_fix_arg_at_na needs no change:
if an argmin/argmax lands on a masked position, the fill must be at least as extreme as
every valid value, so it is the extremum, and it sits at the first unmasked position —
exactly what (~mask).argmax() returns. Verified by brute force against an NA-dropped
reference rather than by argument alone. With skipna=False they now raise the same
ValueError other dtypes do instead of TypeError.

3. sum/prod on object dtype

Same defect in nansum/nanprod: an object NA does not propagate through +/* the
way a float NaN does, so Series([2, None], dtype=object).sum(skipna=False) raised.

Note the asymmetry with min/max: NAs that do support the operation (pd.NA,
Decimal("NaN"), NaT) already propagate correctly and carry more type information
than a bare nan, so those are left alone — the natural reduction is attempted first and
the fill only steps in when it raises. Series([1, pd.NA, 3], dtype=object).sum(skipna=False)
stays <NA>.

4. skipna=False dropped by the groupby fallback

Dtypes with no Cython aggregation fall back to _agg_py_fallbackagg_series(ser, alt)
with alt=np.min, which has no way to receive skipna and silently dropped it:

>>> df = pd.DataFrame({"g": [1, 1], "v": pd.Series(["a", None], dtype="str")})
>>> df.groupby("g")["v"].min(skipna=False)   # wrong, expected NA
g
1    a

npfunc is now a closure that applies skipna. The same fallback backs
DataFrame.min(axis=1) on string columns, so that is fixed by the same change. Reach is
min/max for object/StringDtype/IntervalDtype/other EAs, prod for object and
EAs, and sum for EAs only — object sum has a Cython implementation and already
honoured skipna.

Verification

Beyond the suite: brute-force fuzz against an independent NA-dropped reference; a
systematic mutation sweep over every new source line (every surviving mutation closed and
re-checked); and a differential run of ~63k reduction calls against main with every
difference classified as intended-and-documented or investigated.

Not addressed

mean/std/var/sem on object with skipna=False still raise TypeError. Same
defect, but pre-existing and untouched here; worth a follow-up. Object string sum
with skipna=True also still raises, on the skipna=True path this PR does not modify.

jbrockmendel and others added 3 commits July 28, 2026 11:03
Adds regression tests for the family of duplicate reports fixed by GH#65526:
GH#4147, GH#18588, GH#24109, GH#58707 and GH#61204. Existing coverage was
Series-only, so the 2D take_along_axis branch of _nanminmax that GH#18588 and
GH#61204 report was untested.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…#4147)

_nanminmax gated its NA fill on `dtype == object and skipna`. With skipna=False
the _get_values fallback never builds a mask for object dtype, so NAs were fed
into the comparison as-is: a float NaN propagates through a comparison on its
own, but an object NA does not. min/max either raised TypeError or returned a
value computed as though the NA were absent.

De-gate the object branch so the fill always runs, then null out any slice that
held an NA via _maybe_null_out(min_count=<slice length>), matching float64, str
and datetime64.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Accumulated review fixes across the object-dtype reduction work: idxmin/idxmax
object fill, the groupby _agg_py_fallback skipna closures, nansum/nanprod, the
whole-array short-circuit, and the accompanying tests and whatsnew entries.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
# Conflicts:
#	doc/source/whatsnew/v3.1.0.rst
#	pandas/tests/frame/test_reductions.py
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant