feat: implement pivot for the pyarrow backend - #3816
Open
mkzung wants to merge 2 commits into
Open
Conversation
Closes narwhals-dev#2179. ArrowDataFrame.pivot was not_implemented. pyarrow has no native pivot, so this groups by index + on, then scatters each aggregated cell into its output row and column, matched on Python tuples so it stays correct when index or on hold nulls (a pyarrow join cannot match null keys). It follows polars for column names, order, null handling, and the empty-cell fill. median maps to pyarrow's approximate hash median, so its exact-value test stays xfail for pyarrow (same as group_by median).
first/last are ordered aggregators: pyarrow computes them only in single-threaded execution (use_threads=False), which Table.group_by does not accept before 14.0. Pass the flag only when it is needed and available, mirroring _configure_grouped in group_by.py, and xfail the ordered pivot cases on pyarrow<14. Make _pivot_reshape's schema arguments keyword-only so it stays under the positional-argument limit (PLR0917).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Implements
pivotfor the pyarrow backend (it wasnot_implemented).pyarrow has no native pivot, so it groups by
index+on, then scatters each aggregatedcell into its output row and column. Cells are matched on Python tuples, so it stays correct
when
indexoronhold nulls (a pyarrow join cannot match null keys).It follows polars for column names (
{"a","b"}for multipleon, a{value}{separator}prefix for multiple
values,true/false/nullfor booleans and nulls), column order(occurring
oncombinations by first appearance, or sorted), empty-cell fill (0forsumand
len, null otherwise), null handling infirst/last, and the "not unique" error foraggregate_function=None.medianmaps to pyarrow's approximate hash median (there is no exact one), asgroup_by(...).agg(median())already does, so its exact-value test stays xfail for pyarrow.What type of PR is this? (check all applicable)
Related issues
pivotto Arrow #2179AI assistance
Checklist