ENH: Plotting support for ExtensionArrays

[Julian-Harbeck]

• closes ENH: Plotting support for ExtensionArrays #64300 (Replace xxxx with the GitHub issue number)
• Tests added and passed if fixing a bug or adding a new feature
• All code checks passed.
• Added type annotations to new arguments/methods/functions.
• Added an entry in the latest doc/source/whatsnew/vX.X.X.rst file if fixing a bug or adding a new feature.
• I have reviewed and followed all the contribution guidelines
• If I used AI to develop this pull request, I prompted it to follow AGENTS.md.

Add plotting support for ExtensionArrays:

• Add _is_plotable flag to ExtensionDtype
• Add _get_plot_converter() method to ExtensionDtype to get type and ConversionInterface for matplotlib units registry
• Include EAs in registration of ConversionInterfaces to matplotlib
• Include plotable EA dtypes in _compute_plot_data dtype selection

[Julian-Harbeck]

@jbrockmendel gentle ping, does the proposed API for plotting looks fine to you? Before merging I would also want to add some tests and see if there are any internal EAs that could profit from adapting this API.

[jbrockmendel]

i suspect this will go through plottable_ea_dtypes

[jbrockmendel]

This looks reasonable to me.

How much effort is writing a custom converter? Can you reuse or thin-wrap around one of the existing ones?

[Julian-Harbeck]

How much effort is writing a custom converter? Can you reuse or thin-wrap around one of the existing ones?

I think it is relatively straight forward, see the base implementation of the ConversionInterface:

class ConversionInterface:
    """
    The minimal interface for a converter to take custom data types (or
    sequences) and convert them to values Matplotlib can use.
    """

    @staticmethod
    def axisinfo(unit, axis):
        """Return an `.AxisInfo` for the axis with the specified units."""
        return None

    @staticmethod
    def default_units(x, axis):
        """Return the default unit for *x* or ``None`` for the given axis."""
        return None

    @staticmethod
    def convert(obj, unit, axis):
        """
        Convert *obj* using *unit* for the specified *axis*.

        If *obj* is a sequence, return the converted sequence.  The output must
        be a sequence of scalars that can be used by the numpy array layer.
        """
        return obj

You want to implement default_units to set the unit of an axis from the first data that is plotted on the axis and convert which is used to convert all (following) data sets the default unit of the axis and return a sequence of scalars.

As an example for astropy Quantity you could have a unit of m in the first data set which is then the default unit of the axis and another data set with unit cm which are then converted to m (by essentially dividing by 100) through the convert method. See MplQuantityConverter for the implementation.

I am now second guessing that it might be better to keep plotting for pandas internal extensions in converter.py as this is where TimeConverter, PeriodConverter and DatetimeConverter are defined, also see comment above. Or at least for all ExtensionDtypes that use the same converter classes.

[Julian-Harbeck]

I just saw that I get a UserWarning when trying out this new functionality:

<string>:1: UserWarning: Instantiating UnitsDtype without any arguments.Pass a UnitsDtype instance to silence this warning.

This originates from:

pandas/pandas/core/dtypes/common.py

Lines 1895 to 1907 in 1bba48f

	# registered extension types
	result = registry.find(dtype)
	if result is not None:
	if isinstance(result, type):
	# GH 31356, GH 54592
	warnings.warn(
	f"Instantiating {result.__name__} without any arguments."
	f"Pass a {result.__name__} instance to silence this warning.",
	UserWarning,
	stacklevel=find_stack_level(),
	)
	result = result()
	return result

Call stack:

• _compute_plot_data
• select_dtypes
• check_int_infer_dtype
• infer_dtype_from_object
• pandas_dtype

Specifically the part in infer_dtype_from_object is of interest:

pandas/pandas/core/dtypes/common.py

Lines 1769 to 1775 in 1bba48f

	try:
	dtype = pandas_dtype(dtype)
	except TypeError:
	pass
	if isinstance(dtype, ExtensionDtype):
	return dtype.type

So here we get the UserWarning when calling pandas_dtype but in the ExtensionDtype case all we do with the created instance there is that we call dtype.type directly afterwards which is a property of the dtype and in theory could depend on the instance, but I am not aware of any ExtensionDtypes which are instance depended. Also in the plotting case we are only at the level of checking allowed types and are not handling any data yet so I think there is no way of knowing the flavour of the data at this point.

However earlier in infer_dtype_from_object there is a shortcut if dtype is a type and not an instance but only for subclasses of np.generic:

pandas/pandas/core/dtypes/common.py

Lines 1752 to 1755 in 1bba48f

	if isinstance(dtype, type) and issubclass(dtype, np.generic):
	# Type object from a dtype
	return dtype

Does anyone know why were not also checking for subclasses of ExtensionDtype here? Could be done like so:

if isinstance(dtype, type) and issubclass(dtype, ExtensionDtype):
    # Type object from a dtype

    return dtype.type

[jbrockmendel]

I spent some time on the select_dtypes thing and concluded the thing to do is avoid its usage in the plotting code in favor of something like

mgr = df._mgr._get_data_subset(predicate_for_plottability)
numeric_data = df._constructor_from_mgr(mgr, axes=mgr.axes)

[Julian-Harbeck]

Thank you @jbrockmendel for the idea, I finally found the time to make it work (I think), see changes of latest commit.

As I a had to switch from string representation for dtypes to types directly, I am not sure if I cover all possible ways. Currently I have:

• "datetime" -> np.datetime64
• "datetimetz" -> Timestamp
• "timedelta" -> np.timedelta64
• "object" -> object
• "category" -> CategoricalDtypeType
• "string" -> str

All tests in pandas/tests/plotting seem to work (at least locally), but I am not sure if that actually covers everything, e.g. there was only one test failing before I added the dtype = dtype.numpy_dtype for ArrowDtype line, so Arrow coverage cannot be great.

Also still did not implement the change suggested by Matthew, keeping _is_plottable is a bit more explicit, but I am indifferent on the change.

[jbrockmendel]

what happens with e.g. DecimalDtype? i think that would get selected in main (but isnt tested) but not here

[Julian-Harbeck]

@jbrockmendel I would welcome another round of review :)

I added the BasePlottingTests class for testing EAs for plotting compatibility:

• Includes tests test_plot_on_x_axis and test_plot_on_y_axis, everything should be plottable on x-axis while y-axis needs a specific converter
• Extends ExtensionTests to also inherit from BasePlottingTests
• Added xfail or skip marker to EAs test that cannot be plotted

Also I removed the ArrowDtype -> numpy dtype conversion in the predicate_for_plottability function as some arrow dtypes have an numpy object dtype, but a plottable type (decimal128, time32/64) and instead added more types to cover those.

The following things should still be discussed:

1. I am thinking about moving the registration of the PeriodConverter from the PerdiodDtype._get_plot_converter back to converter.py for the moment. With the current location Period is also added to the types returned by plottable_types, but this has the side effect, that Period is not filtered out through the predicate_for_plottability function anymore as it was before this PR. In theory in my opinion nothing speaks against Period being plottable on the y-axis, but I also could not think of a use case yet and at the moment plotting a Series with PeriodDtype still fails, just later when setting the limits of the y-axis as the PeriodConverter expects that the axis has a freq attribute. I tend to leave that for another PR.
2. For the BasePlottingTests class I was thinking about adding a fixture like skip_if_doesnt_support_2d in Dim2CompatTests, would you prefer that? I already spent some time on trying it out, but had issues with not skipping anything I would like to test. So for now all the EAs that does not support plotting have to xfail the two tests.
3. I added the function _check_plot_data on a suggestion by @jorisvandenbossche to compare the plotted data with the data when manually converted from the series using the registered converter. Currently it is in pandas/tests/extension/base/plotting.py, but I guess there could also be use for it in other plotting tests? Should I move it to pandas/tests/plotting/common.py, same location as the _check_plot_works function?
4. How do you think about testing plotting with different kinds? As the types allowed to be plotted depend on the kind (e.g. for scatter they are extended to [object, CategoricalDtypeType, str]) there could be a fixture specifying the kind supported by each EA.
5. str, binary and Categorical are supported by matplotlib (by now) to be plotted on the y-axis, at the moment pandas filters them out. For now I am xfailing the plot_on_y_axis test for these, but I guess they could be added in? Maybe leave for a future PR?
6. Should this feature by a mentioned in the docs?

[jbrockmendel]

docstring note to return None for non-plottable?

[jbrockmendel]

is doing this as a classmethod going to be a problem in cases where dtype.type might vary across instances? e.g. ArrowDtype?

[jbrockmendel]

should these be xfails or testing for the correct exception?

[jbrockmendel]

i think object being in here means everything matches?

[jbrockmendel]

why are you dropping NA? are EAs with NAs not supported?

[Julian-Harbeck]

The "Minimum Version" CI jobs seems to fail due to an older matplotlib version (3.9.3 according to pyproject.toml) but newer pyparsing version. The PyparsingDeprecationWarning was introduced in pyparsing 3.3.0 and fixed for matplotlib in version 3.11, see the following issue for reference:
matplotlib/matplotlib#29722

I am not sure how to solve this, pin pyparsing to below 3.3.0 if matplotlib is below 3.11? Or just declare a pytest filterwarning on the tests? I guess we do not want to require matplotlib>=3.11 for now as it was only released last month. I suspect the more core issue is that either the pixi minimum-version strategy does not pick up the pyparsing >= 2.3.1 from matplotlib 3.9.3 or that another packages minimum version already requires 3.3.0.

[mroeschke]

Or just declare a pytest filterwarning on the tests?

Yeah I would just use a pytest filterwarning. You can add it in pyproject.toml and add a comment linking to matplotlib/matplotlib#29745

[jbrockmendel]

i dont think this is really an xfail. i think this is the correct behavior and we should be testing with pytest.raises(...)

same for some of the other dtypes like this

[Julian-Harbeck]

@mroeschke after adding the filterwarning to pyproject.toml some other CI jobs failed because pyparsing could not be imported. Therefore I moved it out of there again and first applied it to the BasePlottingTests class or through pytestmark = instead, but as the warning is issued during the import of the matplotlib submodules and not the tests themselves this did not work. So now the latest approach and until now only working one uses warnings.filterwarnings from the standard library.