Skip to content

Fix PandasArrayExtensionDtype._metadata: must be a tuple, not a string - #8377

Open
saket3395 wants to merge 1 commit into
huggingface:mainfrom
saket3395:fix/pandas-array-extension-dtype-metadata
Open

Fix PandasArrayExtensionDtype._metadata: must be a tuple, not a string#8377
saket3395 wants to merge 1 commit into
huggingface:mainfrom
saket3395:fix/pandas-array-extension-dtype-metadata

Conversation

@saket3395

Copy link
Copy Markdown

Summary

Fixes #8375PandasArrayExtensionDtype._metadata is a plain string instead of a tuple, causing AttributeError: 'PandasArrayExtensionDtype' object has no attribute 'v' whenever pandas compares two instances of this dtype (e.g. during a boolean-mask filter on a DataFrame with an array-feature column read back from parquet).

Root cause: pandas.api.extensions.ExtensionDtype expects _metadata to be a tuple of attribute names used for equality comparison. As a plain string "value_type", pandas iterates over it character-by-character ('v', 'a', 'l', 'u', 'e', ...) when comparing dtypes, and crashes looking up an attribute named 'v'.

_metadata = "value_type"

_metadata = ("value_type",)

Verification

Reproduced the underlying bug in isolation — a minimal ExtensionDtype subclass using only pandas, no datasets import required:

import pandas as pd
from pandas.api.extensions import ExtensionDtype

class BadDtype(ExtensionDtype):
    _metadata = "value_type"
    name = "bad"; type = object; na_value = None
    def __init__(self, value_type): self.value_type = value_type
    @classmethod
    def construct_array_type(cls): return None

class GoodDtype(ExtensionDtype):
    _metadata = ("value_type",)
    name = "good"; type = object; na_value = None
    def __init__(self, value_type): self.value_type = value_type
    @classmethod
    def construct_array_type(cls): return None

BadDtype("float64") == BadDtype("float64")   # AttributeError: 'BadDtype' object has no attribute 'v'
GoodDtype("float64") == GoodDtype("float64") # True

This confirms the string-vs-tuple distinction is the actual cause of the reported AttributeError, and that the tuple form resolves it, independent of anything else in datasets.

Test plan

  • Reproduced the exact failure mode (AttributeError: no attribute 'v') in isolation and confirmed the fix resolves it.
  • Didn't run the full datasets reproduction script from the issue (parquet round-trip + Array2D features) in this environment — didn't want to pull in the full datasets/pyarrow stack just to verify a one-character change whose root cause is fully isolated above. Happy to iterate if CI surfaces anything additional.

Fixes huggingface#8375. pandas.api.extensions.ExtensionDtype expects _metadata to
be a tuple of attribute names. As a plain string, pandas iterates over
each character ('v', 'a', 'l', ...) when comparing dtypes (e.g. during
dtype equality checks triggered by boolean-mask filtering), and crashes
with AttributeError on the first character since there's no attribute
named 'v'.

Changed _metadata = "value_type" to _metadata = ("value_type",).

Reproduced the underlying bug in isolation against a minimal
ExtensionDtype subclass (outside the datasets package, using only
pandas) to confirm the string form raises AttributeError on equality
comparison and the tuple form does not.
@kohankhaki

Copy link
Copy Markdown

This is a copy of a PR I have already created: #8376

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

PandasArrayExtensionDtype._metadata should be a tuple, not a string

2 participants