Fix PandasArrayExtensionDtype._metadata: must be a tuple, not a string - #8377
Open
saket3395 wants to merge 1 commit into
Open
Fix PandasArrayExtensionDtype._metadata: must be a tuple, not a string#8377saket3395 wants to merge 1 commit into
saket3395 wants to merge 1 commit into
Conversation
Fixes huggingface#8375. pandas.api.extensions.ExtensionDtype expects _metadata to be a tuple of attribute names. As a plain string, pandas iterates over each character ('v', 'a', 'l', ...) when comparing dtypes (e.g. during dtype equality checks triggered by boolean-mask filtering), and crashes with AttributeError on the first character since there's no attribute named 'v'. Changed _metadata = "value_type" to _metadata = ("value_type",). Reproduced the underlying bug in isolation against a minimal ExtensionDtype subclass (outside the datasets package, using only pandas) to confirm the string form raises AttributeError on equality comparison and the tuple form does not.
|
This is a copy of a PR I have already created: #8376 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes #8375 —
PandasArrayExtensionDtype._metadatais a plain string instead of a tuple, causingAttributeError: 'PandasArrayExtensionDtype' object has no attribute 'v'whenever pandas compares two instances of this dtype (e.g. during a boolean-mask filter on a DataFrame with an array-feature column read back from parquet).Root cause:
pandas.api.extensions.ExtensionDtypeexpects_metadatato be a tuple of attribute names used for equality comparison. As a plain string"value_type", pandas iterates over it character-by-character ('v','a','l','u','e', ...) when comparing dtypes, and crashes looking up an attribute named'v'.→
Verification
Reproduced the underlying bug in isolation — a minimal
ExtensionDtypesubclass using only pandas, nodatasetsimport required:This confirms the string-vs-tuple distinction is the actual cause of the reported
AttributeError, and that the tuple form resolves it, independent of anything else indatasets.Test plan
AttributeError: no attribute 'v') in isolation and confirmed the fix resolves it.datasetsreproduction script from the issue (parquet round-trip +Array2Dfeatures) in this environment — didn't want to pull in the fulldatasets/pyarrowstack just to verify a one-character change whose root cause is fully isolated above. Happy to iterate if CI surfaces anything additional.