Fix PandasArrayExtensionDtype._metadata to use a tuple - #8376
Conversation
ebarkhordar
left a comment
There was a problem hiding this comment.
Confirmed, and the fix reaches further than the issue reports: with _metadata as a string, dtype == dtype and hash(dtype) raise the same AttributeError, so pd.concat on two frames carrying an ArrayXD column fails as well. All of those work at b690a53.
One gap before this closes #8375. The boolean-mask symptom survives for integer ArrayXD columns, one frame lower:
import datasets
features = datasets.Features({"foo": datasets.Array2D(dtype="int32", shape=(2, 2)), "bar": datasets.Value("int64")})
ds = datasets.Dataset.from_dict({"foo": [[[1, 1], [1, 1]], [[2, 2], [2, 2]]], "bar": [0, 1]}, features=features)
df = ds.to_pandas()
df[df.bar == 1]File "src/datasets/features/features.py", line 961, in take
self.dtype.na_value if fill_value is None else np.asarray(fill_value, dtype=self.dtype.value_type)
ValueError: cannot convert float NaN to integer
take_nd calls arr.take(indexer, fill_value=nan, allow_fill=True) for a boolean mask even when the indexer contains no -1, and PandasArrayExtensionArray.take coerces fill_value before it looks at the mask. array[float64] and array[bool] survive because NaN casts into both. Reproduced on pandas 2.3.3 and 3.0.5.
Deferring the coercion until a fill is actually needed clears it:
if allow_fill:
mask = indices == -1
if mask.any():
fill_value = (
self.dtype.na_value if fill_value is None else np.asarray(fill_value, dtype=self.dtype.value_type)
)The snippet above then returns the masked frame, and tests/features/test_array_xd.py stays at 103 passed. A genuine fill on an integer array is still unrepresentable, which is a separate question. Glad to send that as its own PR if you would rather keep this one to the one-liner.
Separately: test_array_xd.py is 103/103 on main too, so nothing pins the dtype today. test_table_to_pandas already builds the frame, and one line there covers the regression:
assert df.foo.dtype == df.foo.dtype
What was wrong
PandasArrayExtensionDtypeset_metadata = "value_type"(a string), but pandas expects a tuple of attribute names, e.g.("value_type",).When pandas compares dtypes (during things like
df[mask]), it loops over_metadataand looks up each entry as an attribute. With a string, it iterates over individual characters ("v","a","l", ...), which raises:AttributeError: 'PandasArrayExtensionDtype' object has no attribute 'v'This breaks boolean indexing on parquet DataFrames that have
array[float64]columns. I specifically hit it when splitting a LeRobot dataset after import.What changed
Fixes #8375