Describe the bug
There's a small typo in PandasArrayExtensionDtype (src/datasets/features/features.py):
Pandas expects _metadata to be a tuple of attribute names, like ("value_type",), not a plain string (API documentation ref). Because it's a string, pandas iterates over each character in "value_type" when comparing dtypes. So it tries to read attributes named v, a, l, u, e, etc, and crashes on the first one:
AttributeError: 'PandasArrayExtensionDtype' object has no attribute 'v'
This shows up whenever:
- Write a parquet file with array columns (e.g.
array[float64] from Array2D/Array3D features)
- Read it back with
pd.read_parquet() after import datasets
- Filter rows with a boolean mask like
df[mask]
I ran into this while splitting a LeRobot dataset, but the bug itself is in datasets. I think the same root cause is discussed in an issue in LeRobot repo: huggingface/lerobot#2445.
Steps to reproduce the bug
You can reproduce the error using the code below:
import tempfile
from pathlib import Path
import datasets
import pandas as pd
from datasets import Features, Array2D
path = Path(tempfile.mkdtemp()) / "test.parquet"
data = {
"episode_index": [0, 0, 1, 1],
"values": [
[[0.0, 0.0, 0.0], [0.0, 0.0, 0.0]],
[[1.0, 1.0, 1.0], [1.0, 1.0, 1.0]],
[[0.0, 0.0, 0.0], [0.0, 0.0, 0.0]],
[[1.0, 1.0, 1.0], [1.0, 1.0, 1.0]],
],
}
features = Features({
"episode_index": datasets.Value("int64"),
"values": Array2D(shape=(2, 3), dtype="float64"),
})
ds = datasets.Dataset.from_dict(data, features=features)
ds.to_parquet(path)
df2 = pd.read_parquet(path)
print(df2["values"].dtype)
mask = df2["episode_index"].isin([0])
df2[mask]
output:
AttributeError: 'PandasArrayExtensionDtype' object has no attribute 'v'
Expected behavior
df[mask] should work on DataFrames with array[float64] columns read from parquet
Environment info
datasets version: 5.0.2.dev0
- Platform: Linux-6.8.0-136-generic-x86_64-with-glibc2.39
- Python version: 3.12.13
huggingface_hub version: 1.26.0
- PyArrow version: 25.0.0
- Pandas version: 3.0.5
fsspec version: 2026.6.0
Describe the bug
There's a small typo in
PandasArrayExtensionDtype(src/datasets/features/features.py):Pandas expects
_metadatato be a tuple of attribute names, like("value_type",), not a plain string (API documentation ref). Because it's a string, pandas iterates over each character in"value_type"when comparing dtypes. So it tries to read attributes namedv,a,l,u,e, etc, and crashes on the first one:This shows up whenever:
array[float64]from Array2D/Array3D features)pd.read_parquet()afterimport datasetsdf[mask]I ran into this while splitting a LeRobot dataset, but the bug itself is in
datasets. I think the same root cause is discussed in an issue in LeRobot repo: huggingface/lerobot#2445.Steps to reproduce the bug
You can reproduce the error using the code below:
output:
Expected behavior
df[mask]should work on DataFrames witharray[float64]columns read from parquetEnvironment info
datasetsversion: 5.0.2.dev0huggingface_hubversion: 1.26.0fsspecversion: 2026.6.0