Skip to content

TST: Add test for writing UUIDs to parquet with pyarrow #61602 #75

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
xirafa wants to merge 28 commits into
base: main
Choose a base branch
from
Open

TST: Add test for writing UUIDs to parquet with pyarrow #61602 #75

Changes from all commits
Commits
Show all changes
28 commits
Select commit Hold shift + click to select a range
3bbeea2
TST: Add test for writing UUIDs to parquet with pyarrow #61602
GiTaDi-CrEaTe May 15, 2026
88d4e28
TST: Fix read_parquet namespace usage
GiTaDi-CrEaTe May 15, 2026
9d98157
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] May 15, 2026
5f4d739
TST: Address reviewer feedback for UUID pyarrow test
GiTaDi-CrEaTe May 15, 2026
b2f9c8b
Merge branch 'main' into tests/io-parquet-uuid-61602
GiTaDi-CrEaTe May 15, 2026
72dc35a
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] May 15, 2026
c53e34c
TST: Implement reviewer feedback and fix formatting
GiTaDi-CrEaTe May 15, 2026
6b7b57f
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] May 15, 2026
1f36ae7
Merge branch 'main' into tests/io-parquet-uuid-61602
GiTaDi-CrEaTe May 16, 2026
e699965
TST: Handle raw bytes fallback for UUIDs on PyArrow nightly/py314
GiTaDi-CrEaTe May 16, 2026
f4f3e4e
Merge branch 'main' into tests/io-parquet-uuid-61602
GiTaDi-CrEaTe May 16, 2026
91f481c
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] May 16, 2026
6332052
TST: Fix line length in comment to pass ruff
GiTaDi-CrEaTe May 16, 2026
8925886
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] May 16, 2026
0139402
TST: Use temp_file fixture instead of tmp_path
GiTaDi-CrEaTe May 17, 2026
9053a75
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] May 17, 2026
bdd3ae3
TST: Replace UUID byte fallback with pytest.xfail for upstream bug
GiTaDi-CrEaTe May 17, 2026
b244666
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] May 17, 2026
764647b
TST: Refactor dynamic xfail to use request.node.add_marker
GiTaDi-CrEaTe May 17, 2026
9fe4745
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] May 17, 2026
07ccaef
TST: Shorten xfail reason string to pass ruff line length
GiTaDi-CrEaTe May 17, 2026
a09deb0
TST: Unconditionally xfail UUID pyarrow test per reviewer request
GiTaDi-CrEaTe Jun 9, 2026
1ad08a5
TST: Fix ruff line length and read_parquet namespace usage
GiTaDi-CrEaTe Jun 10, 2026
214203b
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Jun 10, 2026
c3ae88c
TST: Unconditionally xfail UUID pyarrow test and link upstream Arrow bug
GiTaDi-CrEaTe Jul 1, 2026
a43313d
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Jul 1, 2026
aa6d566
TST: Update xfail reason to reflect stable PyArrow bug
GiTaDi-CrEaTe Jul 9, 2026
e0d6174
TST: Remove comments per reviewer request
GiTaDi-CrEaTe Aug 1, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
17 changes: 17 additions & 0 deletions pandas/tests/io/test_parquet.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,7 @@
from io import BytesIO
import os
import pathlib
import uuid

import numpy as np
import pytest
Expand All @@ -20,6 +21,7 @@
pa_version_under20p0,
)
from pandas.errors import Pandas4Warning
import pandas.util._test_decorators as td

import pandas as pd
import pandas._testing as tm
Expand Down Expand Up @@ -1521,3 +1523,18 @@ def test_invalid_dtype_backend(self, engine, temp_file):
df.to_parquet(temp_file)
with pytest.raises(ValueError, match=msg):
read_parquet(temp_file, dtype_backend="numpy")


@pytest.mark.xfail(
reason="Upstream PyArrow fails to cast FIXED_LEN_BYTE_ARRAY to UUID - GH 61602"

@coderabbitai coderabbitai Bot Jul 29, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
set -euo pipefail

echo "== locate file and target lines =="
wc -l pandas/tests/io/test_parquet.py 2>/dev/null || true
sed -n '1515,1545p' pandas/tests/io/test_parquet.py 2>/dev/null || true

echo "== search related references =="
rg -n "GH 61602|61602|FIXED_LEN_BYTE_ARRAY|UUID|50312|apache/arrow/issues" pandas/tests io pandas -S 2>/dev/null | head -n 120

Repository: autodag-dev/pandas

Length of output: 8295

🌐 Web query:

Apache Arrow FIXED_LEN_BYTE_ARRAY UUID Arrow issue 50312

💡 Result:

Apache Arrow issue 50312 refers to a bug involving the improper handling of UUID extension types during conversion between PyArrow and pandas [1]. Specifically, when a PyArrow Table containing a UUID column is converted to a pandas DataFrame, the data is returned as raw bytes instead of Python uuid.UUID objects [1][2]. Key details regarding this issue include: Root Cause: The issue stems from the UuidType in PyArrow not implementing to_pandas_dtype, which causes Table.to_pandas to fall back to the underlying storage type, fixed_size_binary(16), resulting in bytes [2]. Status: As of July 29, 2026, the issue is open and actively being addressed [1]. A pull request (GH-50325) was initiated to fix this by adding the missing method [2]. However, project maintainers have suggested a more robust, performance-optimized implementation in C++ rather than the initial Python-level fix [3]. Broader Context: This is part of ongoing work to improve UUID and extension type support in PyArrow [4][5]. Other recent related issues have included inconsistencies in Parquet schema reading (where extension types were dropped) [4] and difficulties in constructing UuidArrays from scalars [5]. Users encountering this behavior should follow the progress on GitHub issue 50312 for updates and the final resolution [1].

Citations:

Add the upstream Arrow issue to the xfail reason.

GH 61602 tracks the pandas test, but this PyArrow FIXED_LEN_BYTE_ARRAY → UUID regression is tracked separately as Apache Arrow issue #50312. Include both references so maintainers can determine when this unconditional xfail can be removed.

Proposed fix 
 `@pytest.mark.xfail`(
-    reason="Upstream PyArrow fails to cast FIXED_LEN_BYTE_ARRAY to UUID - GH 61602"
+    reason=(
+        "Upstream PyArrow fails to cast FIXED_LEN_BYTE_ARRAY back to UUID - "
+        "Pandas GH 61602 / Arrow issue: "
+        "https://github.com/apache/arrow/issues/50312"
+    )
 )
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
reason="Upstream PyArrow fails to cast FIXED_LEN_BYTE_ARRAY to UUID - GH 61602"
reason=(
"Upstream PyArrow fails to cast FIXED_LEN_BYTE_ARRAY back to UUID - "
"Pandas GH 61602 / Arrow issue: "
"https://github.com/apache/arrow/issues/50312"
)
🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@pandas/tests/io/test_parquet.py` at line 1529, Update the xfail reason near
the existing GH 61602 reference to also cite Apache Arrow issue `#50312`,
preserving the explanation about PyArrow’s FIXED_LEN_BYTE_ARRAY-to-UUID casting
failure and both issue references.

Source: MCP tools

)
@td.skip_if_no("pyarrow", min_version="24.0.0")
def test_to_parquet_uuid_supported(temp_file):
# GH 61602
expected = pd.DataFrame({"id": [uuid.uuid4(), uuid.uuid4()]})

expected.to_parquet(temp_file, engine="pyarrow")

result = read_parquet(temp_file, engine="pyarrow")

tm.assert_frame_equal(result, expected)