TST: Add test for writing UUIDs to parquet with pyarrow #61602 - #75
TST: Add test for writing UUIDs to parquet with pyarrow #61602#75xirafa wants to merge 27 commits into
Conversation
for more information, see https://pre-commit.ci
for more information, see https://pre-commit.ci
for more information, see https://pre-commit.ci
for more information, see https://pre-commit.ci
for more information, see https://pre-commit.ci
for more information, see https://pre-commit.ci
for more information, see https://pre-commit.ci
for more information, see https://pre-commit.ci
for more information, see https://pre-commit.ci
for more information, see https://pre-commit.ci
📝 WalkthroughWalkthroughAdds a PyArrow parquet test that writes and reads UUID-valued data, asserts exact DataFrame equality, skips unsupported PyArrow versions, and marks the test as an expected failure for the upstream casting issue. ChangesParquet UUID support
Estimated code review effort: 2 (Simple) | ~10 minutes Suggested reviewers: 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@pandas/tests/io/test_parquet.py`:
- Line 1529: Update the xfail reason near the existing GH 61602 reference to
also cite Apache Arrow issue `#50312`, preserving the explanation about PyArrow’s
FIXED_LEN_BYTE_ARRAY-to-UUID casting failure and both issue references.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro Plus
Run ID: 51c9bd28-64a1-48ae-8751-7931be138a70
📒 Files selected for processing (1)
pandas/tests/io/test_parquet.py
|
|
||
|
|
||
| @pytest.mark.xfail( | ||
| reason="Upstream PyArrow fails to cast FIXED_LEN_BYTE_ARRAY to UUID - GH 61602" |
There was a problem hiding this comment.
📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "== locate file and target lines =="
wc -l pandas/tests/io/test_parquet.py 2>/dev/null || true
sed -n '1515,1545p' pandas/tests/io/test_parquet.py 2>/dev/null || true
echo "== search related references =="
rg -n "GH 61602|61602|FIXED_LEN_BYTE_ARRAY|UUID|50312|apache/arrow/issues" pandas/tests io pandas -S 2>/dev/null | head -n 120Repository: autodag-dev/pandas
Length of output: 8295
🌐 Web query:
Apache Arrow FIXED_LEN_BYTE_ARRAY UUID Arrow issue 50312
💡 Result:
Apache Arrow issue 50312 refers to a bug involving the improper handling of UUID extension types during conversion between PyArrow and pandas [1]. Specifically, when a PyArrow Table containing a UUID column is converted to a pandas DataFrame, the data is returned as raw bytes instead of Python uuid.UUID objects [1][2]. Key details regarding this issue include: Root Cause: The issue stems from the UuidType in PyArrow not implementing to_pandas_dtype, which causes Table.to_pandas to fall back to the underlying storage type, fixed_size_binary(16), resulting in bytes [2]. Status: As of July 29, 2026, the issue is open and actively being addressed [1]. A pull request (GH-50325) was initiated to fix this by adding the missing method [2]. However, project maintainers have suggested a more robust, performance-optimized implementation in C++ rather than the initial Python-level fix [3]. Broader Context: This is part of ongoing work to improve UUID and extension type support in PyArrow [4][5]. Other recent related issues have included inconsistencies in Parquet schema reading (where extension types were dropped) [4] and difficulties in constructing UuidArrays from scalars [5]. Users encountering this behavior should follow the progress on GitHub issue 50312 for updates and the final resolution [1].
Citations:
- 1: [Python][Parquet] FIXED_LEN_BYTE_ARRAY fails to cast to UUID on Python 3.14 / Nightly builds apache/arrow#50312
- 2: GH-50312: [Python] Fix UUID extension type round-trip to pandas returning bytes apache/arrow#50325
- 3: https://www.mail-archive.com/github@arrow.apache.org/msg417765.html
- 4: [Python][Parquet] read_schema drops extension types (UUID returned as fixed_size_binary[16]) apache/arrow#48254
- 5: [Python] Cannot construct
UuidArrayfrom list ofUuidScalars apache/arrow#48470
Add the upstream Arrow issue to the xfail reason.
GH 61602 tracks the pandas test, but this PyArrow FIXED_LEN_BYTE_ARRAY → UUID regression is tracked separately as Apache Arrow issue #50312. Include both references so maintainers can determine when this unconditional xfail can be removed.
Proposed fix
`@pytest.mark.xfail`(
- reason="Upstream PyArrow fails to cast FIXED_LEN_BYTE_ARRAY to UUID - GH 61602"
+ reason=(
+ "Upstream PyArrow fails to cast FIXED_LEN_BYTE_ARRAY back to UUID - "
+ "Pandas GH 61602 / Arrow issue: "
+ "https://github.com/apache/arrow/issues/50312"
+ )
)📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| reason="Upstream PyArrow fails to cast FIXED_LEN_BYTE_ARRAY to UUID - GH 61602" | |
| reason=( | |
| "Upstream PyArrow fails to cast FIXED_LEN_BYTE_ARRAY back to UUID - " | |
| "Pandas GH 61602 / Arrow issue: " | |
| "https://github.com/apache/arrow/issues/50312" | |
| ) |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@pandas/tests/io/test_parquet.py` at line 1529, Update the xfail reason near
the existing GH 61602 reference to also cite Apache Arrow issue `#50312`,
preserving the explanation about PyArrow’s FIXED_LEN_BYTE_ARRAY-to-UUID casting
failure and both issue references.
Source: MCP tools
9963fc4 to
dc15178
Compare
2 participants
Mirror of pandas-dev/pandas#65647. Runs the pandas test suite through dagz.
Summary by CodeRabbit