fix: delegate Arrow schema conversion to PyIceberg - #349
Open
konstantinoscs wants to merge 1 commit into
Open
Conversation
|
Important Review skippedAuto reviews are disabled on base/target branches other than the default branch. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Pro Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
Comment |
konstantinoscs
force-pushed
the
feature/tow-2494-native-arrow-schema
branch
2 times, most recently
from
August 13, 2026 17:31
347ec8c to
c8341f9
Compare
konstantinoscs
force-pushed
the
feature/tow-2494-native-arrow-schema
branch
from
August 13, 2026 17:39
c8341f9 to
ced05c6
Compare
konstantinoscs
force-pushed
the
feature/tow-2494-native-arrow-schema
branch
from
August 13, 2026 19:18
ced05c6 to
69a36cc
Compare
konstantinoscs
force-pushed
the
feature/tow-2494-native-arrow-schema
branch
2 times, most recently
from
August 14, 2026 14:26
7d739da to
bd8c7e5
Compare
konstantinoscs
force-pushed
the
feature/tow-2494-native-arrow-schema
branch
from
August 14, 2026 15:34
bd8c7e5 to
14c7cc3
Compare
konstantinoscs
marked this pull request as ready for review
August 17, 2026 11:34
konstantinoscs
force-pushed
the
feature/tow-2494-native-arrow-schema
branch
from
August 17, 2026 11:35
14c7cc3 to
b709205
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this does
This PR passes a
pyarrow.Schemadirectly to PyIceberg when creating a table and removes Tower's custom schema converter.PyIceberg now handles field IDs, nested fields, documentation, nullability, and type validation using its standard public API.
Why
Tower's converter duplicated PyIceberg behavior and could silently lose information, particularly timestamp and decimal precision. The native conversion is easier to maintain and rejects unsupported or lossy schemas clearly.
Tests also create tables through S3 Tables and BYO-style catalog configuration, confirming that those paths still use their existing PyIceberg catalog and never request Tower-managed credentials.
Compatibility
Some schemas that were previously accepted after a lossy conversion now raise PyIceberg's normal validation errors. The reference documentation lists the supported timestamp, time, date, decimal, and timezone behavior.
Testing