Skip to content

fix: delegate Arrow schema conversion to PyIceberg - #349

Open
konstantinoscs wants to merge 1 commit into
feature/tow-2494-iceberg-filter-expressionsfrom
feature/tow-2494-native-arrow-schema
Open

fix: delegate Arrow schema conversion to PyIceberg#349
konstantinoscs wants to merge 1 commit into
feature/tow-2494-iceberg-filter-expressionsfrom
feature/tow-2494-native-arrow-schema

Conversation

@konstantinoscs

@konstantinoscs konstantinoscs commented Aug 13, 2026

Copy link
Copy Markdown
Contributor

What this does

This PR passes a pyarrow.Schema directly to PyIceberg when creating a table and removes Tower's custom schema converter.

PyIceberg now handles field IDs, nested fields, documentation, nullability, and type validation using its standard public API.

Why

Tower's converter duplicated PyIceberg behavior and could silently lose information, particularly timestamp and decimal precision. The native conversion is easier to maintain and rejects unsupported or lossy schemas clearly.

Tests also create tables through S3 Tables and BYO-style catalog configuration, confirming that those paths still use their existing PyIceberg catalog and never request Tower-managed credentials.

Compatibility

Some schemas that were previously accepted after a lossy conversion now raise PyIceberg's normal validation errors. The reference documentation lists the supported timestamp, time, date, decimal, and timezone behavior.

Testing

  • 20 schema-focused tests pass.
  • The complete stack passes 225 tests, with 10 skipped.
  • Black and mypy pass.

@coderabbitai

coderabbitai Bot commented Aug 13, 2026

Copy link
Copy Markdown

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: 2153591e-1183-4321-8bf9-6ec7359abaaa

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review

Comment @coderabbitai help to get the list of available commands.

@konstantinoscs
konstantinoscs force-pushed the feature/tow-2494-native-arrow-schema branch 2 times, most recently from 347ec8c to c8341f9 Compare August 13, 2026 17:31
@konstantinoscs
konstantinoscs force-pushed the feature/tow-2494-native-arrow-schema branch from c8341f9 to ced05c6 Compare August 13, 2026 17:39
@konstantinoscs
konstantinoscs force-pushed the feature/tow-2494-native-arrow-schema branch from ced05c6 to 69a36cc Compare August 13, 2026 19:18
@konstantinoscs
konstantinoscs force-pushed the feature/tow-2494-native-arrow-schema branch 2 times, most recently from 7d739da to bd8c7e5 Compare August 14, 2026 14:26
@konstantinoscs
konstantinoscs force-pushed the feature/tow-2494-native-arrow-schema branch from bd8c7e5 to 14c7cc3 Compare August 14, 2026 15:34
@konstantinoscs
konstantinoscs marked this pull request as ready for review August 17, 2026 11:34
@konstantinoscs
konstantinoscs force-pushed the feature/tow-2494-native-arrow-schema branch from 14c7cc3 to b709205 Compare August 17, 2026 11:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant