Upgrade to arrow 59 - #1
Closed
adriangb wants to merge 1 commit into
Closed
Conversation
adriangb
force-pushed
the
claude/arrow-59-upgrade-vg9o8j
branch
from
June 10, 2026 08:58
bc7fdc0 to
a9acb0a
Compare
Bump arrow, arrow-schema, arrow-cast and the parquet-variant crates to 59.0.0. DataFusion does not yet have a crates.io release built against arrow 59, so a [patch.crates-io] section points the datafusion* crates at apache/datafusion PR #22744 (alamb/update_arrow_59), which updates DataFusion to arrow/parquet 59.0.0. This keeps the whole dependency graph on a single arrow 59 major; drop the patch once a DataFusion release ships arrow 59. parquet-variant 59 changed `variant_get(..., as_type)` to coerce mismatched primitives (matching SQL cast semantics) instead of returning NULL: - numeric -> boolean coerces nonzero to true - float -> integer truncates toward zero Update the affected unit tests and sqllogictests accordingly. https://claude.ai/code/session_01H3jbXkVZam7VH5DpiXczBR
adriangb
force-pushed
the
claude/arrow-59-upgrade-vg9o8j
branch
from
June 10, 2026 09:16
a9acb0a to
2adeba4
Compare
Member
Author
|
Closing this in favor of opening the arrow 59 upgrade against upstream ( The branch is unchanged and already pushed; the upstream PR can be opened here: Generated by Claude Code |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Upgrades
datafusion-variantto arrow 59.arrow,arrow-schema,arrow-castand theparquet-variant*crates to59.0.0.arrow ^58.3), so a[patch.crates-io]section points thedatafusion*crates at apache/datafusion#22744 (alamb/update_arrow_59), which updates DataFusion to arrow/parquet 59.0.0. This keeps the whole dependency graph on a single arrow 59 major — without the patch the resolver pulls in both arrow 58.3 (via DataFusion) and arrow 59 (our direct deps), which fails to compile at the DataFusion ↔ parquet-variant boundary.Behavior change in parquet-variant 59
parquet-variant59 changedvariant_get(.., as_type)to coerce mismatched primitives (matching SQLCASTsemantics) instead of returningNULL:true10.5→10)The affected unit tests and sqllogictests are updated to assert the new coercion behavior, which is closer to the Spark/Databricks parity goal.
Notes
datafusion-contrib/datafusion-variant@main, so it also carries the "Upgrade to DataFusion 54 and arrow 58.3 (Upgrade to DataFusion 54 and arrow 58.3 datafusion-contrib/datafusion-variant#62)" commit that isn't yet on this repo'smain, plus a smallcargo fmtcommit cleaning up blank lines that Upgrade to DataFusion 54 and arrow 58.3 datafusion-contrib/datafusion-variant#62 left behind.cargo test(80 unit tests + sqllogictests),cargo fmt --all -- --check, andcargo clippy -- -D warningsall pass.https://claude.ai/code/session_01H3jbXkVZam7VH5DpiXczBR
Generated by Claude Code