Upgrade to DataFusion 54 and arrow 58.3 - #62
Merged
Conversation
DataFusion 54 removed `as_any` from `ScalarUDFImpl` and bumped arrow to 58.3. Bump the dependency requirements accordingly. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
adriangb
marked this pull request as ready for review
June 8, 2026 22:15
friendlymatthew
merged commit Jun 10, 2026
9e1c846
into
datafusion-contrib:main
1 of 3 checks passed
adriangb
pushed a commit
to pydantic/datafusion-variant
that referenced
this pull request
Jun 10, 2026
These three tests (test_int_scalar_float_value, test_bool_scalar_int_value, test_bool_scalar_variant_array_paths) were left asserting the old NULL behavior in datafusion-contrib#62, but parquet-variant 58.3 already coerces numeric variants (10.5 -> 10, nonzero -> true), so they fail on main. Update the expectations to match the actual coercion behavior. https://claude.ai/code/session_01H3jbXkVZam7VH5DpiXczBR
adriangb
pushed a commit
to pydantic/datafusion-variant
that referenced
this pull request
Jun 10, 2026
The unit tests (test_int_scalar_float_value, test_bool_scalar_int_value, test_bool_scalar_variant_array_paths) and the variant_get_bool / variant_get_int sqllogictest files were left asserting the old NULL behavior in datafusion-contrib#62, but parquet-variant 58.3 already coerces numeric variants (10.5 -> 10, nonzero -> true). Update the expectations to match the actual coercion behavior so the test suite passes on the current dependency versions. https://claude.ai/code/session_01H3jbXkVZam7VH5DpiXczBR
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Bumps
datafusion-variantto DataFusion 54 and arrow 58.3:datafusion/datafusion-sqllogictest53→54arrow/arrow-schema/arrow-cast/parquet-variant*58.1→58.3as_anyfrom everyScalarUDFImplimpl — DataFusion 54 droppedas_anyfrom the
ScalarUDFImpltrait, so the overrides no longer compile (E0407). Thisextends the earlier removal to
src/variant_contains.rs, which was added afterward.Building against vanilla crates.io DataFusion 54.0.0 / arrow 58.3.0 compiles cleanly.
variant_getcast semantics changedarrow 58.3's cast kernel became more permissive for numeric casts, which changes the
behavior of 3
variant_gettests (they currently still assert the old strict-NULLbehavior, so they fail):
test_bool_scalar_int_value_returns_null1→BooleanNULLtruetest_bool_scalar_variant_array_paths3(count) →BooleanNULLtrue(index 1)test_int_scalar_float_value_returns_null10.5→Int64NULLString→bool (e.g.
"alice"→Boolean) still yieldsNULL, so only numeric→bool andfloat→int changed.
@friendlymatthew — could you confirm whether the new permissive-cast behavior is desired?
If so I'll update these 3 test assertions to match; if strict-NULL should be preserved,
that needs a code change in the variant cast path. Left as draft pending your call.
🤖 Generated with Claude Code