chore: bump datafusion to 54.1.0 - #2293
Merged
Merged
Conversation
lonless9
added
run spark tests
Codecov Report❌ Patch coverage is
@@ Coverage Diff @@
## main #2293 +/- ##
==========================================
- Coverage 79.36% 77.69% -1.68%
==========================================
Files 916 916
Lines 176303 177176 +873
==========================================
- Hits 139929 137654 -2275
- Misses 36374 39522 +3148
... and 31 files with indirect coverage changes 🚀 New features to boost your workflow:
|
Spark 3.5.7 Test ReportCommit Information
Test Summary
Test DetailsError CountsPassed Tests Diff(empty) Failed Tests |
Spark 4.1.1 Test ReportCommit Information
Test Summary
Test DetailsError CountsPassed Tests Diff(empty) Failed Tests(truncated) |
Ibis Test ReportCommit Information
Test Summary
Test DetailsError CountsPassed Tests Diff(empty) Failed Tests |
Spark 4.2.0 Test ReportCommit Information
Test Summary
Test DetailsError Counts(truncated) Passed Tests Diff(empty) Failed Tests(truncated) |
…usion-54.1.0 # Conflicts: # crates/sail-execution/src/proto/codec.rs
Contributor
There was a problem hiding this comment.
Pull request overview
Upgrades Sail’s DataFusion dependency stack from 54.0.0 to 54.1.0 and aligns Sail’s Spark-compat layer and plan-serialization to upstream behavior changes, including codec/proto updates for RecursiveQueryExec and refreshed golden plan snapshots.
Changes:
- Bump DataFusion (and related crates) to
54.1.0. - Update physical plan proto/codec to serialize and restore
RecursiveQueryExec.output_schemaand add a regression test. - Extend Spark-compat function tests (
array_compact,width_bucket) and refresh the TPC-DS Q6 plan snapshot.
Reviewed changes
Copilot reviewed 7 out of 8 changed files in this pull request and generated 1 comment.
Show a summary per file
| File | Description |
|---|---|
| python/pysail/tests/spark/function/features/width_bucket.feature | Adds Spark-compat assertion that width_bucket returns BIGINT/long. |
| python/pysail/tests/spark/function/features/array_compact.feature | Adds coverage for array_compact(array(NULL, NULL)) returning an empty array. |
| python/pysail/tests/spark/snapshots/test_tpcds.plan.yaml | Updates TPC-DS Q6 snapshot to match new join rewrite/plan shape. |
| crates/sail-function/src/scalar/array/spark_array_compact.rs | Adjusts array_compact element-null handling for untyped nulls. |
| crates/sail-execution/src/proto/codec.rs | Encodes/decodes RecursiveQueryExec with output_schema and adds a round-trip test. |
| crates/sail-execution/proto/sail/plan/physical.proto | Extends RecursiveQueryExecNode with output_schema. |
| Cargo.toml | Bumps DataFusion crate versions to 54.1.0. |
| Cargo.lock | Lockfile refresh for the DataFusion bump and transitive updates. |
…usion-54.1.0 # Conflicts: # Cargo.lock # python/pysail/tests/spark/function/features/array_compact.feature
Spark 3.5.9 Test ReportCommit Information
Test Summary
Test DetailsError CountsPassed Tests Diff(empty) Failed Tests |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Upgrade DataFusion from 54.0.0 to 54.1.0, covering the release items tracked in apache/datafusion#22547.
Preserve
RecursiveQueryExec.output_schemain physical-plan codec for #23579.Fix
array_compactlogical-null handling (#23196) and coverwidth_bucket's SparkBIGINTreturn type (#23087).Refresh the TPC-DS Q6 snapshot for the valid
LEFT-to-INNERjoin rewrite from #23580.#23296 would need follow-up.