Adapt fork patches to DataFusion 54 - #64
Merged
Merged
Conversation
xudong963
marked this pull request as draft
July 2, 2026 06:20
xudong963
marked this pull request as ready for review
July 2, 2026 09:09
Update sqllogictest plans for the standalone multi-partition TopK fix. Those plans now coalesce partitions before the pushed-down TopK and mark the TopK as preserve_partitioning=false so the final merge receives one globally sorted stream. Make the aggregate dynamic-filter EXPLAIN ANALYZE snapshot ignore the published MIN bound because the partial aggregate can publish a different intermediate bound depending on parallel execution order; the assertion still requires the DynamicFilter and pruning predicate structure. Bump anyhow to 1.0.103 and stabby to 72.1.8 to avoid the current cargo-audit advisory and yanked crate failure. Ignore the current quick-xml advisories and paste maintenance warning in the audit workflow because quick-xml is constrained by the object_store 0.13.x / parquet 58.x stack in DataFusion 54, and paste is a transitive parquet/arrow-flight warning.
Expose BloomFilterStatistics and RowGroupAccessPlanFilter::prune_by_bloom_filters so Atlas reverse parquet scans can reuse DataFusion's bloom-filter predicate evaluation instead of carrying a local copy. Keep the bloom_filter module itself private and re-export only the statistics adapter at the crate root to minimize the fork API surface.
zhuqi-lucas
approved these changes
Jul 3, 2026
zhuqi-lucas
left a comment
zhuqi-lucas
left a comment
There was a problem hiding this comment.
LGTM overall. Most of the delta is legitimate cherry-pick dedup + DF54 API adaptation, and the schema-adapt behavior restore in push_decoder.rs is covered by three focused tests. One non-blocking suggestion inline about test coverage for the new TopK branch in pushdown_sort.rs.
MassivePizza
pushed a commit
that referenced
this pull request
Jul 15, 2026
* Adapt fork patches to DataFusion 54 * Fix CI expectations and audit advisories * Restore parquet schema adaptation * Fix inexact sort pushdown for standalone TopK * Coalesce inexact standalone TopK pushdown * Fix DataFusion 54 CI expectations Update sqllogictest plans for the standalone multi-partition TopK fix. Those plans now coalesce partitions before the pushed-down TopK and mark the TopK as preserve_partitioning=false so the final merge receives one globally sorted stream. Make the aggregate dynamic-filter EXPLAIN ANALYZE snapshot ignore the published MIN bound because the partial aggregate can publish a different intermediate bound depending on parallel execution order; the assertion still requires the DynamicFilter and pruning predicate structure. Bump anyhow to 1.0.103 and stabby to 72.1.8 to avoid the current cargo-audit advisory and yanked crate failure. Ignore the current quick-xml advisories and paste maintenance warning in the audit workflow because quick-xml is constrained by the object_store 0.13.x / parquet 58.x stack in DataFusion 54, and paste is a transitive parquet/arrow-flight warning. * Use SortPreservingMerge for standalone inexact TopK * Expose parquet bloom pruning helpers Expose BloomFilterStatistics and RowGroupAccessPlanFilter::prune_by_bloom_filters so Atlas reverse parquet scans can reuse DataFusion's bloom-filter predicate evaluation instead of carrying a local copy. Keep the bloom_filter module itself private and re-export only the statistics adapter at the crate root to minimize the fork API surface.
MassivePizza
pushed a commit
that referenced
this pull request
Jul 15, 2026
* Adapt fork patches to DataFusion 54 * Fix CI expectations and audit advisories * Restore parquet schema adaptation * Fix inexact sort pushdown for standalone TopK * Coalesce inexact standalone TopK pushdown * Fix DataFusion 54 CI expectations Update sqllogictest plans for the standalone multi-partition TopK fix. Those plans now coalesce partitions before the pushed-down TopK and mark the TopK as preserve_partitioning=false so the final merge receives one globally sorted stream. Make the aggregate dynamic-filter EXPLAIN ANALYZE snapshot ignore the published MIN bound because the partial aggregate can publish a different intermediate bound depending on parallel execution order; the assertion still requires the DynamicFilter and pruning predicate structure. Bump anyhow to 1.0.103 and stabby to 72.1.8 to avoid the current cargo-audit advisory and yanked crate failure. Ignore the current quick-xml advisories and paste maintenance warning in the audit workflow because quick-xml is constrained by the object_store 0.13.x / parquet 58.x stack in DataFusion 54, and paste is a transitive parquet/arrow-flight warning. * Use SortPreservingMerge for standalone inexact TopK * Expose parquet bloom pruning helpers Expose BloomFilterStatistics and RowGroupAccessPlanFilter::prune_by_bloom_filters so Atlas reverse parquet scans can reuse DataFusion's bloom-filter predicate evaluation instead of carrying a local copy. Keep the bloom_filter module itself private and re-export only the statistics adapter at the crate root to minimize the fork API surface.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
Rationale for this change
This PR is the manual fix layer for the fork DataFusion 54 upgrade.
The base branch
branch-54is Apacheupstream/branch-54plus a mechanical cherry-pick of the 9 fork commits fromorigin/branch-53. This head branch contains only the fixes needed for those imported commits to compile and pass checks on DataFusion 54.What changes are included in this PR?
opener.rs; DF54 usesopener/mod.rs.PhysicalPlanDecodeContext.expression_id.Are these changes tested?
YES
Are there any user-facing changes?
No user-facing changes intended. This is an upgrade/fixup PR for the fork branch.