Skip to content

fix: preserve aggregate filter pushdown order - #22926

Open
discord9 wants to merge 5 commits into
apache:mainfrom
discord9:fix/aggregate-filter-pushdown-order
Open

fix: preserve aggregate filter pushdown order#22926
discord9 wants to merge 5 commits into
apache:mainfrom
discord9:fix/aggregate-filter-pushdown-order

Conversation

@discord9

@discord9 discord9 commented Jun 12, 2026

Copy link
Copy Markdown
Contributor

Which issue does this PR close?

Rationale for this change

FilterPushdown maps child pushdown results back to the original parent filters by position.

AggregateExec::gather_filters_for_pushdown was splitting incoming parent filters into safe and unsafe buckets, building the child filter description from the safe filters, and then appending unsupported unsafe filters. For mixed predicates this reordered the parent filter results.

For example, for a predicate above an aggregate such as cnt@2 = 1 AND b@1 = bar, the filter on the grouping column b can be pushed below the aggregate, but the filter on aggregate output cnt must remain above it. If the returned parent filter results are reordered, the optimizer can treat the pushed-down grouping-column filter result as if it belonged to the aggregate-output filter and incorrectly remove the aggregate-output filter.

What changes are included in this PR?

  • Preserve the original parent_filters order in AggregateExec::gather_filters_for_pushdown.
  • Build the child filter description from all parent filters, then mark aggregate-unsafe filters as unsupported in place.
  • Add a regression test for a mixed aggregate-output and grouping-column predicate.
  • Add an SLT regression that disables logical optimizer passes for the mixed predicate so the physical AggregateExec::gather_filters_for_pushdown path is exercised end-to-end.

Are these changes tested?

Yes:

  • cargo fmt --all
  • cargo fmt --all -- --check
  • cargo test -p datafusion --test core_integration physical_optimizer::filter_pushdown::test_pushdown_through_aggregates
  • cargo test --test sqllogictests -- push_down_filter_regression:528
  • cargo test --test sqllogictests -- push_down_filter_regression
  • cargo clippy --all-targets --all-features -- -D warnings

Are there any user-facing changes?

No.

@github-actions github-actions Bot added core Core DataFusion crate physical-plan Changes to the physical-plan crate sqllogictest SQL Logic Tests (.slt) auto detected api change Auto detected API change labels Jun 12, 2026
@discord9

discord9 commented Jul 1, 2026

Copy link
Copy Markdown
Contributor Author

Hi, this PR fixes #22925 and is still waiting for review.

The only failing check is Circular Dependency Check, which appears to fail while compiling cargo-credential rather than from this PR's code. Could someone please rerun it or advise if I should do anything here?

Thanks!

@discord9
discord9 force-pushed the fix/aggregate-filter-pushdown-order branch from 05e5a77 to 3626a96 Compare July 22, 2026 07:28
@github-actions github-actions Bot removed the auto detected api change Auto detected API change label Jul 22, 2026
@codecov-commenter

codecov-commenter commented Jul 22, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 90.47619% with 2 lines in your changes missing coverage. Please review.
✅ Project coverage is 80.70%. Comparing base (ef94a85) to head (4b0e6c6).
⚠️ Report is 1 commits behind head on main.

Files with missing lines Patch % Lines
...tafusion/physical-optimizer/src/filter_pushdown.rs 75.00% 1 Missing ⚠️
datafusion/physical-plan/src/aggregates/mod.rs 94.11% 1 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main   #22926      +/-   ##
==========================================
- Coverage   80.70%   80.70%   -0.01%     
==========================================
  Files        1095     1095              
  Lines      372554   372537      -17     
  Branches   372554   372537      -17     
==========================================
- Hits       300665   300640      -25     
- Misses      53921    53925       +4     
- Partials    17968    17972       +4     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@neilconway neilconway left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for this PR! Overall, this is excellent work: clear diagnosis, minimal and correct fix, and good test coverage. I'm sorry it didn't get reviewed sooner -- a few minor suggestions but it should be good to land shortly.

Comment thread datafusion/physical-plan/src/aggregates/mod.rs Outdated
Comment thread datafusion/physical-plan/src/aggregates/mod.rs Outdated
@discord9
discord9 force-pushed the fix/aggregate-filter-pushdown-order branch from 3626a96 to 4b0e6c6 Compare July 29, 2026 03:54
@github-actions github-actions Bot added the optimizer Optimizer rules label Jul 29, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

core Core DataFusion crate optimizer Optimizer rules physical-plan Changes to the physical-plan crate sqllogictest SQL Logic Tests (.slt)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

AggregateExec filter pushdown can reorder parent filter results

3 participants