Skip to content

fix: port GreptimeDB correctness fixes to DataFusion 54 - #20

Closed
discord9 wants to merge 12 commits into
GreptimeTeam:greptimedb-54.0.0from
discord9:thin-fork/2026-07-16-datafusion-54
Closed

fix: port GreptimeDB correctness fixes to DataFusion 54#20
discord9 wants to merge 12 commits into
GreptimeTeam:greptimedb-54.0.0from
discord9:thin-fork/2026-07-16-datafusion-54

Conversation

@discord9

@discord9 discord9 commented Jul 20, 2026

Copy link
Copy Markdown

Which issue does this PR close?

N/A — issues are disabled on this fork. This maintenance branch is tracked by GreptimeDB PR #8555.

Rationale for this change

GreptimeDB is upgrading from its DataFusion 53.1 maintenance branch to DataFusion 54. Official 54.0.0 already contains most previously tracked fixes, but several correctness fixes either landed after the 54 tag or remain specific to GreptimeDB's query plans.

This PR creates a reviewable, test-first thin fork on the exact official 54.0.0 commit 45d943dfb8699dc9cb9ef2320e955b73e3e6c03b. It carries only fixes with a focused regression and avoids replaying patches already present in 54.

What changes are included in this PR?

Source Correctness behavior retained Commit
GreptimeTeam #5 Preserve mixed aggregate-filter predicate order 7303323513
GreptimeTeam #16 / Apache apache#23357 Account for grouped AVG/MEDIAN accumulator memory 467e7ec99f
GreptimeTeam #14 / Apache apache#23271 Count ragged nested-array cardinality recursively c349e4b388
GreptimeTeam #15 / Apache apache#23306 Bound array_resize by Arrow's maximum array size 9297b40fd4
GreptimeTeam #17 / Apache apache#23395 Avoid duplicating volatile file-scan projections adfae34512
GreptimeTeam #4 / Apache apache#22759 Preserve finer timestamp precision during coercion 557ddad236
GreptimeTeam #13 Make fetch-aware statistics safe and apply them to SortPreservingMergeExec e2be63976c
GreptimeTeam #12 Preserve global LIMIT/OFFSET semantics across partitions d385af852d
DF54 follow-up Keep safe TRY_CAST scalar/array overflow behavior nullable fb7b26d04c
GreptimeTeam #11 Replace unsafe cast unwrapping with exact/range cast preimages 027358559b
DF54 follow-up Return SQL NULL for exact median(NULL) and fall back from grouped accumulation 0f6dcb128
DF54 follow-up Preserve duplicate Substrait root names with deterministic internal qualifiers cee35d94c

Not replayed:

The resulting thin-fork head is cee35d94c03300f1b9a21090828826e61cb67b11.

Are these changes tested?

Yes. Each semantic port was introduced with a focused regression before its implementation. Final validation includes:

  • shared cast-preimage tests, logical optimizer tests, physical unwrap tests, and simplifier tests;
  • optimizer integration coverage and the exact simplify_expr.slt case;
  • aggregate filter order, grouped accumulator memory, ragged cardinality, array_resize, volatile projection, timestamp coercion, fetch-aware statistics, global LIMIT/OFFSET, and safe TRY_CAST regressions;
  • exact-median NULL accumulator and grouped-fallback tests;
  • the complete datafusion-substrait suite, including duplicate root names, null-padded joins, deterministic second round trips, and terminal grouping sets: 249 passed, 6 ignored;
  • cargo clippy --workspace --all-targets --all-features -- -D warnings;
  • workspace format and diff checks.

Are there any user-facing changes?

There is no intentional public API change beyond the DataFusion 54 baseline. The included changes correct query results, optimizer rewrites, statistics, memory accounting, NULL aggregation, Substrait schema reconstruction, and overflow behavior consumed by GreptimeDB.

discord9 added 10 commits July 16, 2026 11:39
Signed-off-by: discord9 <discord9@163.com>
Signed-off-by: discord9 <discord9@163.com>
Signed-off-by: discord9 <discord9@163.com>
Signed-off-by: discord9 <discord9@163.com>
Signed-off-by: discord9 <discord9@163.com>
Signed-off-by: discord9 <discord9@163.com>
Signed-off-by: discord9 <discord9@163.com>
Signed-off-by: discord9 <discord9@163.com>
Signed-off-by: discord9 <discord9@163.com>
Signed-off-by: discord9 <discord9@163.com>
discord9 added 2 commits July 20, 2026 23:53
Signed-off-by: discord9 <discord9@163.com>
Signed-off-by: discord9 <discord9@163.com>
@discord9

Copy link
Copy Markdown
Author

Superseded by #21.

The replacement is based directly on official DataFusion 54.1 (0d1f2ebe2cc97c91b736bc0a160b5b73cf40437a) and replays only the 11 fixes still absent from that release. In particular, it drops this PR's volatile file-scan projection patch because DataFusion 54.1 already contains the adapted upstream fix (c735a49a3). No history on this 54.0 branch was rewritten.

@discord9 discord9 closed this Jul 23, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant