feat!: upgrade DataFusion to 54

[discord9]

I hereby agree to the terms of the GreptimeDB CLA.

Refer to a related PR or issue link (optional)

• DataFusion 54.1 thin fork: GreptimeTeam/datafusion#21, head 43a6b5750
• Scalar-subquery ordering fix: apache/datafusion#23677
• PostgreSQL catalog quoted-regclass fix and 0.18.3 release: datafusion-contrib/datafusion-postgres#396
• Greptime SQL parser 0.62 range-query port: GreptimeTeam/sqlparser-rs@9e9019b

What's changed and what's your intention?

Upgrade GreptimeDB from its DataFusion 53.1 maintenance fork to the official DataFusion 54.1 / Arrow 58 line while retaining the correctness fixes that GreptimeDB still requires.

The final dependency graph is organization-controlled and reproducible:

Dependency	Final revision	Notes
DataFusion	=54.1.0, GreptimeTeam/datafusion@43a6b5750	Exactly 12 Greptime correctness commits above official 54.1.0@0d1f2ebe2
datafusion-pg-catalog	crates.io 0.18.3	Includes quoted/schema-qualified regclass resolution from upstream PR #396
arrow-pg	crates.io 0.15.0	One registry source; the duplicate git source was removed
datafusion-orc	crates.io 0.9.0	DataFusion 54-compatible release
sqlparser	forked 0.62.0@9e9019bb1	Preserves Greptime RANGE/FILL/ALIGN/TO/BY extensions
greptime-proto	a937a6455	Matches the validated main cutoff and retains batching-flow time_ranges

DataFusion 54.1 thin fork

The thin fork starts from the exact official 54.1.0 commit and carries only fixes still needed by GreptimeDB:

Behavior retained	54.1 commit
Preserve mixed aggregate-filter predicate order	a60023a3a
Account for grouped AVG/MEDIAN accumulator memory	e5cfcc3a8
Count ragged nested-array cardinality recursively	062d818d9
Bound array_resize by Arrow's maximum array size	e4231af92
Preserve finer timestamp precision during coercion	a286efa5d
Make fetch-aware statistics safe and apply them to SortPreservingMergeExec	4b31e11d0
Preserve global LIMIT/OFFSET semantics across partitions	33f4161e7
Keep safe TRY_CAST scalar/array overflow nullable	218a14c53
Replace unsafe cast unwrapping with exact/range cast preimages	887df12d5
Return SQL NULL for exact median(NULL) and safely fall back from grouped accumulation	d450ac6b5
Preserve duplicate Substrait root names with deterministic internal qualifiers	33e6bb173
Preserve ordering requirements below ScalarSubqueryExec roots	43a6b5750

The volatile file-scan projection patch from the earlier 54.0 candidate is not replayed because official 54.1 already contains the adapted upstream fix. GreptimeTeam #3 and #19 are also covered by the official release line; #18 remains excluded because no consumer-level regression demonstrates an impact after the statistics fix.

GreptimeDB integration changes

• Port query, PromQL, catalog, table, SQL, function, protocol, and storage consumers to DataFusion 54 / Arrow 58 APIs.
• Mark session-dependent PostgreSQL functions as Stable; make current_schema, current_database, current_schemas, and session_user observe the live query context.
• Replan prepared PostgreSQL statements when catalog/schema context changes while preserving the original protocol contract:
  • decode bound binary parameters using the original prepared plan;
  • require identical inferred parameter types and RowDescription fields before executing a refreshed plan;
  • reject incompatible parameter, result-type, or result-arity changes with SQLSTATE 0A000 instead of reinterpreting bytes or emitting rows under stale metadata.
• Cover quoted and schema-qualified regclass parameters, search-path changes, PostgreSQL attnum ordering, and prepared-plan contract failures through the real extended-query protocol.
• Compare JSON containers semantically in tests so object insertion order from feature unification does not alter assertions.
• Replace brittle distributed-plan strings with structural assertions and update safe widening CAST / TRY_CAST expectations.
• Normalize two SQLness inputs and regenerate the DataFusion 54 result snapshots. Empty query outputs now retain their schema/header instead of collapsing to an untyped empty result.

Latest-main rebase

The branch is rebased onto validated cutoff main@4acd2340c7, with final head fd09114890.

• Linear range: 23 non-merge commits, zero merge commits.
• Strict range-diff preserved all 19 original PR commits; repeated latest-main replays introduced no unexplained patch drift.
• Four signed post-rebase repairs close integration gaps exposed by the final dependency/main graph:
  • 36e0d19d47: restore the single forked sqlparser 0.62 graph;
  • 088b306682: migrate four latest-main PromQL casts to Cast::new;
  • f0e477ee0e: restore greptime-proto@a937a645 from validated main;
  • fd09114890: migrate common-query tests to DataFusion 54 physical-expression/hash APIs.
• The remote update used a ref-specific --force-with-lease against the reviewed old head ff94fb3f1c; no bare --force was used.

Compatibility and limitations

This PR does not intentionally change GreptimeDB public schemas or wire formats. It does change the query-engine dependency boundary and internal plan behavior.

• Full multi-entry PostgreSQL search_path modeling remains an upstream follow-up; the current integration models the first supported schema.
• Typed RangeSelectPartialV1 transport is a separate, unmerged lane and is not part of this PR.
• Mixed-version DataFusion 53/54 CountHash partial states during a rolling upgrade are out of scope. In-flight CountHash aggregations should finish before changing node versions.
• The full release build and persisted VIEW matrices (DF53→54, DF54→53, and DF54→54) are explicitly deferred rather than claimed as completed.

Validation completed

DataFusion thin fork:

• Verified exact ancestry from 54.1.0@0d1f2ebe2 and an exact 12-commit range.
• Selected all-feature package suites: more than 4,412 tests across functions, optimizer, physical expression/plan, and Substrait packages.
• Doctests, workspace all-target/all-feature Clippy, and format/diff checks.
• Rule-level and end-to-end scalar-subquery ordering regressions plus all-target/all-feature datafusion-physical-optimizer Clippy.

Final GreptimeDB head/cutoff:

• Locked Cargo metadata, workspace Rust format, root TOML format, dependency-source audit, and full range git diff --check.
• Locked eight-crate consumer closure for common-function, query, promql, servers, common-datasource, mito2, catalog, and table.
• cargo check --locked --workspace --all-targets --all-features on the final cutoff.
• Common-query physical-expression round trips/fallbacks: 4/4; PromQL direct-OR/dictionary normalization: 8/8.
• common-function: 4/4; query regressions: 2/2; catalog ordering: 2/2.
• Real PostgreSQL cached-plan/regclass/search-path integration: 2/2.
• Focused SQLness selected three standalone/distributed executions. Two identical runs produced no snapshot drift; after the final disjoint main replay, the exact cutoff binary was rebuilt and the same three executions passed again with byte-identical tracked results.

Historical pre-rebase validation also included a global Basic SQLness run: 65 standalone and 5 distributed cases generated 70 expected snapshots once, followed by an identical stable rerun. It was not rerun on the final head because the documented escalation condition did not fire: there was no SQL/result conflict, runner/plan-rendering conflict, unexpected snapshot rewrite, or non-identical focused rerun.

One generated SQLness result retains the runner's standard trailing blank line. The reruns are stable and existing tracked snapshots use the same generator convention.

Draft status / remaining gates

• Rebase onto validated main@4acd2340c7 and publish final head fd09114890.
• Let fresh required CI finish on the new head; checks from ff94fb3f1c are stale.
• Obtain human review for this PR and GreptimeTeam/datafusion#21. DataFusion feat: test http server api #21 is OPEN/CLEAN at 43a6b575, but neither PR currently has an approval.
• Resolve DCO ACTION_REQUIRED for the four retained Ning Sun commits: 0a4817444d, 6518b20ca7, 7ef4f96613, and e84c1ad453. Their author signoffs must not be fabricated.
• Complete the explicitly deferred release-build and persisted/mixed-version compatibility gates before claiming release readiness.

PR Checklist

• I have written the necessary rustdoc comments.
• I have added the necessary unit tests and integration tests.
• This PR requires documentation updates.
• API changes are backward compatible.
• Schema or data changes are backward compatible.

[gemini-code-assist]

Code Review

This pull request upgrades the DataFusion dependency from version 53.1.0 to 54.0.0, which involves extensive refactoring to adapt to API changes such as the removal of explicit as_any() downcasting, updates to the Cast expression structure, and changes to PruningStatistics and partition_statistics signatures. It also introduces a frozen hash_v1 module to preserve compatibility with legacy count_hash states. The review identified critical compilation errors: a missing import for as_largestring_array in the new hash_v1.rs file, and incorrect iterations over the return value of hash_utils::create_hashes in union_distinct_on.rs since it now returns Result<()> instead of the hashes.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

[sunng87]

orc and pg-catalog are our managed libraries, make sure it merges to upstream before merging this.

[sunng87]

I'm working on the pg-catalog upgrade for datafusion 54. will commit to this branch.

[discord9]

Thanks — I see 193726d60a updates pg-catalog to 0.18 and adds the related planner/session compatibility changes. The PR DCO check is currently failing because this commit has no Signed-off-by trailer. Could you please add the sign-off using the DCO app’s accepted flow? I’ll retain and validate this commit while reconciling the latest main; I won’t overwrite the branch update.

[sunng87]

DCO is a non-blocking check, just set it passed.

[discord9]

Thanks, DCO is green now. One remaining managed-dependency setup: GreptimeTeam/datafusion still has main on 53.1, and my direct attempt to create a 54 branch returned HTTP 403 (push=false). Could you create an organization target branch such as greptimedb-54.0.0 from official DataFusion 54.0.0@45d943dfb8699dc9cb9ef2320e955b73e3e6c03b? The validated thin-fork head is discord9/datafusion:thin-fork/2026-07-16-datafusion-54@027358559b74632d3231eea46a34fcc7f7c99fe9; once the target exists I can open the PR and move #8555 to the organization-controlled SHA.

[sunng87]

datafusion-orc is now on 0.9

[discord9]

The pg-catalog 0.18 integration uncovered one remaining upstream compatibility gap during real PostgreSQL-protocol SQLness validation: OID aliases are represented as Int32 + pg.oid_alias metadata, but reverse casts and wire mapping currently expose numeric values (25::regtype::text -> 25, typinput -> 46) instead of PostgreSQL names/types. I filed datafusion-postgres#384 with library-level reproductions. I am fixing the Greptime-owned wire metadata mapping, pg_attribute.attnum ordering, and unstable OID assertions locally; I will not accept the numeric snapshot as the baseline. The symbolic alias-to-text fix belongs upstream and remains a Draft gate.

[discord9]

Updated locally to the managed datafusion-orc 0.9.0 crates.io release. The temporary personal git source is removed; the locked common-datasource and mito2 checks pass. This will be included in the pending latest-main merge commit.

[discord9]

Could you confirm the intended ownership for the DataFusion thin-fork pin itself? The existing dependency used GreptimeTeam/datafusion, but the validated DF54 head is currently discord9/datafusion@027358559b...; I cannot create the requested GreptimeTeam 54 target branch (push=false, direct push returns 403). Is moving this SHA into GreptimeTeam/datafusion a hard pre-merge requirement, or is the immutable personal-fork SHA acceptable for this upgrade? I am treating it as a conservative dependency-ownership gate until clarified.

[discord9]

The managed DataFusion 54 path is now available. I created GreptimeTeam/datafusion:greptimedb-54.0.0 directly from the exact official 54.0.0@45d943dfb... release commit and opened GreptimeTeam/datafusion#20 from the validated thin-fork head 027358559b.... PR #20 is CLEAN and review has been requested from @sunng87. After it merges, #8555 can replace all personal DataFusion URLs with the organization-controlled immutable SHA.

[sunng87]

datafusion 54.1 is out.
I also have a new release for datafusion-pg-catalog 0.18.2 with some bugfix. Consider sending the quote patch to upstream.