chore: rebase Greptime thin fork onto DataFusion 54.1

[discord9]

Which issue does this PR close?

N/A — issues are disabled on this maintenance fork. This replacement is tracked by GreptimeDB PR #8555 and supersedes GreptimeTeam/datafusion#20.

Rationale for this change

GreptimeDB is upgrading from its DataFusion 53.1 maintenance branch to DataFusion 54.1. The first thin-fork proposal (#20) was built on official 54.0.0. Since then, official 54.1.0 shipped 20 backports, including an adapted version of the volatile file-scan projection fix that #20 carried locally.

This replacement starts from the exact official 54.1.0 commit 0d1f2ebe2cc97c91b736bc0a160b5b73cf40437a, drops the now-upstream volatile projection patch, and carries only the 12 correctness fixes still required by GreptimeDB. It keeps the organization branch reviewable instead of making the official 54.1 update appear as part of a PR based on greptimedb-54.0.0.

What changes are included in this PR?

Base:

GreptimeTeam/datafusion:greptimedb-54.1.0
0d1f2ebe2cc97c91b736bc0a160b5b73cf40437a

Retained fixes, replayed with provenance from the validated 54.0 stack:

Source	Correctness behavior retained	Source commit	54.1 commit
GreptimeTeam #5	Preserve mixed aggregate-filter predicate order	7303323513	a60023a3a
GreptimeTeam #16 / Apache apache#23357	Account for grouped AVG/MEDIAN accumulator memory	467e7ec99f	e5cfcc3a8
GreptimeTeam #14 / Apache apache#23271	Count ragged nested-array cardinality recursively	c349e4b388	062d818d9
GreptimeTeam #15 / Apache apache#23306	Bound array_resize by Arrow's maximum array size	9297b40fd4	e4231af92
GreptimeTeam #4 / Apache apache#22759	Preserve finer timestamp precision during coercion	557ddad236	a286efa5d
GreptimeTeam #13	Make fetch-aware statistics safe and apply them to SortPreservingMergeExec	e2be63976c	4b31e11d0
GreptimeTeam #12	Preserve global LIMIT/OFFSET semantics across partitions	d385af852d	33f4161e7
DF54 follow-up	Keep safe TRY_CAST scalar/array overflow behavior nullable	fb7b26d04c	218a14c53
GreptimeTeam #11	Replace unsafe cast unwrapping with exact/range cast preimages	027358559b	887df12d5
DF54 follow-up	Return SQL NULL for exact median(NULL) and fall back from grouped accumulation	0f6dcb128	d450ac6b5
DF54 follow-up	Preserve duplicate Substrait root names with deterministic internal qualifiers	cee35d94c	33e6bb173
Apache apache#23677	Preserve global ORDER BY requirements below ScalarSubqueryExec roots	9eae6da6f	43a6b5750

Explicitly not replayed:

• adfae3451 — official 54.1 contains the adapted fix as c735a49a3.
• GreptimeTeam backport: Allow dynamic filter pushdown for left join #3 and fix: treat NULL sort merge join filters as unmatched #19 — already covered by the official 54 release line.
• GreptimeTeam fix: report sort preserving merge cardinality effect #18 — no consumer-level optimizer regression demonstrates an impact after the fetch-statistics correction.

The resulting candidate head is 43a6b575099e9d6002636814a28654235cdf213a, exactly 12 commits above the official 54.1 base.

Are these changes tested?

Each retained semantic change has focused regression coverage from the original test-first port. The 54.1 replacement additionally requires:

• ancestry and exact 12-commit count from 0d1f2ebe2;
• range-diff against the validated 54.0 stack;
• no thin-fork diff in the official volatile projection implementation, its Cargo files, or its SQL logic test;
• workspace format and diff checks;
• focused tests for the changed optimizer, function, physical-expression, physical-plan, and Substrait crates;
• cargo clippy --all-targets --all-features -- -D warnings;
• DataFusion SQL logic tests, core/CLI tests, and the complete Substrait suite.

Actual completed evidence before submission:

• Verified ancestry from 0d1f2ebe2 and an exact 12-commit range.
• Range-diff maps the original 11 retained patches to the validated 54.0 stack and drops only the now-upstream volatile projection patch; the twelfth commit is the minimal 54.1 adaptation of Apache fix: Capture global ORDER BY requirement under ScalarSubqueryExec root apache/datafusion#23677.
• Confirmed zero thin-fork diff in the official volatile projection implementation, its Cargo manifests, and projection_pushdown.slt.
• cargo fmt --all -- --check
• git diff --check
• Selected all-feature package suites: datafusion-functions (284), datafusion-functions-aggregate (127), datafusion-optimizer (698 unit + 28 integration), datafusion-physical-expr (1,548; 1 ignored), datafusion-physical-plan (1,441), and datafusion-substrait (49 unit + 197 integration; 6 ignored).
• Doctests: 36 passed, 25 ignored.
• Workspace cargo clippy --all-targets --all-features -- -D warnings.
• Scalar-subquery ordering rule-level and end-to-end physical-optimizer regressions, plus all-target/all-feature datafusion-physical-optimizer Clippy.

Are there any user-facing changes?

There is no intentional public API change beyond the official DataFusion 54.1 baseline. The retained patches correct query results, optimizer rewrites, statistics, memory accounting, NULL aggregation, Substrait schema reconstruction, and overflow behavior consumed by GreptimeDB.