Track DataFusion git main via a pinned rev and bump it periodically

[andygrove]

Is your feature request related to a problem or challenge?

Ballista currently depends on published DataFusion crates.io releases (today, 54). Because a full release cycle passes between DataFusion versions, breaking changes in DataFusion only surface in Ballista at upgrade time, all at once, making each upgrade a large and risky change (e.g. the DataFusion 54 upgrade in #1906).

Describe the solution you'd like

Pin the datafusion* crates to a git revision of apache/datafusion main and bump that revision periodically. Tracking unreleased DataFusion continuously means breaking API and behavior changes are caught in small increments, close to when they land upstream, instead of accumulating until the next release.

Each bump would:

• advance the pinned rev in the workspace Cargo.toml,
• port any DataFusion API changes,
• refresh any snapshots/goldens affected by DataFusion planner/metric changes,
• keep cargo test and cargo clippy --all-targets --workspace -- -D warnings green.

Describe alternatives you've considered

Continue upgrading only at DataFusion release boundaries. This keeps the dependency on stable releases but concentrates all breakage into infrequent, large upgrades.

Additional context

Related tracking issue: #1246. First bump implemented in the PR that references this issue.