Broadcast the build side of a SortMergeJoinExec without converting to a hash join

[andygrove]

Background

PR #1904 (closing #1679) added a static-planner optimization that, when one side of a SortMergeJoinExec fits under ballista.optimizer.broadcast_join_threshold_bytes, converts the join to a HashJoinExec(CollectLeft) and reuses the existing broadcast lowering. This avoids the two hash-partitioned shuffles a sort-merge join would otherwise require.

That approach changes the join operator (SMJ → hash join). A distinct, complementary optimization is to keep the SortMergeJoinExec and instead broadcast (replicate) its small build side to every probe task — the probe side stays in its natural partitioning and the merge-join semantics are preserved. This is closer to what Spark does: as @milenkovicm noted on PR #1904, Spark broadcasts the build side of a sort-merge join (observed in TPC-DS Q72 at SF10).

Motivation

• The convert-to-hash-join path forgoes sort-merge's spill safety (the reason Ballista defaults prefer_hash_join = false). It is sound only because the build side is small enough to fit in memory by definition — but the probe side still runs as a hash join. Broadcasting the build side of an SMJ directly would let large joins keep sort-merge execution while still eliminating the small side's shuffle.
• Neither planner currently does this: the static planner converts to a hash join (feat: broadcast small build side of SortMergeJoinExec in the static planner #1904), and the AQE DynamicJoinSelectionExec path promotes to CollectLeft hash join as well — AQE does not broadcast the build side of an SMJ (per @milenkovicm on feat: broadcast small build side of SortMergeJoinExec in the static planner #1904).

Proposed work

1. Spike: determine what a broadcast/replicated build input looks like for SortMergeJoinExec — the build side must still be sorted on the join key and materialized on every probe task, while the probe side keeps its natural partitioning (no join-key repartition).
2. Wire it into both planners (static DefaultDistributedPlanner and the AQE join-selection path), gated by config, and compare against the convert-to-hash-join path from feat: broadcast small build side of SortMergeJoinExec in the static planner #1904.
3. Benchmark under both AQE on and AQE off — TPC-H SF10 and the TPC-DS Q72 case @milenkovicm called out — with row-count verification.

References

• PR feat: broadcast small build side of SortMergeJoinExec in the static planner #1904 — converts a small-side SMJ to a broadcast CollectLeft hash join (the alternative this issue proposes to complement)
• Extend ColocatedJoinRule and BroadcastSmallSideRule to SortMergeJoinExec #1679 — extend broadcast/colocated rules to SortMergeJoinExec
• Broadcast lowering still hash-repartitions the build and probe sides of a CollectLeft join #1905 — broadcast lowering still hash-repartitions both sides of a CollectLeft join (related shuffle-elision follow-up)