feat: serialize ASOF join plans by Xuanwo · Pull Request #23832 · apache/datafusion

Xuanwo · 2026-07-23T06:12:25Z

Which issue does this PR close?

Part of ASOF join support / Specialize Range Joins #318.
Umbrella PR: feat: support ASOF joins #23738.
Depends on feat: add ASOF join logical semantics #23829 (and transitively feat: add ASOF join physical operator #23828).

Rationale for this change

This is the serialization layer of the ASOF JOIN stack. It gives logical and
physical ASOF plans explicit protobuf representations without coupling wire
format review to SQL or DataFrame APIs.

GitHub shows the cumulative stack because fork branches cannot be selected as
upstream PR bases. The isolated diff for this layer is:
Xuanwo/datafusion@xuanwo/asof-logical...xuanwo/asof-proto

What changes are included in this PR?

Add protobuf messages and enum values for logical and physical ASOF joins.
Encode and decode equality keys, ordered match expressions, match direction,
join constraint, and right output indices.
Implement physical serialization through ExecutionPlan::try_to_proto and
AsOfJoinExec::try_from_proto, following the current self-serializing
execution-plan pattern.
Regenerate prost and pbjson sources with the repository generator.
Add logical round trips for all four match directions and a physical
AsOfJoinExec round trip.

Are these changes tested?

Yes:

./datafusion/proto-models/regen.sh
cargo fmt --all
cargo clippy --all-targets --all-features -- -D warnings
cargo test -p datafusion-proto roundtrip_asof_join --all-features

Are there any user-facing changes?

Logical and physical ASOF join plans can be serialized through
datafusion-proto. The additions use new messages and append-only oneof/enum
tags, so existing wire tags are not reused. Generated public Rust enums gain new
variants, however, so downstream exhaustive matches must add arms; this is a
Rust source-compatibility break even though the wire additions are compatible.

This PR should be reviewed and merged after #23829. After its dependencies
squash-merge, this branch will be refreshed from main without rewriting its
published history.

github-actions · 2026-07-23T06:35:57Z

Thank you for opening this pull request!

Reviewer note: cargo-semver-checks reported the current version number is not SemVer-compatible with the changes in this pull request (compared against the base branch).

Details

     Cloning apache/main
    Building datafusion v54.1.0 (current)
       Built [ 109.591s] (current)
     Parsing datafusion v54.1.0 (current)
      Parsed [   0.038s] (current)
    Building datafusion v54.1.0 (baseline)
       Built [ 105.773s] (baseline)
     Parsing datafusion v54.1.0 (baseline)
      Parsed [   0.037s] (baseline)
    Checking datafusion v54.1.0 -> v54.1.0 (no change; assume patch)
     Checked [   0.957s] 223 checks: 223 pass, 30 skip
     Summary no semver update required
    Finished [ 218.301s] datafusion
    Building datafusion-expr v54.1.0 (current)
       Built [  25.830s] (current)
     Parsing datafusion-expr v54.1.0 (current)
      Parsed [   0.080s] (current)
    Building datafusion-expr v54.1.0 (baseline)
       Built [  25.996s] (baseline)
     Parsing datafusion-expr v54.1.0 (baseline)
      Parsed [   0.081s] (baseline)
    Checking datafusion-expr v54.1.0 -> v54.1.0 (no change; assume patch)
     Checked [   1.796s] 223 checks: 222 pass, 1 fail, 0 warn, 30 skip

--- failure enum_variant_added: enum variant added on exhaustive enum ---

Description:
A publicly-visible enum without #[non_exhaustive] has a new variant.
        ref: https://doc.rust-lang.org/cargo/reference/semver.html#enum-variant-new
       impl: https://github.com/obi1kenobi/cargo-semver-checks/tree/v0.49.0/src/lints/enum_variant_added.ron

Failed in:
  variant LogicalPlan:AsOfJoin in /home/runner/work/datafusion/datafusion/datafusion/expr/src/logical_plan/plan.rs:301
  variant LogicalPlan:AsOfJoin in /home/runner/work/datafusion/datafusion/datafusion/expr/src/logical_plan/plan.rs:301

     Summary semver requires new major version: 1 major and 0 minor checks failed
    Finished [  54.776s] datafusion-expr
    Building datafusion-optimizer v54.1.0 (current)
       Built [  26.725s] (current)
     Parsing datafusion-optimizer v54.1.0 (current)
      Parsed [   0.033s] (current)
    Building datafusion-optimizer v54.1.0 (baseline)
       Built [  26.518s] (baseline)
     Parsing datafusion-optimizer v54.1.0 (baseline)
      Parsed [   0.035s] (baseline)
    Checking datafusion-optimizer v54.1.0 -> v54.1.0 (no change; assume patch)
     Checked [   0.239s] 223 checks: 223 pass, 30 skip
     Summary no semver update required
    Finished [  54.359s] datafusion-optimizer
    Building datafusion-physical-plan v54.1.0 (current)
       Built [  36.518s] (current)
     Parsing datafusion-physical-plan v54.1.0 (current)
      Parsed [   0.155s] (current)
    Building datafusion-physical-plan v54.1.0 (baseline)
       Built [  36.273s] (baseline)
     Parsing datafusion-physical-plan v54.1.0 (baseline)
      Parsed [   0.150s] (baseline)
    Checking datafusion-physical-plan v54.1.0 -> v54.1.0 (no change; assume patch)
     Checked [   0.938s] 223 checks: 223 pass, 30 skip
     Summary no semver update required
    Finished [  75.376s] datafusion-physical-plan
    Building datafusion-proto v54.1.0 (current)
       Built [  61.798s] (current)
     Parsing datafusion-proto v54.1.0 (current)
      Parsed [   0.020s] (current)
    Building datafusion-proto v54.1.0 (baseline)
       Built [  62.259s] (baseline)
     Parsing datafusion-proto v54.1.0 (baseline)
      Parsed [   0.021s] (baseline)
    Checking datafusion-proto v54.1.0 -> v54.1.0 (no change; assume patch)
     Checked [   0.341s] 223 checks: 223 pass, 30 skip
     Summary no semver update required
    Finished [ 125.646s] datafusion-proto
    Building datafusion-proto-models v54.1.0 (current)
       Built [  25.350s] (current)
     Parsing datafusion-proto-models v54.1.0 (current)
      Parsed [   0.138s] (current)
    Building datafusion-proto-models v54.1.0 (baseline)
       Built [  25.141s] (baseline)
     Parsing datafusion-proto-models v54.1.0 (baseline)
      Parsed [   0.138s] (baseline)
    Checking datafusion-proto-models v54.1.0 -> v54.1.0 (no change; assume patch)
     Checked [   2.274s] 223 checks: 222 pass, 1 fail, 0 warn, 30 skip

--- failure enum_variant_added: enum variant added on exhaustive enum ---

Description:
A publicly-visible enum without #[non_exhaustive] has a new variant.
        ref: https://doc.rust-lang.org/cargo/reference/semver.html#enum-variant-new
       impl: https://github.com/obi1kenobi/cargo-semver-checks/tree/v0.49.0/src/lints/enum_variant_added.ron

Failed in:
  variant LogicalPlanType:AsOfJoin in /home/runner/work/datafusion/datafusion/datafusion/proto-models/src/generated/prost.rs:83
  variant LogicalPlanType:AsOfJoin in /home/runner/work/datafusion/datafusion/datafusion/proto-models/src/generated/prost.rs:83
  variant PhysicalPlanType:AsOfJoin in /home/runner/work/datafusion/datafusion/datafusion/proto-models/src/generated/prost.rs:1407
  variant PhysicalPlanType:AsOfJoin in /home/runner/work/datafusion/datafusion/datafusion/proto-models/src/generated/prost.rs:1407

     Summary semver requires new major version: 1 major and 0 minor checks failed
    Finished [  54.504s] datafusion-proto-models
    Building datafusion-sql v54.1.0 (current)
       Built [  41.764s] (current)
     Parsing datafusion-sql v54.1.0 (current)
      Parsed [   0.034s] (current)
    Building datafusion-sql v54.1.0 (baseline)
       Built [  41.021s] (baseline)
     Parsing datafusion-sql v54.1.0 (baseline)
      Parsed [   0.035s] (baseline)
    Checking datafusion-sql v54.1.0 -> v54.1.0 (no change; assume patch)
     Checked [   0.345s] 223 checks: 223 pass, 30 skip
     Summary no semver update required
    Finished [  84.424s] datafusion-sql
    Building datafusion-substrait v54.1.0 (current)
       Built [ 332.854s] (current)
     Parsing datafusion-substrait v54.1.0 (current)
      Parsed [   0.019s] (current)
    Building datafusion-substrait v54.1.0 (baseline)
       Built [ 334.107s] (baseline)
     Parsing datafusion-substrait v54.1.0 (baseline)
      Parsed [   0.019s] (baseline)
    Checking datafusion-substrait v54.1.0 -> v54.1.0 (no change; assume patch)
     Checked [   0.286s] 223 checks: 223 pass, 30 skip
     Summary no semver update required
    Finished [ 669.609s] datafusion-substrait

codecov-commenter · 2026-07-23T07:04:11Z

Codecov Report

❌ Patch coverage is 60.50228% with 865 lines in your changes missing coverage. Please review.
✅ Project coverage is 80.55%. Comparing base (e8a65f2) to head (ce2fbc4).
⚠️ Report is 5 commits behind head on main.

Files with missing lines	Patch %	Lines
datafusion/proto-models/src/generated/pbjson.rs	0.00%	341 Missing ⚠️
datafusion/physical-plan/src/joins/asof_join.rs	85.53%	118 Missing and 74 partials ⚠️
datafusion/expr/src/logical_plan/plan.rs	38.70%	100 Missing and 14 partials ⚠️
datafusion/core/src/physical_planner.rs	0.00%	43 Missing ⚠️
datafusion/expr/src/logical_plan/tree_node.rs	0.00%	43 Missing ⚠️
datafusion/proto/src/logical_plan/mod.rs	62.06%	20 Missing and 13 partials ⚠️
datafusion/optimizer/src/analyzer/type_coercion.rs	0.00%	28 Missing ⚠️
datafusion/optimizer/src/push_down_filter.rs	0.00%	24 Missing ⚠️
datafusion/proto-models/src/generated/prost.rs	0.00%	17 Missing ⚠️
datafusion/expr/src/logical_plan/builder.rs	82.45%	1 Missing and 9 partials ⚠️
... and 4 more

Additional details and impacted files

@@            Coverage Diff             @@
##             main   #23832      +/-   ##
==========================================
- Coverage   80.66%   80.55%   -0.12%     
==========================================
  Files        1095     1096       +1     
  Lines      372294   374647    +2353     
  Branches   372294   374647    +2353     
==========================================
+ Hits       300324   301805    +1481     
- Misses      54055    54801     +746     
- Partials    17915    18041     +126

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

# Conflicts: # datafusion/expr/src/logical_plan/plan.rs

# Conflicts: # datafusion/proto/src/physical_plan/mod.rs

Xuanwo added 3 commits July 23, 2026 13:05

feat: add ASOF join physical operator

e546de5

feat: add ASOF join logical semantics

bd58e1d

feat: serialize ASOF join plans

ed2ce92

github-actions Bot added sql SQL Planner logical-expr Logical plan and expressions optimizer Optimizer rules core Core DataFusion crate substrait Changes to the substrait crate proto Related to proto crate physical-plan Changes to the physical-plan crate labels Jul 23, 2026

Xuanwo mentioned this pull request Jul 23, 2026

feat: support ASOF joins #23738

Open

Xuanwo added 3 commits July 23, 2026 14:30

Merge remote-tracking branch 'origin/main' into xuanwo/asof-physical

68ac8ec

Merge branch 'xuanwo/asof-physical' into xuanwo/asof-logical

c8c4813

Merge branch 'xuanwo/asof-logical' into xuanwo/asof-proto

ee41c49

github-actions Bot added the auto detected api change Auto detected API change label Jul 23, 2026

fix: keep generated protobuf code in sync

d36fd27

Xuanwo added 9 commits July 27, 2026 16:22

Merge remote-tracking branch 'origin/main' into xuanwo/asof-physical

73fd599

feat: broadcast ASOF right input

43110a3

Merge branch 'xuanwo/asof-physical' into xuanwo/asof-logical

a15b693

# Conflicts: # datafusion/expr/src/logical_plan/plan.rs

Merge branch 'xuanwo/asof-logical' into xuanwo/asof-proto

88a46e1

# Conflicts: # datafusion/proto/src/physical_plan/mod.rs

fix: remove obsolete ASOF proto import

9556ed3

fix: account shared ASOF build buffers once

339d785

Merge branch 'xuanwo/asof-physical' into xuanwo/asof-logical

a71c19f

fix: align ASOF logical contracts

3a4f995

Merge branch 'xuanwo/asof-logical' into xuanwo/asof-proto

ce2fbc4

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

feat: serialize ASOF join plans - #23832

feat: serialize ASOF join plans#23832
Xuanwo wants to merge 16 commits into
apache:mainfrom
Xuanwo:xuanwo/asof-proto

Xuanwo commented Jul 23, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Jul 23, 2026 •

edited

Loading

Uh oh!

codecov-commenter commented Jul 23, 2026 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

Xuanwo commented Jul 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Which issue does this PR close?

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

Uh oh!

github-actions Bot commented Jul 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codecov-commenter commented Jul 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Xuanwo commented Jul 23, 2026 •

edited

Loading

github-actions Bot commented Jul 23, 2026 •

edited

Loading

codecov-commenter commented Jul 23, 2026 •

edited

Loading