Skip to content

Readiness-aware view exploitation: distinguish predicate-pruned empty MVs from unpopulated ones - #122

Merged
xudong963 merged 4 commits into
datafusion-contrib:mainfrom
zhuqi-lucas:readiness-upstream
Jul 28, 2026
Merged

Readiness-aware view exploitation: distinguish predicate-pruned empty MVs from unpopulated ones#122
xudong963 merged 4 commits into
datafusion-contrib:mainfrom
zhuqi-lucas:readiness-upstream

Conversation

@zhuqi-lucas

Copy link
Copy Markdown
Contributor

Rationale for this change

A materialized view candidate whose scan yields an empty plan is ambiguous. Either the predicate genuinely pruned everything, in which case the empty candidate is the correct answer and dramatically cheaper than scanning the base table (a selective filter on a rarely-matching literal), or the MV simply has not been populated yet, in which case trusting the empty plan silently returns wrong results. A cost function today cannot tell these apart, so the safe policy is to penalize every empty candidate, which also throws away the legitimate predicate-pruned wins.

What changes are included in this PR?

  • RewriteReadiness (Ready / NotReady / Unknown) and Materialized::rewrite_readiness() (default Unknown), letting table providers report lifecycle state. New module rewrite::readiness.
  • CandidateMetadata (Base | Materialized { table_ref, readiness }) carried per branch via a RewriteContext attached to OneOf / OneOfExec, positionally aligned with branch order.
  • Stable branch ordering (prerequisite for the alignment): OneOf::inputs() now returns branches in stored order. It previously re-sorted with LogicalPlan::partial_cmp on every call; since with_exprs_and_inputs rebuilds branches from inputs() output, any optimizer pass that mutates a branch's shape could move the base plan off branches[0], at which point OneOf::schema() exposes a candidate's schema instead of the query's. Candidates are now sorted by table reference at construction (the mv_plans HashMap iteration order was also nondeterministic across processes) and the base plan is pinned at branches[0].
  • Two-gate NotReady filtering: dropped at logical rewrite time, and again at physical planning time after a refresh.
  • Physical-time readiness refresh in plan_extension: by the time the physical planner runs, every candidate's scan() has executed, so providers whose readiness depends on scan-time side effects (e.g. lazy index loading on first scan) are re-consulted for the definitive value. Providers can additionally opt into ReadinessAnnotatedExec, a transparent wrapper carrying readiness captured atomically from the scan's own snapshot; the refresh prefers the annotation over re-sampling, which closes a TOCTOU race (scan sees an unpopulated source and plans empty, a concurrent load publishes the source, the refresh re-samples and wrongly trusts the stale empty plan). The wrapper is stripped before the OneOfExec is constructed so it never impedes later optimizer passes.
  • CostFn signature change: cost functions now receive a CostContext (candidate plan iterator + RewriteContext) instead of a bare iterator, so they can implement readiness-aware policies such as "trust Ready empty candidates, keep penalizing Unknown empty ones".

Are these changes tested?

43 new unit tests: readiness types and annotation/strip round-trips, both filtering gates, the physical-time refresh (including the scan-transition and all-candidates-NotReady short-circuit cases), and branch-order/metadata alignment across with_exprs_and_inputs rebuilds.

Are there any user-facing changes?

  • CostFn is a breaking signature change: existing cost functions change from Fn(Box<dyn Iterator<Item = &dyn ExecutionPlan>>) to Fn(CostContext), where CostContext::into_candidate_plans() returns the same iterator.
  • Plan selection between exactly-tied candidates becomes deterministic (base first, then candidates by table reference).
  • Materialized gains a provided method (rewrite_readiness), default-implemented, so existing implementations are unaffected.

… MVs from unpopulated ones

A materialized view candidate whose scan yields an empty plan is
ambiguous: either the predicate genuinely pruned everything (the empty
result is correct and extremely cheap), or the MV simply has not been
populated yet (trusting the empty plan silently returns wrong results).
Cost functions previously could not tell these apart.

This adds:

- RewriteReadiness (Ready / NotReady / Unknown) and a
  Materialized::rewrite_readiness() method (default: Unknown) so table
  providers can report their lifecycle state.
- CandidateMetadata (Base | Materialized { table_ref, readiness })
  carried per branch via a RewriteContext attached to OneOf/OneOfExec,
  aligned with branch order. To make that alignment hold,
  OneOf::inputs() now returns branches in stored order (previously it
  re-sorted with LogicalPlan::partial_cmp on every call, which could
  move the base branch off branches[0] after a rebuild and expose a
  candidate's schema from OneOf::schema()); candidates are sorted by
  table reference at construction, and the base plan is pinned at
  branches[0].
- Two-gate NotReady filtering: NotReady candidates are dropped at
  logical rewrite time, and again at physical planning time after a
  refresh.
- Physical-time readiness refresh in plan_extension: by the time the
  physical planner runs, every candidate's scan() has executed, so
  providers whose readiness depends on scan-time side effects (lazy
  index loading) are re-consulted. Providers can also opt into
  ReadinessAnnotatedExec, a transparent wrapper that carries readiness
  captured atomically from the scan's own snapshot, which the refresh
  prefers over re-sampling (avoids a TOCTOU race with concurrent index
  publication); the wrapper is stripped before the OneOfExec is built.
- CostFn now receives a CostContext (candidate plans + rewrite
  metadata) instead of a bare plan iterator, so cost functions can
  implement policies like 'trust Ready empty candidates, penalize
  Unknown empty ones'.

43 new unit tests cover the readiness types, annotation and stripping,
the two filtering gates, the refresh, and branch-order/metadata
alignment across plan rebuilds.
Copilot AI review requested due to automatic review settings July 27, 2026 09:33

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

@zhuqi-lucas
zhuqi-lucas marked this pull request as draft July 27, 2026 09:35
- dependencies.rs: LogicalPlan::Window pattern already covers window_expr via ..
- file_metadata.rs: use ? on the else branch instead of explicit return None
The current cargo-deny CLI rejects the singular `license` subcommand:
    error: invalid value 'license' for '[WHICH]...'
Use the plural form so the license check runs.
@zhuqi-lucas
zhuqi-lucas marked this pull request as ready for review July 28, 2026 06:39
@xudong963
xudong963 merged commit dcda49e into datafusion-contrib:main Jul 28, 2026
8 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants