Updatable View Matcher

[wyatt-herkamp]

Allows the set of view-matching candidates to be updated at runtime, instead of being frozen at ViewMatcher::try_new_from_state time.

Motivation

ViewMatcher scanned every catalog once at construction and stored the result in a plain HashMap behind &self. Any materialized view registered, replaced, or dropped afterwards was invisible to query rewriting until the whole ViewMatcher (and therefore the SessionState holding it) was rebuilt. Long-lived services that register MVs dynamically need the candidate set to track the catalog.

What changed

ViewMatcherTable

The map value changes from the tuple (Arc<dyn TableProvider>, SpjNormalForm) to a named struct:

pub struct ViewMatcherTable {
    pub normal_form: Arc<SpjNormalForm>,
    pub table: Arc<dyn TableProvider>,
}

The SpjNormalForm is now behind an Arc so snapshot clones are cheap.

Copy-on-write candidate set

pub struct ViewMatcher {
    mv_plans: RwLock<Arc<HashMap<TableReference, ViewMatcherTable>>>,
    update_lock: futures::lock::Mutex<()>,
}

OptimizerRule::rewrite takes the read lock only long enough to clone the Arc, then does all rewriting against that immutable snapshot — so a concurrent catalog update can never mutate the map out from under an in-flight rewrite, and rewrites never block on updates. Mutations clone the inner HashMap, apply the change, and swap in a new Arc. Adds a parking_lot dependency for the RwLock.

ViewMatchingRewriter correspondingly holds the snapshot (Arc<HashMap<..>>) instead of a &ViewMatcher borrow.

Three ways to maintain the set

Method	Behavior
update_candidates(&state)	Re-scans every catalog and atomically replaces the whole set.
refresh_candidate(&state, table_ref)	Re-analyzes one table and inserts/replaces/removes just its entry. Returns whether the table is now a candidate.
invalidate_candidate(&state, table_ref)	Immediately drops one entry without any analysis. Returns whether one was removed.

refresh_candidate is the cheap path when a single table changed — it analyzes only that table instead of walking every catalog. It removes the entry when the table was dropped, is not a materialized view, or has use_in_query_rewrite = false / no SPJ normal form.

The full-scan and single-table paths share one build_candidate helper, so a candidate is computed identically no matter which entry point produced it.

Correctness details

Table references are resolved. Both refresh_candidate and invalidate_candidate resolve the incoming reference against the session's default catalog/schema before touching the map. The full scan stores fully-qualified keys, so without this a bare "mv" would create a second, duplicate entry rather than updating datafusion.public.mv. Bare, partial, and fully-qualified spellings all address the same entry.

The provider is read from the catalog, not from the caller. refresh_candidate looks the TableProvider up through catalog_list() rather than accepting one as an argument, so an entry always reflects what is actually registered — a caller can't inject a provider that was already replaced.

Mutations are serialized. update_lock (an async mutex, held across the .await on analysis) ensures a slow full rescan cannot overwrite a newer single-table refresh with stale results. The candidate set converges to the catalog state seen by the last completed call.

Failures fail closed. If a refresh errors — catalog lookup failure or analyzer error — the table's candidate is removed and the error returned, rather than leaving a possibly-stale entry live for rewriting. Only the failing table is affected; other candidates are untouched. The table becomes a candidate again on the next successful refresh.

Breaking API changes

• ViewMatcher::mv_plans() returns Arc<HashMap<TableReference, ViewMatcherTable>> (an owned snapshot) instead of &HashMap<TableReference, (Arc<dyn TableProvider>, SpjNormalForm)>. A reference can't be handed out from behind the lock.
• get_potential_mv_candidates_for_table() returns Vec<(TableReference, ViewMatcherTable)> instead of the 3-tuple Vec<(TableReference, Arc<dyn TableProvider>, &SpjNormalForm)>.

Tests

Six new tests in tests_incremental_candidates covering:

• refresh_candidate ignores non-materialized tables and unknown tables — and asserts via Arc::ptr_eq that a no-op does not swap the snapshot at all
• refresh_candidate normalizes table refs: a bare ref updates the fully-qualified entry rather than creating a duplicate; after deregistration any spelling removes it
• invalidate_candidate removes an entry addressed by a bare ref, is a no-op (no snapshot swap) when absent, and a later refresh brings the candidate back
• analyzer failure during refresh removes only the failing table's candidate (via an injected AnalyzerRule that poisons specific plans)
• catalog lookup failure during refresh removes only the failing table's candidate (via a SchemaProvider whose table() can be made to error)

Full suite: 71 lib tests + 1 integration test pass; cargo clippy --tests --all-features -- -D warnings and cargo fmt --check are clean.

Rebase note

Rebased onto branch-54 on top of #55 (RewriteReadiness / per-branch CandidateMetadata). Both changes rewrote the candidate-generation loop in f_down; the merged version keeps #55's NotReady filtering and readiness propagation while reading through ViewMatcherTable and the immutable snapshot. Readiness is still read from the live provider on each rewrite, so #55's lazy-index warming behavior is preserved. Every commit in the branch compiles independently.

[xudong963]

What consistency semantics do we want during a candidate refresh? After the catalog replaces provider A with B, the matcher continues exposing A while refresh_candidate waits for the lock, looks up B, and analyzes it. Is this intentional RCU/stale-while-refresh behavior, where completion of refresh_candidate is the visibility boundary? Or should new queries stop using A immediately and fall back to the original plan until B is ready?

Relatedly, if schema.table(...).await returns an error, A remains available indefinitely. Is stale-on-error also intentional? If so, could we document and test these semantics explicitly?

[wyatt-herkamp]

What consistency semantics do we want during a candidate refresh? After the catalog replaces provider A with B, the matcher continues exposing A while refresh_candidate waits for the lock, looks up B, and analyzes it. Is this intentional RCU/stale-while-refresh behavior, where completion of refresh_candidate is the visibility boundary? Or should new queries stop using A immediately and fall back to the original plan until B is ready?

When a table is added, this will be applied last; no refresh until the table is "ready". When a table is removed, it should be removed from the view matcher first. So we stop showing it as a candidate before we start to remove the rest of the table.

During a change in the table configuration, it gets messy. Any change in the table should, in theory, keep it a valid candidate. As changing the SQL for a materialized view that changes the projection usually results in other issues.
Part of me thinks that a change in a table should cause it to first be dropped as a candidate. Changes get applied internally to the rest of the system. Then we refresh again to say the table is ready.

Relatedly, if schema.table(...).await returns an error, A remains available indefinitely. Is stale-on-error also intentional? If so, could we document and test these semantics explicitly?

I am honestly not sure what the best behavior here is.

Part of me says a failure to refresh the candidate should remove this table and all related tables as valid options. If you have preferred behavior, I would love to hear it.