feat(gfql): native polars multi-source (node-cartesian) MATCH lowering (#1273)

CHANGELOG.md

@@ -12,6 +12,7 @@ This project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.htm
 - **GFQL engine-selection docs (pandas / polars / cuDF / polars-gpu)**: New :doc:`Choosing a GFQL Engine <gfql/engines>` page — a numbers-first, persona-tested guide to the four interchangeable engines. Adds the one-keyword `engine='polars'` speedup (up to ~38× over pandas on real graphs, no GPU), a motivating warm-median comparison table on real public graphs (LiveJournal 35M / Orkut 117M), a decision matrix (workload shape × size × hardware → engine, with the measured ~10K-edge CPU crossover, GPU-work-bound rule, polars-gpu memory-pressure caveat, and GPU-or-error contract), a cuDF-vs-polars-gpu disambiguation (eager-op vs fused-lazy; cuDF is not deprecated), an honest "when *not* to use Polars" section, the differential-parity guarantee, and a methodology + reproducer-script disclosure. Rewrote the top of `gfql/performance.rst` to lead with the engine comparison (de-marketed the prose), wired the new page into the GFQL toctree + recommended paths, and added Polars/polars-gpu to the engine examples in `gfql/quick.rst` and `gfql/about.rst` (previously only pandas/cuDF were documented). Driven by 4-persona doc user-testing (pandas DS, RAPIDS user, perf engineer, skeptical evaluator).
 
 ### Fixed
+- **GFQL polars engine natively runs multi-source (node-cartesian) MATCH (#1273)**: comma-separated disconnected aliases — `g.gfql("MATCH (a {..}), (b {..}) RETURN a.id, b.id")` — previously declined on polars (the `rows(binding_ops)` `node_cartesian` branch hard-NIE'd) while pandas handled them. New `_cartesian_node_bindings_polars` mirrors the pandas oracle `_gfql_cartesian_node_bindings_row_table`: each alias is independently filtered, projected into the per-alias lookup schema (bare `alias` id, `alias.id`, `alias.<prop>`, and pandas' leaked `alias.alias=True` flag incl. property-shadowing), then left-major cross-joined for row-order parity. Scalar/aliased/`count(*)` projections over the cartesian match pandas exactly (verified end-to-end); whole-entity `RETURN a, b` stays an honest NIE (separate projection surface). Two parity-safe declines mirror shapes where pandas itself errors (proven on master, so both engines fail identically, never diverge): an anonymous node op, and ≥4 named aliases (pandas' bare-id merge residue collides). Differential fuzz vs pandas over ~10k lowered cases: 0 disagreements. Tests in `test_engine_polars_binding_rows.py`.
 - **GFQL polars chain — variable-length node aliases are hop-distance gated; the chain is now silent-wrong-free (#1741)**: a node named after a variable-length edge (`g.gfql("MATCH (a)-[*1..2]-(b) RETURN b")`) previously carried its alias regardless of hop distance, so an undirected walk that backtracked into the seed wrongly returned it (pandas correctly excludes it — trail semantics). The polars chain now auto-injects the #1741 hop-distance label (name resolved against the user's node columns via the `reserved_columns` registry, so a user column named like the internal one is never clobbered) and applies pandas' alias `[min_hop, max_hop]` window in `_apply_node_names`. The one shape the gate can't yet cover — a node alias after a **forward/reverse `min_hops>1`** edge, whose labels need pandas' layered backward walk (not ported) — now **declines with an honest `NotImplementedError` (#1748)** instead of returning nodes outside the window; the decline is precise (differential vs pandas: 30/30 named shapes NIE, 30/30 unnamed run and match, 0 over-decline). 4-engine A/B on the stacked build: pandas/cudf 144/144, polars/polars-gpu 112 agree / **0 disagree** / 32 honest-NIE — zero silent-wrong shapes remain on the polars chain. Adapts the mechanism from the retired #1742 (declined undirected varlen+alias, now natively gated). Tests: `TestVarlenAliasHopGate` (pandas-parity across directions, the `*2..3` decline + unnamed-still-runs pins, a column-collision test).
 - **GFQL native polars `label_node_hops` on the plain BFS — correct, direction-dependent hop labels (#1741 groundwork)**: the polars eager hop now emits pandas-parity node hop-distance labels instead of declining `label_node_hops`. The labeling rule is DIRECTION-DEPENDENT (the subtle part, derived by differential fuzz vs the pandas oracle, not by reading pandas alone): forward/reverse label EVERY destination of a hop first-wins (matching pandas `hop.py` `new_node_ids`), so a seed re-entered at hop 1 IS labeled; undirected labels destinations MINUS everything already visited and pre-seeds the seen-set with the seeds, so a seed re-reached by a backtracking walk stays NULL (its shortest-path distance). Two correctness fixes over the first cut: (1) the undirected seed pre-seed must NOT depend on `return_as_wave_front` — a seed filtered out of the frontier by `source_node_match` or suppressed in wave-front mode was being re-labeled (A/B vs pandas over direction × hops × `return_as_wave_front` × `source_node_match` × `destination_node_match` × 10 graphs: was 456/24, now 480/480); (2) a requested `label_node_hops` name that collides with an existing column is redirected to `<name>_1` like pandas' `resolve_label_col`, not the polars left-join auto-suffix `<name>_right`/`DuplicateError`. `label_edge_hops` and `min_hops>1` labels stay honest NIEs. Amplified `test_engine_polars_hop.py` (`TestHopLabelsDifferential` over the seed-filter / wave-front / multi-seed axes, an explicit direction-asymmetry contrast pin, and a collision test).
 - **GFQL polars engine natively runs whole-entity aggregation Cypher (LDBC IC4 shape): HAS_<Label> destination disambiguation + entity identity-key resolution + `key_prefixes` group-by expansion**: three previously honest-NIE polars lowerings, each an exact pandas-parity port: (1) `binding_rows_polars` now applies pandas' `_gfql_disambiguate_has_edge_destination_nodes` candidate-domain rule (an UNLABELED next-node op after a `HAS_<Label>`-typed edge narrows to that label ONLY when candidate node ids collide across labels) instead of declining every labeled-destination pattern; (2) `alias.__gfql_node_id__` (the #1650 whole-entity identity key the aggregation lowering groups by) resolves to the bare `alias` id column (the polars bindings table intentionally omits pandas' join-residue columns — the bare alias column IS the identity key); (3) `group_by(key_prefixes=...)` expands every `<prefix>*` row-table column into the key set exactly like pandas (functionally dependent on the identity key — group sizes unchanged). Result: the official LDBC IC4 (new-topics) query — comma-MATCH, `WITH DISTINCT`, CASE projections, whole-entity `sum` aggregation, HAVING-style `WHERE` — runs natively on polars, parity-exact (harness-verified vs expected rows at SF0.1: ok, 127.8 ms). Regression test `test_polars_rows_entity_groupby.py` (pandas oracle + polars parity on a discriminating fixture). Remaining polars `rows` declines are undirected bounded var-length (IC6/IC11) and shortestPath/zero-hop-unbounded (by design).

graphistry/compute/gfql/lazy/engine/polars/row_pipeline.py

@@ -21,6 +21,7 @@
 if TYPE_CHECKING:
     import polars as pl
     from graphistry.compute.gfql.expr_parser import ExprNode, FunctionCall
+    from graphistry.compute.ast import ASTObject
 
 from graphistry.Plottable import Plottable
 from graphistry.utils.json import JSONVal
@@ -870,6 +871,101 @@ def select_extend_polars(g: Plottable, items: Sequence[SelectItem]) -> Optional[
     return _rewrap(g, out)
 
 
+def _cartesian_node_bindings_polars(
+    g: Plottable,
+    ops: "Sequence[ASTObject]",
+    node_id: Optional[str],
+) -> Optional[Plottable]:
+    """Native polars cross-product for disconnected MATCH aliases (#1273).
+
+    Mirrors the pandas ``_gfql_cartesian_node_bindings_row_table`` oracle: each
+    node alias is independently filtered, projected into the ``_gfql_node_alias_lookup_frame``
+    schema (``alias``, ``alias.node_id``, ``alias.<col>``, plus the leaked named-op
+    FLAG column ``alias.alias = True``), then cross-joined in op order (left-major,
+    matching pandas' ``merge`` order so no ORDER BY is needed for parity). The bare
+    ``node_id`` residue column that pandas carries (``id_x``/``id_y`` merge suffixes)
+    is intentionally dropped: no lowered query references it, and dropping avoids
+    polars cross-join column collisions on 3+ aliases.
+
+    Returns None to DECLINE (caller raises the honest NIE) outside the supported
+    subset: node ``query=`` params, ``alias == node_id`` (pandas' flag column
+    overwrites the id column — no sane shared semantics), and seeded re-entry
+    (already gated by the caller). NO-CHEATING: never bridges to pandas.
+    """
+    import polars as pl
+    from graphistry.compute.ast import ASTNode
+    from graphistry.compute.gfql.lazy import collect as _lazy_collect
+    from .predicates import filter_by_dict_polars
+
+    nodes = g._nodes
+    if nodes is None or node_id is None:  # pragma: no cover - defensive: bindings run post-materialize
+        return None
+    node_id = str(node_id)
+    if node_id not in nodes.columns:  # pragma: no cover - defensive: node_id is the bound id column
+        return None
+
+    aliases = [op._name for op in ops]
+    # Decline outside pandas' RELIABLE zone (empirically derived, keeps parity):
+    #  - anonymous node op: the pandas cartesian raises a spurious schema error on
+    #    an EMPTY result when a bare `()` is present (it drops the id column) rather
+    #    than returning empty — declining avoids that divergence.
+    #  - >3 named aliases: the pandas builder's per-alias bare-id merge residue
+    #    collides on the 4th frame ("Passing 'suffixes' which cause duplicate
+    #    columns"). Both engines must not diverge, so decline to the honest NIE.
+    if any(not isinstance(a, str) for a in aliases):
+        return None
+    named = [a for a in aliases if isinstance(a, str)]
+    if len(named) > 3:
+        return None
+
+    nodes_lf = nodes.lazy()
+    # filter_by_dict_polars is frame-polymorphic (LazyFrame in -> LazyFrame out) but
+    # annotated ``pl.DataFrame``; keep the accumulator loose, as this module does.
+    per_alias: List[Any] = []
+    for op in ops:
+        if not isinstance(op, ASTNode) or op.query is not None:  # pragma: no cover - node_cartesian only routes bare ASTNode ops
+            return None
+        alias = op._name
+        if not isinstance(alias, str):  # pragma: no cover - non-str aliases already declined above
+            return None
+        if alias == node_id:
+            # pandas' named-op flag column overwrites the id column here — neither
+            # engine has sane semantics; decline (mirrors the single-entity
+            # ``rows_binding_ops_polars`` corner).
+            return None
+        try:
+            matched = filter_by_dict_polars(nodes_lf, op.filter_dict)
+        except NotImplementedError:  # pragma: no cover - propagate exotic-predicate NIE unchanged
+            raise
+        except Exception:  # pragma: no cover - defensive: unexpected filter failure declines
+            return None
+        cols = matched.collect_schema().names()
+        # prop_cols excludes node_id and any real column named == alias: the pandas
+        # node execute() leaks a boolean FLAG into a column named ``alias``
+        # (shadowing a same-named real property), which the lookup frame surfaces
+        # as ``alias.alias = True``. Reproduce that exactly.
+        prop_cols = [c for c in cols if c != node_id and c != alias]
+        exprs = [
+            pl.col(node_id).alias(alias),
+            pl.col(node_id).alias(f"{alias}.{node_id}"),
+            pl.lit(True).alias(f"{alias}.{alias}"),
+        ]
+        exprs.extend(pl.col(c).alias(f"{alias}.{c}") for c in prop_cols)
+        per_alias.append(matched.select(exprs))
+
+    if not per_alias:  # pragma: no cover - defensive: ops is non-empty so per_alias is too
+        return None
+    state = per_alias[0]
+    for frame in per_alias[1:]:
+        # Left-major cross join → same row order as the pandas constant-key merge.
+        state = state.join(frame, how="cross")
+    try:
+        out_df = _lazy_collect(state)
+    except pl.exceptions.SchemaError:  # pragma: no cover - defensive: cross-join schema clash declines
+        return None
+    return _rewrap(g, out_df)
+
+
 def binding_rows_polars(
     g: Plottable,
     binding_ops: Sequence[Dict[str, JSONVal]],
@@ -921,7 +1017,8 @@ def _names(lf: pl.LazyFrame) -> List[str]:
     # for malformed op sequences / duplicate aliases — same error as pandas.
     RowPipelineMixin._gfql_validate_binding_ops(ops)
     if RowPipelineMixin._gfql_binding_ops_mode(ops) == "node_cartesian":
-        return None  # MATCH (a), (b) cross joins: deferred (rare; own schema study)
+        # MATCH (a), (b), ... disconnected node aliases: native cross-product (#1273).
+        return _cartesian_node_bindings_polars(g, ops, node_id)
     if RowPipelineMixin._gfql_is_shortest_path_scalar_binding_ops(ops):
         return None  # shortestPath scalar contract: BFS/native backends, pandas-only
 

graphistry/tests/compute/gfql/test_engine_polars_binding_rows.py

@@ -81,6 +81,13 @@ def _assert_parity(query, *, order_sensitive=False):
     "MATCH (a)-[*2..2]->(b) RETURN count(*) AS c",           # exactly-k
     "MATCH (a)-[:F*1..2]->(b) RETURN count(*) AS c",         # typed var-length
     "MATCH (a)-[*1..2]->(b)-[]->(c) RETURN count(*) AS c",   # var-length + fixed hop
+    # node-only cartesian: disconnected multi-source MATCH (#1273). <=3 named aliases.
+    "MATCH (a), (b) RETURN a.id AS ai, b.id AS bi ORDER BY ai, bi",
+    "MATCH (a {kind: 'a'}), (b {kind: 'b'}) RETURN a.id AS ai, b.id AS bi",
+    "MATCH (a {kind: 'a'}), (b) RETURN a.id AS ai, a.age AS aa, b.id AS bi",
+    "MATCH (a {kind: 'a'}), (b {kind: 'b'}) WHERE a.age > 20 RETURN a.id AS ai, b.id AS bi",
+    "MATCH (a), (b), (c {kind: 'a'}) RETURN a.id AS ai, b.id AS bi, c.id AS ci ORDER BY ai, bi, ci",
+    "MATCH (a {kind: 'z'}), (b) RETURN a.id AS ai, b.id AS bi",   # empty (no kind=z) keeps schema
 ]
 
 # Outside the MVP subset: must raise NotImplementedError (honest NIE, no bridge,
@@ -89,6 +96,10 @@ def _assert_parity(query, *, order_sensitive=False):
     "MATCH (a)-[*]->(b) RETURN count(*) AS c",               # unbounded var-length
     "MATCH (a)-[*1..2]-(b) RETURN count(*) AS c",            # undirected var-length
     "MATCH (a)-[e]->(b) WHERE a.age < b.age RETURN a.id",    # cross-alias same-path WHERE
+    # cartesian outside the pandas-reliable subset (#1273): pandas itself errors or
+    # is fragile here, so polars declines rather than diverge.
+    "MATCH (a), (b), (c), (d) RETURN a.id, b.id, c.id, d.id",  # >3 named aliases
+    "MATCH (a), (b), () RETURN a.id, b.id",                    # anonymous companion
 ]
 
 
@@ -121,6 +132,50 @@ def test_polars_binding_rows_raw_table_meaningful_cols():
     pd.testing.assert_frame_equal(a, b, check_dtype=False)
 
 
+def test_polars_cartesian_binding_rows_raw_meaningful_cols():
+    """Raw node-only cartesian rows(binding_ops): polars carries the same meaningful
+    per-alias schema as pandas (bare ``alias`` id, ``alias.id``, ``alias.<prop>``,
+    plus the leaked named-op flag ``alias.alias = True``), values equal to pandas."""
+    bo = serialize_binding_ops([n(name="a"), n(name="b")])
+    rpd = BASE.gfql([rows(binding_ops=bo)], engine="pandas")._nodes
+    rpl = BASE.gfql([rows(binding_ops=bo)], engine="polars")._nodes
+    assert "polars" in type(rpl).__module__
+    expected = {"a", "a.id", "a.age", "a.kind", "a.a", "b", "b.id", "b.age", "b.kind", "b.b"}
+    assert expected <= set(rpl.columns)
+    assert set(rpl.columns) <= set(rpd.columns)  # no columns pandas lacks
+    key = ["a", "b"]
+    a = rpd[list(rpl.columns)].sort_values(key).reset_index(drop=True)
+    b = rpl.to_pandas().sort_values(key).reset_index(drop=True)
+    pd.testing.assert_frame_equal(a, b, check_dtype=False)
+
+
+def test_polars_cartesian_alias_name_collides_with_property():
+    """A node property named the same as a MATCH alias is shadowed by the leaked
+    named-op flag (``alias.alias = True``) on BOTH engines — polars mirrors the
+    pandas quirk exactly rather than surfacing the real property value."""
+    nodes = pd.DataFrame({"id": [0, 1, 2], "kind": ["a", "b", "a"], "a": [10, 20, 30], "b": [1, 2, 3]})
+    g = graphistry.nodes(nodes, "id").edges(pd.DataFrame({"s": [0], "d": [1]}), "s", "d")
+    q = "MATCH (a {kind: 'a'}), (b {kind: 'b'}) RETURN a.id AS ai, a.a AS aa, b.id AS bi, b.b AS bb"
+    rpd = g.gfql(q, engine="pandas")._nodes.reset_index(drop=True)
+    rpl = g.gfql(q, engine="polars")._nodes.to_pandas().reset_index(drop=True)
+    assert list(rpd["aa"]) == [True, True] and list(rpd["bb"]) == [True, True]  # flag, not 10/30
+    pd.testing.assert_frame_equal(
+        rpd.sort_values(["ai", "bi"]).reset_index(drop=True),
+        rpl[rpd.columns.tolist()].sort_values(["ai", "bi"]).reset_index(drop=True),
+        check_dtype=False,
+    )
+
+
+def test_polars_cartesian_multiplicity_three_aliases():
+    """Three-alias cross product has |a|*|b|*|c| rows in left-major order on polars,
+    identical to pandas."""
+    q = "MATCH (a {kind: 'a'}), (b {kind: 'b'}), (c {kind: 'a'}) RETURN a.id AS ai, b.id AS bi, c.id AS ci"
+    rpd = BASE.gfql(q, engine="pandas")._nodes.reset_index(drop=True)
+    rpl = BASE.gfql(q, engine="polars")._nodes.to_pandas().reset_index(drop=True)
+    assert len(rpl) == 3 * 2 * 3  # kind a = {0,2,4}, kind b = {1,3}
+    pd.testing.assert_frame_equal(rpd, rpl[rpd.columns.tolist()], check_dtype=False)  # order-exact
+
+
 def test_binding_rows_projection_pushdown_skips_unused_props():
     """#1711: a query referencing no node properties (count(*)) attaches ZERO
     property columns to the binding table; one referencing only b's property
@@ -248,7 +303,19 @@ def test_polars_binding_rows_decline_branches_direct():
     assert binding_rows_polars(seeded, bo) is None
 
     g = graphistry.nodes(pl.from_pandas(NODES), "id").edges(pl.from_pandas(EDGES), "s", "d")
-    assert binding_rows_polars(g, serialize_binding_ops([n(name="a"), n(name="b")])) is None
+    # node-only cartesian (#1273) is now natively supported for <=3 named aliases;
+    # it declines only outside the pandas-reliable subset:
+    #  - anonymous node op (pandas raises a spurious schema error on empty results)
+    assert binding_rows_polars(g, serialize_binding_ops([n(name="a"), n()])) is None
+    #  - >3 named aliases (pandas' bare-id merge residue collides on the 4th frame)
+    assert binding_rows_polars(
+        g, serialize_binding_ops([n(name="a"), n(name="b"), n(name="c"), n(name="d")])
+    ) is None
+    #  - an alias named exactly like the bound node-id column ("id"): pandas' leaked
+    #    flag column would overwrite the id column — no sane shared semantics, so decline
+    assert binding_rows_polars(g, serialize_binding_ops([n(name="id"), n(name="b")])) is None
+    # ...but 2-3 named aliases now lower natively (returns a row table, not None)
+    assert binding_rows_polars(g, serialize_binding_ops([n(name="a"), n(name="b")])) is not None
     assert binding_rows_polars(g, serialize_binding_ops([n(name="a", query="id > 0"), e_forward(), n(name="b")])) is None
     assert binding_rows_polars(g, serialize_binding_ops([n(name="a"), e_forward(source_node_match={"kind": "a"}), n(name="b")])) is None
     assert binding_rows_polars(g, serialize_binding_ops([n(name="a"), e_forward(label_seeds=True), n(name="b")])) is None