fix(gfql): gate polars varlen node aliases by hop distance (refs #1741)

CHANGELOG.md

@@ -12,6 +12,7 @@ This project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.htm
 - **GFQL engine-selection docs (pandas / polars / cuDF / polars-gpu)**: New :doc:`Choosing a GFQL Engine <gfql/engines>` page — a numbers-first, persona-tested guide to the four interchangeable engines. Adds the one-keyword `engine='polars'` speedup (up to ~38× over pandas on real graphs, no GPU), a motivating warm-median comparison table on real public graphs (LiveJournal 35M / Orkut 117M), a decision matrix (workload shape × size × hardware → engine, with the measured ~10K-edge CPU crossover, GPU-work-bound rule, polars-gpu memory-pressure caveat, and GPU-or-error contract), a cuDF-vs-polars-gpu disambiguation (eager-op vs fused-lazy; cuDF is not deprecated), an honest "when *not* to use Polars" section, the differential-parity guarantee, and a methodology + reproducer-script disclosure. Rewrote the top of `gfql/performance.rst` to lead with the engine comparison (de-marketed the prose), wired the new page into the GFQL toctree + recommended paths, and added Polars/polars-gpu to the engine examples in `gfql/quick.rst` and `gfql/about.rst` (previously only pandas/cuDF were documented). Driven by 4-persona doc user-testing (pandas DS, RAPIDS user, perf engineer, skeptical evaluator).
 
 ### Fixed
+- **GFQL polars chain — variable-length node aliases are hop-distance gated; the chain is now silent-wrong-free (#1741)**: a node named after a variable-length edge (`g.gfql("MATCH (a)-[*1..2]-(b) RETURN b")`) previously carried its alias regardless of hop distance, so an undirected walk that backtracked into the seed wrongly returned it (pandas correctly excludes it — trail semantics). The polars chain now auto-injects the #1741 hop-distance label (name resolved against the user's node columns via the `reserved_columns` registry, so a user column named like the internal one is never clobbered) and applies pandas' alias `[min_hop, max_hop]` window in `_apply_node_names`. The one shape the gate can't yet cover — a node alias after a **forward/reverse `min_hops>1`** edge, whose labels need pandas' layered backward walk (not ported) — now **declines with an honest `NotImplementedError` (#1748)** instead of returning nodes outside the window; the decline is precise (differential vs pandas: 30/30 named shapes NIE, 30/30 unnamed run and match, 0 over-decline). 4-engine A/B on the stacked build: pandas/cudf 144/144, polars/polars-gpu 112 agree / **0 disagree** / 32 honest-NIE — zero silent-wrong shapes remain on the polars chain. Adapts the mechanism from the retired #1742 (declined undirected varlen+alias, now natively gated). Tests: `TestVarlenAliasHopGate` (pandas-parity across directions, the `*2..3` decline + unnamed-still-runs pins, a column-collision test).
 - **GFQL native polars `label_node_hops` on the plain BFS — correct, direction-dependent hop labels (#1741 groundwork)**: the polars eager hop now emits pandas-parity node hop-distance labels instead of declining `label_node_hops`. The labeling rule is DIRECTION-DEPENDENT (the subtle part, derived by differential fuzz vs the pandas oracle, not by reading pandas alone): forward/reverse label EVERY destination of a hop first-wins (matching pandas `hop.py` `new_node_ids`), so a seed re-entered at hop 1 IS labeled; undirected labels destinations MINUS everything already visited and pre-seeds the seen-set with the seeds, so a seed re-reached by a backtracking walk stays NULL (its shortest-path distance). Two correctness fixes over the first cut: (1) the undirected seed pre-seed must NOT depend on `return_as_wave_front` — a seed filtered out of the frontier by `source_node_match` or suppressed in wave-front mode was being re-labeled (A/B vs pandas over direction × hops × `return_as_wave_front` × `source_node_match` × `destination_node_match` × 10 graphs: was 456/24, now 480/480); (2) a requested `label_node_hops` name that collides with an existing column is redirected to `<name>_1` like pandas' `resolve_label_col`, not the polars left-join auto-suffix `<name>_right`/`DuplicateError`. `label_edge_hops` and `min_hops>1` labels stay honest NIEs. Amplified `test_engine_polars_hop.py` (`TestHopLabelsDifferential` over the seed-filter / wave-front / multi-seed axes, an explicit direction-asymmetry contrast pin, and a collision test).
 - **GFQL polars engine natively runs whole-entity aggregation Cypher (LDBC IC4 shape): HAS_<Label> destination disambiguation + entity identity-key resolution + `key_prefixes` group-by expansion**: three previously honest-NIE polars lowerings, each an exact pandas-parity port: (1) `binding_rows_polars` now applies pandas' `_gfql_disambiguate_has_edge_destination_nodes` candidate-domain rule (an UNLABELED next-node op after a `HAS_<Label>`-typed edge narrows to that label ONLY when candidate node ids collide across labels) instead of declining every labeled-destination pattern; (2) `alias.__gfql_node_id__` (the #1650 whole-entity identity key the aggregation lowering groups by) resolves to the bare `alias` id column (the polars bindings table intentionally omits pandas' join-residue columns — the bare alias column IS the identity key); (3) `group_by(key_prefixes=...)` expands every `<prefix>*` row-table column into the key set exactly like pandas (functionally dependent on the identity key — group sizes unchanged). Result: the official LDBC IC4 (new-topics) query — comma-MATCH, `WITH DISTINCT`, CASE projections, whole-entity `sum` aggregation, HAVING-style `WHERE` — runs natively on polars, parity-exact (harness-verified vs expected rows at SF0.1: ok, 127.8 ms). Regression test `test_polars_rows_entity_groupby.py` (pandas oracle + polars parity on a discriminating fixture). Remaining polars `rows` declines are undirected bounded var-length (IC6/IC11) and shortestPath/zero-hop-unbounded (by design).
 - **GFQL connected OPTIONAL MATCH no longer crashes on polars frames (engine-polymorphic seed extraction)**: `_optional_arm_start_nodes` (gfql_unified.py) applied pandas-only ops (`.dropna()`/`.drop_duplicates()`/`.rename(columns=)`/boolean-mask indexing) to the joined binding rows, so an IS7-shaped `MATCH ... OPTIONAL MATCH ...` on `engine='polars'` crashed with `AttributeError: 'DataFrame' object has no attribute 'dropna'` before reaching the row pipeline (hit by LDBC SNB interactive-short-7 via the official query text). Now FIXED end-to-end: the seed extraction branches on frame type (`drop_nulls().unique().rename({...})` + `filter(is_in)` on polars), the optional-arm dispatch skips the pandas-only seeded-rows contract on polars (the seed restriction is a pruning hint — arm rows are left-outer-joined on the shared aliases, so unseeded is semantically identical, and `where`-arms already ran unseeded), and the arm join/alias-synthesis block gains a polars twin (`is_in` semi-join prune, `join(how='left')`, null-preserving alias synthesis). Result: **connected OPTIONAL MATCH now runs natively on polars** (previously pandas-only), oracle-exact on the IS7 shape; the simple-CASE null-literal equality (`CASE x WHEN null`, lowered as the `__cypher_case_eq__(x, null)` marker) now lowers natively to `is_null()` on polars (pandas-parity null-mask semantics; the general `CASE x WHEN v` form still declines honestly — it carries pandas' bool/numeric cross-dtype rules), so the full LDBC IS7 query INCLUDING its `CASE r WHEN null` flag projection runs natively on polars, oracle-exact. The polars arm is also PRUNED like the pandas one: since the polars bindings-row path declines `start_nodes` by contract, the same restriction is expressed as an id-membership filter injected into the arm's first (shared) node op — measured 9.2× on the LDBC SF0.1 harness (IS7 message-replies cypher 710.8→77.2 ms). pandas/cuDF paths byte-identical. Regression tests `test_optional_match_polars_frames.py` (pandas oracle, polars native end-to-end, polars no-pandas-ism contract).

graphistry/compute/gfql/lazy/engine/polars/chain.py

@@ -21,6 +21,7 @@
 from .dtypes import is_lazy, colnames, endpoint_ids
 from .degrees import get_degrees_polars, get_indegrees_polars, get_outdegrees_polars
 from .predicates import filter_by_dict_polars
+from .reserved_columns import CHAIN_NODE_HOP
 
 
 def _semi(df, ids_df, df_col, id_col):
@@ -68,7 +69,56 @@ def _restore_edge_dtypes(edges, src, dst, restore):
     return edges.with_columns([pl.col(src).cast(sdt), pl.col(dst).cast(ddt)])
 
 
-def _exec(op: ASTObject, g: Plottable, prev_wf, target_wf, intermediate_universe=None) -> Plottable:
+# Auto-injected hop-distance label (#1741). pandas' chain asks the hop for node hop labels
+# whenever an edge op is variable-length or output-sliced (ast.py:621-625 needs_auto_labels) and
+# then gates node ALIASES by that distance (chain.py:456-501). Without the labels the polars chain
+# had no distance to gate on, so `MATCH (a)-[*1..2]-(b)` over-flagged a backtracked-to seed.
+#
+# The concrete column name is resolved once per chain against the user's node columns via
+# generate_safe_column_name (see _chain_traversal_polars), so a user column literally named
+# `__gfql_chain_node_hop__` can't be clobbered (would otherwise crash on the int/str compare in
+# the gate). This base string (declared in reserved_columns.py, the per-engine symbol registry)
+# is only the seed + the fallback for callers that don't resolve.
+_AUTO_NODE_HOP: str = CHAIN_NODE_HOP
+
+
+def _auto_node_hop_col(op: ASTObject, name: str = _AUTO_NODE_HOP) -> Optional[str]:
+    """The auto-label column for an edge op, or None when labels are unnecessary/unsupported."""
+    if not isinstance(op, ASTEdge):
+        return None
+    if op.min_hops is not None and op.min_hops > 1:
+        # min_hops>1 labels come from the layered backward walk, not yet ported -> asking for
+        # them would turn a currently-native chain into an NIE. Gap tracked on #1741/#1748.
+        return None
+    needs = (
+        (op.min_hops is not None and op.min_hops > 0)
+        or op.output_min_hops is not None
+        or op.output_max_hops is not None
+        or op.prune_to_endpoints
+    )
+    return name if needs else None
+
+
+def _alias_hop_bounds(op: ASTEdge) -> Tuple[int, Optional[int]]:
+    """pandas' (min_hop, max_hop) alias window for a node op preceded by edge `op` (chain.py:477-499)."""
+    min_hop = (
+        op.output_min_hops if op.output_min_hops is not None
+        else (op.min_hops if op.min_hops is not None else (op.hops if op.hops is not None else 1))
+    )
+    max_hop = (
+        op.output_max_hops if op.output_max_hops is not None
+        else (op.max_hops if op.max_hops is not None else op.hops)
+    )
+    if op.to_fixed_point:
+        max_hop = None
+    return min_hop, max_hop
+
+
+def _exec(op: ASTObject, g: Plottable, prev_wf: Optional[Any], target_wf: Optional[Any],
+          intermediate_universe: Optional[Any] = None,
+          auto_hop_col: str = _AUTO_NODE_HOP) -> Plottable:
+    # prev_wf/target_wf/intermediate_universe are polars wavefront frames (DataFrame|LazyFrame)
+    # or None; typed Optional[Any] to match this module's frame-annotation convention.
     import polars as pl
 
     node_col = g._node
@@ -100,7 +150,7 @@ def _exec(op: ASTObject, g: Plottable, prev_wf, target_wf, intermediate_universe
             source_node_query=op.source_node_query,
             destination_node_query=op.destination_node_query,
             edge_query=op.edge_query,
-            label_node_hops=op.label_node_hops,
+            label_node_hops=op.label_node_hops or _auto_node_hop_col(op, auto_hop_col),
             label_edge_hops=op.label_edge_hops,
             label_seeds=op.label_seeds,
             output_min_hops=op.output_min_hops,
@@ -243,7 +293,7 @@ def _combine_nodes(g, steps):
     return g._nodes.join(ids, on=node_col, how="semi")
 
 
-def _apply_node_names(out, g, steps):
+def _apply_node_names(out, g, steps, auto_hop_col: str = _AUTO_NODE_HOP):
     """Tag node aliases on the FINAL node frame (after endpoint materialization). A node carries
     the alias iff it matched the named step in the backward-PRUNED frame (dead-end matches
     excluded) AND, when followed by an edge step, participates in that edge's PRUNED edges.
@@ -259,6 +309,22 @@ def _apply_node_names(out, g, steps):
         if op._name not in colnames(g_step._nodes):
             continue
         named = g_step._nodes.filter(pl.col(op._name)).select(pl.col(node_col)).unique()
+        # #1741 hop-distance gate: a node named AFTER a variable-length edge carries the alias
+        # only if its hop distance lands inside that edge's [min_hop, max_hop] window (pandas
+        # chain.py:456-501). An unlabeled node (null distance) always fails — which is how the
+        # seed of an undirected `*1..2` walk loses the alias when the walk backtracks into it.
+        if idx > 0:
+            prev_op, _prev_step = step_list[idx - 1]
+            # The distance travels WITH the wavefront, so it lands on this node step's own frame.
+            if isinstance(prev_op, ASTEdge) and auto_hop_col in colnames(g_step._nodes):
+                min_hop, max_hop = _alias_hop_bounds(prev_op)
+                hop = pl.col(auto_hop_col)
+                in_window = hop.is_not_null() & (hop >= min_hop)
+                if max_hop is not None:
+                    in_window = in_window & (hop <= max_hop)
+                named = named.join(
+                    g_step._nodes.filter(in_window).select(pl.col(node_col)).unique(),
+                    on=node_col, how="semi")
         if idx + 1 < len(step_list):
             next_op, next_step = step_list[idx + 1]
             if isinstance(next_op, ASTEdge) and next_step._edges is not None and (is_lazy(next_step._edges) or next_step._edges.height > 0):
@@ -569,6 +635,24 @@ def _chain_traversal_polars(self: Plottable, ops, start_nodes: Optional[Any] = N
             "include_zero_hop_seed, prune_to_endpoints) require engine='pandas'."
         )
 
+    # issue #1748: forward/reverse min_hops>1 gets its node hop-labels from pandas' layered
+    # backward walk, which is not ported — so a node named AFTER such an edge cannot be hop-gated
+    # (_auto_node_hop_col returns None for min_hops>1), and its alias would include nodes OUTSIDE
+    # the [min_hop, max_hop] window. The node/edge SETS stay correct; only the projected alias is
+    # wrong. Decline that specific shape (honest NIE) instead of emitting silently-wrong rows —
+    # min_hops>1 chains WITHOUT a following node alias keep running. Adapted from the retired
+    # #1742 decline pattern (which did the same for undirected, now natively gated by #1741).
+    for _i in range(1, len(ops)):
+        _prev, _cur = ops[_i - 1], ops[_i]
+        if (isinstance(_prev, ASTEdge) and _prev.direction in ("forward", "reverse")
+                and _prev.min_hops is not None and _prev.min_hops > 1
+                and isinstance(_cur, ASTNode) and _cur._name is not None):
+            raise NotImplementedError(
+                "polars chain engine: a node alias after a forward/reverse variable-length edge "
+                "with min_hops>1 is not yet hop-gated (would tag nodes outside the hop window — "
+                "issue #1748); use engine='pandas' for this query"
+            )
+
     edge_ops = [op for op in ops if isinstance(op, ASTEdge)]
     # Undirected edges in multi-edge chains: NATIVE for single-hop (backward pass threads BOTH
     # endpoints — see override below) and fixed-length multi-hop (generic backward hop +
@@ -681,11 +765,17 @@ def _plain_edge(op):
     node_col, src, dst = g._node, g._source, g._destination
     assert node_col is not None and src is not None and dst is not None
 
+    # Resolve the #1741 auto hop-label column against the user's node columns ONCE, so it can't
+    # collide with a real column (a clash would clobber user data and crash the int/str gate).
+    # Same helper the endpoint block below uses for NORD/EORD.
+    from graphistry.compute.util import generate_safe_column_name as _gsafe
+    auto_hop_col = _gsafe(_AUTO_NODE_HOP, g._nodes, prefix="__gfql_", suffix="__")
+
     # Forward pass.
     g_stack: List[Plottable] = []
     for i, op in enumerate(ops):
         prev = start_nodes if i == 0 else g_stack[-1]._nodes
-        g_stack.append(_exec(op, g, prev, None))
+        g_stack.append(_exec(op, g, prev, None, auto_hop_col=auto_hop_col))
 
     # Backward pass.
     g_rev: List[Plottable] = []
@@ -705,7 +795,8 @@ def _plain_edge(op):
         # use_fast_backward (full g._nodes).
         _iu = g_step._nodes if (isinstance(op, ASTEdge) and not op.is_simple_single_hop()) else None
         g_step_full = g_step.nodes(g._nodes, g._node)
-        rev = _exec(op.reverse(), g_step_full, prev_wf, target_wf, intermediate_universe=_iu)
+        rev = _exec(op.reverse(), g_step_full, prev_wf, target_wf, intermediate_universe=_iu,
+                    auto_hop_col=auto_hop_col)
         # Undirected single-hop backward threading: the generic hop returns a ONE-SIDED
         # (TO-side) wavefront; pandas' fast backward branch (chain.py:1090-1098) threads BOTH
         # endpoints of surviving edges. One-sided drops an intermediate node reachable only as
@@ -748,7 +839,7 @@ def _plain_edge(op):
                     if prev_src is not None else None
                 )
                 g_sub = g.edges(g_step._edges, src, dst, edge=g._edge)
-                edge_steps.append((op, _exec(op, g_sub, prev_wf, None)))
+                edge_steps.append((op, _exec(op, g_sub, prev_wf, None, auto_hop_col=auto_hop_col)))
             else:
                 edge_steps.append((op, g_step))
 
@@ -776,7 +867,7 @@ def _plain_edge(op):
     extra = g_lz._nodes.join(missing, on=node_col, how="semi")
     final_nodes = pl.concat([final_nodes, extra], how="diagonal_relaxed").unique(
         subset=[node_col], maintain_order=True)
-    final_nodes = _apply_node_names(final_nodes, g_lz, steps_lz)
+    final_nodes = _apply_node_names(final_nodes, g_lz, steps_lz, auto_hop_col=auto_hop_col)
 
     final_nodes = final_nodes.sort(NORD).drop(NORD)
     final_edges = final_edges.sort(EORD).drop(EORD)

graphistry/compute/gfql/lazy/engine/polars/reserved_columns.py

@@ -0,0 +1,28 @@
+"""Reserved internal column-name bases for the native polars GFQL engine.
+
+Central, per-engine registry of the ``__gfql_*__`` base names the polars executor
+reserves for internal/temporary columns — declared in one place rather than inlined as
+string literals across the executor. Every base follows the repo-wide reserved pattern
+from :mod:`graphistry.compute.gfql.identifiers`
+(``INTERNAL_COLUMN_PREFIX``/``INTERNAL_COLUMN_SUFFIX`` → ``__gfql_*__``).
+
+Convention:
+- One module-level constant per reserved base, ``UPPER_SNAKE``, value ``__gfql_<role>__``.
+- At use sites, pass the base to
+  ``generate_safe_column_name(base, frame, prefix=INTERNAL_COLUMN_PREFIX, suffix=INTERNAL_COLUMN_SUFFIX)``
+  so a user column of the same name is never clobbered.
+- New polars-engine internal columns SHOULD be added here rather than inlined.
+
+Follow-up (tracked, non-blocking): migrate the remaining inline literals in
+``chain.py`` (``__gfql_norder__``/``__gfql_eorder__``) and ``hop_eager.py``
+(``__gfql_from__``/``__gfql_to__``/``__gfql_nid__``/``__gfql_eid__``/``__gfql_hop__``/
+``__gfql_node_hop__``) into this module so the whole engine shares one registry.
+"""
+from graphistry.compute.gfql.identifiers import (
+    INTERNAL_COLUMN_PREFIX,
+    INTERNAL_COLUMN_SUFFIX,
+)
+
+#: chain.py — auto-injected hop-distance label used to gate a node alias that follows a
+#: variable-length edge (#1741). Resolved against the user's node columns at each use.
+CHAIN_NODE_HOP: str = f"{INTERNAL_COLUMN_PREFIX}chain_node_hop{INTERNAL_COLUMN_SUFFIX}"

graphistry/tests/compute/gfql/coverage_baselines/ci-polars-py3.12.json

@@ -13,6 +13,7 @@
     "graphistry/compute/gfql/lazy/engine/polars/pattern_apply.py": 67.0,
     "graphistry/compute/gfql/lazy/engine/polars/predicates.py": 83.0,
     "graphistry/compute/gfql/lazy/engine/polars/projection.py": 65.0,
+    "graphistry/compute/gfql/lazy/engine/polars/reserved_columns.py": 90.0,
     "graphistry/compute/gfql/lazy/engine/polars/row_pipeline.py": 79.0,
     "graphistry/compute/gfql/lazy/engine/polars/search.py": 79.0
   }