diff --git a/CHANGELOG.md b/CHANGELOG.md index 63041b7291..1b386c5869 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -9,6 +9,7 @@ This project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.htm ### Added +- **GFQL Cypher `EXISTS { }` pattern-existence subqueries (openCypher-standard), native on all four engines**: `WHERE EXISTS { (n)-[:R]->() }` and `WHERE NOT EXISTS { (n)--() }` now parse and run — the declarative prune-isolated building blocks for the streamgl-viz filter pipeline. An `EXISTS` body reuses the existing pattern-predicate lowering wholesale (`semi_apply_mark` / `anti_semi_apply` row ops), so pandas/cuDF worked immediately; the polars engine gains NATIVE lowerings for the semi-apply family (correlated key sets computed by the polars chain executor's named-flag columns; order-preserving `is_in` joins) plus `rows(binding_ops=...)` for the single-entity row table — previously all honest-NIE. Aliases introduced inside the braces are existentially scoped (`EXISTS { (n)--(m) }` allowed with `m` unbound outside — bare pattern predicates keep the conservative guard), inline property maps work, and the one supported inner `WHERE` form is endpoint inequality — `EXISTS { (n)--(m) WHERE m <> n }`, the drop-self-loop prune-isolated flavor (pandas/cuDF filter the correlated bindings; polars excludes self-loop edges, which is exactly the `m <> n` witness). Both prune flavors are oracle-pinned in the conformance matrix on a self-loop discriminator graph, 4-engine parity-or-NIE. Honest declines with clear errors: `EXISTS` in RETURN/WITH projections, general inner `WHERE`, multi-pattern bodies, full `MATCH..RETURN` subquery bodies, multi-alias correlation on polars. - **GFQL polars execution config is Python-settable and live**: `set_cpu_streaming(bool)` and `set_gpu_executor('in-memory'|'streaming')` in `graphistry.compute.gfql.lazy` (plus the public `GPU_EXECUTORS` options and `GpuExecutor` type) set the CPU-streaming / GPU-executor knobs from Python. They resolve **Python override > environment variable > default**, read **live** per collect — previously these were env-only (`GFQL_POLARS_CPU_STREAMING` / `GFQL_POLARS_GPU_EXECUTOR`) and frozen at import, so neither a Python setting nor a post-import env change took effect. `None` resets a setter to env/default. - **GFQL engine conversion honors the `validate`/`warn` convention**: `Engine.df_to_engine(df, engine, *, validate=, warn=)` threads the repo-wide `validate` (`'strict'`/`'strict-fast'`/`'autofix'`; `True`→strict, `False`→autofix) + `warn` protocol into the pandas→polars and pandas→cuDF converters. On a mixed-type object column that Arrow/polars/cuDF cannot represent, `strict` raises (`NotImplementedError` for polars, `ArrowConversionError` for cuDF) and `autofix` coerces the column to string and warns — the same convention as `plot()`/`upload()`. Each engine keeps its established default (polars `strict` = parity-or-raise; cuDF `autofix` = its shipped best-effort coercion, now `warn`-suppressible). - **GFQL Cypher numeric functions + `toLower`/`toUpper`/`lower`/`upper` (openCypher/neo4j/GQL-standard)**: added the standard scalar functions `floor`, `ceil` (alias `ceiling`, per Cypher 25 GQL conformance and the GQL grammar's `CEIL|CEILING`), `round(x)` / `round(x, precision)`, and `toLower` / `toUpper` plus their GQL-conformance aliases `lower` / `upper` (ISO GQL §20.24 character-string functions; neo4j accepts both spellings) — the idiomatic case-insensitive compare `WHERE toLower(n.name) = 'bob'` — across the Cypher `WHERE`/`RETURN` expression surface. Evaluated natively on pandas/cuDF and polars (differential-parity tested). **`round` follows neo4j's documented tie-breaking** (standards-vetted against the neo4j manual): precision 0 (and the 1-arg form) rounds ties toward **positive infinity** (`round(-1.5)` → `-1.0`, `round(2.5)` → `3.0`), and precision > 0 rounds ties **away from zero** (HALF_UP: `round(-1.55, 1)` → `-1.6`) — not the numpy/polars half-to-even defaults, which give wrong answers on ties. The `round(x, precision, mode)` 3-arg form is not yet supported. Complements the already-supported `abs`/`sqrt`/`sign` and chained comparisons (`WHERE 1 < n.age < 65`). The `^` exponentiation operator is deferred: standards vetting settled it as **left-associative** (the openCypher TCK pins left, and neo4j's shipped Cypher 5/25 parser left-folds `Pow` — the manual's "right to left" row is a docs bug), but the current conformance corpus marks it reject-expected, so re-adding it is a coordinated corpus change. `LIKE`/`ILIKE`/`BETWEEN` remain intentionally unimplemented (verified absent from both Cypher and ISO GQL — GQL's only `LIKE` is the unrelated `CREATE GRAPH TYPE … LIKE g` DDL). diff --git a/docs/source/gfql/cypher.rst b/docs/source/gfql/cypher.rst index f5c8e9ead7..761bc89df4 100644 --- a/docs/source/gfql/cypher.rst +++ b/docs/source/gfql/cypher.rst @@ -271,6 +271,21 @@ WHERE Forms - Positive relationship-existence pattern predicates such as ``WHERE (n)-[:R]->()`` and variable-length existence checks such as ``WHERE (n)-[*]-()`` and ``WHERE (n)-[:R*2]->()``. +- ``EXISTS { }`` pattern-existence subqueries (openCypher-standard) in + WHERE position, e.g. ``WHERE EXISTS { (n)-[:R]->() }`` and ``WHERE NOT EXISTS + { (n)--() }`` — the declarative prune-isolated building blocks. Aliases + introduced inside the braces are existentially scoped (``EXISTS { (n)--(m) }`` + is fine even when ``m`` is unbound outside), inline property maps work + (``EXISTS { (n)-[:R {w: 1}]->() }``), and the one supported inner ``WHERE`` + form is endpoint inequality: ``EXISTS { (n)--(m) WHERE m <> n }`` — "has a + neighbor other than itself", i.e. the drop-self-loop prune-isolated flavor. + Runs natively on all four engines. Not yet supported (clear errors): + ``EXISTS`` in RETURN/WITH projections, general inner ``WHERE`` clauses, + multi-pattern bodies, full ``MATCH .. RETURN`` subquery bodies, and ``EXISTS`` + inside ``GRAPH { }`` pipelines. For prune-isolated in GRAPH STATE (nodes AND + edges back), use edge patterns instead: ``GRAPH { MATCH (a)-[e]-(b) }`` keeps + every edge-touching node with ALL its edges (self-loops included); add + ``WHERE a.id <> b.id`` for the drop-self-loop variant. - Pattern predicates can be combined with row predicates in the current boolean subset, including ``AND`` / ``OR`` / ``XOR`` and ``NOT`` forms. diff --git a/graphistry/compute/ast.py b/graphistry/compute/ast.py index 7f53199a62..df858e4b3b 100644 --- a/graphistry/compute/ast.py +++ b/graphistry/compute/ast.py @@ -1662,19 +1662,22 @@ def anti_semi_apply( *, binding_ops: List[Dict[str, Any]], join_aliases: Sequence[str], + neq: Optional[Sequence[str]] = None, ) -> ASTCall: """Filter active rows by removing rows matching a correlated pattern. ``binding_ops`` encodes the pattern to evaluate as bindings rows. ``join_aliases`` names shared aliases used as anti-join keys. + ``neq`` optionally names two pattern aliases whose bindings must DIFFER + (``EXISTS { (n)--(m) WHERE m <> n }`` — the viz drop-self prune flavor). """ - return ASTCall( - "anti_semi_apply", - { - "binding_ops": binding_ops, - "join_aliases": list(join_aliases), - }, - ) + params: Dict[str, Any] = { + "binding_ops": binding_ops, + "join_aliases": list(join_aliases), + } + if neq is not None: + params["neq"] = list(neq) + return ASTCall("anti_semi_apply", params) def semi_apply_mark( @@ -1682,6 +1685,7 @@ def semi_apply_mark( binding_ops: List[Dict[str, Any]], join_aliases: Sequence[str], out_col: str, + neq: Optional[Sequence[str]] = None, ) -> ASTCall: """Annotate active rows with a correlated pattern-existence boolean. @@ -1689,14 +1693,14 @@ def semi_apply_mark( ``join_aliases`` names shared aliases used as join keys. ``out_col`` receives a bool marker where True means the pattern matched. """ - return ASTCall( - "semi_apply_mark", - { - "binding_ops": binding_ops, - "join_aliases": list(join_aliases), - "out_col": out_col, - }, - ) + params: Dict[str, Any] = { + "binding_ops": binding_ops, + "join_aliases": list(join_aliases), + "out_col": out_col, + } + if neq is not None: + params["neq"] = list(neq) + return ASTCall("semi_apply_mark", params) def join_apply( diff --git a/graphistry/compute/gfql/call/support.py b/graphistry/compute/gfql/call/support.py index b77da77126..5637a7fd2d 100644 --- a/graphistry/compute/gfql/call/support.py +++ b/graphistry/compute/gfql/call/support.py @@ -83,6 +83,17 @@ def is_projection_items(v: object) -> bool: return True +def is_string_pair(v: object) -> bool: + """Exactly two non-empty strings — the semi-apply family's ``neq`` endpoint pair + (wave-1: a 1-item list crashed late with IndexError; a 3-item list silently + filtered on the first pair).""" + return ( + isinstance(v, list) + and len(v) == 2 + and all(isinstance(item, str) and item != "" for item in v) + ) + + def is_non_empty_list_of_strings(v: object) -> bool: return isinstance(v, list) and len(v) > 0 and all(isinstance(item, str) for item in v) diff --git a/graphistry/compute/gfql/call/validation.py b/graphistry/compute/gfql/call/validation.py index 3d4778ae3c..93d571c922 100644 --- a/graphistry/compute/gfql/call/validation.py +++ b/graphistry/compute/gfql/call/validation.py @@ -39,6 +39,7 @@ is_list_of_strings, is_list_or_dict, is_non_empty_list_of_strings, + is_string_pair, is_non_empty_string, is_non_negative_int_like, is_string, @@ -395,22 +396,24 @@ def _semi_apply_mark_added_node_cols(params: Dict[str, object]) -> Set[str]: ), 'anti_semi_apply': _safelist_entry( - {'binding_ops', 'join_aliases'}, + {'binding_ops', 'join_aliases', 'neq'}, required_params={'binding_ops', 'join_aliases'}, param_validators={ 'binding_ops': is_list_of_dicts, 'join_aliases': is_non_empty_list_of_strings, + 'neq': is_string_pair, }, description='Filter active rows by anti-semi joining against correlated binding rows', ), 'semi_apply_mark': _safelist_entry( - {'binding_ops', 'join_aliases', 'out_col'}, + {'binding_ops', 'join_aliases', 'out_col', 'neq'}, required_params={'binding_ops', 'join_aliases', 'out_col'}, param_validators={ 'binding_ops': is_list_of_dicts, 'join_aliases': is_non_empty_list_of_strings, 'out_col': is_non_empty_string, + 'neq': is_string_pair, }, description='Annotate active rows with correlated pattern-existence booleans', schema_effects=_schema_effects(adds_node_cols=_semi_apply_mark_added_node_cols), diff --git a/graphistry/compute/gfql/cypher/ast.py b/graphistry/compute/gfql/cypher/ast.py index b700e7b1e4..44af2046c8 100644 --- a/graphistry/compute/gfql/cypher/ast.py +++ b/graphistry/compute/gfql/cypher/ast.py @@ -147,6 +147,13 @@ class WherePatternPredicate: # anti-semi-join lowering instead of intersect-MATCH. Default False keeps # all existing single-positive / multi-positive callers unchanged. negated: bool = False + # viz-filter L1: "exists" when built from an EXISTS { } subquery — its pattern + # aliases are EXISTENTIALLY quantified (locals allowed); "bare" keeps the + # conservative no-new-aliases guard for bare pattern predicates. + pattern_origin: str = "bare" + # viz-filter L1: endpoint-inequality constraint from `EXISTS { (n)--(m) WHERE + # m <> n }` — the viz drop-self prune-isolated flavor. (alias_a, alias_b). + pattern_neq: Optional[Tuple[str, str]] = None WhereTerm = Union[WherePredicate, WherePatternPredicate] @@ -183,6 +190,10 @@ class BooleanExpr: atom_text: Optional[str] = None atom_span: Optional[SourceSpan] = None pattern: Optional[Tuple[PatternElement, ...]] = None + # viz-filter L1 (EXISTS { } leaves): existential-origin marker + optional + # endpoint-inequality; both flow into WherePatternPredicate at lift time. + pattern_origin: str = "bare" + pattern_neq: Optional[Tuple[str, str]] = None @dataclass(frozen=True) diff --git a/graphistry/compute/gfql/cypher/lowering.py b/graphistry/compute/gfql/cypher/lowering.py index 885339014e..c326eb7002 100644 --- a/graphistry/compute/gfql/cypher/lowering.py +++ b/graphistry/compute/gfql/cypher/lowering.py @@ -5763,7 +5763,9 @@ def _where_expr_tree_pattern_predicates(expr: BooleanExpr) -> List[WherePatternP line=cur.span.line, column=cur.span.column, ) - out.append(WherePatternPredicate(pattern=cur.pattern, span=cur.span, negated=False)) + out.append(WherePatternPredicate( + pattern=cur.pattern, span=cur.span, negated=False, + pattern_origin=cur.pattern_origin, pattern_neq=cur.pattern_neq)) continue if cur.left is not None: stack.append(cur.left) @@ -5799,7 +5801,10 @@ def _lower_pattern_predicate_to_row_marker( ) introduced_aliases = sorted(alias for alias in predicate_aliases if alias not in alias_targets) - if introduced_aliases: + if introduced_aliases and predicate.pattern_origin != "exists": + # EXISTS { } subquery aliases are EXISTENTIALLY quantified (locals) — the + # bindings table projects them away; bare pattern predicates keep the + # conservative guard (viz-filter L1). raise _unsupported( "Cypher WHERE pattern predicates cannot introduce new aliases in this phase", field="where", @@ -5830,6 +5835,7 @@ def _lower_pattern_predicate_to_row_marker( binding_ops=serialize_binding_ops(pattern_ops), join_aliases=shared_aliases, out_col=out_col, + neq=predicate.pattern_neq, ) @@ -5870,7 +5876,9 @@ def _rewrite(expr: BooleanExpr) -> BooleanExpr: marker_col = _fresh_marker_col(expr.span) marker_ops.append( _lower_pattern_predicate_to_row_marker( - WherePatternPredicate(pattern=expr.pattern, span=expr.span, negated=False), + WherePatternPredicate( + pattern=expr.pattern, span=expr.span, negated=False, + pattern_origin=expr.pattern_origin, pattern_neq=expr.pattern_neq), alias_targets=alias_targets, params=params, out_col=marker_col, @@ -5936,7 +5944,10 @@ def _lower_negated_pattern_predicate_to_row_filter( ) introduced_aliases = sorted(alias for alias in predicate_aliases if alias not in alias_targets) - if introduced_aliases: + if introduced_aliases and predicate.pattern_origin != "exists": + # EXISTS { } subquery aliases are EXISTENTIALLY quantified (locals) — the + # bindings table projects them away; bare pattern predicates keep the + # conservative guard (viz-filter L1). raise _unsupported( "Cypher WHERE pattern predicates cannot introduce new aliases in this phase", field="where", @@ -5966,6 +5977,7 @@ def _lower_negated_pattern_predicate_to_row_filter( return anti_semi_apply( binding_ops=serialize_binding_ops(pattern_ops), join_aliases=shared_aliases, + neq=predicate.pattern_neq, ) diff --git a/graphistry/compute/gfql/cypher/parser.py b/graphistry/compute/gfql/cypher/parser.py index fe41df3a0a..e692fd5d40 100644 --- a/graphistry/compute/gfql/cypher/parser.py +++ b/graphistry/compute/gfql/cypher/parser.py @@ -4,7 +4,7 @@ from dataclasses import dataclass, replace from functools import lru_cache import re -from typing import Any, Callable, List, Literal, Optional, Protocol, Sequence, Tuple, Type, Union, cast +from typing import Any, Callable, List, Literal, NoReturn, Optional, Protocol, Sequence, Tuple, Type, Union, cast from graphistry.compute.exceptions import ErrorCode, GFQLSyntaxError, GFQLValidationError from graphistry.compute.gfql.cypher.ast import ( @@ -265,6 +265,7 @@ | list_comprehension | list_literal | map_literal + | EXISTS_SUBQUERY -> exists_subquery_atom | WHERE_PATTERN -> pattern_atom | "(" expr ")" -> grouped_expr @@ -358,6 +359,7 @@ NUMBER: /[+-]?(?:0[xX][0-9A-Fa-f]+|0[oO][0-7]+|(?:\d+\.\d+(?:[eE][+-]?\d+)?|\.\d+(?:[eE][+-]?\d+)?|\d+(?:[eE][+-]?\d+)?))/ INT: /[0-9]+/ STRING : /'(?:\\.|[^'\\])*'|"(?:\\.|[^"\\])*"/ +EXISTS_SUBQUERY.3: /(?i:EXISTS)(?:\s|\/\*(?:[^*]|\*(?!\/))*\*\/)*\{(?:[^{}]|\{[^{}]*\})*\}/ WHERE_PATTERN: /\([^)\n]*\)\s*(?:<--|-->|--|<-\[[^\]\n]*\]-|-\[[^\]\n]*\]->|-\[[^\]\n]*\]-)\s*\([^)\n]*\)(?:\s*(?:<--|-->|--|<-\[[^\]\n]*\]-|-\[[^\]\n]*\]->|-\[[^\]\n]*\]-)\s*\([^)\n]*\))*?(?:\s+AND\s+\([^)\n]*\)\s*(?:<--|-->|--|<-\[[^\]\n]*\]-|-\[[^\]\n]*\]->|-\[[^\]\n]*\]-)\s*\([^)\n]*\)(?:\s*(?:<--|-->|--|<-\[[^\]\n]*\]-|-\[[^\]\n]*\]->|-\[[^\]\n]*\]-)\s*\([^)\n]*\))*)*/ REL_FWD_SIMPLE: /-->/ REL_REV_SIMPLE: /<--/ @@ -501,10 +503,14 @@ def _build_where_with_pattern_lift( pattern_preds: List[WherePatternPredicate] = [] for leaf in pattern_leaves: assert leaf.pattern is not None, "pattern_atom invariant: pattern payload always set" - pattern_preds.append(WherePatternPredicate(pattern=leaf.pattern, span=leaf.span, negated=False)) + pattern_preds.append(WherePatternPredicate( + pattern=leaf.pattern, span=leaf.span, negated=False, + pattern_origin=leaf.pattern_origin, pattern_neq=leaf.pattern_neq)) for leaf in negated_pattern_leaves: assert leaf.pattern is not None, "pattern_atom invariant: pattern payload always set" - pattern_preds.append(WherePatternPredicate(pattern=leaf.pattern, span=leaf.span, negated=True)) + pattern_preds.append(WherePatternPredicate( + pattern=leaf.pattern, span=leaf.span, negated=True, + pattern_origin=leaf.pattern_origin, pattern_neq=leaf.pattern_neq)) new_expr_tree = _rebuild_and_tree(other_conjuncts) if new_expr_tree is None: return WhereClause(predicates=tuple(pattern_preds), expr_tree=None, span=span) @@ -1188,6 +1194,83 @@ def pattern_atom(self, meta: Any, items: Sequence[Any]) -> BooleanExpr: assert tree is not None # gated above by `if not pattern_item_texts` return tree + def exists_subquery_atom(self, meta: Any, items: Sequence[Any]) -> BooleanExpr: + """``EXISTS { }`` — openCypher pattern-existence subquery, WHERE + position. Produces the SAME ``BooleanExpr(op="pattern")`` leaf as a bare + pattern predicate, so the whole existing lift/marker/semi-apply lowering + applies unchanged (``NOT EXISTS`` composes via the NOT tier -> + anti_semi_apply). v1 subset: a single fixed-shape pattern body (inline + ``{prop: val}`` maps fine); inner WHERE clauses, multi-pattern bodies, and + full MATCH..RETURN subquery bodies decline with a clear error.""" + if len(items) != 1: + raise _to_syntax_error("Invalid EXISTS subquery", line=meta.line, column=meta.column) + span = _span_from_meta(meta) + text = str(items[0]) + raw_inner = text[text.index("{") + 1:text.rindex("}")] + inner = raw_inner.strip() + # keyword/comma scans run on MASKED text (quoted/backticked/commented + # segments blanked; length-preserving so indices align with the real + # text) — a property string like {name: 'WHERE it hurts'} must not + # falsely decline (wave-1 I4). + lstrip_off = len(raw_inner) - len(raw_inner.lstrip()) + masked_inner = _mask_quoted_backticked_and_commented_for_scan(raw_inner)[ + lstrip_off:lstrip_off + len(inner)] + + def _decline(what: str) -> NoReturn: + raise GFQLValidationError( + ErrorCode.E108, + f"EXISTS {{ ... }} {what} is outside the currently supported GFQL Cypher subset", + field="where", + value=inner[:80], + suggestion="Use a single fixed-length pattern inside EXISTS { }, e.g. EXISTS { (n)--() }.", + line=span.line, + column=span.column, + language="cypher", + ) + + if re.search(r"\bMATCH\b|\bRETURN\b", masked_inner, re.IGNORECASE): + _decline("with a full subquery body") + # viz-filter L1 drop-self flavor: EXISTS { (n)--(m) WHERE m <> n } — the ONE + # supported inner-WHERE form (endpoint inequality); anything else declines. + neq: Optional[Tuple[str, str]] = None + m_where = re.search(r"\bWHERE\b", masked_inner, flags=re.IGNORECASE) + if m_where is not None: + cond = inner[m_where.end():].strip() + m_neq = re.fullmatch(r"([A-Za-z_]\w*)\s*(?:<>|!=)\s*([A-Za-z_]\w*)", cond) + if m_neq is None: + _decline("with an inner WHERE clause (only `a <> b` endpoint inequality is supported)") + neq = (m_neq.group(1), m_neq.group(2)) + inner = inner[:m_where.start()].strip() + masked_inner = masked_inner[:m_where.start()].rstrip() + depth = 0 + for ch in masked_inner: + if ch in "([{": + depth += 1 + elif ch in ")]}": + depth -= 1 + elif ch == "," and depth == 0: + _decline("with a multi-pattern body") + pattern_item_texts = [m.group(0).strip() for m in _WHERE_PATTERN_ITEM_RE.finditer(inner)] + if len(pattern_item_texts) != 1 or pattern_item_texts[0] != inner: + _decline("body (expected a single relationship pattern)") + pattern_pred = self._parse_single_where_pattern_predicate_text(inner, span) + if neq is not None: + endpoint_names = { + el.variable # both PatternElement members declare variable + for el in (pattern_pred.pattern[0], pattern_pred.pattern[-1]) + } + if set(neq) != endpoint_names or neq[0] == neq[1]: + _decline("inner WHERE must compare the pattern's two endpoint aliases") + return BooleanExpr( + op="pattern", + span=span, + atom_text=inner, + atom_span=span, + pattern=pattern_pred.pattern, + pattern_origin="exists", + pattern_neq=neq, + ) + def _wrap_as_boolean_atom(self, operand: Any, enclosing_meta: Any) -> BooleanExpr: """Coerce a parsed expression operand into a ``BooleanExpr`` atom. @@ -1973,14 +2056,32 @@ def graph_query_standalone( r""" not\s*\(\s*\(\s*[^)\n]*\)\s* # not(() (?:<--|-->|--|<-\[[^\]\n]*\]-|-\[[^\]\n]*\]->|-\[[^\]\n]*\]-) - | - not\s+exists\s*\{ # not exists { - | - exists\s*\{ # exists { """, re.IGNORECASE | re.VERBOSE, ) +# EXISTS { } inside a GRAPH { } pipeline: the graph-state compiler does not wire +# row_pre_filters, so the EXISTS filter was SILENTLY DROPPED (dgx-repro'd: all +# nodes returned) — fail fast instead. ANCHORED: the grammar only admits GRAPH +# constructors at the very start of the query, so `n.graph`/:Graph labels in a +# plain MATCH cannot false-positive (wave-2 W2-1). Post-USE row stages still +# decline (honest over-reject). +_EXISTS_IN_GRAPH_PIPELINE_RE = re.compile( + r"\A\s*GRAPH\b", re.IGNORECASE +) +_EXISTS_ANYWHERE_RE = re.compile(r"\bexists\s*(?:/\*[\s\S]*?\*/\s*)*\{", re.IGNORECASE) + +# EXISTS { } in a PROJECTION (RETURN/WITH segment, i.e. before the next clause +# keyword): unsupported — marker columns are only wired into WHERE. WHERE-position +# EXISTS parses via the EXISTS_SUBQUERY token instead (viz-filter L1). +_EXISTS_IN_PROJECTION_RE = re.compile( + r"\b(?:RETURN|WITH)\b(?:(?!\b(?:MATCH|WHERE|UNWIND|UNION|CALL)\b).)*?" + r"\bnot\s+exists\s*(?:/\*[\s\S]*?\*/\s*)*\{" + r"|\b(?:RETURN|WITH)\b(?:(?!\b(?:MATCH|WHERE|UNWIND|UNION|CALL)\b).)*?" + r"\bexists\s*(?:/\*[\s\S]*?\*/\s*)*\{", + re.IGNORECASE | re.DOTALL, +) + def _mask_quoted_backticked_and_commented_for_scan(expr: str) -> str: """Replace quoted/backticked/commented segments with spaces before regex scans.""" @@ -2033,13 +2134,31 @@ def _mask_quoted_backticked_and_commented_for_scan(expr: str) -> str: def _check_unsupported_syntax_patterns(query: str) -> None: """Detect known-but-unsupported Cypher syntax and raise a clear error.""" - if _PATTERN_EXISTENCE_RE.search(_mask_quoted_backticked_and_commented_for_scan(query)): + masked = _mask_quoted_backticked_and_commented_for_scan(query) + if _PATTERN_EXISTENCE_RE.search(masked): raise _to_unsupported( - "Pattern existence expressions (e.g., not((a)-[:R]-(b)) or exists { ... }) " + "Pattern existence expressions of the form not((a)-[:R]-(b)) " "are not yet supported in the local Cypher compiler", field="expression", value="pattern_existence", ) + if _EXISTS_IN_PROJECTION_RE.search(masked): + raise _to_unsupported( + "Pattern existence expressions (exists { ... }) are not yet supported in " + "RETURN/WITH projections in the local Cypher compiler (WHERE position is supported)", + field="expression", + value="pattern_existence", + ) + if _EXISTS_IN_GRAPH_PIPELINE_RE.search(masked) and _EXISTS_ANYWHERE_RE.search(masked): + raise _to_unsupported( + "Pattern existence expressions (exists { ... }) are not yet supported inside " + "GRAPH { } pipelines (the filter would be silently dropped). For graph-state " + "prune-isolated use edge patterns: GRAPH { MATCH (a)-[e]-(b) } keeps every " + "edge-touching node with ALL its edges; add WHERE a.id <> b.id for the " + "drop-self-loop variant", + field="expression", + value="pattern_existence", + ) def parse_cypher(query: str) -> Union[CypherQuery, CypherUnionQuery, CypherGraphQuery]: diff --git a/graphistry/compute/gfql/lazy/engine/polars/chain.py b/graphistry/compute/gfql/lazy/engine/polars/chain.py index 10d15c0d21..ed38b68c39 100644 --- a/graphistry/compute/gfql/lazy/engine/polars/chain.py +++ b/graphistry/compute/gfql/lazy/engine/polars/chain.py @@ -351,12 +351,39 @@ def _run_calls_polars(g_cur, calls, start_nodes, base_graph, middle): def _try_native_row_op(g_cur, op): """Run a row-pipeline call natively on polars, or return None to defer (NIE).""" from graphistry.Engine import Engine - from .row_pipeline import select_polars, with_columns_polars, order_by_polars, group_by_polars, unwind_polars, where_rows_polars + from .row_pipeline import ( + select_polars, with_columns_polars, order_by_polars, group_by_polars, + unwind_polars, where_rows_polars, + ) + from .pattern_apply import ( + rows_binding_ops_polars, semi_apply_mark_polars, anti_semi_apply_polars, + ) fn = getattr(op, "function", None) if _call_native_on_polars(op): # frame ops (rows/limit/skip/distinct/drop_cols) — engine-polymorphic return op.execute(g=g_cur, prev_node_wavefront=None, target_wave_front=None, engine=Engine.POLARS) + # correlated pattern-existence family (EXISTS { } / pattern predicates): native + # via chain_polars-computed key sets; unsupported shapes return None -> honest NIE. + if ( + fn == "rows" + and op.params.get("binding_ops") is not None + and op.params.get("alias_endpoints") is None + and op.params.get("source") is None + ): + return rows_binding_ops_polars(g_cur, op.params["binding_ops"]) + if fn == "semi_apply_mark": + # required params are safelist-validated — direct indexing (an or-default + # here could only mask an unvalidated call); neq is the optional one. + return semi_apply_mark_polars( + g_cur, op.params["binding_ops"], op.params["join_aliases"], + op.params["out_col"], neq=op.params.get("neq"), + ) + if fn == "anti_semi_apply": + return anti_semi_apply_polars( + g_cur, op.params["binding_ops"], op.params["join_aliases"], + neq=op.params.get("neq"), + ) if fn in ("select", "return_"): return select_polars(g_cur, op.params.get("items", [])) if fn == "with_": diff --git a/graphistry/compute/gfql/lazy/engine/polars/pattern_apply.py b/graphistry/compute/gfql/lazy/engine/polars/pattern_apply.py new file mode 100644 index 0000000000..ed23aa27bb --- /dev/null +++ b/graphistry/compute/gfql/lazy/engine/polars/pattern_apply.py @@ -0,0 +1,236 @@ +"""Native polars lowering for the correlated pattern-existence row-op family +(``EXISTS { }`` subqueries / bare pattern predicates): ``rows(binding_ops=...)``, +``semi_apply_mark``, ``anti_semi_apply``. Split from row_pipeline.py (which is the +expression/projection lowering core) per the degrees.py module-per-family precedent. + +NO-CHEATING contract as everywhere in this engine: unsupported shapes return None +and the boundary lane raises an honest NotImplementedError — never a pandas bridge. +""" +from __future__ import annotations + +from typing import TYPE_CHECKING, Dict, List, Optional, Sequence +from graphistry.utils.json import JSONVal + +from graphistry.Plottable import Plottable + +if TYPE_CHECKING: + import polars as pl + from graphistry.compute.ast import ASTObject + +from .row_pipeline import _active_table, _rewrap + + +def _rows_base_graph(g: Plottable) -> Plottable: + """The ORIGINAL graph threaded by the boundary lane (`_gfql_rows_base_graph` + setattr convention, mirrored from the pandas lane at compute/chain.py) — the one + sanctioned dynamic read, contained here; falls back to the current graph.""" + base = getattr(g, "_gfql_rows_base_graph", None) + return base if base is not None else g + + +def _binding_ast_ops(binding_ops: Sequence[Dict[str, JSONVal]]) -> Optional[List["ASTObject"]]: + """Deserialize the semi-apply family's serialized binding ops; None on failure.""" + from graphistry.compute.ast import from_json as ast_from_json + try: + return [ast_from_json(op_json, validate=False) for op_json in binding_ops] + except Exception: + return None + + +def rows_binding_ops_polars(g: Plottable, binding_ops: Sequence[Dict[str, JSONVal]]) -> Optional[Plottable]: + """Native ``rows(binding_ops=[...])`` for the SINGLE named-Node case — the shape the + boundary rewrite emits for a one-entity MATCH (the EXISTS pipeline's left table). + Mirrors the pandas ``_gfql_node_alias_lookup_frame`` layout exactly: + ``[node_id, alias, alias.node_id, alias....]`` in source column order. + Anything else (multi-op, edge ops, unnamed, node query=) declines (None -> NIE).""" + import polars as pl + from graphistry.compute.ast import ASTNode as _ASTNode + from .predicates import filter_by_dict_polars + ops = _binding_ast_ops(binding_ops) + if ops is None or len(ops) != 1 or not isinstance(ops[0], _ASTNode): + return None + op = ops[0] + alias = op._name + if not isinstance(alias, str) or op.query: + return None + base_graph = _rows_base_graph(g) + node_id = base_graph._node or "id" # Plottable._node: Optional[str] + nodes = base_graph._nodes + if nodes is None or node_id not in nodes.columns: + return None + start_nodes = getattr(g, "_gfql_start_nodes", None) + if start_nodes is not None: + # start-nodes constrain the scan like the pandas prev_node_wavefront does. + # Normalize to EAGER: an eager.join(LazyFrame) raises and would silently + # decline the whole wavefront-constrained case (wave-2 W2-3). + sn = start_nodes.collect() if isinstance(start_nodes, pl.LazyFrame) else start_nodes + try: + nodes = nodes.join(sn.select(node_id).unique(), on=node_id, how="semi") + except Exception: + return None + try: + matched = filter_by_dict_polars(nodes, getattr(op, "filter_dict", None)) + except NotImplementedError: + raise + except Exception: + return None + if matched is None: + return None + if alias == node_id: + # pandas' named-op flag column OVERWRITES the id column in this corner — + # neither engine has sane semantics; decline honestly (wave-1 I1). + return None + # Mirror the pandas twin's quirks (wave-1 I3): ASTNode.execute leaks the + # named-op FLAG column into its source frame, so pandas emits a fabricated + # `alias.alias = True` column (shadowing any real property of that name) — + # emit the same. (Column ORDER differs slightly; the parity sig sorts columns.) + prop_cols = [c for c in matched.columns if c != node_id and c != alias] + exprs = [ + pl.col(node_id), + pl.col(node_id).alias(alias), + pl.col(node_id).alias(f"{alias}.{node_id}"), + pl.lit(True).alias(f"{alias}.{alias}"), + ] + exprs.extend(pl.col(c).alias(f"{alias}.{c}") for c in prop_cols) + lookup = matched.select(exprs) + return _rewrap(g, lookup) + + +def _pattern_alias_keys_polars( + g: Plottable, binding_ops: Sequence[Dict[str, JSONVal]], alias: str, neq: Optional[Sequence[str]] = None +) -> Optional["pl.DataFrame"]: + """Ids of ``alias``'s nodes that participate in the (single) pattern — the semi-apply + right side — computed by running the binding chain NATIVELY via ``chain_polars`` on the + base graph and reading the named-op flag column (the chain's backward prune makes the + flag exactly 'a full pattern match exists through this node'). v1: [Node, Edge, Node] + single-hop shapes only; the join alias must be a NAMED endpoint. None declines (NIE).""" + import polars as pl + from graphistry.compute.ast import ASTNode as _ASTNode, ASTEdge as _ASTEdge + ops = _binding_ast_ops(binding_ops) + if ( + ops is None or len(ops) != 3 + or not isinstance(ops[0], _ASTNode) or not isinstance(ops[1], _ASTEdge) + or not isinstance(ops[2], _ASTNode) + ): + return None + n0, edge_op, n2 = ops[0], ops[1], ops[2] # locals: mypy narrows these, not ops[i] + if edge_op.hops not in (None, 1) or edge_op.to_fixed_point: + return None + if edge_op.min_hops not in (None, 1) or edge_op.max_hops not in (None, 1): + return None + if alias not in (n0._name, n2._name): + return None + if n0.query or n2.query: + return None + base_graph = _rows_base_graph(g) + node_id = base_graph._node or "id" # Plottable._node: Optional[str] + if neq: + # EXISTS { (n)--(m) WHERE m <> n } — for the single-edge shape, endpoint + # inequality == "witnessed by a NON-self-loop edge": pre-drop self loops and + # reuse the same key computation. Any other neq spelling declines. + endpoint_names = {n0._name, n2._name} + if set(neq) != endpoint_names or len(set(neq)) != 2: + return None + edges = base_graph._edges + src, dst = base_graph._source, base_graph._destination + if ( + edges is None + or not isinstance(src, str) or src not in edges.columns + or not isinstance(dst, str) or dst not in edges.columns + ): + return None + base_graph = base_graph.edges(edges.filter(pl.col(src) != pl.col(dst))) + from .chain import chain_polars # local: chain imports this module at call time + from graphistry.compute.exceptions import GFQLValidationError + try: + out = chain_polars(base_graph, list(ops)) + except NotImplementedError: + return None + except GFQLValidationError: + # e.g. chain's duplicate-alias guard on EXISTS { (n)--(n) } — decline to + # honest NIE rather than a non-NIE crash (wave-1 T4); pandas may answer. + return None + out_nodes = out._nodes + if out_nodes is None or alias not in out_nodes.columns or node_id not in out_nodes.columns: + # empty match -> no keys (an empty id frame, NOT a decline) + if out_nodes is not None and node_id in out_nodes.columns: + return out_nodes.select(node_id).head(0) + return None + return ( + out_nodes.filter(pl.col(alias).fill_null(False)) + .select(node_id) + .unique() + ) + + +def _semi_apply_join_col(left: "pl.DataFrame", alias: str, node_id: str) -> Optional[str]: + """Mirror the pandas join-column choice: prefer ``alias.node_id``, else bare ``alias``.""" + for cand in (f"{alias}.{node_id}", alias): + if cand in left.columns: + return cand + return None + + +def semi_apply_mark_polars( + g: Plottable, binding_ops: Sequence[Dict[str, JSONVal]], join_aliases: Sequence[str], out_col: str, + neq: Optional[Sequence[str]] = None, +) -> Optional[Plottable]: + """Native polars ``semi_apply_mark``: boolean existence marker per active row. + v1: exactly ONE join alias (two-alias correlation needs PAIR bindings, which the + flag-column route cannot express — declines to honest NIE).""" + import polars as pl + if len(join_aliases) != 1 or not isinstance(out_col, str) or not out_col: + return None + alias = join_aliases[0] + left = _active_table(g) + if left is None: + return None + base_graph = _rows_base_graph(g) + node_id = base_graph._node or "id" # Plottable._node: Optional[str] + join_col = _semi_apply_join_col(left, alias, node_id) + if join_col is None: + return None + keys = _pattern_alias_keys_polars(g, binding_ops, alias, neq) + if keys is None: + return None + key_series = keys.get_column(keys.columns[0]) + if key_series.null_count() > 0: + # pandas merge matches NaN==NaN join keys; is_in does not — decline the + # pathological null-id case rather than silently diverge (wave-1 E1). + return None + # is_in (not a join): row ORDER preserved trivially; a null LEFT key marks + # False, same as the pandas merge-no-match path (null -> fill_null(False)). + marked = left.with_columns( + pl.col(join_col).is_in(key_series).fill_null(False).alias(out_col) + ) + return _rewrap(g, marked) + + +def anti_semi_apply_polars( + g: Plottable, binding_ops: Sequence[Dict[str, JSONVal]], join_aliases: Sequence[str], + neq: Optional[Sequence[str]] = None, +) -> Optional[Plottable]: + """Native polars ``anti_semi_apply``: drop active rows whose alias node participates + in the pattern (NOT EXISTS / negated pattern predicate). Same v1 bounds as the mark.""" + if len(join_aliases) != 1: + return None + alias = join_aliases[0] + left = _active_table(g) + if left is None: + return None + base_graph = _rows_base_graph(g) + node_id = base_graph._node or "id" # Plottable._node: Optional[str] + join_col = _semi_apply_join_col(left, alias, node_id) + if join_col is None: + return None + keys = _pattern_alias_keys_polars(g, binding_ops, alias, neq) + if keys is None: + return None + key_series = keys.get_column(keys.columns[0]) + import polars as pl + if key_series.null_count() > 0: + return None # see semi_apply_mark_polars: NaN==NaN merge semantics (wave-1 E1) + # filter (not an anti-join): order preserved; a null LEFT key row SURVIVES like + # the pandas merge-no-match path (is_in null -> fill_null(False) -> not_ -> True). + kept = left.filter(pl.col(join_col).is_in(key_series).fill_null(False).not_()) + return _rewrap(g, kept) diff --git a/graphistry/compute/gfql/lazy/engine/polars/row_pipeline.py b/graphistry/compute/gfql/lazy/engine/polars/row_pipeline.py index 31927f8aa7..34e6dfa72c 100644 --- a/graphistry/compute/gfql/lazy/engine/polars/row_pipeline.py +++ b/graphistry/compute/gfql/lazy/engine/polars/row_pipeline.py @@ -790,3 +790,4 @@ def unwind_polars(g: Plottable, expr: str, as_: str = "value") -> Optional[Plott values = [it.value for it in node.items if isinstance(it, Literal)] rhs = pl.DataFrame({as_: values}) return _rewrap(g, table.join(rhs, how="cross")) + diff --git a/graphistry/compute/gfql/row/pipeline.py b/graphistry/compute/gfql/row/pipeline.py index 04d9a22d04..9f80da31b5 100644 --- a/graphistry/compute/gfql/row/pipeline.py +++ b/graphistry/compute/gfql/row/pipeline.py @@ -4149,10 +4149,33 @@ def _gfql_apply_right_df(self, binding_ops: List[Dict[str, Any]]) -> Tuple[Any, right_rows = _RowPipelineAdapter(cast("Plottable", base_graph))._gfql_binding_ops_row_table(binding_ops) return base_graph, right_rows._nodes + @staticmethod + def _gfql_apply_neq_filter(right_df: Any, neq: Optional[List[str]], node_id_col: str) -> Any: + """``EXISTS { ... WHERE a <> b }``: keep binding rows whose two alias node ids + DIFFER (the viz drop-self prune-isolated flavor). Column resolution mirrors the + join-column candidates (``alias.node_id`` then bare ``alias``).""" + if not neq or right_df is None or len(right_df) == 0: + return right_df + cols: List[str] = [] + for alias in neq: + cand = next((c for c in (f"{alias}.{node_id_col}", alias) if c in right_df.columns), None) + if cand is None: + raise GFQLValidationError( + ErrorCode.E108, + "EXISTS inner WHERE inequality references an alias missing from the pattern bindings", + field="where", + value=list(neq), + suggestion="Compare the pattern's endpoint aliases, e.g. EXISTS { (n)--(m) WHERE m <> n }.", + language="cypher", + ) + cols.append(cand) + return right_df.loc[right_df[cols[0]] != right_df[cols[1]]] + def anti_semi_apply( self, binding_ops: List[Dict[str, Any]], join_aliases: List[str], + neq: Optional[List[str]] = None, ) -> "Plottable": """Row anti-semi-join against rows produced by ``binding_ops``.""" self._gfql_validate_apply_args("anti_semi_apply", binding_ops, join_aliases) @@ -4169,6 +4192,10 @@ def anti_semi_apply( if not isinstance(node_id_col, str) or node_id_col == "": node_id_col = "id" + right_df = self._gfql_apply_neq_filter(right_df, neq, node_id_col) + if right_df is None or len(right_df) == 0: + return self._gfql_row_table(left_df.copy()) + join_cols, missing_aliases = self._gfql_shared_join_cols(left_df, right_df, join_aliases, node_id_col) if not join_cols: self._gfql_raise_missing_join_cols( @@ -4192,6 +4219,7 @@ def semi_apply_mark( binding_ops: List[Dict[str, Any]], join_aliases: List[str], out_col: str, + neq: Optional[List[str]] = None, ) -> "Plottable": """Annotate each active row with correlated pattern-match existence.""" self._gfql_validate_apply_args("semi_apply_mark", binding_ops, join_aliases, out_col=out_col) @@ -4216,6 +4244,12 @@ def semi_apply_mark( if not isinstance(node_id_col, str) or node_id_col == "": node_id_col = "id" + right_df = self._gfql_apply_neq_filter(right_df, neq, node_id_col) + if right_df is None or len(right_df) == 0: + out_df = left_df.copy() + out_df[out_col] = False + return self._gfql_row_table(out_df) + join_cols, missing_aliases = self._gfql_shared_join_cols(left_df, right_df, join_aliases, node_id_col) if not join_cols: self._gfql_raise_missing_join_cols( diff --git a/graphistry/tests/compute/gfql/coverage_baselines/ci-polars-py3.12.json b/graphistry/tests/compute/gfql/coverage_baselines/ci-polars-py3.12.json index 195fae3ce3..6ba869bf0e 100644 --- a/graphistry/tests/compute/gfql/coverage_baselines/ci-polars-py3.12.json +++ b/graphistry/tests/compute/gfql/coverage_baselines/ci-polars-py3.12.json @@ -1,5 +1,5 @@ { - "_note": "Per-file line-coverage floors for the native POLARS engine (profile gfql-polars, bin/test-polars.sh). COMPLETE list required (audit fails un-baselined target files). Floors set conservatively (~8% below dgx-measured; measured: chain 94.3 row_pipeline 87.9 predicates 91 projection 73.2 hop 95.1 hop_eager 98.3 degrees 90.7 dtypes 100 lazy/__init__ 98.6). Raise as coverage grows.", + "_note": "Per-file line-coverage floors for the native POLARS engine (profile gfql-polars, bin/test-polars.sh). COMPLETE list required (audit fails un-baselined target files). Floors set conservatively (~8% below dgx-measured; measured: chain 94.3 row_pipeline 87.9 predicates 91 projection 73.2 hop 95.1 hop_eager 98.3 degrees 90.7 dtypes 100 lazy/__init__ 98.6). pattern_apply 75.4 (2026-07-05, module split from row_pipeline). Raise as coverage grows.", "files": { "graphistry/compute/gfql/lazy/__init__.py": 90.0, "graphistry/compute/gfql/lazy/engine/__init__.py": 90.0, @@ -9,6 +9,7 @@ "graphistry/compute/gfql/lazy/engine/polars/dtypes.py": 92.0, "graphistry/compute/gfql/lazy/engine/polars/hop.py": 87.0, "graphistry/compute/gfql/lazy/engine/polars/hop_eager.py": 90.0, + "graphistry/compute/gfql/lazy/engine/polars/pattern_apply.py": 67.0, "graphistry/compute/gfql/lazy/engine/polars/predicates.py": 83.0, "graphistry/compute/gfql/lazy/engine/polars/projection.py": 65.0, "graphistry/compute/gfql/lazy/engine/polars/row_pipeline.py": 79.0 diff --git a/graphistry/tests/compute/gfql/cypher/test_grammar_invariants.py b/graphistry/tests/compute/gfql/cypher/test_grammar_invariants.py index 0117f5455a..2974f8c512 100644 --- a/graphistry/tests/compute/gfql/cypher/test_grammar_invariants.py +++ b/graphistry/tests/compute/gfql/cypher/test_grammar_invariants.py @@ -76,6 +76,8 @@ # WHERE: generic expression forms (stay on expr_tree) "MATCH (n) WHERE n.a > 3 OR n.b = 1 RETURN n", "MATCH (n) WHERE n.a =~ 'x' OR n.b =~ 'y' RETURN n", + "MATCH (n) WHERE EXISTS { (n)-[:R]->() } RETURN n", + "MATCH (n) WHERE NOT EXISTS { (n)--(m) } RETURN n", "MATCH (n) WHERE n.x = 1 XOR n.y = 2 RETURN n", "MATCH (n) WHERE NOT n.deleted = true RETURN n", "MATCH (n) WHERE (n.x = 1 AND n.y = 2) OR n.z = 3 RETURN n", diff --git a/graphistry/tests/compute/gfql/cypher/test_lowering.py b/graphistry/tests/compute/gfql/cypher/test_lowering.py index ecc985e4ca..79936509f4 100644 --- a/graphistry/tests/compute/gfql/cypher/test_lowering.py +++ b/graphistry/tests/compute/gfql/cypher/test_lowering.py @@ -1967,6 +1967,75 @@ def test_lower_match_query_emits_row_marker_for_single_positive_where_pattern() assert [op.get("type") for op in binding_ops] == ["Node", "Edge", "Node"] +def test_lower_match_query_exists_subquery_emits_same_marker_as_bare_pattern() -> None: + """viz-filter L1: WHERE EXISTS { } lowers to the IDENTICAL + semi_apply_mark shape as the bare pattern predicate (same leaf, zero new ops); + inline block comments and property maps inside the pattern are fine.""" + for q in [ + "MATCH (n) WHERE EXISTS { (n)-[:R]->() } RETURN n", + "MATCH (n) WHERE exists/*inline*/{ (n)-[:R]->() } RETURN n", + "MATCH (n) WHERE EXISTS { (n)-[:R {w: 1}]->() } RETURN n", + ]: + lowered = lower_match_query(_parse_query(q)) + assert len(lowered.row_pre_filters) == 1, q + marker = lowered.row_pre_filters[0] + assert isinstance(marker, ASTCall), q + assert marker.function == "semi_apply_mark", q + assert marker.params.get("join_aliases") == ["n"], q + binding_ops = marker.params.get("binding_ops") + assert [op.get("type") for op in binding_ops] == ["Node", "Edge", "Node"], q + + +def test_lower_match_query_not_exists_subquery_emits_anti_semi_apply() -> None: + """viz-filter L1: NOT EXISTS { } composes through the NOT tier into + anti_semi_apply — the declarative prune-isolated building block.""" + lowered = lower_match_query( + _parse_query("MATCH (n) WHERE NOT EXISTS { (n)-[:R]->() } RETURN n.id AS id") + ) + assert len(lowered.row_pre_filters) == 1 + anti = lowered.row_pre_filters[0] + assert isinstance(anti, ASTCall) + assert anti.function == "anti_semi_apply" + assert anti.params.get("join_aliases") == ["n"] + + +def test_lower_exists_drop_self_flavor_emits_neq_semi_apply() -> None: + """viz-filter L1 drop-self prune-isolated: EXISTS { (n)--(m) WHERE m <> n } — the + existential local alias m is ALLOWED (bindings project it away) and the endpoint + inequality rides the op as neq=[m, n] (bindings-table filter / self-loop-edge + exclusion on polars).""" + lowered = lower_match_query( + _parse_query("MATCH (n) WHERE EXISTS { (n)--(m) WHERE m <> n } RETURN n.id AS id") + ) + assert len(lowered.row_pre_filters) == 1 + marker = lowered.row_pre_filters[0] + assert isinstance(marker, ASTCall) + assert marker.function == "semi_apply_mark" + assert marker.params.get("join_aliases") == ["n"] + assert sorted(marker.params.get("neq") or []) == ["m", "n"] + + anti = lower_match_query( + _parse_query("MATCH (n) WHERE NOT EXISTS { (n)--(m) WHERE m <> n } RETURN n.id AS id") + ).row_pre_filters[0] + assert isinstance(anti, ASTCall) + assert anti.function == "anti_semi_apply" + assert sorted(anti.params.get("neq") or []) == ["m", "n"] + + +def test_exists_subquery_unsupported_bodies_decline_clearly() -> None: + """viz-filter L1 v1 boundaries: inner WHERE, multi-pattern bodies, and full + MATCH..RETURN subquery bodies decline with a clear message (never a wrong answer).""" + for q, phrase in [ + ("MATCH (a) WHERE EXISTS { (a)--(m) WHERE m.x > 1 } RETURN a", "inner WHERE"), + ("MATCH (a) WHERE EXISTS { (a)--(), (a)-->() } RETURN a", "multi-pattern"), + ("MATCH (a) WHERE EXISTS { MATCH (a)--(b) RETURN b } RETURN a", "subquery body"), + ("MATCH (a) WHERE EXISTS { (a)--(b) WHERE a <> c } RETURN a", "endpoint aliases"), + ]: + with pytest.raises(GFQLValidationError) as exc_info: + _parse_query(q) + assert phrase in exc_info.value.message, (q, exc_info.value.message) + + def test_lower_match_query_emits_row_anti_semi_filter_for_negated_where_pattern() -> None: lowered = lower_match_query(_parse_query("MATCH (n) WHERE NOT (n)-[:R]->() RETURN n.id AS id")) @@ -6573,12 +6642,12 @@ def test_string_cypher_executes_xor_between_bounded_reverse_and_forward_where_pa "MATCH (a) RETURN exists/*inline*/{ (a)-[:KNOWS]-() } AS has", "MATCH (a) RETURN not exists { (a)-[:KNOWS]-() } AS no", "MATCH (a) RETURN not/*inline*/exists/*inline*/{ (a)-[:KNOWS]-() } AS no", - "MATCH (a) WHERE exists { (a)-[:KNOWS]-() } RETURN a.id", - "MATCH (a) WHERE exists/*inline*/{ (a)-[:KNOWS]-() } RETURN a.id", ], ) def test_string_cypher_failfast_rejects_pattern_existence(query: str) -> None: - """#998: pattern existence expressions fail-fast with clear message.""" + """#998: not((..)) pattern-expression spellings and RETURN/WITH-position + EXISTS {{ }} fail-fast with a clear message. WHERE-position EXISTS {{ }} is + SUPPORTED since viz-filter L1 (see the exists_subquery lowering tests).""" graph = _mk_empty_graph() with pytest.raises(GFQLValidationError) as exc_info: graph.gfql(query) diff --git a/graphistry/tests/compute/gfql/cypher/test_parser.py b/graphistry/tests/compute/gfql/cypher/test_parser.py index 3f78c096a0..d7d67e56e8 100644 --- a/graphistry/tests/compute/gfql/cypher/test_parser.py +++ b/graphistry/tests/compute/gfql/cypher/test_parser.py @@ -1050,8 +1050,6 @@ def test_parse_does_not_treat_pattern_existence_lexemes_inside_comments_or_liter @pytest.mark.parametrize( "query", [ - "MATCH (n) WHERE exists { (n)-[:R]->() } RETURN n", - "MATCH (n) WHERE not exists { (n)-[:R]->() } RETURN n", "MATCH (n) WHERE not((n)-[:R]->()) RETURN n", "MATCH (n) WHERE not((n:Admin)-[:R]->()) RETURN n", "MATCH (n) WHERE not((n)-[:R {w: 1}]->()) RETURN n", @@ -1060,18 +1058,39 @@ def test_parse_does_not_treat_pattern_existence_lexemes_inside_comments_or_liter "MATCH (n) WHERE not((n)--()) RETURN n", "MATCH (n) WHERE not((n)-[:R*]->()) RETURN n", "MATCH (n) WHERE not((n)-[r:R]->()) RETURN n", - "MATCH (n) WHERE exists/*inline*/{ (n)-[:R]->() } RETURN n", - "MATCH (n) WHERE not/*inline*/exists/*inline*/{ (n)-[:R]->() } RETURN n", "MATCH (n) WHERE not/*inline*/((n)-[:R]->()) RETURN n", - "MATCH (n) WHERE exists { (n)-[:R]->() } /* keep rejecting */ RETURN n", "// leading comment\nMATCH (n) WHERE not((n)-[:R]->()) RETURN n", ], ) def test_parse_still_rejects_true_pattern_existence_expressions(query: str) -> None: + """not((..)) spellings keep the fail-fast; WHERE-position EXISTS {{ }} is + SUPPORTED since viz-filter L1 (positive cases below).""" with pytest.raises(GFQLValidationError, match="Pattern existence expressions"): _parse_query(query) +@pytest.mark.parametrize( + "query", + [ + "MATCH (n) WHERE exists { (n)-[:R]->() } RETURN n", + "MATCH (n) WHERE not exists { (n)-[:R]->() } RETURN n", + "MATCH (n) WHERE exists/*inline*/{ (n)-[:R]->() } RETURN n", + "MATCH (n) WHERE not/*inline*/exists/*inline*/{ (n)-[:R]->() } RETURN n", + "MATCH (n) WHERE exists { (n)-[:R]->() } /* trailing comment */ RETURN n", + # wave-2 pins: a property STRING containing a clause keyword must not decline + # (keyword scans run on masked text); `graph` as a property name must not trip + # the anchored GRAPH-pipeline gate. + "MATCH (n) WHERE exists { (n)-[:R {name: 'WHERE it hurts'}]->() } RETURN n", + "MATCH (n) WHERE n.graph = 1 AND exists { (n)-[:R]->() } RETURN n", + ], +) +def test_parse_accepts_where_position_exists_subqueries(query: str) -> None: + """viz-filter L1: WHERE-position EXISTS {{ }} parses (RETURN/WITH position and + not((..)) spellings keep their fail-fast).""" + parsed = parse_cypher(query) + assert isinstance(parsed, CypherQuery) + + @pytest.mark.parametrize( "query", [ diff --git a/graphistry/tests/compute/gfql/cypher/test_where_row_boolean_policy_boundary.py b/graphistry/tests/compute/gfql/cypher/test_where_row_boolean_policy_boundary.py index 650900c349..79b5db77a3 100644 --- a/graphistry/tests/compute/gfql/cypher/test_where_row_boolean_policy_boundary.py +++ b/graphistry/tests/compute/gfql/cypher/test_where_row_boolean_policy_boundary.py @@ -214,7 +214,6 @@ def test_issue_1219_policy_boundary_comment_lexemes_do_not_trip_unsupported_gate "MATCH (n) WHERE NOT((n)<-[:R]-()) RETURN n", "MATCH (n) WHERE NOT((n)--()) RETURN n", "MATCH (n) WHERE NOT((n)-[:R*]->()) RETURN n", - "MATCH (n) WHERE exists { (n)-[:R]->() } RETURN n", ], ) def test_issue_1219_policy_boundary_pattern_existence_forms_still_rejected(query: str) -> None: diff --git a/graphistry/tests/compute/gfql/test_conformance_ledger.py b/graphistry/tests/compute/gfql/test_conformance_ledger.py index ecd957cedb..6e0706f176 100644 --- a/graphistry/tests/compute/gfql/test_conformance_ledger.py +++ b/graphistry/tests/compute/gfql/test_conformance_ledger.py @@ -138,8 +138,8 @@ def _exercised_function_names() -> set: "unwind": "list explode; native on polars chain, NIE via call()/DAG executor; not asserted via call(). TODO.", "group_by": "grouped aggregation; native on polars chain, NIE via call()/DAG executor; not asserted via call(). TODO.", "count_table": "count(*) short-circuit fast path (table height / source-mask sum); native frame op emitted by the cypher lowering, exercised as a labeled subject via _ROW_OP_CASES + the count_all_nodes/edges cypher cases, not via a direct call() consistency label. TODO.", - "semi_apply_mark": "correlated EXISTS-mark; row-pipeline op honest-NIE under polars; not asserted. TODO.", - "anti_semi_apply": "anti-semi correlated filter; row-pipeline op honest-NIE under polars; not asserted. TODO.", + "semi_apply_mark": "correlated EXISTS-mark; NATIVE on polars (viz-filter L1), exercised implicitly by the matrix EXISTS cypher cases; no direct labeled subject.", + "anti_semi_apply": "anti-semi correlated filter; NATIVE on polars (viz-filter L1), exercised implicitly by the matrix NOT-EXISTS cypher case; no direct labeled subject.", "join_apply": "correlated row join; row-pipeline op honest-NIE under polars; not asserted. TODO.", # Plottable-method calls: no native polars impl; pandas/cuDF only -> no-silent-bridge NIE under polars. # Class covered by test_engine_polars_no_silent_call_bridge (hypergraph representative); each TODO individually. @@ -194,8 +194,8 @@ def _exercised_function_names() -> set: ROW_OP_KNOWN_UNCOVERED: dict[str, str] = { # honest NIE — correlated-subquery ops with no native polars lowering (_try_native_row_op returns None) - "semi_apply_mark": "honest NIE — correlated EXISTS-mark op has no native polars lowering. TODO: add an explicit NIE-assertion case.", - "anti_semi_apply": "honest NIE — correlated anti-semi op has no native polars lowering. TODO: add an explicit NIE-assertion case.", + "semi_apply_mark": "native single-join-alias polars lowering (viz-filter L1); multi-alias correlation still declines NIE; exercised via cypher EXISTS cases, no labeled rowop subject.", + "anti_semi_apply": "native single-join-alias polars lowering (viz-filter L1); multi-alias correlation still declines NIE; exercised via cypher NOT-EXISTS case, no labeled rowop subject.", "join_apply": "honest NIE — correlated join op has no native polars lowering. TODO: add an explicit NIE-assertion case.", } diff --git a/graphistry/tests/compute/gfql/test_engine_polars_conformance_matrix.py b/graphistry/tests/compute/gfql/test_engine_polars_conformance_matrix.py index 806cfc7b6f..40a34c817f 100644 --- a/graphistry/tests/compute/gfql/test_engine_polars_conformance_matrix.py +++ b/graphistry/tests/compute/gfql/test_engine_polars_conformance_matrix.py @@ -233,6 +233,13 @@ def _cypher_expression_queries(): ("regex_ci_fold_unsafe_declines", "MATCH (n) WHERE n.name =~ '(?i)\\\\D+' RETURN n.id AS id"), ("regex_lookbehind_declines", "MATCH (n) WHERE n.name =~ 'node(?<=e).*' RETURN n.id AS id"), ("regex_composed_or", "MATCH (n) WHERE n.name =~ 'node.1' OR n.num > 1000 RETURN n.id AS id"), + # viz-filter L1: EXISTS { } pattern subqueries — semi_apply_mark/anti_semi_apply + # run NATIVELY on polars (chain-flag key sets); parity-or-NIE 4-engine. + ("exists_neighbor", "MATCH (n) WHERE EXISTS { (n)-->() } RETURN n.id AS id"), + ("exists_neighbor_undirected", "MATCH (n) WHERE EXISTS { (n)--() } RETURN n.id AS id"), + ("not_exists_neighbor", "MATCH (n) WHERE NOT EXISTS { (n)--() } RETURN n.id AS id"), + ("exists_far_node_prop", "MATCH (n) WHERE EXISTS { (n)-->({num: 50}) } RETURN n.id AS id"), + ("exists_composed_and", "MATCH (n) WHERE EXISTS { (n)-->() } AND n.num < 50 RETURN n.id AS id"), ] @@ -382,6 +389,122 @@ def test_scalar_fns_null_cells_parity(): _assert_invariant(g, q, f"nullcells {q}") +def test_exists_polars_internal_decline_branches(): + """Coverage pins for the honest-decline branches of the polars semi-apply family + (each returns None -> the boundary lane raises NIE): non-single/unnamed/query=/ + alias==node_id rows shapes; non-endpoint join alias; multi-hop edges; neq aliases + that aren't the endpoints (wave-2 / changed-line gate).""" + import polars as pl + from graphistry.compute.ast import n as _n, e_forward as _ef + from graphistry.compute.chain import serialize_binding_ops + from graphistry.compute.gfql.lazy.engine.polars.pattern_apply import ( + rows_binding_ops_polars, + _pattern_alias_keys_polars, + ) + import pandas as pd + nd = pd.DataFrame({"id": [0, 1], "x": [1, 2]}) + ed = pd.DataFrame({"s": [0], "d": [1], "eid": [0]}) + gpl = graphistry.nodes(pl.from_pandas(nd), "id").edges( + pl.from_pandas(ed), "s", "d").bind(edge="eid") + assert rows_binding_ops_polars(gpl, []) is None + assert rows_binding_ops_polars( + gpl, serialize_binding_ops([_n(name="n"), _n(name="m")])) is None + assert rows_binding_ops_polars(gpl, serialize_binding_ops([_n()])) is None + assert rows_binding_ops_polars( + gpl, serialize_binding_ops([_n(name="n", query="x > 1")])) is None + assert rows_binding_ops_polars( + gpl, serialize_binding_ops([_n(name="id")])) is None # alias == node-id col + bos = serialize_binding_ops([_n(name="n"), _ef(), _n(name="m")]) + assert _pattern_alias_keys_polars(gpl, bos, "zzz") is None # not an endpoint + assert _pattern_alias_keys_polars( + gpl, serialize_binding_ops([_n(name="n"), _ef(hops=2), _n()]), "n") is None + assert _pattern_alias_keys_polars(gpl, bos, "n", neq=["n", "zzz"]) is None + assert _pattern_alias_keys_polars(gpl, bos[:2], "n") is None # not Node/Edge/Node + + +def test_prune_isolated_graph_shape_all_engines(): + """LEO steer 2026-07-05: viz needs the FULL GRAPH back — nodes AND edges. THE + working patterns (dgx-probed): `GRAPH { MATCH (a)-[e]-(b) }` = keep-self + prune-isolated (self-edged + mutually-connected nodes with ALL their edges; + true edgeless nodes dropped); `+ WHERE a.id <> b.id` = drop-self variant (note: + also drops self-EDGES of kept nodes — panel-algebra §3's E₂ keeps them; the + exact algebra form needs a two-stage pipeline, L3). EXISTS inside GRAPH { } is + REJECTED fail-fast — it was SILENTLY DROPPED (dgx-repro'd all-nodes wrong + answer), and node-only GRAPH matches return zero edges.""" + import pandas as pd + from graphistry.compute.exceptions import GFQLValidationError + nd = pd.DataFrame({"id": [0, 1, 2, 3, 4]}) + # 0<->1 mutually connected (2 directed edges), 4 self-loop-only, 2+3 edgeless + ed = pd.DataFrame({"s": [0, 1, 4], "d": [1, 0, 4], "eid": [0, 1, 2]}) + g = graphistry.nodes(nd, "id").edges(ed, "s", "d").bind(edge="eid") + oracle = { + "GRAPH { MATCH (a)-[e]-(b) }": ([0, 1, 4], [0, 1, 2]), + "GRAPH { MATCH (a)-[e]-(b) WHERE a.id <> b.id }": ([0, 1], [0, 1]), + } + for q, (exp_nodes, exp_edges) in oracle.items(): + out = g.gfql(q, engine="pandas") + assert sorted(_to_pd(out._nodes)["id"].tolist()) == exp_nodes, q + assert sorted(_to_pd(out._edges)["eid"].tolist()) == exp_edges, \ + f"{q}: kept nodes must keep ALL their edges" + _assert_invariant(g, "GRAPH { MATCH (a)-[e]-(b) }", "prune-graph-shape keep-self") + # drop-self uses cross-entity same-path WHERE: polars/polars-gpu decline honestly; + # cuDF has a PRE-EXISTING TypeError in the same-path executor (verified + # byte-identical on the base tree — repo debt predating this PR, plan-tracked). + q2 = "GRAPH { MATCH (a)-[e]-(b) WHERE a.id <> b.id }" + base = _run(g, q2, "pandas") + for eng in _NONPANDAS_ENGINES: + res = _run(g, q2, eng) + if res[0] == "ok": + assert res == base, f"prune-graph-shape drop-self[{eng}]: silent divergence" + elif eng == "cudf": + assert res[0] == "nie" or res == ("err", "TypeError"), \ + f"cudf drop-self: expected NIE or the pre-existing same-path TypeError, got {res}" + else: + assert res[0] == "nie", f"{eng} drop-self must decline honestly, got {res}" + with pytest.raises(GFQLValidationError, match="GRAPH"): + g.gfql("GRAPH { MATCH (n) WHERE EXISTS { (n)--() } }", engine="pandas") + + +def test_exists_native_and_boundary_pins_polars(): + """Wave-1 T1/T2/T4 pins: (a) EXISTS stays NATIVE on polars — parity-or-NIE alone + would let a silent NIE regression pass; (b) multi-alias correlation DECLINES on + polars (pair bindings inexpressible in the key-set route) while pandas answers; + (c) self-alias EXISTS { (n)--(n) } never non-NIE crashes (chain's duplicate-alias + guard is caught into a decline).""" + g = _graph(4) + for q in [ + "MATCH (n) WHERE EXISTS { (n)-->() } RETURN n.id AS id", + "MATCH (n) WHERE NOT EXISTS { (n)--() } RETURN n.id AS id", + ]: + assert _run(g, q, "polars")[0] == "ok", f"EXISTS must stay NATIVE on polars: {q}" + q_multi = "MATCH (a)-->(b) WHERE EXISTS { (a)-->(b) } RETURN a.id AS id" + assert _run(g, q_multi, "pandas")[0] == "ok" + assert _run(g, q_multi, "polars")[0] == "nie", "multi-alias correlation must decline on polars (not crash or diverge)" + _assert_invariant(g, q_multi, "exists-multi-alias") + _assert_invariant(g, "MATCH (n) WHERE EXISTS { (n)--(n) } RETURN n.id AS id", "exists-self-alias") + + +def test_exists_prune_isolated_flavors_all_engines(): + """viz-filter L1 acceptance kernel: BOTH prune-isolated flavors of the panel algebra + (research/panel-algebra.md §3) on a discriminating graph — node 3 is fully isolated, + node 4 has ONLY a self-loop. keep-self (EXISTS {(n)--()}) keeps 4; drop-self + (EXISTS {(n)--(m) WHERE m<>n}) drops 4; 4-engine parity-or-NIE.""" + import pandas as pd + nd = pd.DataFrame({"id": [0, 1, 2, 3, 4]}) + ed = pd.DataFrame({"s": [0, 1, 4], "d": [1, 2, 4], "eid": [0, 1, 2]}) + g = graphistry.nodes(nd, "id").edges(ed, "s", "d").bind(edge="eid") + oracle = { + "MATCH (n) WHERE EXISTS { (n)--() } RETURN n.id AS id": [0, 1, 2, 4], + "MATCH (n) WHERE NOT EXISTS { (n)--() } RETURN n.id AS id": [3], + "MATCH (n) WHERE EXISTS { (n)--(m) WHERE m <> n } RETURN n.id AS id": [0, 1, 2], + "MATCH (n) WHERE NOT EXISTS { (n)--(m) WHERE m <> n } RETURN n.id AS id": [3, 4], + } + for q, expected in oracle.items(): + pdf = _to_pd(g.gfql(q, engine="pandas")._nodes) + assert sorted(pdf["id"].tolist()) == expected, f"oracle drift: {q}" + _assert_invariant(g, q, f"prune-isolated {q}") + + def test_tolower_non_string_declines_never_fabricates(): """toLower/toUpper on a non-string column must never return values (neo4j: type error). Regression (#1675 wave-1, dgx-repro'd): the scalar fallback broadcast the lowercased @@ -463,7 +586,8 @@ def _rowop_exercised(): """ROW_PIPELINE_CALLS ops with a labeled SUBJECT here (importable for the ledger): `with_` (with_extend*/in-membership), `unwind` (unwind_* native+NIE), and the _ROW_OP_CASES ops (chain+dag). Ops exercised only implicitly via cypher text (RETURN->select etc.) stay ledger - waivers; only semi_apply_mark/anti_semi_apply/join_apply remain honest-NIE waivers now.""" + waivers; semi_apply_mark/anti_semi_apply are now NATIVE on polars and exercised implicitly by + the EXISTS { } cypher cases (exists_neighbor etc.); join_apply remains the honest-NIE waiver.""" return { "with_", "unwind", "rows", "skip", "limit", "distinct", "drop_cols", diff --git a/graphistry/tests/compute/gfql/test_row_pipeline_ops.py b/graphistry/tests/compute/gfql/test_row_pipeline_ops.py index fc8e5b305d..cea97cd7b5 100644 --- a/graphistry/tests/compute/gfql/test_row_pipeline_ops.py +++ b/graphistry/tests/compute/gfql/test_row_pipeline_ops.py @@ -2496,6 +2496,20 @@ def test_row_pipeline_semi_apply_mark_validation(self): assert adds_node_cols({"out_col": ""}) == set() self._assert_valid("semi_apply_mark", {"binding_ops": [], "join_aliases": ["n"], "out_col": "__hit__"}) + # wave-2 pin: neq must be EXACTLY a string pair (a 1-item list crashed late + # with IndexError; a 3-item list silently filtered on the first pair) + self._assert_valid( + "semi_apply_mark", + {"binding_ops": [], "join_aliases": ["n"], "out_col": "__hit__", "neq": ["m", "n"]}, + ) + self._assert_e201( + "semi_apply_mark", + {"binding_ops": [], "join_aliases": ["n"], "out_col": "__hit__", "neq": ["n"]}, + ) + self._assert_e201( + "semi_apply_mark", + {"binding_ops": [], "join_aliases": ["n"], "out_col": "__hit__", "neq": ["a", "b", "c"]}, + ) self._assert_e201( "semi_apply_mark", {"binding_ops": [{"type": "Node"}], "join_aliases": [], "out_col": "__hit__"},