Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,7 @@ This project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.htm
- **GFQL Polars-CPU streaming collect (opt-in, large traversals)**: `GFQL_POLARS_CPU_STREAMING=1` runs the polars-CPU lazy collects (`hop`/`chain`) on the polars **streaming** executor instead of the default in-memory collect. Benchmarked ~1.04–1.11× faster on big multi-hop traversals (10M nodes / 80M edges: 20.0→18.0 s) and parity-identical, but ~0.86× (slower) on small/interactive sizes (streaming overhead) — so it is **opt-in, default off** (no change to default behavior). Use for large batch traversals where CPU is the target.

### Fixed
- **GFQL Polars engine `contains` honors `regex=`/`flags=`**: the native polars `filter_by_dict` lowering of the `Contains` predicate always used `str.contains(..., literal=False)`, so a **literal** request (`contains(pat, regex=False)`) was still regex-interpreted — a pattern with a metacharacter over-matched (e.g. `contains('a.c', regex=False)` matched `'abc'`), diverging from pandas/cuDF. It also dropped `flags=`. Now `regex=False` lowers to a literal match (`literal=True`; case-insensitive literal folds both sides, matching pandas' result), and regex mode maps `case=`/`flags=` (IGNORECASE/MULTILINE/DOTALL/VERBOSE) to a Rust-regex inline flag prefix (`(?ims…)`). +1 engine-parametrized differential-parity test.
- **GFQL `engine='polars-gpu'` silent CPU fallback removed (NO-CHEATING)**: the GPU collect used `pl.GPUEngine(raise_on_fail=False)`, so any plan node the cudf_polars backend can't execute would silently run **on CPU** and still be reported as a `polars-gpu` result — making `engine='polars-gpu'` indistinguishable from `engine='polars'` whenever the plan isn't fully GPU-capable (a benchmark showing near-identical `polars`/`polars-gpu` timings is exactly this tell). Flipped to `raise_on_fail=True` and translate the cudf_polars failure into a clear `NotImplementedError` pointing at `engine='polars'` for native CPU. `polars-gpu` is now **GPU-or-error**: any timing it produces is real on-device work, never CPU mislabeled as GPU. Verified on dgx-spark (LiveJournal 35M): the seeded-frontier `hop`/2-hop chain plan executes fully on GPU without raising (nvidia-smi 92% util during the loop), so existing GPU timings are unchanged — only the honesty guarantee is added. +1 regression test (the GPU-collect error path is translated, not swallowed).
- **GFQL Polars chain dtype-mismatched join keys**: a chain (`MATCH (a)-[]->(b)…`) crashed under `engine='polars'` with `SchemaError: datatypes of join keys don't match` when an edge endpoint column's dtype differed from the node-id dtype across the int↔float boundary — e.g. a null in a `source`/`destination` column promotes it to `float64` while integer node ids stay `int64`. The hop already aligned join keys; the chain fast paths + combine did not. Now aligns endpoint join keys to the node-id dtype for the traversal and restores the original endpoint dtype on the output (matching pandas). No-op when dtypes already match.
- **GFQL `chain()` OTel span placement**: the `gfql.chain` OpenTelemetry span (`@otel_traced`) had landed on the internal `_try_chain_fast_path` probe instead of the public `chain()` — so `chain()` lost its span and the span recorded the wrong function/attributes. Moved the decorator back onto `chain()`.
Expand Down
30 changes: 28 additions & 2 deletions graphistry/compute/gfql/lazy/engine/polars/predicates.py
Original file line number Diff line number Diff line change
Expand Up @@ -42,6 +42,22 @@ def _cmp_expr(col_expr, op, val):
return None


def _inline_regex_flag_prefix(case: bool, flags: int) -> str:
"""Translate a pandas-style ``re`` ``flags`` int + ``case`` bool into a Rust-regex
inline flag prefix like ``(?im)`` (polars' regex engine honors inline flags). Empty
when nothing applies. Keeps the polars regex lowering faithful to the pandas one."""
letters = ""
if not case or (flags & re.IGNORECASE):
letters += "i"
if flags & re.MULTILINE:
letters += "m"
if flags & re.DOTALL:
letters += "s"
if flags & re.VERBOSE:
letters += "x"
return f"(?{letters})" if letters else ""


def predicate_to_expr(col: str, pred: ASTPredicate):
"""Lower an ASTPredicate to a polars boolean expression, or None if unsupported."""
import polars as pl
Expand Down Expand Up @@ -86,8 +102,18 @@ def predicate_to_expr(col: str, pred: ASTPredicate):

if name == "Contains" and hasattr(pred, "pat") and isinstance(pred.pat, str):
case = getattr(pred, "case", True)
pat = pred.pat if case else f"(?i){pred.pat}"
return c.str.contains(pat, literal=False)
regex = getattr(pred, "regex", True)
flags = getattr(pred, "flags", 0)
if not regex:
# Literal substring: must NOT be regex-interpreted, or a metacharacter
# (``.``/``*``/``(`` …) over-matches. polars has no literal+case flag, so
# fold both sides for the case-insensitive literal (matches pandas' result).
if case:
return c.str.contains(pred.pat, literal=True)
return c.str.to_lowercase().str.contains(pred.pat.lower(), literal=True)
# Regex: mirror pandas' ``case``/``flags`` via a Rust-regex inline flag prefix.
prefix = _inline_regex_flag_prefix(case, flags)
return c.str.contains(f"{prefix}{pred.pat}", literal=False)

if name in ("Startswith", "Endswith") and hasattr(pred, "pat") and isinstance(pred.pat, str):
if getattr(pred, "case", True):
Expand Down
22 changes: 22 additions & 0 deletions graphistry/tests/compute/gfql/test_engine_polars_chain.py
Original file line number Diff line number Diff line change
Expand Up @@ -104,6 +104,28 @@ def test_polars_chain_parity(cname):
assert _named(gp, nm) == _named(gl, nm), f"alias[{nm}] mismatch [{cname}]"


@pytest.mark.parametrize("label,pred", [
("literal dot", contains("a.c", regex=False)), # literal: no metachar
("regex dot", contains("a.c", regex=True)), # '.' is a wildcard
("ci literal", contains("A.C", regex=False, case=False)), # literal + case-insensitive
("ci regex", contains("A.C", regex=True, case=False)), # regex + case-insensitive
])
def test_polars_contains_regex_and_case_parity(label, pred):
"""B1: polars Contains must honor ``regex=``/``flags=``/``case=`` like pandas — a
literal contains with a regex metacharacter must NOT over-match (before the fix the
polars lowering always used ``literal=False``, so ``contains('a.c', regex=False)``
matched 'abc'). Differential parity vs the pandas oracle."""
nodes = pd.DataFrame({"id": ["p", "q", "r", "s", "t"],
"name": ["a.c", "abc", "AxC", "a.c.d", "zzz"]})
edges = pd.DataFrame({"s": ["p", "q", "r", "s"], "d": ["q", "r", "s", "t"]})
g = graphistry.nodes(nodes, "id").edges(edges, "s", "d")
ch = [n({"name": pred})]
gp = g.chain(ch, engine="pandas")
gl = g.chain(ch, engine="polars")
assert "polars" in type(gl._nodes).__module__
assert _nset(gp) == _nset(gl), f"[{label}] pandas {_nset(gp)} != polars {_nset(gl)}"


# ---- Randomized differential fuzzer (the CHANGELOG-advertised fuzzer) ----

def _rand_node(rng):
Expand Down
Loading