From 52a8c7df1fc8ae3fb22e49e1339c1b4a5f0923cd Mon Sep 17 00:00:00 2001 From: Leo Meyerovich Date: Sun, 19 Jul 2026 12:26:52 -0700 Subject: [PATCH 1/2] perf(gfql): kill per-call O(E) terms in seeded chains (keep-mask + augmentation caches) IS1 profiling found two linear-with-E costs per seeded Cypher call on a resident graph: the index path's simple-equality keep-mask rebuilt via a full column compare (eq_str), and the polars chain's per-call with_row_index O(E) copy whose fresh identity also defeated downstream id-keyed caches. Both now recycle-safe caches keyed on the resident frame (weakref.finalize eviction, the _pl_nan_to_null pattern). Profiled: both terms eliminated; 274 tests pass. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01Y6dQEcjdazEnzuvuwf73ZL --- CHANGELOG.md | 1 + graphistry/compute/gfql/index/traverse.py | 29 ++++++++++-- .../compute/gfql/lazy/engine/polars/chain.py | 27 ++++++++++- .../gfql/lazy/engine/polars/row_pipeline.py | 46 ++++++++++++++++++- 4 files changed, 98 insertions(+), 5 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 729502ef07..8b5046d07c 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -19,6 +19,7 @@ This project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.htm - **GFQL polars engine natively runs multi-source (node-cartesian) MATCH (#1273)**: comma-separated disconnected aliases — `g.gfql("MATCH (a {..}), (b {..}) RETURN a.id, b.id")` — previously declined on polars (the `rows(binding_ops)` `node_cartesian` branch hard-NIE'd) while pandas handled them. New `_cartesian_node_bindings_polars` mirrors the pandas oracle `_gfql_cartesian_node_bindings_row_table`: each alias is independently filtered, projected into the per-alias lookup schema (bare `alias` id, `alias.id`, `alias.`, and pandas' leaked `alias.alias=True` flag incl. property-shadowing), then left-major cross-joined for row-order parity. Scalar/aliased/`count(*)` projections over the cartesian match pandas exactly (verified end-to-end); whole-entity `RETURN a, b` stays an honest NIE (separate projection surface). Two parity-safe declines mirror shapes where pandas itself errors (proven on master, so both engines fail identically, never diverge): an anonymous node op, and ≥4 named aliases (pandas' bare-id merge residue collides). Differential fuzz vs pandas over ~10k lowered cases: 0 disagreements. Tests in `test_engine_polars_binding_rows.py`. - **GFQL polars chain — variable-length node aliases are hop-distance gated; the chain is now silent-wrong-free (#1741)**: a node named after a variable-length edge (`g.gfql("MATCH (a)-[*1..2]-(b) RETURN b")`) previously carried its alias regardless of hop distance, so an undirected walk that backtracked into the seed wrongly returned it (pandas correctly excludes it — trail semantics). The polars chain now auto-injects the #1741 hop-distance label (name resolved against the user's node columns via the `reserved_columns` registry, so a user column named like the internal one is never clobbered) and applies pandas' alias `[min_hop, max_hop]` window in `_apply_node_names`. The one shape the gate can't yet cover — a node alias after a **forward/reverse `min_hops>1`** edge, whose labels need pandas' layered backward walk (not ported) — now **declines with an honest `NotImplementedError` (#1748)** instead of returning nodes outside the window; the decline is precise (differential vs pandas: 30/30 named shapes NIE, 30/30 unnamed run and match, 0 over-decline). 4-engine A/B on the stacked build: pandas/cudf 144/144, polars/polars-gpu 112 agree / **0 disagree** / 32 honest-NIE — zero silent-wrong shapes remain on the polars chain. Adapts the mechanism from the retired #1742 (declined undirected varlen+alias, now natively gated). Tests: `TestVarlenAliasHopGate` (pandas-parity across directions, the `*2..3` decline + unnamed-still-runs pins, a column-collision test). - **GFQL native polars `label_node_hops` on the plain BFS — correct, direction-dependent hop labels (#1741 groundwork)**: the polars eager hop now emits pandas-parity node hop-distance labels instead of declining `label_node_hops`. The labeling rule is DIRECTION-DEPENDENT (the subtle part, derived by differential fuzz vs the pandas oracle, not by reading pandas alone): forward/reverse label EVERY destination of a hop first-wins (matching pandas `hop.py` `new_node_ids`), so a seed re-entered at hop 1 IS labeled; undirected labels destinations MINUS everything already visited and pre-seeds the seen-set with the seeds, so a seed re-reached by a backtracking walk stays NULL (its shortest-path distance). Two correctness fixes over the first cut: (1) the undirected seed pre-seed must NOT depend on `return_as_wave_front` — a seed filtered out of the frontier by `source_node_match` or suppressed in wave-front mode was being re-labeled (A/B vs pandas over direction × hops × `return_as_wave_front` × `source_node_match` × `destination_node_match` × 10 graphs: was 456/24, now 480/480); (2) a requested `label_node_hops` name that collides with an existing column is redirected to `_1` like pandas' `resolve_label_col`, not the polars left-join auto-suffix `_right`/`DuplicateError`. `label_edge_hops` and `min_hops>1` labels stay honest NIEs. Amplified `test_engine_polars_hop.py` (`TestHopLabelsDifferential` over the seed-filter / wave-front / multi-seed axes, an explicit direction-asymmetry contrast pin, and a collision test). +- **GFQL seeded Cypher/chain queries no longer pay two per-call O(E) costs on a resident graph (typed keep-mask + edge-frame augmentation caches)**: profiling the official LDBC IS1 point query on polars found the linear-with-edge-count terms that made Cypher-surface latency GROW with scale: (1) the #1658 index path rebuilt its simple-equality edge keep-mask via a full column compare every seeded hop (`_build_edge_keep_mask` — the profiled `eq_str` O(E) scan), and (2) the polars chain executor rebuilt `with_row_index` on the edge frame every call — an O(E) copy whose fresh object identity ALSO defeated every downstream id-keyed cache. Both are now recycle-safe caches keyed on the resident frame (weakref.finalize eviction, the proven `_pl_nan_to_null` clean-cache pattern): one augmented frame per resident edge frame (stable identity for all downstream caches) and one keep-mask per (frame, engine, scalar edge_match). Profiled: both O(E) terms eliminated from the warm path (`eq_str`/`with_row_index` gone). Parity: 274 tests (index 4-engine + polars conformance) unchanged. - **GFQL polars engine natively runs whole-entity aggregation Cypher (LDBC IC4 shape): HAS_