feat(gfql): polars-native connected OPTIONAL MATCH (fixes pandas-ism crash + unseeded-arm dispatch + polars join twin) by lmeyerov · Pull Request #1736 · graphistry/pygraphistry

lmeyerov · 2026-07-19T17:50:23Z

What

Stack PR #3 (base: #1734 feat/gfql-typed-edge-index-coverage -> #1735 -> master). Fixes the first of the two polars blockers found by the 2026-07-19 official-Cypher LDBC baseline (plans/gfql-benchmark-numbers/cypherbase-sf01/):

MATCH ... OPTIONAL MATCH ... (LDBC IS7 / message-replies shape) on engine='polars' crashed with AttributeError: 'DataFrame' object has no attribute 'dropna' — _optional_arm_start_nodes in gfql_unified.py ran pandas-only frame ops (.dropna()/.drop_duplicates()/.rename(columns=)/boolean-mask indexing) on the joined binding rows, which are polars frames on the polars engine.

Fix

Branch on frame type in the seed extraction: polars uses drop_nulls().unique().rename({...}) + filter(col.is_in(...)); the pandas/cuDF path is unchanged. After this the polars row pipeline can still honestly decline the full IS7 query (parity-or-error rows NIE — the second blocker, next stack PR), but never crashes with a pandas-ism.

Validation (dgx, 26.02-gfql-polars container)

Minimal IS7-shaped repro: AttributeError before -> clean run to the row pipeline after (traceback-verified at gfql_unified.py:380).
test_runtime_physical_cutover.py + test_engine_polars_cypher_conformance.py: 167 passed.
New test_optional_match_polars_frames.py: pandas oracle (row values + knows-flag) + polars no-pandas-ism contract. 2 passed.

Why it matters

LDBC cypher-surface coverage on polars is 6/20 with 5 more queries blocked only by the polars rows NIE; this crash was the one dishonest failure (a bug, not a decline). Fixing it is a prerequisite for the rows native-lowering PR that unlocks the polars Cypher lane.

🤖 Generated with Claude Code

https://claude.ai/code/session_01Y6dQEcjdazEnzuvuwf73ZL

lmeyerov · 2026-07-19T17:57:44Z

Extended beyond the dropna fix — connected OPTIONAL MATCH is now polars-NATIVE end-to-end, not just crash-free:

Engine-polymorphic seed extraction (the original AttributeError fix at gfql_unified.py:380).
Unseeded-arm dispatch on polars: start_nodes is a pruning hint (arm rows are left-outer-joined on shared aliases; where-arms already ran unseeded), but it forced the polars bindings-row path into its bounded-reentry pandas-only decline → NIE on queries polars can run natively. Polars engines now dispatch the arm unseeded.
Polars twin of the arm join/alias-synthesis block: is_in semi-join prune, join(how='left'), null-preserving alias synthesis.

Probe-verified on dgx: IS7-shaped MATCH ...OPTIONAL MATCH ... RETURN ... ORDER BY returns oracle-exact ordered rows on engine='polars' (previously: AttributeError crash; after fix 1 alone: honest NIE). Full IS7's CASE r WHEN null projection still declines honestly (select NIE — polars CASE lowering is a separate feature, next stack PR). 184 tests pass (cutover + polars conformance + logical planner + 3 new regression tests). Perf follow-up noted in-code: optional seeded pruning for polars via first-alias membership injection if arm fan-out ever dominates.

lmeyerov · 2026-07-19T18:19:42Z

Third commit: full LDBC IS7 now runs natively on polars, oracle-exact — the last blocker was the simple-CASE marker __cypher_case_eq__(x, null) (from CASE r WHEN null THEN false ELSE true END) declining in the polars function lowerer. Lowered the null-literal forms as is_null() (exact pandas null-mask parity); the general CASE x WHEN v form still declines honestly (it carries pandas' bool/numeric cross-dtype rules — separate follow-up). Regression test upgraded to a discriminating graph (creator KNOWS one replier, not the other → flags [False, True] pinned on pandas AND polars). dgx validation: 184 cutover/conformance/planner tests + 2 CASE mixed-dtype tests pass.

lmeyerov · 2026-07-19T18:23:22Z

Fourth commit — the noted perf follow-up, measured: polars OPTIONAL MATCH arm pruning via first-alias id-membership injection. The polars arm previously ran unseeded (correct but computed the full pattern before the outer join). The same restriction start_nodes provides on pandas is now expressed engine-natively: the shared first alias's bound ids become a membership filter on the arm's first node op (guarded: only when that op has no query and no existing id constraint; unseeded fallback otherwise). LDBC SF0.1 harness: IS7 message-replies cypher on polars 710.8 → 77.2 ms (9.2×) — vs ast-lane 39.6 ms and matched Kuzu 24.2 ms. 184 cutover/conformance/planner tests pass. PR now = 4 commits: crash fix → polars-native arm+join → CASE-null lowering → arm pruning.

Review of PR #1736 asked for amplification of the 3 IS7-shaped regression tests. Add a 14-test oracle-gated parity matrix for the polars-native connected OPTIONAL MATCH lowering: every behavioral test diffs engine='polars' against the engine='pandas' oracle (normalized sorted row sets) plus hand-derived expected rows on deliberately asymmetric fixtures (unique score/weight per node/edge so mis-joins produce wrong values). Strict-parity (shape runs natively today; regression to NIE or wrong rows fails): partial matched/unmatched seeds, zero-match arm via CASE null flag, arm WHERE keeping rows, duplicate seed ids, empty seed set, arm fan-out multiplicity, null-carrying arm value column, string ids, pruning fallback (arm first op id-constrained), pruning engaged (120-person arm space >> 2 seeds, decoy rows must not leak). Parity-or-honest-NIE (shape declines today by the parity-or-error contract; silent wrong results or pandas-ism crash fails): zero-match arm + property projection (NIE 'select'), two OPTIONAL arms (NIE 'rows'), arm WHERE filtering all rows, float ids/endpoints (dtype-divergent join key). No real polars/pandas divergence found (0 xfails). 17/17 in 4.3s, clean under -W error::FutureWarning; ruff/mypy clean. Round-001 amplification artifacts under plans/gfql-optional-match-polars-amplification/ (local; plans/ is gitignored by repo policy). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Y6dQEcjdazEnzuvuwf73ZL

…crash -> works/honest-NIE) _optional_arm_start_nodes applied pandas-only frame ops to the joined binding rows, crashing IS7-shaped MATCH ... OPTIONAL MATCH queries on engine='polars' with AttributeError before the row pipeline could run (or honestly decline). Branch on frame type: polars uses drop_nulls/unique/rename + filter(is_in); pandas/cuDF path unchanged. Validated on dgx: repro fixed, 167 cutover + polars-conformance tests pass, 2 new regression tests. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Y6dQEcjdazEnzuvuwf73ZL

…atch + polars join twin) The optional-arm dispatch passed start_nodes (a pruning hint) which the polars bindings-row path declines by contract (bounded-reentry, pandas-only), forcing an NIE on queries polars can run; and the arm join/alias-synthesis block was pandas-only. Skip seeding on polars engines (semantics preserved by the left-outer join on shared aliases) and add a polars twin for the join block. IS7-shaped MATCH...OPTIONAL MATCH now runs natively on polars, oracle-exact. Validated on dgx: 184 tests pass (cutover/conformance/planner + 3 new). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Y6dQEcjdazEnzuvuwf73ZL

… WHEN null -> is_null) The Cypher simple-CASE marker __cypher_case_eq__(x, null) declined on polars, NIE-ing any query with a CASE x WHEN null projection (LDBC IS7's knows-flag). Lower the null-literal forms as the other side's null mask, exactly matching the pandas evaluator; the general form (pandas bool/numeric cross-dtype rules) still declines honestly. Full IS7 now runs natively on polars, oracle-exact (discriminating true/false flag test). Validated dgx: 184 + 2 CASE tests pass. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Y6dQEcjdazEnzuvuwf73ZL

…ship (9.2x on LDBC IS7) The polars arm ran unseeded (start_nodes declines by contract on the polars bindings-row path), computing the full pattern before the outer join. Express the same pruning as an id-membership filter injected into the arm's first (shared) node op — only when that op has no query and no existing id constraint; falls back to unseeded otherwise. Harness-measured: IS7 message-replies cypher on polars 710.8 -> 77.2 ms. 184 tests pass. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Y6dQEcjdazEnzuvuwf73ZL

Review of PR #1736 asked for amplification of the 3 IS7-shaped regression tests. Add a 14-test oracle-gated parity matrix for the polars-native connected OPTIONAL MATCH lowering: every behavioral test diffs engine='polars' against the engine='pandas' oracle (normalized sorted row sets) plus hand-derived expected rows on deliberately asymmetric fixtures (unique score/weight per node/edge so mis-joins produce wrong values). Strict-parity (shape runs natively today; regression to NIE or wrong rows fails): partial matched/unmatched seeds, zero-match arm via CASE null flag, arm WHERE keeping rows, duplicate seed ids, empty seed set, arm fan-out multiplicity, null-carrying arm value column, string ids, pruning fallback (arm first op id-constrained), pruning engaged (120-person arm space >> 2 seeds, decoy rows must not leak). Parity-or-honest-NIE (shape declines today by the parity-or-error contract; silent wrong results or pandas-ism crash fails): zero-match arm + property projection (NIE 'select'), two OPTIONAL arms (NIE 'rows'), arm WHERE filtering all rows, float ids/endpoints (dtype-divergent join key). No real polars/pandas divergence found (0 xfails). 17/17 in 4.3s, clean under -W error::FutureWarning; ruff/mypy clean. Round-001 amplification artifacts under plans/gfql-optional-match-polars-amplification/ (local; plans/ is gitignored by repo policy). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Y6dQEcjdazEnzuvuwf73ZL

…ed-line gate) The PR's gfql_unified.py polars branches were exercised only by its own test file, which no coverage lane ran (changed-line gate saw 28.75%). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Y6dQEcjdazEnzuvuwf73ZL

lmeyerov changed the title ~~fix(gfql): engine-polymorphic OPTIONAL MATCH seed extraction (polars AttributeError -> works/honest-NIE)~~ feat(gfql): polars-native connected OPTIONAL MATCH (fixes pandas-ism crash + unseeded-arm dispatch + polars join twin) Jul 19, 2026

This was referenced Jul 19, 2026

feat(gfql): polars whole-entity aggregation Cypher — HAS_label disambiguation + identity-key + key_prefixes (LDBC IC4 native) #1737

Merged

[EXPERIMENTAL] perf(gfql): seeded-chain fast path — slices 1-2 shipped, slice 3 (lean combine) in design #1738

Draft

lmeyerov force-pushed the feat/gfql-typed-edge-index-coverage branch from 31b07ba to 13d2442 Compare July 19, 2026 20:29

lmeyerov force-pushed the fix/gfql-optional-match-polars-frames branch from da41fb1 to 5e6d307 Compare July 19, 2026 20:29

lmeyerov marked this pull request as ready for review July 19, 2026 20:32

lmeyerov mentioned this pull request Jul 19, 2026

perf(gfql): identity-stable _pl_nan_to_null + clean-cache (kills O(E) per-hop NaN re-scan on polars) #1735

Merged

lmeyerov force-pushed the feat/gfql-typed-edge-index-coverage branch 2 times, most recently from 8df8249 to 11d0bab Compare July 19, 2026 21:08

lmeyerov force-pushed the fix/gfql-optional-match-polars-frames branch from 5e6d307 to e513b8c Compare July 19, 2026 21:14

lmeyerov and others added 6 commits July 19, 2026 14:33

lmeyerov force-pushed the fix/gfql-optional-match-polars-frames branch from e513b8c to 2da6ba6 Compare July 19, 2026 21:33

Base automatically changed from feat/gfql-typed-edge-index-coverage to master July 19, 2026 23:24

lmeyerov merged commit dbc6f62 into master Jul 19, 2026
75 checks passed

lmeyerov deleted the fix/gfql-optional-match-polars-frames branch July 19, 2026 23:25

This was referenced Jul 20, 2026

perf(gfql): real q1-q9 OLAP lowering on Polars #1714

Open

perf(gfql): kill per-call O(E) terms in seeded Cypher/chain queries (#1755 slice 1) #1757

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

feat(gfql): polars-native connected OPTIONAL MATCH (fixes pandas-ism crash + unseeded-arm dispatch + polars join twin)#1736

feat(gfql): polars-native connected OPTIONAL MATCH (fixes pandas-ism crash + unseeded-arm dispatch + polars join twin)#1736
lmeyerov merged 6 commits into
masterfrom
fix/gfql-optional-match-polars-frames

lmeyerov commented Jul 19, 2026

Uh oh!

lmeyerov commented Jul 19, 2026

Uh oh!

lmeyerov commented Jul 19, 2026

Uh oh!

lmeyerov commented Jul 19, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Conversation

lmeyerov commented Jul 19, 2026

What

Fix

Validation (dgx, 26.02-gfql-polars container)

Why it matters

Uh oh!

lmeyerov commented Jul 19, 2026

Uh oh!

lmeyerov commented Jul 19, 2026

Uh oh!

lmeyerov commented Jul 19, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant