feat(gfql): add native Polars OLAP foundation#1728
Merged
Conversation
…dcast (#1707) + count(non-active node alias) routing (#1708) Two correctness fixes that let graph-benchmark q1-q9 run as real GFQL Cypher (no dataframe shortcuts) on pandas/cuDF, and remove a polars silent-wrong answer. `RETURN count(*)` (and any all-scalar-literal projection) returned constant 1 instead of the true count. Cypher WITH/RETURN preserves row cardinality, but polars DataFrame.select of an all-scalar projection collapses to 1 row, so the synthetic __cypher_group__=1 for keyless count(*) made the downstream group_by count see 1 row. select_polars now broadcasts an all-scalar projection to the frame height via with_columns(...).select(names). +5 parity + 2 value-regression tests. `MATCH (a)-[e]->(b) RETURN b.id, count(a)` (graph-bench q1 in-degree) failed with "one MATCH source alias at a time" — the aggregate referenced the other endpoint alias, routing to the single-alias table instead of the bindings-row table. _lower_general_row_projection now forces the bindings source for count(<bare node alias>) (non-distinct) over a non-active pattern node alias — sound on the bindings table (count of matched paths per group = in-degree). Narrow: property/compound cross-source aggregates keep the conservative fail-fast. +3 tests (incl. cuDF). Regression sweep (cypher/ + polars engine + row-pipeline): 2177 passed, 11 skipped, 15 xfailed. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012rRBmezaBEi2CJgDGkMNgR
Every chain()/Cypher call on a graph without an edge-id binding attached a synthetic per-edge index via reset_index + rename, which deep-copied AND block-consolidated the entire edge frame (~70ms @2m edges) — even for node-only queries, and repeatedly per boundary-call sub-chain. Now: shallow copy + assign the frame index as the id column — identical id values (any index type), no O(E) data copy. The column is internal-only and dropped on every exit path, so only uniqueness semantics matter (unchanged). Measured @500k nodes / 2M edges (pandas): 1-row point RETURN 95.7 -> 13.5 ms (7x) full-scan RETURN a 134 -> 55 ms (2.4x) seeded 1-hop 192 -> ~170 ms polars unchanged (own chain path) Scaling (1-row RETURN): 15.6@50k/100@500k -> 6.8@50k/13.5@500k/42@1M. Residual O(N/E) tail (backward-pass hydration merges, endpoints reconciliation) remains tracked in #1670/#887; a hydration-skip variant was evaluated and rejected (row-order semantics risk for ~1.5ms). Suites: compute 4301 passed + gfql 3014 passed (9 pre-existing umap/dask env failures, verified identical pre-fix via stash). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012rRBmezaBEi2CJgDGkMNgR
Same fix as the chain.py edge-index attach (8765c0bd), applied to hop.py — the traversal hot path. reset_index(drop=False) + rename deep-copied + block-consolidated the whole post-filter edge frame (~80ms @2m edges) on every hop, even seeded ones. Now a shallow copy + assigning the frame index as the id column: identical id values (dedup/join key only, never used positionally — verified no downstream .iloc/.loc/.index dependence), no O(E) copy. Consistent ~10-25ms lower per raw hop/chain traversal @500k/2M pandas; the Cypher RETURN traversal is still dominated by binding materialization (tracked #887). No user-frame contamination; hop + chain + gfql suites green (152 + 3014 passed; 1 pre-existing umap env failure unrelated). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012rRBmezaBEi2CJgDGkMNgR
…d patterns (#1709) Native polars bindings-row tables — the rows(binding_ops=...) op emitted by Cypher multi-alias lowering. Unblocks traversal-shaped Cypher on polars: graph-benchmark q1/q2 (top-k in-degree), q8/q9 (fixed 2-hop counts), multi-alias property projections, cross-alias grouped aggregates. - binding_rows_polars: seed filter -> per-edge orient (fwd/rev/undirected, no-dedupe concat) + inner join -> node-filter semi join -> per-alias left-join assembly (alias.{col} node props, edge_alias.{col} payload, bare alias id cols). Same meaningful schema as pandas; pandas' internal join-residue columns intentionally not replicated. - select_extend_polars: native with_(extend=True) (bindings-path aggregate lowering emits it; previously NIE'd). - group_by key_prefixes guard: whole-row bindings grouping would have been silently dropped by the native dispatch once rows() stopped NIE-ing -> now declines honestly (latent wrong-answer trap). - DECLINES (None -> honest NIE, NO-CHEATING): var-length/multihop edges, shortestPath scalar bindings, node query=/edge query/endpoint-match params, hop labels, HAS_-label disambiguation, seeded re-entry contexts, node-cartesian mode, alias_endpoints variant, join-key dtype divergence. Conformance harness: _round_floats now renders non-bool numeric (incl clean-coercible object) columns as float64 on both sides — pandas' hop internals upcast int64->float64 on the 2nd alias (engine artifact; polars faithfully keeps Int64), so the astype(str) gate failed on '20.0' vs '20' for equal values. Same non-semantic noise class as the existing summation-order rounding and null-repr normalization; genuine value differences still fail. Tests: +20 (parity incl q1/q8/q9 shapes, NIE assertions, raw-table schema, path multiplicity, undirected self-loop); DEFERRED->supported moves in the row-pipeline + conformance corpora. gfql suite 3036 passed; compute 4323 passed (9 pre-existing umap/dask env failures). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012rRBmezaBEi2CJgDGkMNgR
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012rRBmezaBEi2CJgDGkMNgR
…raph-bench q3) Extends the native polars rows(binding_ops) builder (c9b178ad) to bounded directed variable-length segments (-[*1..k]->, -[:TYPE*i..k]->, exactly-k): iterative pair joins hop-by-hop, one row per distinct edge sequence (Cypher path multiplicity; pairs not deduped so parallel edges multiply per hop — matching pandas _gfql_multihop_binding_rows), zero-hop rows for min 0, same min/max defaults as pandas (bare hops=k means exactly-k). Unblocks the graph-benchmark q3 shape on polars: MATCH (a:Person)-[:FOLLOWS*1..2]->(b:Person) WHERE ... RETURN avg(b.age) verified parity vs pandas (values + count) on typed-label replicas. Still honestly deferred (NIE): unbounded [*] (needs fixed-point + termination error), undirected var-length (immediate-backtrack avoidance not ported), aliased var-length edges (pandas rejects outright). Non-aggregate var-length projections route via the traversal chain (separate pre-existing Phase-1 single-hop limit) and keep their honest NIE. Tests: +5 parity (q3 shape, exactly-k, typed, var+fixed compound, seeded avg), DEFERRED repointed at unbounded/undirected. gfql suite 3042 passed. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012rRBmezaBEi2CJgDGkMNgR
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012rRBmezaBEi2CJgDGkMNgR
…erty joins (#1711) The Cypher lowering now computes which node aliases have their PROPERTIES referenced downstream and threads them as rows(binding_ops, attach_prop_aliases=...); both binding builders (pandas _gfql_connected_bindings_row_frame_from_state and polars binding_rows_polars) skip the O(N) property left-join for aliases not listed — their free bare id column suffices. count(*) attaches zero property columns; count(a) needs only a's bare id; RETURN b.id attaches only b. Conservative gate (attach-all default when unsure): only computed for the simple single-MATCH shape with no WITH stages, no WHERE anywhere, no repeated node alias (bound-identity `n.__gfql_node_id__` refs a RETURN-text walk can't see, #1490), and no collect() (carry/reentry hidden columns, #1413). The referenced set itself is exact (_expr_match_alias_usage non-aggregate refs = property/whole-entity uses). Primary win: unblocks the polars-gpu lazy-build (a large unpruned property-attached intermediate made GPU q8 regress 19×; pruning → the join-chain the de-risk probe clocked at ~12ms on GPU). Also trims polars-cpu q8 (244→185ms). pandas 2-hop is dominated by the state-build merges, not the property attach, so it's ~unchanged there (its bottleneck is separate). Tests: +2 (pushdown skips unused props on both engines; end-to-end parity). gfql 3042 + compute 4331 passed (9 pre-existing umap/dask env failures). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012rRBmezaBEi2CJgDGkMNgR
…1709/#1711) binding_rows_polars now builds ONE deferred pl.LazyFrame and collects once on the active target (CPU / GPU via collect()). Under engine='polars-gpu' the whole join chain + property attach runs on cudf_polars in a single GPU collect. Paired with #1711 projection-pushdown (which prunes the unused property joins that otherwise bloat the GPU intermediate), this delivers the polars-gpu OLAP win WITHOUT the regression the lazy-build-alone caused. Column access uses .collect_schema().names() (schema-only, no data). The var-length loop drops the eager .height early-break (empty lazy joins are identical; the break was an optimization). NO-CHEATING preserved: a GPU-incapable plan node makes collect() raise NotImplementedError (honest NIE), never a silent CPU fallback. Validated on dgx (safe_run, all values correct) @100k nodes/1M edges: q8 2-hop count polars-gpu: 4294ms (lazy-alone regression) -> 54ms (79x); now the FASTEST engine (vs polars-cpu 90, cudf 439, pandas 1122). q1 top-k polars-gpu 30ms (fastest). #1711 also cut all engines on q8 (pandas 4386->1122, cudf 973->439, polars 244->90). (One transient GPU-JIT spike at 50k — non-monotonic 9/311/54ms across sizes.) gfql suite 3044 passed. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012rRBmezaBEi2CJgDGkMNgR
…rk q5/q6/q7 shape) The benchmark shortcuts bypassed the GFQL engine, so the multi-part MATCH...WHERE...WITH <subset> MATCH (alias)... shape (graph-benchmark q5/q6/q7) had NO real-engine correctness test — which is how the silent wrong answer in #1712 went uncaught. Adds the missing xfail(strict)-locked correctness test: a >1-node filtered subset carried via WITH must restrict the re-matched alias (numPersons=2, not 3), with and without collect. Flips to pass when #1712 lands; strict xfail forces re-enabling then. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012rRBmezaBEi2CJgDGkMNgR
…d-alias filter (#1712 silent wrong) Two manifestations of the same silent wrong answer — a connected pattern sharing a node alias failed to intersect the two patterns' constraints, returning unfiltered counts (graph-benchmark q2/q5/q6/q7 shape). Both fixed: 1. Comma-form `MATCH (p)-[R1]->(i),(p)-[R2]->(c) WHERE i.k='x'`: the connected-match-join only applied a WHERE in the `expr_tree` form and SILENTLY DROPPED the structured `predicates` form (a plain `WHERE i.k='x'`). `_where_clause_expr_text` already renders both forms; relax the gate and NIE honestly if unrenderable rather than drop. 2. WITH-form `MATCH (p)-[R1]->(i) WHERE i.k='x' WITH p MATCH (p)-[R2]->(c)`: the bounded-reentry seed (the prefix-filtered `p`) was computed and passed as chain start_nodes, but never wired to `_gfql_start_nodes` for an all-call binding-ops chain (only the traversal->suffix-call boundary path set it), so the binding builder re-matched `p` from the whole graph. Wire it in `_execute_compiled_query_chain_non_union` when the chain is a rows(binding_ops) pipeline. Both correct on pandas and cuDF now (numPersons=2, not 3). This unblocks the benchmark q5/q6/q7 (expressible as the comma-form with tolower). Tests: the previously xfail(strict)-locked subset-carry test now passes (WITH + WITH-collect forms) + a comma-form intersection test. gfql 3047 + compute 4334 passed (9 pre-existing umap/dask env failures). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012rRBmezaBEi2CJgDGkMNgR
…raph-bench q3/q4) `RETURN c.city, avg(p.age)` (group key on one alias, aggregate over another) NIE'd with the misleading "one MATCH source alias at a time". Now routes to the bindings-row table (which materializes every alias, one row per matched path — a standard GROUP BY). Extends the #1708 force-bindings gate from bare-alias count to any CLEAN grouped aggregate func(<alias>.<prop>) (avg/sum/min/max/count) over a non-active node alias. Guarded for soundness: only CLEAN agg/non-agg separation is admitted — an aggregate nested inside a larger expression (`me.age + count(you.age)`, or `age + count(...)` in ORDER BY) keeps the conservative fail-fast (the rejects-unsound-multi-source-overlap contract), detected structurally via _expr_has_aggregate / _is_pure_aggregate_call over RETURN + top-level ORDER BY. Also fixes a latent #1711 bug this exposed: _binding_prop_alias_set dropped an alias referenced only via a property inside an aggregate (`avg(p.age)` needs p's join) — it now includes property-access aliases from inside aggregates (_expr_property_access_node_aliases), and reads the top-level query.order_by (ReturnClause has no order_by attr). avg/sum/count(property) correct on pandas + cuDF (covers q3 avg, q4 count). min/max still NIE honestly (not multiplicity-sensitive, so they don't reach the force-bindings gate — a separate, non-benchmark-blocking gap). +3 tests. gfql 3050 + compute 4337 passed (9 pre-existing umap/dask env failures). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012rRBmezaBEi2CJgDGkMNgR
lmeyerov
marked this pull request as ready for review
July 16, 2026 15:19
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Replacement stack for #1714 (rung 1/4).
Scope: semantic fixes plus native Polars fixed/variable-length binding rows and projection pushdown.
Size: 15 files, +1013/-32 (1,045 changed lines).
Validation: reconstructed from the manually reviewed #1714 tree; local focused/static gates pass. Benchmark runners/results remain private.