Skip to content

feat(gfql): native polars WITH...MATCH re-traversal reentry (IC6/IC11 core, #1709-adjacent)#1751

Merged
lmeyerov merged 6 commits into
masterfrom
feat/gfql-with-match-reentry-polars
Jul 21, 2026
Merged

feat(gfql): native polars WITH...MATCH re-traversal reentry (IC6/IC11 core, #1709-adjacent)#1751
lmeyerov merged 6 commits into
masterfrom
feat/gfql-with-match-reentry-polars

Conversation

@lmeyerov

@lmeyerov lmeyerov commented Jul 20, 2026

Copy link
Copy Markdown
Contributor

What

A MATCH that re-traverses from a preceding WITH's projected/aggregated bindings — the structural core of LDBC IC6/IC11 — previously declined on polars (Cypher MATCH after WITH could not recover carried node identities from the prefix stage) while pandas handled it. This makes the WITH→MATCH re-seed native.

Root cause (two pandas-only gaps):

  1. The native polars projector never emitted the _cypher_entity_projection_meta side-channel (carried alias id/ids) that the bounded-reentry executor reads to re-seed the next MATCH → immediate decline.
  2. The reentry executor's id-handling + the seeded binding pipeline were pandas-only (dropna/.loc/safe_merge).

Fix: the projector emits the meta for whole-entity node projections; the executor and binding pipeline are made engine-aware (polars is_not_null/filter/order-preserving left-join; a semi-join of the first alias to the carried ids).

Verification (independent — re-run by the reviewing agent)

  • End-to-end A/B vs pandas, match: ... WITH DISTINCT friend MATCH (friend)-[:HAS_CREATOR]-(post) RETURN count(post) → 2; RETURN post.id → 3.
  • Differential fuzz vs pandas over ~3500 random graphs × WITH→MATCH shapes (DISTINCT/filtered/ORDER-BY prefixes × fwd/rev/undirected re-MATCH × entity/property/count/count(*)/DISTINCT RETURN): 0 disagreements, 0 crashes.
  • test_engine_polars_with_match_reentry.py (new) + test_engine_polars_binding_rows.py: 60 passed (independently re-run). Pandas reentry/cypher suites: 152 passed, 0 regressions. ruff clean; mypy at master baseline; changed-line coverage 100%.

Still honest-NIE (clean decline, not silent-wrong)

  • A WITH that also carries a scalar column into the trailing MATCH (hidden-payload threading still pandas-only) — was an AttributeError crash, now a clean NIE.
  • Duplicate carried ids / node-cartesian trailing patterns under a seed (preserve pandas path-multiplicity / avoid silent-wrong).

Off master (touches projection.py, reentry/execution.py, row_pipeline.py — the last overlaps #1750's function-neighborhood; trivial rebase). Advances IC6/IC11 coverage.

🤖 Generated with Claude Code

This was referenced Jul 20, 2026
Merged
Merged
lmeyerov added a commit that referenced this pull request Jul 20, 2026
@lmeyerov @claude
refactor(gfql): type reentry/projection helpers with SeriesT/DataFram…
c257996 
…eT, drop call-site casts (review)

Review feedback on #1751: the engine-agnostic reentry helpers were (s: Any) -> Any with
unchecked cast(SeriesT/DataFrameT, ...) at call sites. Now typed with the repo frame
aliases (SeriesT/DataFrameT; object for the type-guard); the genuinely-dynamic polars
branches carry a localized # type: ignore at the dispatch point instead of an opaque Any
return + caller cast -> the three call-site casts are removed. projection.py entity_meta
dict -> Dict[str, Dict[str, Any]]. mypy: 0 new errors; ruff clean; 19 reentry tests pass.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Y6dQEcjdazEnzuvuwf73ZL
@lmeyerov
lmeyerov force-pushed the feat/gfql-with-match-reentry-polars branch from c6a11db to c257996 Compare July 20, 2026 23:47
lmeyerov and others added 5 commits July 21, 2026 01:06
@lmeyerov @claude
feat(gfql): native polars WITH...MATCH re-traversal reentry (IC6/IC11…
d112d7a 
… core)

A MATCH re-traversing from a preceding WITH's projected/aggregated bindings previously
declined on polars ("could not recover carried node identities"). Fixes two pandas-only
gaps: the native projector now emits the _cypher_entity_projection_meta side-channel the
bounded-reentry executor reads to re-seed; the executor id-handling + seeded binding
pipeline are engine-aware (polars is_not_null/filter/order-preserving join; semi-join to
carried ids). RETURN entity/property/count/count(*)/DISTINCT over WITH->MATCH now native,
pandas-parity. Differential fuzz ~3500 graphs: 0 disagreements, 0 crashes. Scalar-column
carry + duplicate-id / cartesian-under-seed stay honest NIEs (were crashes/silent-wrong).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Y6dQEcjdazEnzuvuwf73ZL
@lmeyerov @claude
test(gfql): lower pandas-lane execution.py coverage floor 84.75->83.5
1c470ad 
WITH...MATCH reentry added polars-specific branches to the shared reentry/execution.py.
Those are covered by the polars test lane (changed-line 100%) but unreachable by the
pandas core lane, which legitimately drops its per-file coverage to 83.81%. Lower the
pandas-lane floor to match; the polars branches keep real coverage via test-polars.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Y6dQEcjdazEnzuvuwf73ZL
@lmeyerov @claude
refactor(gfql): type reentry/projection helpers with SeriesT/DataFram…
c3d0b80 
…eT, drop call-site casts (review)

Review feedback on #1751: the engine-agnostic reentry helpers were (s: Any) -> Any with
unchecked cast(SeriesT/DataFrameT, ...) at call sites. Now typed with the repo frame
aliases (SeriesT/DataFrameT; object for the type-guard); the genuinely-dynamic polars
branches carry a localized # type: ignore at the dispatch point instead of an opaque Any
return + caller cast -> the three call-site casts are removed. projection.py entity_meta
dict -> Dict[str, Dict[str, Any]]. mypy: 0 new errors; ruff clean; 19 reentry tests pass.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Y6dQEcjdazEnzuvuwf73ZL
@lmeyerov @claude
refactor(gfql): move engine-agnostic frame/series helpers to Engine.py
c1c63f8 
The reentry executor accumulated a cluster of generic pandas/cuDF/polars
dispatch helpers (null mask, series/frame filter, order-preserving left join,
constant-column assign, drop-columns, row-as-mapping, series-to-pylist) that
have no reentry semantics and belong with safe_merge / df_concat / df_unique in
the engine layer. gfql_unified.py had already re-implemented series_to_pylist,
proving the leak. Move the eight primitives to graphistry.Engine (public names,
no underscore), drop the reentry-local defs, and consolidate the duplicate onto
the more-defensive version. SeriesT/DataFrameT typing preserved via a
TYPE_CHECKING import + string annotations so Engine.py (imported very early)
never triggers graphistry.compute package init at runtime (would be circular).

No behavior change: byte-identical dispatch; reentry suite 19/19, cypher/entity
subset green, ruff clean, mypy-neutral.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Y6dQEcjdazEnzuvuwf73ZL
@lmeyerov @claude
test(engine): cover moved frame/series primitives; widen polars lane …
d15d1a0 
…cov to graphistry

The frame-helper move added dispatch code to graphistry/Engine.py, whose polars
branches were exercised only via the reentry tests but NOT measured — the polars
lane covered only `--cov=graphistry/compute`, so Engine.py (outside compute/)
showed 0% on its polars branches and dragged changed-line coverage to 75%.

Fix both halves: (1) add a dedicated unit test for all nine primitives across
pandas and polars (polars cases guarded by _HAS_POLARS + registered in
POLARS_TEST_FILES) plus series_to_pylist's arrow/pandas/tolist fallbacks and
raise-paths; (2) widen the polars lane to `--cov=graphistry` (the #1727 pattern)
so Engine.py's polars branches are measured. Audit-safe: coverage_audit's polars
profile only enforces graphistry/compute/gfql/lazy/**, so Engine.py is not floored.

Reproduced the exact changed-line gate locally (combined core+gfql+polars
artifacts): 75.00% -> 98.96%, Engine.py changed lines 100%.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Y6dQEcjdazEnzuvuwf73ZL
@lmeyerov
lmeyerov force-pushed the feat/gfql-with-match-reentry-polars branch from e8ae0aa to d15d1a0 Compare July 21, 2026 08:07
@lmeyerov @claude
test(gfql): end-to-end parity for the seeded binding-rows WITH->MATCH…
f3e4f13 
… path

The row_pipeline.py change (WITH re-entry seed for binding_rows_polars) was
covered only by direct-call unit tests asserting the seed is applied + the
decline branches; nothing drove a real cypher WITH->MATCH whose trailing MATCH
routes through the MULTI-ALIAS binding-table path (vs the single-alias hop path
the differential fuzz exercises). Add an e2e differential-parity test for a
two-fresh-alias trailing MATCH with a multi-alias RETURN, with a monkeypatch
guard asserting the seeded binding_rows path is actually hit so coverage can't
silently regress.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Y6dQEcjdazEnzuvuwf73ZL
@lmeyerov
lmeyerov merged commit faf3184 into master Jul 21, 2026
77 checks passed
@lmeyerov
lmeyerov deleted the feat/gfql-with-match-reentry-polars branch July 21, 2026 08:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@lmeyerov