perf(gfql): cardinality-aware lean combine for seeded chains (#1755)

[lmeyerov]

What

Cardinality-aware lean combine for seeded chains (#1755, "abstraction tax" lever 1).

The two-pass chain executor (_chain_impl backward pass + combine_steps) reconciles wavefront boundaries with full-node-frame safe_merge/pandas.merge calls even when the seeded result is a single row, so a seeded 1-hop pays a whole-frame join. Three hot merges scan the full node/edge frame regardless of frontier size:

• backward pass: safe_merge(g._nodes, wavefront[[id]], how='inner') (2×/op)
• combine tail: safe_merge(out_df, g._nodes/_edges, how='left')

When the small side is ≥4× smaller than the full frame, the byte-identical result is obtained by an isin membership filter (no join-indexer / block-consolidation machinery):

• inner-merge of a unique full frame against a unique single-key wavefront ≡ isin in full-frame order;
• a how='left' merge discards unmatched right rows anyway, so pre-shrinking the right side is byte-identical.

Gating (narrow by design)

• pandas-only (_lean_engine_ok): cuDF's GPU hash-join is already sub-ms so an isin pre-pass is net-negative there; polars takes its own engine path.
• Engages only on cardinality gap (≥4×) and right-side column-set == [id] and id-uniqueness, so the bulk / multi-source / fan-out paths are untouched.
• GFQL_LEAN_COMBINE=0 restores the legacy full-frame merges (used by the differential parity harness).

Evidence

Parity (local, pandas): test_chain_lean_combine.py — 11 tests: gate units + lean-on vs lean-off byte-identical node+edge frames across seeded node lookup, seeded 1-hop expand (creator / both endpoints), native typed chain, plus a non-vacuity guard asserting the lean path actually engages. mypy clean; typed SeriesT/DataFrameT.

A/B (dgx-spark, 26.02-gfql-polars, 50k persons / 200k messages, seeded 1-hop, warm-median warmup3/repeat7):

engine	parity	rows	cypher off→on	native typed chain off→on
pandas	✓ byte-identical	1	50.27 → 43.83 ms (1.15×)	29.37 → 23.44 ms (1.25×)
cudf	✓ byte-identical	1	65.63 → 65.54 ms (1.00× no-op)	40.36 → 40.30 ms (1.00× no-op)

cuDF is untouched (empirically confirms the pandas-only gate → no cross-engine regression).

Note on magnitude: the prototype measured ~1.6× with the resident #1658 index (#1676, not yet on master) making the hop cheap so combine dominated total time. On master as-is the seeded hop still scans edges, so combine is a smaller fraction and the win is 1.15–1.25×; it grows once the resident index lands. This PR is a correctness-safe, parity-exact increment — not the full ≤1ms fix (which additionally needs the typed-filter lever + a seeded single-hop fast-path that bypasses the two-pass combine entirely).

🤖 Generated with Claude Code

[lmeyerov]

review-skill pass (findings mode) — no correctness blockers

Ran the repo review skill. No correctness defects survived verification — the isin ≡ merge equivalence is sound (unique/single-column guards prevent fan-out; left/within-key order preserved; the null-key case was checked empirically and pandas matches NaN keys in merge, so isin agrees). Confirmed combine_steps receives engine_concrete, so both the backward-pass intersect and the tail prefilter actually engage on pandas.

Low-severity, non-blocking (deferred):

• robustness: the isin ≡ inner-merge equivalence relies on pandas merging null keys (holds here; a null node-id is pathological, and GFQL_LEAN_COMBINE=0 + parity harness backstop it). Optional: decline when key_frame[key] has nulls to make it version-independent.
• test-coverage: the non-vacuity engagement guard covers _lean_intersect_full but not the tail _lean_prefilter_right (which has a standalone correctness unit test). Optional: extend the spy.
• style: import os as _os mid-module — a top-level import os would be more idiomatic.

Verdict: landable incremental parity-safe perf fix. Leaving open for human review — not self-merging.