feat(gfql): physical adjacency indexes for O(degree) seeded traversal#1658
Merged
Conversation
lmeyerov
added a commit
that referenced
this pull request
Jun 28, 2026
…(takeover) Took over the seeded-traversal CSR index (#1658). A 2-agent adversarial review (architecture + correctness) reproduced multiple wrong-answer divergences from the un-indexed scan oracle that the original parity scenarios missed (the cost gate masks them unless >=3 distinct seed keys hit the index path). All fixed; the index now matches scan or falls back — never a wrong answer. +6 engine-parametrized differential regression tests. BLOCKERs: - B1 max_hops silently ignored: the coverage gate omitted max_hops, so the index ran 'hops' (default 1) while the scan ran 'max_hops or hops'. Now resolve eff_hops = max_hops or hops in maybe_index_hop. - B2 duplicate node ids corrupted the node-id index: build treated a per-row (length-E) row_positions array as per-key (length-U). Now collapse to the first row per key + refuse (None) on non-unique ids; gfql_index_all skips node_id gracefully; traverse falls back when the output would carry duplicate id rows (the scan dedups output nodes by id). - B3 endpoints absent from the node table were dropped: the scan synthesizes a row per edge endpoint; the index only materialized existing rows. Now detect missing ids and fall back to scan. IMPORTANTs: - I4 node-row order: node-id-index materialization returned sorted-by-id order; now sort row positions ascending to preserve the .nodes table order. - I5 id(df) fingerprint staleness: an object id can be recycled after GC, so a new same-shape frame could pass validation -> stale index -> wrong answer. Now hold a strong ref + validate by object identity (recycle-proof). - I6 frontier dtype narrowing: an int64 id seed cast to int32 keys wraps and false- matches. Now promote both sides to a common dtype (never narrow). Latent integration bugs surfaced under force policy: - the polars/cudf index hooks passed un-converted pandas seeds to col_to_array (which assumes engine-native frames) -> AttributeError. Now normalize seeds via df_to_engine at the index entry. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
lmeyerov
force-pushed
the
dev/gfql-seeded-traversal-index
branch
from
June 28, 2026 17:41
446d419 to
1e4cde8
Compare
lmeyerov
force-pushed
the
dev/gfql-lazy-gpu
branch
from
June 28, 2026 20:00
194671a to
e5321e4
Compare
lmeyerov
added a commit
that referenced
this pull request
Jun 28, 2026
…(takeover) Took over the seeded-traversal CSR index (#1658). A 2-agent adversarial review (architecture + correctness) reproduced multiple wrong-answer divergences from the un-indexed scan oracle that the original parity scenarios missed (the cost gate masks them unless >=3 distinct seed keys hit the index path). All fixed; the index now matches scan or falls back — never a wrong answer. +6 engine-parametrized differential regression tests. BLOCKERs: - B1 max_hops silently ignored: the coverage gate omitted max_hops, so the index ran 'hops' (default 1) while the scan ran 'max_hops or hops'. Now resolve eff_hops = max_hops or hops in maybe_index_hop. - B2 duplicate node ids corrupted the node-id index: build treated a per-row (length-E) row_positions array as per-key (length-U). Now collapse to the first row per key + refuse (None) on non-unique ids; gfql_index_all skips node_id gracefully; traverse falls back when the output would carry duplicate id rows (the scan dedups output nodes by id). - B3 endpoints absent from the node table were dropped: the scan synthesizes a row per edge endpoint; the index only materialized existing rows. Now detect missing ids and fall back to scan. IMPORTANTs: - I4 node-row order: node-id-index materialization returned sorted-by-id order; now sort row positions ascending to preserve the .nodes table order. - I5 id(df) fingerprint staleness: an object id can be recycled after GC, so a new same-shape frame could pass validation -> stale index -> wrong answer. Now hold a strong ref + validate by object identity (recycle-proof). - I6 frontier dtype narrowing: an int64 id seed cast to int32 keys wraps and false- matches. Now promote both sides to a common dtype (never narrow). Latent integration bugs surfaced under force policy: - the polars/cudf index hooks passed un-converted pandas seeds to col_to_array (which assumes engine-native frames) -> AttributeError. Now normalize seeds via df_to_engine at the index entry. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
lmeyerov
force-pushed
the
dev/gfql-seeded-traversal-index
branch
from
June 28, 2026 20:00
1e4cde8 to
dec68fc
Compare
lmeyerov
force-pushed
the
dev/gfql-lazy-gpu
branch
from
June 28, 2026 20:27
e5321e4 to
5c60a46
Compare
lmeyerov
added a commit
that referenced
this pull request
Jun 28, 2026
…(takeover) Took over the seeded-traversal CSR index (#1658). A 2-agent adversarial review (architecture + correctness) reproduced multiple wrong-answer divergences from the un-indexed scan oracle that the original parity scenarios missed (the cost gate masks them unless >=3 distinct seed keys hit the index path). All fixed; the index now matches scan or falls back — never a wrong answer. +6 engine-parametrized differential regression tests. BLOCKERs: - B1 max_hops silently ignored: the coverage gate omitted max_hops, so the index ran 'hops' (default 1) while the scan ran 'max_hops or hops'. Now resolve eff_hops = max_hops or hops in maybe_index_hop. - B2 duplicate node ids corrupted the node-id index: build treated a per-row (length-E) row_positions array as per-key (length-U). Now collapse to the first row per key + refuse (None) on non-unique ids; gfql_index_all skips node_id gracefully; traverse falls back when the output would carry duplicate id rows (the scan dedups output nodes by id). - B3 endpoints absent from the node table were dropped: the scan synthesizes a row per edge endpoint; the index only materialized existing rows. Now detect missing ids and fall back to scan. IMPORTANTs: - I4 node-row order: node-id-index materialization returned sorted-by-id order; now sort row positions ascending to preserve the .nodes table order. - I5 id(df) fingerprint staleness: an object id can be recycled after GC, so a new same-shape frame could pass validation -> stale index -> wrong answer. Now hold a strong ref + validate by object identity (recycle-proof). - I6 frontier dtype narrowing: an int64 id seed cast to int32 keys wraps and false- matches. Now promote both sides to a common dtype (never narrow). Latent integration bugs surfaced under force policy: - the polars/cudf index hooks passed un-converted pandas seeds to col_to_array (which assumes engine-native frames) -> AttributeError. Now normalize seeds via df_to_engine at the index entry. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
lmeyerov
force-pushed
the
dev/gfql-seeded-traversal-index
branch
from
June 28, 2026 20:27
a5e70f7 to
9cbe4d7
Compare
lmeyerov
added a commit
that referenced
this pull request
Jun 28, 2026
…(takeover) Took over the seeded-traversal CSR index (#1658). A 2-agent adversarial review (architecture + correctness) reproduced multiple wrong-answer divergences from the un-indexed scan oracle that the original parity scenarios missed (the cost gate masks them unless >=3 distinct seed keys hit the index path). All fixed; the index now matches scan or falls back — never a wrong answer. +6 engine-parametrized differential regression tests. BLOCKERs: - B1 max_hops silently ignored: the coverage gate omitted max_hops, so the index ran 'hops' (default 1) while the scan ran 'max_hops or hops'. Now resolve eff_hops = max_hops or hops in maybe_index_hop. - B2 duplicate node ids corrupted the node-id index: build treated a per-row (length-E) row_positions array as per-key (length-U). Now collapse to the first row per key + refuse (None) on non-unique ids; gfql_index_all skips node_id gracefully; traverse falls back when the output would carry duplicate id rows (the scan dedups output nodes by id). - B3 endpoints absent from the node table were dropped: the scan synthesizes a row per edge endpoint; the index only materialized existing rows. Now detect missing ids and fall back to scan. IMPORTANTs: - I4 node-row order: node-id-index materialization returned sorted-by-id order; now sort row positions ascending to preserve the .nodes table order. - I5 id(df) fingerprint staleness: an object id can be recycled after GC, so a new same-shape frame could pass validation -> stale index -> wrong answer. Now hold a strong ref + validate by object identity (recycle-proof). - I6 frontier dtype narrowing: an int64 id seed cast to int32 keys wraps and false- matches. Now promote both sides to a common dtype (never narrow). Latent integration bugs surfaced under force policy: - the polars/cudf index hooks passed un-converted pandas seeds to col_to_array (which assumes engine-native frames) -> AttributeError. Now normalize seeds via df_to_engine at the index entry. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
lmeyerov
force-pushed
the
dev/gfql-seeded-traversal-index
branch
from
June 28, 2026 20:46
9cbe4d7 to
073eb80
Compare
lmeyerov
force-pushed
the
dev/gfql-lazy-gpu
branch
from
June 28, 2026 20:51
36cf1fd to
7207dc8
Compare
lmeyerov
added a commit
that referenced
this pull request
Jun 28, 2026
…(takeover) Took over the seeded-traversal CSR index (#1658). A 2-agent adversarial review (architecture + correctness) reproduced multiple wrong-answer divergences from the un-indexed scan oracle that the original parity scenarios missed (the cost gate masks them unless >=3 distinct seed keys hit the index path). All fixed; the index now matches scan or falls back — never a wrong answer. +6 engine-parametrized differential regression tests. BLOCKERs: - B1 max_hops silently ignored: the coverage gate omitted max_hops, so the index ran 'hops' (default 1) while the scan ran 'max_hops or hops'. Now resolve eff_hops = max_hops or hops in maybe_index_hop. - B2 duplicate node ids corrupted the node-id index: build treated a per-row (length-E) row_positions array as per-key (length-U). Now collapse to the first row per key + refuse (None) on non-unique ids; gfql_index_all skips node_id gracefully; traverse falls back when the output would carry duplicate id rows (the scan dedups output nodes by id). - B3 endpoints absent from the node table were dropped: the scan synthesizes a row per edge endpoint; the index only materialized existing rows. Now detect missing ids and fall back to scan. IMPORTANTs: - I4 node-row order: node-id-index materialization returned sorted-by-id order; now sort row positions ascending to preserve the .nodes table order. - I5 id(df) fingerprint staleness: an object id can be recycled after GC, so a new same-shape frame could pass validation -> stale index -> wrong answer. Now hold a strong ref + validate by object identity (recycle-proof). - I6 frontier dtype narrowing: an int64 id seed cast to int32 keys wraps and false- matches. Now promote both sides to a common dtype (never narrow). Latent integration bugs surfaced under force policy: - the polars/cudf index hooks passed un-converted pandas seeds to col_to_array (which assumes engine-native frames) -> AttributeError. Now normalize seeds via df_to_engine at the index entry. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
lmeyerov
force-pushed
the
dev/gfql-seeded-traversal-index
branch
from
June 28, 2026 20:51
073eb80 to
bb4d7e3
Compare
lmeyerov
force-pushed
the
dev/gfql-lazy-gpu
branch
from
June 28, 2026 21:02
7207dc8 to
8591416
Compare
lmeyerov
added a commit
that referenced
this pull request
Jun 28, 2026
…(takeover) Took over the seeded-traversal CSR index (#1658). A 2-agent adversarial review (architecture + correctness) reproduced multiple wrong-answer divergences from the un-indexed scan oracle that the original parity scenarios missed (the cost gate masks them unless >=3 distinct seed keys hit the index path). All fixed; the index now matches scan or falls back — never a wrong answer. +6 engine-parametrized differential regression tests. BLOCKERs: - B1 max_hops silently ignored: the coverage gate omitted max_hops, so the index ran 'hops' (default 1) while the scan ran 'max_hops or hops'. Now resolve eff_hops = max_hops or hops in maybe_index_hop. - B2 duplicate node ids corrupted the node-id index: build treated a per-row (length-E) row_positions array as per-key (length-U). Now collapse to the first row per key + refuse (None) on non-unique ids; gfql_index_all skips node_id gracefully; traverse falls back when the output would carry duplicate id rows (the scan dedups output nodes by id). - B3 endpoints absent from the node table were dropped: the scan synthesizes a row per edge endpoint; the index only materialized existing rows. Now detect missing ids and fall back to scan. IMPORTANTs: - I4 node-row order: node-id-index materialization returned sorted-by-id order; now sort row positions ascending to preserve the .nodes table order. - I5 id(df) fingerprint staleness: an object id can be recycled after GC, so a new same-shape frame could pass validation -> stale index -> wrong answer. Now hold a strong ref + validate by object identity (recycle-proof). - I6 frontier dtype narrowing: an int64 id seed cast to int32 keys wraps and false- matches. Now promote both sides to a common dtype (never narrow). Latent integration bugs surfaced under force policy: - the polars/cudf index hooks passed un-converted pandas seeds to col_to_array (which assumes engine-native frames) -> AttributeError. Now normalize seeds via df_to_engine at the index entry. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
lmeyerov
force-pushed
the
dev/gfql-seeded-traversal-index
branch
from
June 28, 2026 21:02
bb4d7e3 to
648f10d
Compare
lmeyerov
force-pushed
the
dev/gfql-lazy-gpu
branch
from
June 28, 2026 21:59
8591416 to
3d47a07
Compare
lmeyerov
added a commit
that referenced
this pull request
Jun 28, 2026
…(takeover) Took over the seeded-traversal CSR index (#1658). A 2-agent adversarial review (architecture + correctness) reproduced multiple wrong-answer divergences from the un-indexed scan oracle that the original parity scenarios missed (the cost gate masks them unless >=3 distinct seed keys hit the index path). All fixed; the index now matches scan or falls back — never a wrong answer. +6 engine-parametrized differential regression tests. BLOCKERs: - B1 max_hops silently ignored: the coverage gate omitted max_hops, so the index ran 'hops' (default 1) while the scan ran 'max_hops or hops'. Now resolve eff_hops = max_hops or hops in maybe_index_hop. - B2 duplicate node ids corrupted the node-id index: build treated a per-row (length-E) row_positions array as per-key (length-U). Now collapse to the first row per key + refuse (None) on non-unique ids; gfql_index_all skips node_id gracefully; traverse falls back when the output would carry duplicate id rows (the scan dedups output nodes by id). - B3 endpoints absent from the node table were dropped: the scan synthesizes a row per edge endpoint; the index only materialized existing rows. Now detect missing ids and fall back to scan. IMPORTANTs: - I4 node-row order: node-id-index materialization returned sorted-by-id order; now sort row positions ascending to preserve the .nodes table order. - I5 id(df) fingerprint staleness: an object id can be recycled after GC, so a new same-shape frame could pass validation -> stale index -> wrong answer. Now hold a strong ref + validate by object identity (recycle-proof). - I6 frontier dtype narrowing: an int64 id seed cast to int32 keys wraps and false- matches. Now promote both sides to a common dtype (never narrow). Latent integration bugs surfaced under force policy: - the polars/cudf index hooks passed un-converted pandas seeds to col_to_array (which assumes engine-native frames) -> AttributeError. Now normalize seeds via df_to_engine at the index entry. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
lmeyerov
force-pushed
the
dev/gfql-seeded-traversal-index
branch
from
June 28, 2026 22:00
648f10d to
8759fd4
Compare
lmeyerov
added a commit
that referenced
this pull request
Jun 29, 2026
…(takeover) Took over the seeded-traversal CSR index (#1658). A 2-agent adversarial review (architecture + correctness) reproduced multiple wrong-answer divergences from the un-indexed scan oracle that the original parity scenarios missed (the cost gate masks them unless >=3 distinct seed keys hit the index path). All fixed; the index now matches scan or falls back — never a wrong answer. +6 engine-parametrized differential regression tests. BLOCKERs: - B1 max_hops silently ignored: the coverage gate omitted max_hops, so the index ran 'hops' (default 1) while the scan ran 'max_hops or hops'. Now resolve eff_hops = max_hops or hops in maybe_index_hop. - B2 duplicate node ids corrupted the node-id index: build treated a per-row (length-E) row_positions array as per-key (length-U). Now collapse to the first row per key + refuse (None) on non-unique ids; gfql_index_all skips node_id gracefully; traverse falls back when the output would carry duplicate id rows (the scan dedups output nodes by id). - B3 endpoints absent from the node table were dropped: the scan synthesizes a row per edge endpoint; the index only materialized existing rows. Now detect missing ids and fall back to scan. IMPORTANTs: - I4 node-row order: node-id-index materialization returned sorted-by-id order; now sort row positions ascending to preserve the .nodes table order. - I5 id(df) fingerprint staleness: an object id can be recycled after GC, so a new same-shape frame could pass validation -> stale index -> wrong answer. Now hold a strong ref + validate by object identity (recycle-proof). - I6 frontier dtype narrowing: an int64 id seed cast to int32 keys wraps and false- matches. Now promote both sides to a common dtype (never narrow). Latent integration bugs surfaced under force policy: - the polars/cudf index hooks passed un-converted pandas seeds to col_to_array (which assumes engine-native frames) -> AttributeError. Now normalize seeds via df_to_engine at the index entry. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
lmeyerov
force-pushed
the
dev/gfql-seeded-traversal-index
branch
from
June 29, 2026 03:06
8759fd4 to
9014160
Compare
lmeyerov
force-pushed
the
dev/gfql-lazy-gpu
branch
from
June 29, 2026 03:14
16d5beb to
eea6c96
Compare
lmeyerov
added a commit
that referenced
this pull request
Jun 29, 2026
…(takeover) Took over the seeded-traversal CSR index (#1658). A 2-agent adversarial review (architecture + correctness) reproduced multiple wrong-answer divergences from the un-indexed scan oracle that the original parity scenarios missed (the cost gate masks them unless >=3 distinct seed keys hit the index path). All fixed; the index now matches scan or falls back — never a wrong answer. +6 engine-parametrized differential regression tests. BLOCKERs: - B1 max_hops silently ignored: the coverage gate omitted max_hops, so the index ran 'hops' (default 1) while the scan ran 'max_hops or hops'. Now resolve eff_hops = max_hops or hops in maybe_index_hop. - B2 duplicate node ids corrupted the node-id index: build treated a per-row (length-E) row_positions array as per-key (length-U). Now collapse to the first row per key + refuse (None) on non-unique ids; gfql_index_all skips node_id gracefully; traverse falls back when the output would carry duplicate id rows (the scan dedups output nodes by id). - B3 endpoints absent from the node table were dropped: the scan synthesizes a row per edge endpoint; the index only materialized existing rows. Now detect missing ids and fall back to scan. IMPORTANTs: - I4 node-row order: node-id-index materialization returned sorted-by-id order; now sort row positions ascending to preserve the .nodes table order. - I5 id(df) fingerprint staleness: an object id can be recycled after GC, so a new same-shape frame could pass validation -> stale index -> wrong answer. Now hold a strong ref + validate by object identity (recycle-proof). - I6 frontier dtype narrowing: an int64 id seed cast to int32 keys wraps and false- matches. Now promote both sides to a common dtype (never narrow). Latent integration bugs surfaced under force policy: - the polars/cudf index hooks passed un-converted pandas seeds to col_to_array (which assumes engine-native frames) -> AttributeError. Now normalize seeds via df_to_engine at the index entry. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
lmeyerov
force-pushed
the
dev/gfql-seeded-traversal-index
branch
from
June 29, 2026 03:14
9014160 to
ca5dfab
Compare
lmeyerov
force-pushed
the
dev/gfql-lazy-gpu
branch
from
June 29, 2026 23:08
eea6c96 to
38a08e7
Compare
lmeyerov
added a commit
that referenced
this pull request
Jun 29, 2026
…(takeover) Took over the seeded-traversal CSR index (#1658). A 2-agent adversarial review (architecture + correctness) reproduced multiple wrong-answer divergences from the un-indexed scan oracle that the original parity scenarios missed (the cost gate masks them unless >=3 distinct seed keys hit the index path). All fixed; the index now matches scan or falls back — never a wrong answer. +6 engine-parametrized differential regression tests. BLOCKERs: - B1 max_hops silently ignored: the coverage gate omitted max_hops, so the index ran 'hops' (default 1) while the scan ran 'max_hops or hops'. Now resolve eff_hops = max_hops or hops in maybe_index_hop. - B2 duplicate node ids corrupted the node-id index: build treated a per-row (length-E) row_positions array as per-key (length-U). Now collapse to the first row per key + refuse (None) on non-unique ids; gfql_index_all skips node_id gracefully; traverse falls back when the output would carry duplicate id rows (the scan dedups output nodes by id). - B3 endpoints absent from the node table were dropped: the scan synthesizes a row per edge endpoint; the index only materialized existing rows. Now detect missing ids and fall back to scan. IMPORTANTs: - I4 node-row order: node-id-index materialization returned sorted-by-id order; now sort row positions ascending to preserve the .nodes table order. - I5 id(df) fingerprint staleness: an object id can be recycled after GC, so a new same-shape frame could pass validation -> stale index -> wrong answer. Now hold a strong ref + validate by object identity (recycle-proof). - I6 frontier dtype narrowing: an int64 id seed cast to int32 keys wraps and false- matches. Now promote both sides to a common dtype (never narrow). Latent integration bugs surfaced under force policy: - the polars/cudf index hooks passed un-converted pandas seeds to col_to_array (which assumes engine-native frames) -> AttributeError. Now normalize seeds via df_to_engine at the index entry. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
lmeyerov
force-pushed
the
dev/gfql-seeded-traversal-index
branch
from
June 29, 2026 23:08
ca5dfab to
b65ca7f
Compare
lmeyerov
force-pushed
the
dev/gfql-lazy-gpu
branch
from
July 1, 2026 01:20
38a08e7 to
c5bc2fd
Compare
lmeyerov
added a commit
that referenced
this pull request
Jul 1, 2026
…(takeover) Took over the seeded-traversal CSR index (#1658). A 2-agent adversarial review (architecture + correctness) reproduced multiple wrong-answer divergences from the un-indexed scan oracle that the original parity scenarios missed (the cost gate masks them unless >=3 distinct seed keys hit the index path). All fixed; the index now matches scan or falls back — never a wrong answer. +6 engine-parametrized differential regression tests. BLOCKERs: - B1 max_hops silently ignored: the coverage gate omitted max_hops, so the index ran 'hops' (default 1) while the scan ran 'max_hops or hops'. Now resolve eff_hops = max_hops or hops in maybe_index_hop. - B2 duplicate node ids corrupted the node-id index: build treated a per-row (length-E) row_positions array as per-key (length-U). Now collapse to the first row per key + refuse (None) on non-unique ids; gfql_index_all skips node_id gracefully; traverse falls back when the output would carry duplicate id rows (the scan dedups output nodes by id). - B3 endpoints absent from the node table were dropped: the scan synthesizes a row per edge endpoint; the index only materialized existing rows. Now detect missing ids and fall back to scan. IMPORTANTs: - I4 node-row order: node-id-index materialization returned sorted-by-id order; now sort row positions ascending to preserve the .nodes table order. - I5 id(df) fingerprint staleness: an object id can be recycled after GC, so a new same-shape frame could pass validation -> stale index -> wrong answer. Now hold a strong ref + validate by object identity (recycle-proof). - I6 frontier dtype narrowing: an int64 id seed cast to int32 keys wraps and false- matches. Now promote both sides to a common dtype (never narrow). Latent integration bugs surfaced under force policy: - the polars/cudf index hooks passed un-converted pandas seeds to col_to_array (which assumes engine-native frames) -> AttributeError. Now normalize seeds via df_to_engine at the index entry. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
lmeyerov
commented
Jul 8, 2026
| g: Any, | ||
| query: object, | ||
| *, | ||
| index_policy: str = "use", |
lmeyerov
commented
Jul 8, 2026
| FrameLike = DataFrameT | ||
|
|
||
|
|
||
| class ArrayLike(Protocol): |
lmeyerov
commented
Jul 8, 2026
lmeyerov
commented
Jul 8, 2026
| # Type variable for return type preservation in predicates | ||
| T = TypeVar('T') | ||
|
|
||
| class ArrayLike(Protocol): |
pull Bot
pushed a commit
to admariner/pygraphistry
that referenced
this pull request
Jul 16, 2026
…ngages on polars (graphistry#1726) _pl_nan_to_null returned a fresh polars frame whenever any Float32/Float64 column existed (keyed on column presence, not actual NaN), churning frame identity so the graphistry#1658 CSR index's `source_ref is df` check failed -> polars/polars-gpu direct hops always fell back to the O(E) scan. Fix: probe is_nan().any() per float column (dtype-gated) and return the SAME frame object when no NaN is present; rebuild only NaN-carrying columns when present. Value-identical (fill_nan(None) is a no-op on NaN-free columns); only object identity changes -> the resident index now engages. Verified on the real SNB graph: polars flipped 120/120-scan -> 54-index/12-scan (remaining 12 = cost-gate, separate). Adds pure-polars TestPlNanToNull. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Myz4YLr6FhjyH7h7dHHTBV
This was referenced Jul 19, 2026
lmeyerov
added a commit
that referenced
this pull request
Jul 19, 2026
…edges The native polars lazy chain executor (lazy/engine/polars/chain.py) attaches its synthetic edge id via with_row_index -> a fresh edge frame -> the #1658 identity guard missed, so typed-edge chains scanned on polars/polars-gpu (untyped already engaged). Apply the same rebind_edges fix at the polars augmentation site. Now ALL FOUR engines engage the index for typed AND untyped seeded chain/Cypher hops. Measured (dgx, 500k/4M/4-types, index==scan): polars typed 1.3x / untyped 5.4x; polars-gpu typed engaged / untyped 12.4x. Broadened the engagement test to all engines; 138 index tests pass (109 existing + 29 new), 4-engine. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Y6dQEcjdazEnzuvuwf73ZL
lmeyerov
added a commit
that referenced
this pull request
Jul 19, 2026
…das/cuDF (incl. typed edges) A seeded hop expressed as a gfql()/Cypher chain (not a direct g.hop) always scanned on pandas/cuDF even with a resident index: _chain_impl attaches a synthetic per-edge id column (copy(deep=False)+assign) -> a fresh edge-frame object -> the index's `source_ref is df` identity guard missed. - registry.rebind_edges: re-point edge adjacency indexes at the augmented frame (safe: shallow copy preserves src/dst by value; fingerprint unchanged by an added column). Called in chain.py right after the edge-index augmentation. - Typed edges: relax _hop_is_index_coverable for simple scalar-equality edge_match on the wavefront path; index_seeded_hop applies the edge predicate to CSR-matched rows each hop, parity-exact with the scan's filter_edges_by_dict. - Predicate/membership edge_match, edge_query, and the direct-hop path stay on scan. Measured (dgx, 500k/4M/4-types, warm median, index==scan): pandas typed 2.1x / untyped 2.7x; cuDF typed 1.9x / untyped 2.0x. polars/polars-gpu untyped already engage via the native lazy executor; typed polars is a tracked follow-up. 27 new 4-engine differential + engagement tests; existing 109-test index suite green. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Y6dQEcjdazEnzuvuwf73ZL
lmeyerov
added a commit
that referenced
this pull request
Jul 19, 2026
…edges The native polars lazy chain executor (lazy/engine/polars/chain.py) attaches its synthetic edge id via with_row_index -> a fresh edge frame -> the #1658 identity guard missed, so typed-edge chains scanned on polars/polars-gpu (untyped already engaged). Apply the same rebind_edges fix at the polars augmentation site. Now ALL FOUR engines engage the index for typed AND untyped seeded chain/Cypher hops. Measured (dgx, 500k/4M/4-types, index==scan): polars typed 1.3x / untyped 5.4x; polars-gpu typed engaged / untyped 12.4x. Broadened the engagement test to all engines; 138 index tests pass (109 existing + 29 new), 4-engine. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Y6dQEcjdazEnzuvuwf73ZL
lmeyerov
added a commit
that referenced
this pull request
Jul 19, 2026
…das/cuDF (incl. typed edges) A seeded hop expressed as a gfql()/Cypher chain (not a direct g.hop) always scanned on pandas/cuDF even with a resident index: _chain_impl attaches a synthetic per-edge id column (copy(deep=False)+assign) -> a fresh edge-frame object -> the index's `source_ref is df` identity guard missed. - registry.rebind_edges: re-point edge adjacency indexes at the augmented frame (safe: shallow copy preserves src/dst by value; fingerprint unchanged by an added column). Called in chain.py right after the edge-index augmentation. - Typed edges: relax _hop_is_index_coverable for simple scalar-equality edge_match on the wavefront path; index_seeded_hop applies the edge predicate to CSR-matched rows each hop, parity-exact with the scan's filter_edges_by_dict. - Predicate/membership edge_match, edge_query, and the direct-hop path stay on scan. Measured (dgx, 500k/4M/4-types, warm median, index==scan): pandas typed 2.1x / untyped 2.7x; cuDF typed 1.9x / untyped 2.0x. polars/polars-gpu untyped already engage via the native lazy executor; typed polars is a tracked follow-up. 27 new 4-engine differential + engagement tests; existing 109-test index suite green. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Y6dQEcjdazEnzuvuwf73ZL
lmeyerov
added a commit
that referenced
this pull request
Jul 19, 2026
…edges The native polars lazy chain executor (lazy/engine/polars/chain.py) attaches its synthetic edge id via with_row_index -> a fresh edge frame -> the #1658 identity guard missed, so typed-edge chains scanned on polars/polars-gpu (untyped already engaged). Apply the same rebind_edges fix at the polars augmentation site. Now ALL FOUR engines engage the index for typed AND untyped seeded chain/Cypher hops. Measured (dgx, 500k/4M/4-types, index==scan): polars typed 1.3x / untyped 5.4x; polars-gpu typed engaged / untyped 12.4x. Broadened the engagement test to all engines; 138 index tests pass (109 existing + 29 new), 4-engine. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Y6dQEcjdazEnzuvuwf73ZL
This was referenced Jul 21, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stack position
Stacks on #1667 (polars-engine followups). Full spine:
master → #1666 (3VL) → #1660 (polars engine + GPU target; #1655 folded in) → #1667 (followups) → **this PR**.Restacked 2026-07-02: the followups PR proved independent of this index layer, so the most
experimental PR (this one) moved to the top of the spine — reviewable against the final
engine surface its hooks land on. Additive — no change to default (un-indexed) behavior
on any engine.
What this does
Opt-in, pay-as-you-go CSR adjacency / node-id indexes → seeded traversal becomes
O(degree), flat in graph size. Sidecar over edge row positions (never reorders
.edges/.nodes), fingerprint-validated by object IDENTITY (a.edges()rebind safelyinvalidates → scan fallback, never a wrong answer). Three uniform surfaces (Python /
Cypher DDL
CREATE/DROP/SHOW GFQL INDEX/ JSON wire) +gfql(..., index_policy=...).Engine-polymorphic: numpy host arrays for pandas/polars, cupy on-device for cudf.
Hooked at every seeded scan site; falls back to scan for uncovered features
(edge/source/dest match,
target_wave_front,min_hops>1, labeling, missing/duplicatenode ids, missing endpoints).
A 2-agent adversarial review reproduced wrong-answer divergences from the un-indexed
scan oracle that the original parity scenarios missed (the cost gate masks them unless
≥3 distinct seed keys hit the index path). All fixed — the index now matches scan or
falls back, never a wrong answer (+6 engine-parametrized differential regression tests):
max_hopswas silently ignored (hop(max_hops=3)ran 1 hop) → now honored.id(df)fingerprint recycling, int32/int64 seed narrowing, and apolars/cudf seed-conversion crash under
force→ all fixed.is_indeprecation → order-preserving semi-join;DROP GFQL INDEX <custom-name>silent no-op → resolve-by-name + raise.Results (dgx-spark, deg-8, warm median) — OUR reproduced numbers
Every benchmarked cell is GUARDED: the timing is discarded unless (a) the index path
was actually taken (
index_trace) AND (b) the index result == the scan result. No cellfailed either guard.
Seeded warm latency is FLAT in N (O(degree)) while the scan is O(E):
(10× more edges → pandas index latency 0.124 → 0.122 ms = flat, while the scan grows 10× = O(E). That is the O(degree) proof.)
cuDF / polars-GPU are flat but floored ~3 ms (cudf) / sub-ms (polars-gpu) by GPU kernel
launch — selective traversal is an indexing problem, not a compute one (CPU wins it).
vs graph DBs (0.8M, identical graph/seed, matched answer counts):
(Adversarial note: the original neo4j SEL2 query was exactly-2-hop — 70 rows vs the
1–2-hop 78 — and undercounted; re-run with
*1..2for a fair 78-row comparison.)Tests
graphistry/tests/compute/gfql/index/test_index.py: 75 passed on dgx (pandas/cudf/polars/polars-gpu), incl. 6 new adversarial regression tests (max_hops, dup ids, missing
endpoint, node order, mixed dtype, stale rebind) + drop-by-custom-name.
🤖 Generated with Claude Code
Review notes
dc4f9da3):DROP GFQL INDEX [IF EXISTS]semantics wired end-to-end (wiremissing_okwas serialized-but-ignored; plain DROP of a missing index now raises, IF EXISTS is a no-op, +test); dead utility code removed; internal review-wave codes scrubbed from production comments; real docstrings on theComputeMixinindex API incl. the previously-undocumentedgfql(index_policy=...); benchmark catalog added tobenchmarks/gfql/README.md(which harness pair backs the published vs-Kuzu claims); bulk-OLAP polars-gpu cell fixed to GPU-or-error.