From 18dc21f491dbdde38ebb57511c4e4fde456844a5 Mon Sep 17 00:00:00 2001 From: Leo Meyerovich Date: Sat, 26 Jul 2025 00:22:04 -0700 Subject: [PATCH 1/7] docs(gfql): add Let bindings to Language Spec, Quick Reference, and 10 Minutes tutorial - Add Let operations grammar (let_op, binding, chain_ref) to EBNF in language.md - Update high-level discussion from 'Chains' to 'Chains and DAGs' - Add 'Let Bindings (DAG Patterns)' section to quick.rst with examples - Add 'Advanced: Let Bindings for Reusable Patterns' to 10 minutes tutorial These changes make Let bindings discoverable in the key user-facing docs. --- docs/source/gfql/about.rst | 22 ++++++++++++++++++++++ docs/source/gfql/quick.rst | 22 ++++++++++++++++++++++ docs/source/gfql/spec/language.md | 12 +++++++++--- 3 files changed, 53 insertions(+), 3 deletions(-) diff --git a/docs/source/gfql/about.rst b/docs/source/gfql/about.rst index f9f633ec23..d47eef4ac2 100644 --- a/docs/source/gfql/about.rst +++ b/docs/source/gfql/about.rst @@ -338,11 +338,33 @@ Additional parameters enable controlling options such as the execution `engine` Conclusion and Next Steps ------------------------- +9. Advanced: Let Bindings for Reusable Patterns +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +For complex analysis requiring reusable components, use Let bindings to create DAG patterns: + +**Example: Multi-step investigation with named components** + +:: + + investigation = g.chain_let({ + 'suspects': n({'risk_score': gt(8)}), + 'contacts': ref('suspects').chain([e_undirected(), n()]), + 'evidence': ref('contacts').chain([e_forward({'type': 'transaction'}), n()]) + }) + +**Explanation:** + +- `chain_let()` creates named bindings that can reference each other. +- `ref('suspects')` references the named suspects pattern. +- Enables complex investigations with reusable, composable parts. + Congratulations! You've covered the basics of GFQL in just 10 minutes. You've learned how to: - Query and filter nodes and edges using GFQL. - Chain multiple hops and apply advanced predicates. - Leverage GPU acceleration for high-performance graph querying. +- Create reusable patterns with Let bindings for complex analysis. - Integrate GFQL with graph algorithms and visualization tools. **Next Steps:** diff --git a/docs/source/gfql/quick.rst b/docs/source/gfql/quick.rst index 23694998ed..4cf0aaded5 100644 --- a/docs/source/gfql/quick.rst +++ b/docs/source/gfql/quick.rst @@ -201,6 +201,28 @@ Combined Examples n(query="status == 'active'") ]) +Let Bindings (DAG Patterns) +---------------------------- + +- **Basic Let syntax:** + + .. code-block:: python + + g.chain_let({ + 'persons': n({'type': 'person'}), + 'friends': ref('persons').chain([e_forward({'rel': 'friend'}), n()]) + }) + +- **Complex analysis with reusable components:** + + .. code-block:: python + + g.chain_let({ + 'suspects': n({'risk_score': gt(7)}), + 'contacts': ref('suspects').chain([e_undirected(), n()]), + 'final': ref('contacts').chain([n({'active': True})]) + }) + GPU Acceleration ---------------- diff --git a/docs/source/gfql/spec/language.md b/docs/source/gfql/spec/language.md index 65fae87b4b..52ea9a9025 100644 --- a/docs/source/gfql/spec/language.md +++ b/docs/source/gfql/spec/language.md @@ -45,10 +45,11 @@ GFQL programs are declarative graph-to-graph transformations: - Enable use cases like search, filter, enrich, and traverse - Express *what* to find (ex: Cypher), not *how* to find it (ex: Gremlin) -#### Chains +#### Chains and DAGs Path pattern expressions for matching graph structures: -- Express graph patterns as sequences of node and edge matching operations +- **Chains**: Express linear graph patterns as sequences of node and edge matching operations +- **DAGs**: Use Let bindings to define reusable named operations and complex directed acyclic patterns - Similar to Cypher patterns but decomposed into composable steps - Define paths through the graph: start nodes → edges → end nodes - Each operation refines the pattern match based on previous results @@ -87,7 +88,12 @@ query ::= chain chain ::= "[" operation ("," operation)* "]" (* Operations *) -operation ::= node_matcher | edge_matcher +operation ::= node_matcher | edge_matcher | let_op | chain_ref + +(* Let bindings for DAG patterns *) +let_op ::= "let(" "{" binding ("," binding)* "}" ")" +binding ::= identifier ":" operation +chain_ref ::= "ref(" identifier ("," "[" operation ("," operation)* "]")? ")" (* Node Matcher *) node_matcher ::= "n(" node_params? ")" From ae4b20c9b380c1a4333039579db963143725086f Mon Sep 17 00:00:00 2001 From: Leo Meyerovich Date: Sat, 26 Jul 2025 00:43:01 -0700 Subject: [PATCH 2/7] docs(gfql): add Let bindings to Cypher mapping and translation guide MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - Add comprehensive WITH → Let section to cypher_mapping.md with Python/Wire Protocol examples - Add Complex Pattern Reuse section to translate.rst comparing SQL/Pandas/Cypher/GFQL - Remove WITH clause from 'Not Supported' section - Show GFQL DAG advantages over Cypher linear dependencies 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude --- docs/source/gfql/spec/cypher_mapping.md | 81 ++++++++++++++++++++++++- docs/source/gfql/translate.rst | 75 +++++++++++++++++++++++ 2 files changed, 154 insertions(+), 2 deletions(-) diff --git a/docs/source/gfql/spec/cypher_mapping.md b/docs/source/gfql/spec/cypher_mapping.md index f936358348..d0e6eb3582 100644 --- a/docs/source/gfql/spec/cypher_mapping.md +++ b/docs/source/gfql/spec/cypher_mapping.md @@ -201,6 +201,84 @@ analysis = (trans_df **Note:** Wire protocol returns the filtered graph; aggregations require client-side processing. +## WITH Clause Mapping: Let Bindings + +Cypher's `WITH` clause for intermediate variables maps to GFQL's Let bindings for reusable patterns. + +### Basic WITH Pattern + +**Cypher:** +```cypher +MATCH (u:User)-[:FRIEND]->(f) +WITH u, count(f) as friend_count +MATCH (u)-[:TRANSACTION]->(t:Transaction) +WHERE friend_count > 5 +``` + +**Python:** +```python +g.chain_let({ + 'social_users': n({'type': 'User'}).chain([e_forward({'type': 'FRIEND'}), n()]), + 'high_social': ref('social_users', [n({'friend_count': gt(5)})]), + 'transactions': ref('high_social').chain([e_forward({'type': 'TRANSACTION'}), n({'type': 'Transaction'})]) +}) +``` + +**Wire Protocol:** +```json +{"type": "Let", "bindings": { + "social_users": {"type": "Chain", "chain": [ + {"type": "Node", "filter_dict": {"type": "User"}}, + {"type": "Edge", "direction": "forward", "edge_match": {"type": "FRIEND"}}, + {"type": "Node"} + ]}, + "high_social": {"type": "ChainRef", "ref": "social_users", "chain": [ + {"type": "Node", "filter_dict": {"friend_count": {"type": "GT", "val": 5}}} + ]}, + "transactions": {"type": "ChainRef", "ref": "high_social", "chain": [ + {"type": "Edge", "direction": "forward", "edge_match": {"type": "TRANSACTION"}}, + {"type": "Node", "filter_dict": {"type": "Transaction"}} + ]} +}} +``` + +### Pattern Reuse + +**Cypher:** +```cypher +MATCH (p:Person {risk_score: > 8}) +WITH p as suspects +MATCH (suspects)-[:CONNECTED]-(contacts) +WITH suspects, contacts +MATCH (contacts)-[:TRANSACTION]->(evidence) +``` + +**Python:** +```python +g.chain_let({ + 'suspects': n({'type': 'Person', 'risk_score': gt(8)}), + 'contacts': ref('suspects').chain([e_undirected({'type': 'CONNECTED'}), n()]), + 'evidence': ref('contacts').chain([e_forward({'type': 'TRANSACTION'}), n()]) +}) +``` + +**Wire Protocol:** +```json +{"type": "Let", "bindings": { + "suspects": {"type": "Node", "filter_dict": {"type": "Person", "risk_score": {"type": "GT", "val": 8}}}, + "contacts": {"type": "ChainRef", "ref": "suspects", "chain": [ + {"type": "Edge", "direction": "undirected", "edge_match": {"type": "CONNECTED"}}, + {"type": "Node"} + ]}, + "evidence": {"type": "ChainRef", "ref": "contacts", "chain": [ + {"type": "Edge", "direction": "forward", "edge_match": {"type": "TRANSACTION"}}, + {"type": "Node"} + ]} +}} +``` + +**Note:** GFQL Let bindings provide more flexibility than Cypher WITH - patterns can reference multiple previous bindings and form complex DAG structures. + ## DataFrame Operations Mapping | Cypher Feature | Python DataFrame Operation | Notes | @@ -226,8 +304,7 @@ analysis = (trans_df ## Not Supported - `OPTIONAL MATCH` - No equivalent (would need outer joins) - `CREATE`, `DELETE`, `SET` - GFQL is read-only -- `WITH` clauses - Requires intermediate variables -- Multiple `MATCH` patterns - Use separate chains or joins +- Multiple disconnected `MATCH` patterns - Use separate chains or joins ## Best Practices diff --git a/docs/source/gfql/translate.rst b/docs/source/gfql/translate.rst index 86bedd4b16..8df6b2040d 100644 --- a/docs/source/gfql/translate.rst +++ b/docs/source/gfql/translate.rst @@ -543,14 +543,89 @@ This example builds on the previous one, showing how **GFQL** handles parallel e --- +Complex Pattern Reuse and DAG Structures +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +**Objective**: Execute investigations with reusable named patterns and complex DAG dependencies. +**SQL** +.. code-block:: sql + -- SQL requires multiple CTEs or temp tables for complex pattern reuse + WITH suspects AS ( + SELECT id FROM nodes WHERE risk_score > 8 + ), + contacts AS ( + SELECT DISTINCT e.dst as id + FROM suspects s + JOIN edges e ON s.id = e.src + WHERE e.type = 'connected' + ), + evidence AS ( + SELECT DISTINCT e.dst as id + FROM contacts c + JOIN edges e ON c.id = e.src + WHERE e.type = 'transaction' + ) + SELECT * FROM nodes n + WHERE n.id IN (SELECT id FROM evidence); +**Pandas** +.. code-block:: python + + # Pandas requires intermediate variables and merges + suspects = nodes_df[nodes_df['risk_score'] > 8] + + contacts = edges_df[ + (edges_df['src'].isin(suspects['id'])) & + (edges_df['type'] == 'connected') + ]['dst'].unique() + + evidence = edges_df[ + (edges_df['src'].isin(contacts)) & + (edges_df['type'] == 'transaction') + ]['dst'].unique() + + result = nodes_df[nodes_df['id'].isin(evidence)] +**Cypher** + +.. code-block:: cypher + // Cypher WITH clauses for intermediate results + MATCH (p:Person) WHERE p.risk_score > 8 + WITH collect(p) as suspects + MATCH (suspects)-[:CONNECTED]-(contacts) + WITH suspects, collect(contacts) as contact_nodes + MATCH (contact_nodes)-[:TRANSACTION]->(evidence) + RETURN evidence; + +**GFQL** + +.. code-block:: python + + from graphistry import n, e_forward, e_undirected, ref, gt + + # Reusable named patterns with DAG dependencies + investigation = g.chain_let({ + 'suspects': n({'risk_score': gt(8)}), + 'contacts': ref('suspects').chain([e_undirected({'type': 'connected'}), n()]), + 'evidence': ref('contacts').chain([e_forward({'type': 'transaction'}), n()]) + }) + + # Access any binding results + suspects_df = investigation._nodes[investigation._nodes['suspects']] + evidence_df = investigation._nodes[investigation._nodes['evidence']] + +**Explanation**: + +- **SQL/Pandas**: Require verbose intermediate variables and complex joins for pattern reuse +- **Cypher**: WITH clauses provide some reuse but limited to linear dependencies +- **GFQL**: Let bindings enable true DAG patterns where any binding can reference multiple previous bindings, providing both clarity and performance benefits through query optimization + +--- GFQL Functions and Equivalents ------------------------------ From 42b5d04370104b4c1ccaa8a164d02c1caf6470b0 Mon Sep 17 00:00:00 2001 From: Leo Meyerovich Date: Sat, 26 Jul 2025 01:22:43 -0700 Subject: [PATCH 3/7] =?UTF-8?q?docs(gfql):=20update=20ChainRef=20=E2=86=92?= =?UTF-8?q?=20Ref=20terminology=20in=20specification=20docs?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Updates documentation to match ASTChainRef → ASTRef code refactoring: Language Specification: - Update EBNF grammar: chain_ref → ref_op - Consistent terminology in grammar definitions Python Embedding: - Update section header: "ChainRef" → "Ref" Wire Protocol: - Update JSON type: "ChainRef" → "Ref" in all examples - Update Python class references: ASTChainRef → ASTRef - Update best practices: "ChainRef validation" → "Ref validation" All documentation now uses consistent "Ref" terminology matching the ASTRef code changes from commit ccdc8677. 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude --- docs/source/gfql/spec/language.md | 4 +- docs/source/gfql/spec/python_embedding.md | 262 ++++++++++++++++ docs/source/gfql/spec/wire_protocol.md | 350 +++++++++++++++++++++- 3 files changed, 613 insertions(+), 3 deletions(-) diff --git a/docs/source/gfql/spec/language.md b/docs/source/gfql/spec/language.md index 52ea9a9025..afd18c211c 100644 --- a/docs/source/gfql/spec/language.md +++ b/docs/source/gfql/spec/language.md @@ -88,12 +88,12 @@ query ::= chain chain ::= "[" operation ("," operation)* "]" (* Operations *) -operation ::= node_matcher | edge_matcher | let_op | chain_ref +operation ::= node_matcher | edge_matcher | let_op | ref_op (* Let bindings for DAG patterns *) let_op ::= "let(" "{" binding ("," binding)* "}" ")" binding ::= identifier ":" operation -chain_ref ::= "ref(" identifier ("," "[" operation ("," operation)* "]")? ")" +ref_op ::= "ref(" identifier ("," "[" operation ("," operation)* "]")? ")" (* Node Matcher *) node_matcher ::= "n(" node_params? ")" diff --git a/docs/source/gfql/spec/python_embedding.md b/docs/source/gfql/spec/python_embedding.md index cf20db1741..04c24831e2 100644 --- a/docs/source/gfql/spec/python_embedding.md +++ b/docs/source/gfql/spec/python_embedding.md @@ -53,6 +53,268 @@ nodes_df = result._nodes # Filtered nodes DataFrame edges_df = result._edges # Filtered edges DataFrame ``` +## DAG Patterns with Let Bindings + +GFQL supports directed acyclic graph (DAG) patterns using Let bindings, which allow you to define named graph operations that can reference each other. + +### Let Bindings + +```python +from graphistry import let, ref, n, e_forward + +# Define DAG patterns with named bindings +result = g.gfql(let({ + 'persons': n({'type': 'person'}), + 'adults': ref('persons', [n({'age': ge(18)})]), + 'connections': ref('adults', [ + e_forward({'type': 'knows'}), + ref('adults') # Find connections between adults + ]) +})) + +# Access individual binding results +persons_df = result._nodes[result._nodes['persons']] +adults_df = result._nodes[result._nodes['adults']] +connection_edges = result._edges[result._edges['connections']] +``` + +### Ref (Reference to Named Bindings) + +The `ref()` function creates references to named bindings within a Let: + +```python +# Basic reference - just the binding result +result = g.gfql(let({ + 'base': n({'status': 'active'}), + 'extended': ref('base') # Just references 'base' +})) + +# Reference with additional operations +result = g.gfql(let({ + 'suspects': n({'risk_score': gt(80)}), + 'lateral_movement': ref('suspects', [ + e_forward({'type': 'ssh', 'failed_attempts': gt(5)}), + n({'type': 'server'}) + ]) +})) +``` + +### Complex DAG Patterns + +```python +# Multi-level analysis pattern +result = g.gfql(let({ + # Find high-value accounts + 'high_value': n({'balance': gt(100000)}), + + # Find transactions from high-value accounts + 'high_value_txns': ref('high_value', [ + e_forward({'type': 'transaction', 'amount': gt(10000)}) + ]), + + # Find recipients of high-value transactions + 'recipients': ref('high_value_txns', [n()]), + + # Find second-hop connections + 'network': ref('recipients', [ + e_forward({'type': 'transaction'}, hops=2) + ]) +})) +``` + +### RemoteGraph (Load Remote Datasets) + +```python +from graphistry import remote_dataset + +# Load a public dataset +remote_g = remote_dataset('public-dataset-id') +result = remote_g.gfql([n({'type': 'user'})]) + +# Load a private dataset with authentication +remote_g = remote_dataset('private-dataset-id', token='auth-token') + +# Use remote dataset in Let bindings +result = g.gfql(let({ + 'remote_data': remote_dataset('dataset-123'), + 'filtered': ref('remote_data', [n({'active': True})]) +})) +``` + +## Call Operations + +GFQL supports calling Plottable methods through the `call()` function, providing a safe way to invoke graph transformations and analysis operations. + +### Basic Call Usage + +```python +from graphistry import call + +# Calculate node degrees +result = g.gfql([ + n({'type': 'person'}), + call('get_degrees', { + 'col': 'centrality', + 'col_in': 'in_centrality', + 'col_out': 'out_centrality' + }) +]) + +# Access degree columns +degree_df = result._nodes[['centrality', 'in_centrality', 'out_centrality']] +``` + +### Graph Analysis Operations + +```python +# PageRank computation +result = g.gfql([ + call('compute_cugraph', { + 'alg': 'pagerank', + 'out_col': 'pagerank_score', + 'params': {'alpha': 0.85} + }) +]) + +# Community detection +result = g.gfql([ + call('compute_cugraph', { + 'alg': 'louvain', + 'out_col': 'community' + }) +]) + +# Topological analysis +result = g.gfql([ + call('get_topological_levels', { + 'level_col': 'topo_level', + 'allow_cycles': True + }) +]) +``` + +### Layout Operations + +```python +# GPU-accelerated layout +result = g.gfql([ + call('layout_cugraph', { + 'layout': 'force_atlas2', + 'params': { + 'iterations': 500, + 'outbound_attraction_distribution': True + } + }) +]) + +# Graphviz layouts +result = g.gfql([ + call('layout_graphviz', { + 'prog': 'dot', + 'directed': True + }) +]) +``` + +### Filtering and Transformation + +```python +# Complex filtering +result = g.gfql([ + call('filter_nodes_by_dict', { + 'filter_dict': {'type': 'server', 'critical': True} + }), + call('hop', { + 'hops': 2, + 'direction': 'forward', + 'edge_match': {'port': 22} + }) +]) + +# Node transformations +result = g.gfql([ + call('collapse', { + 'column': 'department', + 'self_edges': False + }) +]) +``` + +### Visual Encoding + +```python +# Encode visual properties +result = g.gfql([ + call('encode_point_color', { + 'column': 'risk_score', + 'palette': ['green', 'yellow', 'red'], + 'as_continuous': True + }), + call('encode_point_size', { + 'column': 'importance', + 'categorical_mapping': { + 'low': 10, + 'medium': 20, + 'high': 30 + } + }) +]) +``` + +### Call with Let Bindings + +```python +from graphistry import let, ref, call + +# Combine Let bindings with Call operations +result = g.gfql(let({ + 'high_risk': n({'risk_score': gt(80)}), + 'connected': ref('high_risk', [ + e_forward({'type': 'communicates'}) + ]), + 'analyzed': call('compute_cugraph', { + 'alg': 'pagerank', + 'out_col': 'influence' + }) +})) +``` + +### Available Call Methods + +Call operations are restricted to a safelist of Plottable methods: + +- **Graph Analysis**: `get_degrees`, `get_indegrees`, `get_outdegrees`, `compute_cugraph`, `compute_igraph`, `get_topological_levels` +- **Filtering**: `filter_nodes_by_dict`, `filter_edges_by_dict`, `hop`, `drop_nodes`, `keep_nodes` +- **Transformation**: `collapse`, `prune_self_edges`, `materialize_nodes` +- **Layout**: `layout_cugraph`, `layout_igraph`, `layout_graphviz`, `fa2_layout` +- **Visual Encoding**: `encode_point_color`, `encode_edge_color`, `encode_point_size`, `encode_point_icon` +- **Metadata**: `name`, `description` + +### Call Validation + +Call operations are validated at multiple levels: + +1. **Function validation**: Only safelist methods allowed +2. **Parameter validation**: Type checking for all parameters +3. **Schema validation**: Ensures required columns exist + +```python +try: + result = g.gfql([ + call('dangerous_method', {}) # Raises E303: not in safelist + ]) +except GFQLTypeError as e: + print(f"Error: {e.message}") + +# Parameter type validation +try: + result = g.gfql([ + call('hop', {'hops': 'two'}) # Raises E201: wrong type + ]) +except GFQLTypeError as e: + print(f"Error: {e.message}") +``` + ## Engine Selection GFQL supports multiple execution engines: diff --git a/docs/source/gfql/spec/wire_protocol.md b/docs/source/gfql/spec/wire_protocol.md index beacfdf20f..04faf0b6c0 100644 --- a/docs/source/gfql/spec/wire_protocol.md +++ b/docs/source/gfql/spec/wire_protocol.md @@ -43,7 +43,7 @@ All GFQL wire protocol messages are JSON objects with a `type` field that identi ### Type Identification Each object includes a `type` field: -- Operations: `"Node"`, `"Edge"`, `"Chain"` +- Operations: `"Node"`, `"Edge"`, `"Chain"`, `"Let"`, `"Ref"`, `"RemoteGraph"`, `"Call"` - Predicates: `"GT"`, `"LT"`, `"IsIn"`, etc. - Temporal values: `"datetime"`, `"date"`, `"time"` @@ -135,6 +135,233 @@ chain([ } ``` +### Let Bindings (DAG Patterns) + +**Python**: +```python +ASTLet({ + 'persons': n({'type': 'Person'}), + 'adults': ASTRef('persons', [n({'age': ge(18)})]), + 'connections': ASTRef('adults', [ + e_forward({'type': 'knows'}), + ASTRef('adults') + ]) +}) +``` + +**Wire Format**: +```json +{ + "type": "Let", + "bindings": { + "persons": { + "type": "Node", + "filter_dict": {"type": "Person"} + }, + "adults": { + "type": "Ref", + "ref": "persons", + "chain": [{ + "type": "Node", + "filter_dict": { + "age": {"type": "GE", "val": 18} + } + }] + }, + "connections": { + "type": "Ref", + "ref": "adults", + "chain": [ + { + "type": "Edge", + "direction": "forward", + "edge_match": {"type": "knows"} + }, + { + "type": "Ref", + "ref": "adults", + "chain": [] + } + ] + } + } +} +``` + +### Ref (Reference to Named Binding) + +**Python**: +```python +ASTRef('base_pattern', [ + e_forward({'status': 'active'}), + n({'verified': True}) +]) +``` + +**Wire Format**: +```json +{ + "type": "Ref", + "ref": "base_pattern", + "chain": [ + { + "type": "Edge", + "direction": "forward", + "edge_match": {"status": "active"} + }, + { + "type": "Node", + "filter_dict": {"verified": true} + } + ] +} +``` + +### RemoteGraph (Load Remote Dataset) + +**Python**: +```python +ASTRemoteGraph('dataset-123', token='auth-token') +``` + +**Wire Format**: +```json +{ + "type": "RemoteGraph", + "dataset_id": "dataset-123", + "token": "auth-token" +} +``` + +Without token (public dataset): +```json +{ + "type": "RemoteGraph", + "dataset_id": "public-dataset-456" +} +``` + +### Call Operation + +**Python**: +```python +ASTCall('get_degrees', { + 'col': 'centrality', + 'col_in': 'in_centrality', + 'col_out': 'out_centrality' +}) +``` + +**Wire Format**: +```json +{ + "type": "Call", + "function": "get_degrees", + "params": { + "col": "centrality", + "col_in": "in_centrality", + "col_out": "out_centrality" + } +} +``` + +#### Call Operation Examples + +**PageRank computation**: +```json +{ + "type": "Call", + "function": "compute_cugraph", + "params": { + "alg": "pagerank", + "out_col": "pagerank_score", + "params": {"alpha": 0.85} + } +} +``` + +**Graph layout**: +```json +{ + "type": "Call", + "function": "layout_cugraph", + "params": { + "layout": "force_atlas2", + "params": { + "iterations": 500, + "outbound_attraction_distribution": true, + "edge_weight_influence": 1.0 + } + } +} +``` + +**Node filtering**: +```json +{ + "type": "Call", + "function": "filter_nodes_by_dict", + "params": { + "filter_dict": { + "type": "person", + "active": true + } + } +} +``` + +**Complex hop traversal**: +```json +{ + "type": "Call", + "function": "hop", + "params": { + "hops": 3, + "direction": "forward", + "edge_match": {"type": "transfer"}, + "destination_node_match": {"account_type": "checking"} + } +} +``` + +#### Available Call Methods + +The following Plottable methods are available through Call operations: + +**Graph Analysis**: +- `get_degrees`: Calculate node degrees +- `get_indegrees`: Calculate in-degrees only +- `get_outdegrees`: Calculate out-degrees only +- `compute_cugraph`: Run GPU algorithms (pagerank, louvain, etc.) +- `compute_igraph`: Run CPU algorithms +- `get_topological_levels`: Analyze DAG structure + +**Filtering & Transformation**: +- `filter_nodes_by_dict`: Filter nodes by attributes +- `filter_edges_by_dict`: Filter edges by attributes +- `hop`: Traverse graph with complex conditions +- `drop_nodes`: Remove specified nodes +- `keep_nodes`: Keep only specified nodes +- `collapse`: Merge nodes by attribute +- `prune_self_edges`: Remove self-loops +- `materialize_nodes`: Generate node table from edges + +**Layout**: +- `layout_cugraph`: GPU-accelerated layouts +- `layout_igraph`: CPU-based layouts +- `layout_graphviz`: Graphviz layouts (dot, neato, etc.) +- `fa2_layout`: ForceAtlas2 layout + +**Visual Encoding**: +- `encode_point_color`: Map node values to colors +- `encode_edge_color`: Map edge values to colors +- `encode_point_size`: Map node values to sizes +- `encode_point_icon`: Map node values to icons + +**Metadata**: +- `name`: Set visualization name +- `description`: Set visualization description + ## Predicate Serialization ### Comparison Predicates @@ -325,6 +552,121 @@ g.chain([ } ``` +### Complex DAG Pattern + +**Python**: +```python +g.gfql(ASTLet({ + 'suspicious_ips': n({'risk_score': gt(80)}), + 'lateral_movement': ASTRef('suspicious_ips', [ + e_forward({'type': 'ssh', 'failed_attempts': gt(5)}), + n({'type': 'server'}) + ]), + 'escalation': ASTRef('lateral_movement', [ + e_forward({'type': 'privilege_change'}), + n({'admin': True}) + ]) +})) +``` + +**Wire Format**: +```json +{ + "type": "Let", + "bindings": { + "suspicious_ips": { + "type": "Node", + "filter_dict": { + "risk_score": {"type": "GT", "val": 80} + } + }, + "lateral_movement": { + "type": "Ref", + "ref": "suspicious_ips", + "chain": [ + { + "type": "Edge", + "direction": "forward", + "edge_match": { + "type": "ssh", + "failed_attempts": {"type": "GT", "val": 5} + } + }, + { + "type": "Node", + "filter_dict": {"type": "server"} + } + ] + }, + "escalation": { + "type": "Ref", + "ref": "lateral_movement", + "chain": [ + { + "type": "Edge", + "direction": "forward", + "edge_match": {"type": "privilege_change"} + }, + { + "type": "Node", + "filter_dict": {"admin": true} + } + ] + } + } +} +``` + +### DAG Pattern with Call Operations + +**Python**: +```python +g.gfql(ASTLet({ + 'high_value': n({'amount': gt(100000)}), + 'connected': ASTRef('high_value', [ + e_forward({'type': 'transfer'}, hops=2) + ]), + 'analyzed': ASTCall('compute_cugraph', { + 'alg': 'pagerank', + 'out_col': 'influence_score' + }) +})) +``` + +**Wire Format**: +```json +{ + "type": "Let", + "bindings": { + "high_value": { + "type": "Node", + "filter_dict": { + "amount": {"type": "GT", "val": 100000} + } + }, + "connected": { + "type": "Ref", + "ref": "high_value", + "chain": [ + { + "type": "Edge", + "direction": "forward", + "edge_match": {"type": "transfer"}, + "hops": 2 + } + ] + }, + "analyzed": { + "type": "Call", + "function": "compute_cugraph", + "params": { + "alg": "pagerank", + "out_col": "influence_score" + } + } + } +} +``` ## Best Practices @@ -333,6 +675,12 @@ g.chain([ 3. **Handle timezones consistently**: Include timezone for datetime values when precision matters (defaults to UTC) 4. **Validate before sending**: Use JSON Schema validation 5. **Handle unknown fields**: Ignore unrecognized fields for compatibility +6. **Let bindings**: Define bindings in dependency order (referenced names must be defined first) +7. **Ref validation**: Ensure referenced names exist in the Let binding scope +8. **RemoteGraph security**: Protect authentication tokens in transit and storage +9. **Call operations**: Only use function names from the safelist +10. **Parameter validation**: Ensure Call parameters match expected types +11. **Error handling**: Call operations may fail if schema requirements aren't met ## See Also From 41858cc966d1c18c9e563c637d8bd52d2623bbe6 Mon Sep 17 00:00:00 2001 From: Leo Meyerovich Date: Sat, 26 Jul 2025 16:02:48 -0700 Subject: [PATCH 4/7] docs(gfql): Add comprehensive Let sequencing documentation and examples MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - Add "Sequencing Programs with Let" section to 10-minute guide with PageRank example - Add Let sequencing example to Overview -> Quick Examples - Add comprehensive Let examples to Remote Mode documentation - Update all chain_let references to .let() syntax - Update all .chain() calls to .gfql() in documentation - Move Let Bindings section before Predicates in Quick Reference - Update GFQL Language Spec to allow both chain and let at top level - Add Query Structure section to Wire Protocol before operations - Add Let example to hop_and_chain_graph_pattern_mining notebook - Update Call Operations headers to include "and Let Bindings" - Add Let+Call example to gfql_remote notebook - Add Python vs pure GFQL comparison in "Combining GFQL" section - Add forward reference from Multi-hop to Complex Pattern Reuse 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude --- LET_SEQUENCING_DOCS_TASKS.md | 149 ++++++++++++++++++ demos/gfql/gfql_remote.ipynb | 78 ++++++++- .../hop_and_chain_graph_pattern_mining.ipynb | 44 +++++- docs/source/10min.rst | 44 +++++- docs/source/gfql/about.rst | 49 +++++- docs/source/gfql/overview.rst | 47 +++++- docs/source/gfql/quick.rst | 63 ++++---- docs/source/gfql/remote.rst | 102 ++++++++++++ docs/source/gfql/spec/cypher_mapping.md | 4 +- docs/source/gfql/spec/language.md | 118 +++++++++++++- docs/source/gfql/spec/python_embedding.md | 2 +- docs/source/gfql/spec/wire_protocol.md | 63 ++++++++ docs/source/gfql/translate.rst | 25 +-- 13 files changed, 727 insertions(+), 61 deletions(-) create mode 100644 LET_SEQUENCING_DOCS_TASKS.md diff --git a/LET_SEQUENCING_DOCS_TASKS.md b/LET_SEQUENCING_DOCS_TASKS.md new file mode 100644 index 0000000000..a366446698 --- /dev/null +++ b/LET_SEQUENCING_DOCS_TASKS.md @@ -0,0 +1,149 @@ +# Let Sequencing Documentation Tasks + +## Raw Task List + +1. **10-minute GFQL Guide** + - Add sequencing GFQL programs section with `let` + - Include PageRank example: compute PageRank, then explore 2 hops from high-scoring nodes + - Location: Find appropriate place for ninja edit + +2. **Overview -> Quick Examples** + - Add similar `let` sequencing example + - Show the power of composing operations + +3. **Remote Mode Documentation** + - Add `let` examples (more important here due to no Python escape hatch) + - Show how to sequence operations in pure GFQL + +4. **Fix chain_let References** + - Find all "chain_let" references in docs + - Update to ".let()" syntax + - Verify correctness across all docs + +5. **Update .chain() Syntax** + - Find all .chain(...) references + - Update to .gfql(Chain(...)) + - OR: If implicit coercion .gfql([...]) => .gfql(Chain([...])) is supported: + - Do code scan to verify + - Add pytest if needed + - Document the coercion clearly + +6. **GFQL Quick Reference** + - Move .let() earlier in the reference + - Place before predicates section + - Show as fundamental composition tool + +7. **GFQL Language Spec** + - Fix top-level to show both chain and let as allowed + - Add chain+let section before Operations section + - Ensure proper hierarchy + +8. **Wire Protocol Documentation** + - Add chain+let section before operations + - Show protocol representation + +9. **hop_and_chain_graph_pattern_mining Notebook** + - Add `let` example + - Validate with docs build that runs notebooks + +10. **call_operations Documentation** + - Update title to include "and Let bindings" + - Show how Call and Let work together + +11. **gfql_remote Notebook** + - Add let+call example + - Show remote execution patterns + +12. **Multi-hop Section Updates** + - Add forward reference to "Complex Pattern Reuse" + - Improve flow between sections + +13. **"Between" Section** + - Has "Complex Pattern Reuse" + - Fix chain_let references + +14. **Combining GFQL Section** + - Start early with Python vs pure GFQL equivalents + - Show the relationship clearly + +## Reordered by PR + +All tasks should be on **PR #708** (top of docs stack): "docs(gfql): Comprehensive Let bindings and Call operations documentation" + +## Granular Implementation Steps + +### Phase 1: Preparation +1. Switch to PR #708 branch +2. Pull latest changes +3. Create working list of all files to modify + +### Phase 2: Find and Fix chain_let References +4. Search for all "chain_let" occurrences in docs +5. List each file and line number +6. Update each to ".let()" syntax +7. Verify context makes sense + +### Phase 3: Update .chain() Syntax +8. Search for all .chain() calls in docs +9. Check if implicit coercion is supported in code +10. If yes: Document coercion behavior +11. If no: Update all to .gfql(Chain(...)) + +### Phase 4: Core Documentation Updates +12. Find 10-minute GFQL guide file +13. Add "Sequencing Programs with Let" section +14. Write PageRank + 2-hop example +15. Find Overview -> Quick Examples +16. Add let sequencing example there +17. Find Remote Mode docs +18. Add comprehensive let examples for remote + +### Phase 5: Reference Documentation +19. Find GFQL Quick Reference +20. Restructure to introduce .let() before predicates +21. Find GFQL Language Spec +22. Update top-level grammar to include let +23. Add chain+let section before Operations +24. Find Wire Protocol docs +25. Add chain+let protocol section before ops + +### Phase 6: Notebook Updates +26. Find hop_and_chain_graph_pattern_mining.ipynb +27. Add let example with pattern reuse +28. Find gfql_remote.ipynb +29. Add let+call remote example +30. Update notebook outputs + +### Phase 7: Cross-references and Titles +31. Find Multi-hop section +32. Add forward reference to Complex Pattern Reuse +33. Find call_operations docs +34. Update title to include "and Let Bindings" +35. Find Combining GFQL section +36. Add early Python vs GFQL comparison + +### Phase 8: Validation +37. Run flake8/ruff linting +38. Run mypy type checking +39. Run docs build locally +40. Run notebook tests +41. Review all changes + +### Phase 9: Commit and Push +42. Stage all changes +43. Create detailed commit message +44. Push to PR #708 + +## Validation Checklist +- [ ] All chain_let replaced with .let() +- [ ] All .chain() updated appropriately +- [ ] PageRank example works in 10-min guide +- [ ] Remote examples are comprehensive +- [ ] Quick Ref has proper ordering +- [ ] Language Spec has let at top level +- [ ] Notebooks execute without errors +- [ ] Cross-references are correct +- [ ] No lint errors +- [ ] No type errors +- [ ] Docs build succeeds +- [ ] Notebook tests pass \ No newline at end of file diff --git a/demos/gfql/gfql_remote.ipynb b/demos/gfql/gfql_remote.ipynb index 11440662cd..2a0a39a032 100644 --- a/demos/gfql/gfql_remote.ipynb +++ b/demos/gfql/gfql_remote.ipynb @@ -1401,6 +1401,82 @@ "metadata": {}, "outputs": [], "source": [] + }, + { + "cell_type": "markdown", + "id": "fs1pabrqfaj", + "source": "## Combining Let Bindings with Call Operations\n\nLet bindings in GFQL allow you to create named intermediate results and compose complex operations. When combined with call operations in remote mode, you can orchestrate sophisticated graph analyses entirely on the server, minimizing data transfer and leveraging server-side GPU acceleration.", + "metadata": {} + }, + { + "cell_type": "markdown", + "id": "bs7rghntlp", + "source": "### Example 1: PageRank Analysis with Filtering\n\nThis example demonstrates using let bindings to:\n1. Compute PageRank scores\n2. Filter high-value nodes\n3. Extract subgraphs around important nodes\n4. Return results for visualization", + "metadata": {} + }, + { + "cell_type": "code", + "id": "wurwk0xplp", + "source": "# Create a more complex graph for demonstration\ncomplex_edges = pd.DataFrame({\n 's': ['a', 'b', 'c', 'd', 'e', 'f', 'a', 'b', 'c', 'd'],\n 'd': ['b', 'c', 'd', 'e', 'f', 'a', 'c', 'd', 'e', 'f'],\n 'weight': [1, 2, 1, 3, 1, 2, 1, 2, 1, 1],\n 'type': ['follow', 'mention', 'follow', 'follow', 'mention', 'follow', 'mention', 'follow', 'follow', 'mention']\n})\n\ng_complex = graphistry.edges(complex_edges, 's', 'd').upload()\nprint(f\"Uploaded graph with {len(complex_edges)} edges\")", + "metadata": {}, + "execution_count": null, + "outputs": [] + }, + { + "cell_type": "code", + "id": "mwr3948llv", + "source": "%%time\n\n# Define a complex query using let bindings with call operations\npagerank_analysis_query = {\n 'let': {\n # Step 1: Compute PageRank scores\n 'with_pagerank': {\n 'call': {\n 'method': 'compute_pagerank',\n 'args': [],\n 'kwargs': {}\n }\n },\n \n # Step 2: Filter nodes with high PageRank scores\n 'important_nodes': {\n 'pipe': [\n {'var': 'with_pagerank'},\n {\n 'n': {\n 'filter': {\n 'gte': [\n {'col': 'pagerank'},\n 0.15 # Threshold for \"important\" nodes\n ]\n }\n }\n }\n ]\n },\n \n # Step 3: Get 1-hop neighborhoods of important nodes\n 'important_neighborhoods': {\n 'pipe': [\n {'var': 'important_nodes'},\n {'e_undirected': {'hops': 1}},\n {'n': {}}\n ]\n }\n },\n \n # Return the final result\n 'in': {'var': 'important_neighborhoods'}\n}\n\n# Execute the query remotely\nresult = g_complex.chain_remote([pagerank_analysis_query])\n\nprint(f\"Result has {len(result._nodes)} nodes and {len(result._edges)} edges\")\nprint(\"\\nNodes with PageRank scores:\")\nprint(result._nodes)", + "metadata": {}, + "execution_count": null, + "outputs": [] + }, + { + "cell_type": "markdown", + "id": "jdzoowghv2q", + "source": "### Example 2: Multi-Stage Analysis with Different Edge Types\n\nThis example shows how to use let bindings to analyze different edge types separately and combine the results:", + "metadata": {} + }, + { + "cell_type": "code", + "id": "40m2wl0yn3m", + "source": "%%time\n\n# Analyze different edge types and combine results\nedge_type_analysis = {\n 'let': {\n # Analyze follow edges\n 'follow_network': {\n 'pipe': [\n {\n 'e_undirected': {\n 'filter': {\n 'eq': [\n {'col': 'type'},\n 'follow'\n ]\n }\n }\n },\n {'n': {}}\n ]\n },\n \n # Compute centrality on follow network\n 'follow_centrality': {\n 'pipe': [\n {'var': 'follow_network'},\n {\n 'call': {\n 'method': 'compute_degree_centrality',\n 'args': [],\n 'kwargs': {}\n }\n }\n ]\n },\n \n # Find mention patterns\n 'mention_edges': {\n 'e_undirected': {\n 'filter': {\n 'eq': [\n {'col': 'type'},\n 'mention'\n ]\n }\n }\n },\n \n # Get nodes that are both highly connected and frequently mentioned\n 'influential_nodes': {\n 'pipe': [\n {'var': 'follow_centrality'},\n {\n 'n': {\n 'filter': {\n 'gte': [\n {'col': 'degree_centrality'},\n 0.5\n ]\n }\n }\n },\n {'var': 'mention_edges'},\n {'n': {}}\n ]\n }\n },\n \n 'in': {'var': 'influential_nodes'}\n}\n\ninfluential_result = g_complex.chain_remote([edge_type_analysis])\n\nprint(f\"Found {len(influential_result._nodes)} influential nodes\")\nprint(f\"Connected by {len(influential_result._edges)} edges\")\nprint(\"\\nInfluential nodes with centrality scores:\")\nprint(influential_result._nodes)", + "metadata": {}, + "execution_count": null, + "outputs": [] + }, + { + "cell_type": "markdown", + "id": "5y02h8p7mkk", + "source": "### Example 3: Conditional Analysis with Let Bindings\n\nThis example demonstrates using let bindings to perform conditional analysis based on graph properties:", + "metadata": {} + }, + { + "cell_type": "code", + "id": "gmy30drvbts", + "source": "%%time\n\n# Complex analysis with multiple algorithms\ncomprehensive_analysis = {\n 'let': {\n # Base graph with all computations\n 'enriched_graph': {\n 'pipe': [\n {'call': {'method': 'compute_pagerank', 'args': [], 'kwargs': {}}},\n {'call': {'method': 'compute_degree_centrality', 'args': [], 'kwargs': {}}},\n # Add edge weight normalization\n {\n 'call': {\n 'method': 'edges',\n 'args': [],\n 'kwargs': {\n 'normalize': {\n 'weight': {\n 'min': 0,\n 'max': 1\n }\n }\n }\n }\n }\n ]\n },\n \n # Find bridge nodes (high betweenness)\n 'bridge_nodes': {\n 'pipe': [\n {'var': 'enriched_graph'},\n {\n 'n': {\n 'filter': {\n 'and': [\n {'gte': [{'col': 'pagerank'}, 0.1]},\n {'lte': [{'col': 'degree_centrality'}, 0.7]}\n ]\n }\n }\n }\n ]\n },\n \n # Find hub nodes (high degree centrality)\n 'hub_nodes': {\n 'pipe': [\n {'var': 'enriched_graph'},\n {\n 'n': {\n 'filter': {\n 'gte': [{'col': 'degree_centrality'}, 0.7]\n }\n }\n }\n ]\n },\n \n # Get connections between bridges and hubs\n 'critical_paths': {\n 'pipe': [\n {'var': 'bridge_nodes'},\n {'e_forward': {'to_nodes': {'var': 'hub_nodes'}}},\n {'n': {}}\n ]\n }\n },\n \n 'in': {'var': 'critical_paths'}\n}\n\n# Execute remotely with GPU acceleration\ncritical_paths_result = g_complex.chain_remote([comprehensive_analysis], engine='cudf')\n\nprint(f\"Critical paths network: {len(critical_paths_result._nodes)} nodes, {len(critical_paths_result._edges)} edges\")\n\n# Check if we got results\nif len(critical_paths_result._nodes) > 0:\n print(\"\\nCritical path nodes:\")\n print(critical_paths_result._nodes)\nelse:\n print(\"\\nNo critical paths found with current thresholds\")", + "metadata": {}, + "execution_count": null, + "outputs": [] + }, + { + "cell_type": "markdown", + "id": "wbt02937wz", + "source": "### Example 4: Visualization-Ready Analysis\n\nThis example shows how to prepare data for visualization by enriching it with multiple metrics and creating a focused subgraph:", + "metadata": {} + }, + { + "cell_type": "code", + "id": "4glzmgi1u3s", + "source": "%%time\n\n# Prepare visualization-ready data with all enrichments\nviz_prep_query = {\n 'let': {\n # Compute all metrics\n 'with_metrics': {\n 'pipe': [\n {'call': {'method': 'compute_pagerank', 'args': [], 'kwargs': {}}},\n {'call': {'method': 'compute_degree_centrality', 'args': [], 'kwargs': {}}},\n # Add node colors based on PageRank\n {\n 'call': {\n 'method': 'nodes',\n 'args': [],\n 'kwargs': {\n 'assign': {\n 'node_color': {\n 'case': [\n {\n 'when': {'gte': [{'col': 'pagerank'}, 0.2]},\n 'then': 65280 # Green for high PageRank\n },\n {\n 'when': {'gte': [{'col': 'pagerank'}, 0.15]},\n 'then': 16776960 # Yellow for medium\n }\n ],\n 'else': 16711680 # Red for low\n },\n 'node_size': {\n 'mul': [\n {'col': 'degree_centrality'},\n 50 # Scale factor\n ]\n }\n }\n }\n }\n }\n ]\n },\n \n # Add edge styling based on type and weight\n 'styled_graph': {\n 'pipe': [\n {'var': 'with_metrics'},\n {\n 'call': {\n 'method': 'edges',\n 'args': [],\n 'kwargs': {\n 'assign': {\n 'edge_color': {\n 'case': [\n {\n 'when': {'eq': [{'col': 'type'}, 'follow']},\n 'then': 255 # Blue for follows\n }\n ],\n 'else': 16711935 # Magenta for mentions\n },\n 'edge_weight': {\n 'col': 'weight'\n }\n }\n }\n }\n }\n ]\n },\n \n # Focus on top nodes and their connections\n 'viz_subgraph': {\n 'pipe': [\n {'var': 'styled_graph'},\n {\n 'n': {\n 'filter': {\n 'or': [\n {'gte': [{'col': 'pagerank'}, 0.15]},\n {'gte': [{'col': 'degree_centrality'}, 0.6]}\n ]\n }\n }\n },\n {'e_undirected': {'hops': 1}},\n {'n': {}}\n ]\n }\n },\n \n 'in': {'var': 'viz_subgraph'}\n}\n\n# Get visualization-ready data\nviz_result = g_complex.chain_remote([viz_prep_query])\n\nprint(f\"Visualization subgraph: {len(viz_result._nodes)} nodes, {len(viz_result._edges)} edges\")\nprint(\"\\nNodes with visualization attributes:\")\nprint(viz_result._nodes)\nprint(\"\\nEdges with styling:\")\nprint(viz_result._edges)\n\n# Ready to visualize\n# viz_result.plot() # Uncomment to create visualization", + "metadata": {}, + "execution_count": null, + "outputs": [] + }, + { + "cell_type": "markdown", + "id": "p1o68dnaemk", + "source": "### Key Benefits of Let Bindings with Remote Calls\n\n1. **Server-Side Orchestration**: All operations happen on the server, minimizing data transfer\n2. **Named Intermediate Results**: Create readable, reusable steps in complex analyses\n3. **GPU Acceleration**: Leverage server GPU for compute-intensive operations like PageRank\n4. **Composability**: Build complex workflows from simple building blocks\n5. **Efficiency**: Avoid redundant computations by reusing named results\n\nWhen working with large graphs, this approach is particularly powerful as it allows you to:\n- Perform multiple analyses without downloading intermediate results\n- Chain together different algorithms and filters\n- Prepare visualization-ready data entirely on the server\n- Return only the final, filtered results you need", + "metadata": {} } ], "metadata": { @@ -1424,4 +1500,4 @@ }, "nbformat": 4, "nbformat_minor": 5 -} +} \ No newline at end of file diff --git a/demos/more_examples/graphistry_features/hop_and_chain_graph_pattern_mining.ipynb b/demos/more_examples/graphistry_features/hop_and_chain_graph_pattern_mining.ipynb index e07f836f9d..10a3f88326 100644 --- a/demos/more_examples/graphistry_features/hop_and_chain_graph_pattern_mining.ipynb +++ b/demos/more_examples/graphistry_features/hop_and_chain_graph_pattern_mining.ipynb @@ -17,7 +17,7 @@ "This tutorial demonstrates how to use PyGraphistry's `hop()` and `gfql()` methods for graph pattern mining and traversal.\n", "\n", "**Key concepts:**\n", - "- `g.hop()`: Filter by source node \u2192 edge \u2192 destination node patterns\n", + "- `g.hop()`: Filter by source node → edge → destination node patterns\n", "- `g.gfql()`: Chain multiple node and edge filters for complex patterns\n", "- Predicates: Use comparisons, string matching, and other filters\n", "- Result labeling: Name intermediate results for analysis\n", @@ -312,7 +312,7 @@ " \n", " \n", "\n", - "

475 rows \u00d7 7 columns

\n", + "

475 rows × 7 columns

\n", "\n", "
\n", "\n", @@ -1530,13 +1530,49 @@ ] }, { - "cell_type": "code", + "cell_type": "markdown", "execution_count": null, "metadata": { "id": "w3w4RRYkWXKo" }, "outputs": [], - "source": [] + "source": "## 7. Pattern Reuse with Let Bindings\n\nThe `let` operator allows you to define named graph patterns that can be referenced multiple times in your query. This is particularly useful for:\n- Creating reusable pattern components\n- Building complex patterns from simpler building blocks\n- Avoiding repetition in pattern definitions\n\nLet's explore how to use `let` bindings for finding triangles and other complex patterns." + }, + { + "cell_type": "code", + "source": "# Finding triangles using let bindings\n# Define a reusable pattern for high-influence nodes (top 30% pagerank)\ntop_30_pr = g2._nodes.pagerank.quantile(0.7)\n\n# Find triangles of high-influence members\ng_triangles = g2.gfql([\n {\n 'let': {\n # Define a pattern for high-influence nodes\n 'influential': n({'pagerank': ge(top_30_pr)}),\n # Define a pattern for strong connections\n 'strong_edge': e_undirected({'weight': ge(0.01)})\n }\n },\n # Use the defined patterns to find triangles\n {'pattern': 'influential', 'name': 'node_a'},\n {'pattern': 'strong_edge'},\n {'pattern': 'influential', 'name': 'node_b'},\n {'pattern': 'strong_edge'},\n {'pattern': 'influential', 'name': 'node_c'},\n {'pattern': 'strong_edge'},\n {'pattern': 'influential', 'name': 'node_a'} # Close the triangle\n])\n\nprint(f\"Found {len(g_triangles._nodes)} nodes in triangles\")\nprint(f\"Found {len(g_triangles._edges)} edges in triangles\")\n\n# Visualize the triangles\ng_triangles.encode_point_color('community_infomap', as_categorical=True).plot()", + "metadata": {}, + "execution_count": null, + "outputs": [] + }, + { + "cell_type": "markdown", + "source": "### Finding Community Bridge Patterns with Let\n\nLet's use `let` to define reusable patterns for finding members who bridge different communities:", + "metadata": {} + }, + { + "cell_type": "code", + "source": "# Find members who bridge communities using let bindings\ng_community_bridges = g2.gfql([\n {\n 'let': {\n # Pattern for community 0 members\n 'community_0': n({'community_infomap': 0}),\n # Pattern for community 1 members \n 'community_1': n({'community_infomap': 1}),\n # Pattern for community 2 members\n 'community_2': n({'community_infomap': 2}),\n # Pattern for any edge\n 'any_edge': e_undirected()\n }\n },\n # Find paths from community 0 to community 1 through community 2\n {'pattern': 'community_0', 'name': 'start'},\n {'pattern': 'any_edge'},\n {'pattern': 'community_2', 'name': 'bridge'},\n {'pattern': 'any_edge'},\n {'pattern': 'community_1', 'name': 'end'}\n])\n\nprint(f\"Found {len(g_community_bridges._nodes)} nodes in bridging pattern\")\nbridges = g_community_bridges._nodes[g_community_bridges._nodes.bridge]\nprint(f\"Community 2 members acting as bridges: {list(bridges.title.values)}\")\n\n# Visualize with bridge nodes highlighted\ng_community_bridges.encode_point_color(\n 'bridge',\n as_categorical=True,\n categorical_mapping={\n True: 'red',\n False: 'lightgray'\n }\n).encode_point_size('bridge', categorical_mapping={True: 80, False: 40}).plot()", + "metadata": {}, + "execution_count": null, + "outputs": [] + }, + { + "cell_type": "markdown", + "source": "### Complex Pattern Composition with Let\n\nLet's create more sophisticated patterns by composing smaller patterns:", + "metadata": {} + }, + { + "cell_type": "code", + "source": "# Find star patterns around influential nodes\n# A star pattern is where one central node connects to multiple others\n\ng_star_patterns = g2.gfql([\n {\n 'let': {\n # Very influential nodes (top 10%)\n 'very_influential': n({'pagerank': ge(g2._nodes.pagerank.quantile(0.9))}),\n # Moderately influential nodes (top 50%)\n 'moderately_influential': n({'pagerank': ge(g2._nodes.pagerank.quantile(0.5))}),\n # Strong bidirectional connection\n 'strong_connection': e_undirected({'weight': ge(0.02)})\n }\n },\n # Find star patterns: very influential center connected to multiple moderately influential nodes\n {'pattern': 'very_influential', 'name': 'center'},\n {'pattern': 'strong_connection'},\n {'pattern': 'moderately_influential', 'name': 'spoke1'},\n # Return to center\n e_undirected(),\n {'pattern': 'very_influential', 'name': 'center'},\n {'pattern': 'strong_connection'},\n {'pattern': 'moderately_influential', 'name': 'spoke2'},\n # Return to center again\n e_undirected(),\n {'pattern': 'very_influential', 'name': 'center'},\n {'pattern': 'strong_connection'},\n {'pattern': 'moderately_influential', 'name': 'spoke3'}\n])\n\nprint(f\"Found {len(g_star_patterns._nodes)} nodes in star patterns\")\ncenters = g_star_patterns._nodes[g_star_patterns._nodes.center]\nprint(f\"Central nodes: {list(centers.title.unique())[:5]}...\") # Show first 5\n\n# Visualize with centers highlighted\ng_star_patterns.encode_point_color(\n 'center',\n as_categorical=True,\n categorical_mapping={\n True: 'gold',\n False: 'lightblue'\n }\n).encode_point_size(\n 'center',\n categorical_mapping={True: 100, False: 50}\n).plot()", + "metadata": {}, + "execution_count": null, + "outputs": [] + }, + { + "cell_type": "markdown", + "source": "### Benefits of Let Bindings\n\nThe `let` operator provides several advantages:\n\n1. **Reusability**: Define a pattern once and use it multiple times\n2. **Readability**: Give meaningful names to complex patterns\n3. **Maintainability**: Change pattern definitions in one place\n4. **Composability**: Build complex patterns from simpler components\n\nThis makes it easier to explore and mine complex graph patterns in your data!", + "metadata": {} } ], "metadata": { diff --git a/docs/source/10min.rst b/docs/source/10min.rst index 1108d988f4..2bd7c1e802 100644 --- a/docs/source/10min.rst +++ b/docs/source/10min.rst @@ -229,7 +229,7 @@ Suppose you want to focus on attacks that started with the "MS08067 (NetAPI)" vu .. code-block:: python - g2 = g1.chain([ + g2 = g1.gfql([ n(), e(edge_query="vulnName == 'MS08067 (NetAPI)' & `time(max)` > 1421430000"), n(), @@ -241,6 +241,48 @@ Suppose you want to focus on attacks that started with the "MS08067 (NetAPI)" vu This GFQL query filters the edges based on the vulnerability name and time, then returns the matching nodes and edges for visualization. +Sequencing Programs with Let +~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +For more complex analyses, GFQL's ``let`` feature allows you to create named, reusable graph patterns that can reference each other. This is particularly powerful for multi-step graph algorithms and investigations. + +**Example: PageRank-Guided Exploration** + +Imagine you want to find important nodes using PageRank, then explore everything within 2 hops of the high-scoring nodes: + +.. code-block:: python + + # Run PageRank and explore high-scoring neighborhoods + g_analysis = g1.let({ + # Step 1: Compute PageRank for all nodes + 'ranked': g1.compute_pagerank(columns=['pagerank']), + + # Step 2: Filter to high PageRank nodes (top influencers) + 'influencers': ref('ranked').gfql([ + n(node_query='pagerank > 0.02') + ]), + + # Step 3: Get 2-hop neighborhoods around influencers + 'influence_zone': ref('influencers').gfql([ + n(), + e(hops=2), + n() + ]) + }) + + # Visualize the influence zones with PageRank-based sizing + g_analysis['influence_zone'].encode_point_size('pagerank').plot() + +This example demonstrates how ``let`` enables you to: + +1. **Sequence operations**: Each step builds on previous results +2. **Name intermediate results**: Makes complex queries readable and debuggable +3. **Combine algorithms with traversals**: Mix graph algorithms (PageRank) with pattern matching +4. **Create reusable analysis pipelines**: Save and share investigation patterns + +The ``let`` syntax is especially powerful in remote mode where you can't use Python escape hatches, allowing you to express complex graph programs entirely in GFQL. + + Utilizing Hypergraphs --------------------- diff --git a/docs/source/gfql/about.rst b/docs/source/gfql/about.rst index d47eef4ac2..4898ad2f85 100644 --- a/docs/source/gfql/about.rst +++ b/docs/source/gfql/about.rst @@ -228,6 +228,51 @@ GFQL integrates seamlessly with the PyData ecosystem, allowing you to combine it 8. Combining GFQL with Graph Algorithms ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +GFQL can be combined with graph algorithms in two ways: using Python escape hatches or pure GFQL with `let` bindings. + +**Python Escape Hatch Approach:** + +:: + + # Traditional Python approach - requires local execution + # Step 1: Filter graph + g_filtered = g.gfql([n({'type': 'person'}), e(), n()]) + + # Step 2: Compute PageRank (Python escape) + g_with_pr = g_filtered.compute_pagerank(columns=['pagerank']) + + # Step 3: Filter high PageRank nodes (Python escape) + high_pr_nodes = g_with_pr._nodes[g_with_pr._nodes['pagerank'] > 0.02] + g_high_pr = g_with_pr.nodes(high_pr_nodes) + + # Step 4: Get neighborhoods + g_result = g_high_pr.gfql([n(), e(hops=2), n()]) + +**Pure GFQL Approach with Let:** + +:: + + # Pure GFQL - can run entirely on remote GPU + g_result = g.let({ + # Step 1: Filter to persons + 'persons': n({'type': 'person'}), + + # Step 2: Compute PageRank + 'ranked': ref('persons').call('compute_pagerank', {'columns': ['pagerank']}), + + # Step 3: Filter high PageRank nodes + 'influencers': ref('ranked').gfql([n(node_query='pagerank > 0.02')]), + + # Step 4: Get 2-hop neighborhoods + 'influence_zones': ref('influencers').gfql([n(), e(hops=2), n()]) + })['influence_zones'] + +The pure GFQL approach with `let` is especially powerful for: +- **Remote execution**: Entire computation stays on the GPU server +- **Composability**: Named intermediate results can be reused +- **Readability**: Clear step-by-step logic +- **Performance**: No data movement between steps + **Example: Compute PageRank on the resulting graph** :: @@ -347,7 +392,7 @@ For complex analysis requiring reusable components, use Let bindings to create D :: - investigation = g.chain_let({ + investigation = g.let({ 'suspects': n({'risk_score': gt(8)}), 'contacts': ref('suspects').chain([e_undirected(), n()]), 'evidence': ref('contacts').chain([e_forward({'type': 'transaction'}), n()]) @@ -355,7 +400,7 @@ For complex analysis requiring reusable components, use Let bindings to create D **Explanation:** -- `chain_let()` creates named bindings that can reference each other. +- `let()` creates named bindings that can reference each other. - `ref('suspects')` references the named suspects pattern. - Enables complex investigations with reusable, composable parts. diff --git a/docs/source/gfql/overview.rst b/docs/source/gfql/overview.rst index bbedaf9414..57daff0300 100644 --- a/docs/source/gfql/overview.rst +++ b/docs/source/gfql/overview.rst @@ -64,7 +64,7 @@ Example: Find all nodes where the `type` is `"person"`. from graphistry import n - people_nodes_df = g.chain([ n({"type": "person"}) ])._nodes + people_nodes_df = g.gfql([ n({"type": "person"}) ])._nodes print('Number of person nodes:', len(people_nodes_df)) **Visualize 2-Hop Edge Sequences with an Attribute** @@ -75,7 +75,7 @@ Example: Find 2-hop paths where edges have `"interesting": True`. from graphistry import n, e_forward - g_2_hops = g.chain([n(), e_forward({"interesting": True}, hops=2) ]) + g_2_hops = g.gfql([n(), e_forward({"interesting": True}, hops=2) ]) g_2_hops.plot() **Find Nodes 1-2 Hops Away and Label Each Hop** @@ -86,7 +86,7 @@ Example: Find nodes up to 2 hops away from node `"a"` and label each hop. from graphistry import n, e_undirected - g_2_hops = g.chain([ + g_2_hops = g.gfql([ n({g._node: "a"}), e_undirected(name="hop1"), e_undirected(name="hop2") @@ -106,12 +106,12 @@ Example: Find recent transactions using temporal predicates. import pandas as pd # Find transactions after a specific date - recent = g.chain([ + recent = g.gfql([ n(edge_match={"timestamp": gt(pd.Timestamp("2023-01-01"))}) ]) # Find transactions in a date range during business hours - business_hours_txns = g.chain([ + business_hours_txns = g.gfql([ n(edge_match={ "date": between(date(2023, 6, 1), date(2023, 6, 30)), "time": between(time(9, 0), time(17, 0)) @@ -126,7 +126,7 @@ Example: Find transaction nodes between two kinds of risky nodes. from graphistry import n, e_forward, e_reverse - g_risky = g.chain([ + g_risky = g.gfql([ n({"risk1": True}), e_forward(to_fixed_point=True), n({"type": "transaction"}, name="hit"), @@ -144,7 +144,7 @@ Example: Filter nodes and edges by multiple types. from graphistry import n, e_forward, e_reverse, is_in - g_filtered = g.chain([ + g_filtered = g.gfql([ n({"type": is_in(["person", "company"])}), e_forward({"e_type": is_in(["owns", "reviews"])}, to_fixed_point=True), n({"type": is_in(["transaction", "account"])}, name="hit"), @@ -154,6 +154,39 @@ Example: Filter nodes and edges by multiple types. hits = g_filtered._nodes[ g_filtered._nodes["hit"] == True ] print('Number of filtered hits:', len(hits)) +**Sequence Complex Analysis with Let** + +Example: Use ``let`` to create reusable named patterns for multi-step graph analysis. + +.. code-block:: python + + from graphistry import n, e_forward, ref + + # PageRank-guided analysis: find influencers and their impact zones + analysis = g.let({ + # Compute centrality metrics + 'ranked': g.compute_pagerank(columns=['pagerank']), + + # Find high-influence nodes + 'influencers': ref('ranked').gfql([ + n(node_query='pagerank > 0.01') + ]), + + # Analyze their immediate networks + 'influence_network': ref('influencers').gfql([ + n(), + e_forward(hops=2), + n(name='influenced') + ]) + }) + + # Access results + influential_nodes = analysis['influencers']._nodes + print(f'Found {len(influential_nodes)} influencers') + + # Visualize the influence network + analysis['influence_network'].encode_point_size('pagerank').plot() + Leveraging GPU Acceleration ~~~~~~~~~~~~~~~~~~~~~~~~~~~~ diff --git a/docs/source/gfql/quick.rst b/docs/source/gfql/quick.rst index 4cf0aaded5..487f1dd091 100644 --- a/docs/source/gfql/quick.rst +++ b/docs/source/gfql/quick.rst @@ -127,6 +127,28 @@ Edge Matchers - :class:`e_reverse `: Same as :class:`e_forward `, but traverses in reverse. - :class:`e `: Traverses edges regardless of direction. +Let Bindings (DAG Patterns) +---------------------------- + +- **Basic Let syntax:** + + .. code-block:: python + + g.let({ + 'persons': n({'type': 'person'}), + 'friends': ref('persons').gfql([e_forward({'rel': 'friend'}), n()]) + }) + +- **Complex analysis with reusable components:** + + .. code-block:: python + + g.let({ + 'suspects': n({'risk_score': gt(7)}), + 'contacts': ref('suspects').gfql([e_undirected(), n()]), + 'final': ref('contacts').gfql([n({'active': True})]) + }) + Predicates ----------- @@ -153,7 +175,7 @@ Combined Examples .. code-block:: python - g.chain([ + g.gfql([ n({"type": "person"}), e_forward({"status": "active"}), n({"type": "transaction"}) @@ -163,7 +185,7 @@ Combined Examples .. code-block:: python - g.chain([ + g.gfql([ n({"id": "start_node"}, name="start"), e_forward(name="edge1"), n({"level": 2}, name="middle"), @@ -175,7 +197,7 @@ Combined Examples .. code-block:: python - g.chain([ + g.gfql([ n({"status": "infected"}), e_forward(to_fixed_point=True), n(name="reachable") @@ -185,7 +207,7 @@ Combined Examples .. code-block:: python - g.chain([ + g.gfql([ n({"type": is_in(["server", "database"])}), e_undirected({"protocol": "TCP"}, hops=3), n(query="risk_level >= 8") @@ -195,33 +217,12 @@ Combined Examples .. code-block:: python - g.chain([ + g.gfql([ n(query="age > 30 and country == 'USA'"), e_forward(edge_query="weight > 5"), n(query="status == 'active'") ]) -Let Bindings (DAG Patterns) ----------------------------- - -- **Basic Let syntax:** - - .. code-block:: python - - g.chain_let({ - 'persons': n({'type': 'person'}), - 'friends': ref('persons').chain([e_forward({'rel': 'friend'}), n()]) - }) - -- **Complex analysis with reusable components:** - - .. code-block:: python - - g.chain_let({ - 'suspects': n({'risk_score': gt(7)}), - 'contacts': ref('suspects').chain([e_undirected(), n()]), - 'final': ref('contacts').chain([n({'active': True})]) - }) GPU Acceleration ---------------- @@ -230,7 +231,7 @@ GPU Acceleration .. code-block:: python - g.chain([...], engine='cudf') + g.gfql([...], engine='cudf') - **Example with cuDF DataFrames:** @@ -242,7 +243,7 @@ GPU Acceleration n_gdf = cudf.from_pandas(node_df) g = graphistry.nodes(n_gdf, 'node_id').edges(e_gdf, 'src', 'dst') - g.chain([...], engine='cudf') + g.gfql([...], engine='cudf') Remote Mode ----------- @@ -420,7 +421,7 @@ Examples at a Glance .. code-block:: python - g.chain([ + g.gfql([ n({g._node: "Alice"}), e_undirected(hops=3), n({g._node: "Bob"}) @@ -442,7 +443,7 @@ Examples at a Glance .. code-block:: python - g.chain([ + g.gfql([ n({"community": "A"}), e_undirected(hops=2), n({"community": "B"}, name="bridge_nodes") @@ -452,7 +453,7 @@ Examples at a Glance .. code-block:: python - g.chain([ + g.gfql([ n(query="age >= 18"), e_forward(edge_query="interaction == 'message'"), n(query="location == 'NYC'") diff --git a/docs/source/gfql/remote.rst b/docs/source/gfql/remote.rst index ad3f040cd9..85b9aa5886 100644 --- a/docs/source/gfql/remote.rst +++ b/docs/source/gfql/remote.rst @@ -208,3 +208,105 @@ Run Python on an existing graph, return JSON obj = g.remote_python_json(first_n_edges_shape) assert obj['num_edges'] == 10 + + +Using Let for Complex Remote Queries +------------------------------------ + +The ``let`` feature is particularly powerful in remote mode where you cannot use Python escape hatches. It allows you to express complex multi-step graph programs entirely in GFQL. + +Basic Let Usage +~~~~~~~~~~~~~~~ + +.. code-block:: python + + from graphistry import n, e_forward, ref + + # Complex analysis with named, reusable patterns + analysis = g1.gfql_remote({ + # Find suspicious accounts + 'suspicious': n({'risk_score': {'$gt': 0.8}}), + + # Get their transaction network + 'tx_network': ref('suspicious').gfql([ + n(), + e_forward({'type': 'transaction'}), + n() + ]), + + # Find high-value transactions in that network + 'high_value': ref('tx_network').gfql([ + e({'amount': {'$gt': 10000}}) + ]) + }) + + # Access individual results + suspicious_accounts = analysis['suspicious'] + high_value_txns = analysis['high_value'] + +PageRank-Guided Remote Analysis +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Combine graph algorithms with pattern matching in a single remote query: + +.. code-block:: python + + # Run PageRank and explore influential neighborhoods + investigation = g1.gfql_remote({ + # Compute centrality metrics remotely + 'ranked': g1.compute_pagerank(columns=['pagerank']), + + # Find top influencers + 'influencers': ref('ranked').gfql([ + n(node_query='pagerank > 0.02') + ]), + + # Get 2-hop neighborhoods + 'influence_zones': ref('influencers').gfql([ + n(), + e_forward(hops=2), + n(name='influenced') + ]), + + # Find transactions between influencers + 'influencer_txns': ref('influencers').gfql([ + n(), + e_forward({'type': 'transaction'}), + n({'_n': ref('influencers')._nodes.index}) + ]) + }, output='influence_zones') # Return only the influence zones + + # Visualize with PageRank-based sizing + investigation.encode_point_size('pagerank').plot() + +Remote-Only Operations +~~~~~~~~~~~~~~~~~~~~~ + +Some operations are only practical in remote mode due to data size: + +.. code-block:: python + + # Large-scale pattern mining + patterns = g1.gfql_remote({ + # Find all triangles (computationally intensive) + 'triangles': g1.gfql([ + n(name='a'), + e_forward(), + n(name='b'), + e_forward(), + n(name='c'), + e_forward(), + n({'_n': ref('a')._nodes.index}) + ]), + + # Filter to specific triangle types + 'fraud_triangles': ref('triangles').gfql([ + n({'a': True, 'type': 'account'}), + e({'type': 'transaction'}), + n({'b': True, 'type': 'merchant'}), + e({'type': 'payment'}), + n({'c': True, 'type': 'account'}) + ]) + }, engine='cudf') # Force GPU for performance + + print(f"Found {len(patterns['fraud_triangles']._edges)} fraud triangles") diff --git a/docs/source/gfql/spec/cypher_mapping.md b/docs/source/gfql/spec/cypher_mapping.md index d0e6eb3582..c7a0198793 100644 --- a/docs/source/gfql/spec/cypher_mapping.md +++ b/docs/source/gfql/spec/cypher_mapping.md @@ -217,7 +217,7 @@ WHERE friend_count > 5 **Python:** ```python -g.chain_let({ +g.let({ 'social_users': n({'type': 'User'}).chain([e_forward({'type': 'FRIEND'}), n()]), 'high_social': ref('social_users', [n({'friend_count': gt(5)})]), 'transactions': ref('high_social').chain([e_forward({'type': 'TRANSACTION'}), n({'type': 'Transaction'})]) @@ -255,7 +255,7 @@ MATCH (contacts)-[:TRANSACTION]->(evidence) **Python:** ```python -g.chain_let({ +g.let({ 'suspects': n({'type': 'Person', 'risk_score': gt(8)}), 'contacts': ref('suspects').chain([e_undirected({'type': 'CONNECTED'}), n()]), 'evidence': ref('contacts').chain([e_forward({'type': 'TRANSACTION'}), n()]) diff --git a/docs/source/gfql/spec/language.md b/docs/source/gfql/spec/language.md index afd18c211c..c4ce1551c9 100644 --- a/docs/source/gfql/spec/language.md +++ b/docs/source/gfql/spec/language.md @@ -82,11 +82,14 @@ Type system matching modern data formats: :caption: GFQL Grammar in Extended Backus-Naur Form (* Entry point *) -query ::= chain +query ::= chain | let_query (* Chain - path pattern expression *) chain ::= "[" operation ("," operation)* "]" +(* Let query - DAG pattern at top level *) +let_query ::= "{" binding ("," binding)* "}" + (* Operations *) operation ::= node_matcher | edge_matcher | let_op | ref_op @@ -367,6 +370,119 @@ people_nodes = result._nodes.query("people == True") This pattern is essential for extracting specific subsets from complex graph traversals. +## Call Operations, Let Bindings, and Security + +### Call Operations and Let Bindings + +GFQL supports calling Plottable methods through the `call()` operation, providing controlled access to graph transformation and analysis capabilities: + +```python +call(function: str, params: dict) -> ASTCall +``` + +Call operations enable: +- Graph algorithms (PageRank, community detection) +- Layout computations (ForceAtlas2, Graphviz) +- Data transformations (filtering, collapsing) +- Visual encodings (color, size, icons) + +### Safelist Architecture + +For security and stability, Call operations are restricted to a predefined safelist of methods. This prevents: +- Arbitrary code execution +- Access to filesystem or network operations +- Modification of global state +- Unsafe graph operations + +#### Safelist Categories + +**Graph Analysis** +- `get_degrees`, `get_indegrees`, `get_outdegrees`: Calculate node degrees +- `compute_cugraph`: Run GPU algorithms (pagerank, louvain, etc.) +- `compute_igraph`: Run CPU algorithms +- `get_topological_levels`: Analyze DAG structure + +**Filtering & Transformation** +- `filter_nodes_by_dict`, `filter_edges_by_dict`: Filter by attributes +- `hop`: Traverse graph with conditions +- `drop_nodes`, `keep_nodes`: Node selection +- `collapse`: Merge nodes by attribute +- `prune_self_edges`: Remove self-loops +- `materialize_nodes`: Generate node table + +**Layout** +- `layout_cugraph`: GPU-accelerated layouts +- `layout_igraph`: CPU-based layouts +- `layout_graphviz`: Graphviz layouts +- `fa2_layout`: ForceAtlas2 layout + +**Visual Encoding** +- `encode_point_color`, `encode_edge_color`: Color mapping +- `encode_point_size`: Size mapping +- `encode_point_icon`: Icon mapping + +**Metadata** +- `name`: Set visualization name +- `description`: Set visualization description + +### Parameter Validation + +Call operations enforce strict parameter validation: + +1. **Type Validation**: Parameters must match expected types + ```python + call('hop', {'hops': 2}) # Valid: integer + call('hop', {'hops': 'two'}) # Error: E201 type mismatch + ``` + +2. **Required Parameters**: Missing required parameters raise errors + ```python + call('filter_nodes_by_dict', {}) # Error: E105 missing filter_dict + ``` + +3. **Unknown Parameters**: Extra parameters are rejected + ```python + call('hop', {'steps': 2}) # Error: E303 unknown parameter + ``` + +### Schema Effects + +Many Call operations modify the graph schema by adding columns: + +```python +# get_degrees adds degree columns +call('get_degrees', { + 'col': 'degree', + 'col_in': 'in_degree', + 'col_out': 'out_degree' +}) +# Schema effect: adds 3 new node columns + +# compute_cugraph adds result column +call('compute_cugraph', { + 'alg': 'pagerank', + 'out_col': 'pr_score' +}) +# Schema effect: adds 'pr_score' node column +``` + +### Security Considerations + +1. **No Direct Code Execution**: Call operations cannot execute arbitrary Python code +2. **No System Access**: Methods cannot access filesystem, network, or system resources +3. **Validated Parameters**: All inputs are type-checked and validated +4. **Deterministic Effects**: Operations have predictable, documented effects +5. **No State Mutation**: Operations return new graphs without modifying originals + +### Error Handling + +Call operations use GFQL's standard error codes: +- **E303**: Function not in safelist +- **E105**: Missing required parameter +- **E201**: Parameter type mismatch +- **E303**: Unknown parameter +- **E301**: Required column not found (runtime) + ## Best Practices 1. **Use specific filters early**: Filter nodes before traversing edges diff --git a/docs/source/gfql/spec/python_embedding.md b/docs/source/gfql/spec/python_embedding.md index 04c24831e2..45d065eea7 100644 --- a/docs/source/gfql/spec/python_embedding.md +++ b/docs/source/gfql/spec/python_embedding.md @@ -141,7 +141,7 @@ result = g.gfql(let({ })) ``` -## Call Operations +## Call Operations and Let Bindings GFQL supports calling Plottable methods through the `call()` function, providing a safe way to invoke graph transformations and analysis operations. diff --git a/docs/source/gfql/spec/wire_protocol.md b/docs/source/gfql/spec/wire_protocol.md index 04faf0b6c0..f5d59bda42 100644 --- a/docs/source/gfql/spec/wire_protocol.md +++ b/docs/source/gfql/spec/wire_protocol.md @@ -50,6 +50,69 @@ Each object includes a `type` field: This enables unambiguous deserialization and validation. +## Query Structure + +GFQL queries can be expressed as either Chains (linear patterns) or Let queries (DAG patterns with named bindings). + +### Chain Queries + +Chains represent linear graph patterns as sequences of operations: + +**Python**: +```python +g.gfql([n({"type": "person"}), e_forward(), n({"type": "company"})]) +``` + +**Wire Format**: +```json +{ + "type": "Chain", + "ops": [ + {"type": "Node", "filter_dict": {"type": "person"}}, + {"type": "Edge", "direction": "forward"}, + {"type": "Node", "filter_dict": {"type": "company"}} + ] +} +``` + +### Let Queries (DAG Patterns) + +Let queries enable complex patterns with named bindings and references: + +**Python**: +```python +g.gfql({ + "suspects": n({"risk_score": gt(8)}), + "contacts": ref("suspects").gfql([e(), n()]), + "transactions": ref("contacts").gfql([e_forward({"type": "transaction"})]) +}) +``` + +**Wire Format**: +```json +{ + "type": "Let", + "bindings": { + "suspects": {"type": "Node", "filter_dict": {"risk_score": {"type": "GT", "val": 8}}}, + "contacts": { + "type": "Ref", + "ref": "suspects", + "chain": [ + {"type": "Edge", "direction": "undirected"}, + {"type": "Node"} + ] + }, + "transactions": { + "type": "Ref", + "ref": "contacts", + "chain": [ + {"type": "Edge", "direction": "forward", "filter_dict": {"type": "transaction"}} + ] + } + } +} +``` + ## Operation Serialization ### Node Operation diff --git a/docs/source/gfql/translate.rst b/docs/source/gfql/translate.rst index 8df6b2040d..912f60234e 100644 --- a/docs/source/gfql/translate.rst +++ b/docs/source/gfql/translate.rst @@ -59,11 +59,11 @@ Finding Nodes with Specific Properties from graphistry import n # df[['id', 'type', ...]] - g.chain([ n({"type": "person"}) ])._nodes + g.gfql([ n({"type": "person"}) ])._nodes **Explanation**: -- **GFQL**: `n({"type": "person"})` filters nodes where `type` is `"person"`. `g.chain([...])` applies this filter to the graph `g`, and `._nodes` retrieves the resulting nodes. The performance is similar to that of Pandas (CPU) or cuDF (GPU). +- **GFQL**: `n({"type": "person"})` filters nodes where `type` is `"person"`. `g.gfql([...])` applies this filter to the graph `g`, and `._nodes` retrieves the resulting nodes. The performance is similar to that of Pandas (CPU) or cuDF (GPU). --- @@ -165,7 +165,7 @@ Performing Multi-Hop Traversals from graphistry import n, e_forward # df[['id', ...]] - g.chain([ + g.gfql([ n({g._node: "Alice"}), e_forward(), e_forward(), n(name='m') ])._nodes.query('m') @@ -173,6 +173,9 @@ Performing Multi-Hop Traversals - **GFQL**: Starts at node `"Alice"`, performs two forward hops, and obtains nodes two steps away. Results are in `nodes_df`. Building on the expressive and performance benefits of the previous 1-hop example, it begins adding the parallel path finding benefits of GFQL over Cypher, which benefits both CPU and GPU usage. +.. note:: + For more complex multi-hop patterns with reusable components, see the :ref:`Complex Pattern Reuse and DAG Structures` section below, which demonstrates using ``let`` to create named, composable graph traversals. + --- Filtering Edges and Nodes with Conditions @@ -208,7 +211,7 @@ Filtering Edges and Nodes with Conditions from graphistry import e_forward # df[['src', 'dst', 'weight', ...]] - g.chain([ e_forward(edge_query='weight > 0.5') ])._edges + g.gfql([ e_forward(edge_query='weight > 0.5') ])._edges **Explanation**: @@ -349,7 +352,7 @@ All Paths and Connectivity # g._edges: df[['src', 'dst', ...]] # g._nodes: df[['id', ...]] - g.chain([ + g.gfql([ n({"id": "Alice"}), e_forward( source_node_query='type == "person"', @@ -437,7 +440,7 @@ Time-Windowed Graph Analytics .. code-block:: python past_week = pd.Timestamp.now() - pd.Timedelta(7) - g.chain([ + g.gfql([ n({"id": {"$in": ["Alice", "Bob"]}}), e_forward(edge_query=f'timestamp >= "{past_week}"'), n({"id": {"$in": ["Alice", "Bob"]}}) @@ -488,7 +491,7 @@ Parallel Pathfinding from graphistry import n, e_forward # g._nodes: cudf.DataFrame[['src', 'dst', ...]] - g.chain([ + g.gfql([ n({"id": "Alice"}), e_forward(to_fixed_point=False), n({"id": is_in(["Bob", "Charlie"])}) @@ -527,7 +530,7 @@ GPU Execution from graphistry import n, e_forward # Executing pathfinding queries in parallel - g.chain([ + g.gfql([ n({"id": "Alice"}), e_forward(to_fixed_point=False), n({"id": is_in(["Bob", "Charlie"])}) @@ -609,10 +612,10 @@ Complex Pattern Reuse and DAG Structures from graphistry import n, e_forward, e_undirected, ref, gt # Reusable named patterns with DAG dependencies - investigation = g.chain_let({ + investigation = g.let({ 'suspects': n({'risk_score': gt(8)}), - 'contacts': ref('suspects').chain([e_undirected({'type': 'connected'}), n()]), - 'evidence': ref('contacts').chain([e_forward({'type': 'transaction'}), n()]) + 'contacts': ref('suspects').gfql([e_undirected({'type': 'connected'}), n()]), + 'evidence': ref('contacts').gfql([e_forward({'type': 'transaction'}), n()]) }) # Access any binding results From e6fc31708644e5a9de24bd85c533856b7aa9abd7 Mon Sep 17 00:00:00 2001 From: Leo Meyerovich Date: Sun, 27 Jul 2025 17:58:01 -0700 Subject: [PATCH 5/7] Apply PR #709b: GFQL documentation updates --- demos/gfql/gfql_remote.ipynb | 78 +------------- .../hop_and_chain_graph_pattern_mining.ipynb | 44 +------- docs/source/10min.rst | 44 +------- docs/source/gfql/about.rst | 67 ------------ docs/source/gfql/overview.rst | 47 ++------ docs/source/gfql/quick.rst | 43 ++------ docs/source/gfql/remote.rst | 102 ------------------ docs/source/gfql/spec/cypher_mapping.md | 81 +------------- docs/source/gfql/spec/language.md | 21 ++-- docs/source/gfql/spec/python_embedding.md | 4 +- docs/source/gfql/spec/wire_protocol.md | 97 +++-------------- docs/source/gfql/translate.rst | 94 ++-------------- 12 files changed, 58 insertions(+), 664 deletions(-) diff --git a/demos/gfql/gfql_remote.ipynb b/demos/gfql/gfql_remote.ipynb index 2a0a39a032..11440662cd 100644 --- a/demos/gfql/gfql_remote.ipynb +++ b/demos/gfql/gfql_remote.ipynb @@ -1401,82 +1401,6 @@ "metadata": {}, "outputs": [], "source": [] - }, - { - "cell_type": "markdown", - "id": "fs1pabrqfaj", - "source": "## Combining Let Bindings with Call Operations\n\nLet bindings in GFQL allow you to create named intermediate results and compose complex operations. When combined with call operations in remote mode, you can orchestrate sophisticated graph analyses entirely on the server, minimizing data transfer and leveraging server-side GPU acceleration.", - "metadata": {} - }, - { - "cell_type": "markdown", - "id": "bs7rghntlp", - "source": "### Example 1: PageRank Analysis with Filtering\n\nThis example demonstrates using let bindings to:\n1. Compute PageRank scores\n2. Filter high-value nodes\n3. Extract subgraphs around important nodes\n4. Return results for visualization", - "metadata": {} - }, - { - "cell_type": "code", - "id": "wurwk0xplp", - "source": "# Create a more complex graph for demonstration\ncomplex_edges = pd.DataFrame({\n 's': ['a', 'b', 'c', 'd', 'e', 'f', 'a', 'b', 'c', 'd'],\n 'd': ['b', 'c', 'd', 'e', 'f', 'a', 'c', 'd', 'e', 'f'],\n 'weight': [1, 2, 1, 3, 1, 2, 1, 2, 1, 1],\n 'type': ['follow', 'mention', 'follow', 'follow', 'mention', 'follow', 'mention', 'follow', 'follow', 'mention']\n})\n\ng_complex = graphistry.edges(complex_edges, 's', 'd').upload()\nprint(f\"Uploaded graph with {len(complex_edges)} edges\")", - "metadata": {}, - "execution_count": null, - "outputs": [] - }, - { - "cell_type": "code", - "id": "mwr3948llv", - "source": "%%time\n\n# Define a complex query using let bindings with call operations\npagerank_analysis_query = {\n 'let': {\n # Step 1: Compute PageRank scores\n 'with_pagerank': {\n 'call': {\n 'method': 'compute_pagerank',\n 'args': [],\n 'kwargs': {}\n }\n },\n \n # Step 2: Filter nodes with high PageRank scores\n 'important_nodes': {\n 'pipe': [\n {'var': 'with_pagerank'},\n {\n 'n': {\n 'filter': {\n 'gte': [\n {'col': 'pagerank'},\n 0.15 # Threshold for \"important\" nodes\n ]\n }\n }\n }\n ]\n },\n \n # Step 3: Get 1-hop neighborhoods of important nodes\n 'important_neighborhoods': {\n 'pipe': [\n {'var': 'important_nodes'},\n {'e_undirected': {'hops': 1}},\n {'n': {}}\n ]\n }\n },\n \n # Return the final result\n 'in': {'var': 'important_neighborhoods'}\n}\n\n# Execute the query remotely\nresult = g_complex.chain_remote([pagerank_analysis_query])\n\nprint(f\"Result has {len(result._nodes)} nodes and {len(result._edges)} edges\")\nprint(\"\\nNodes with PageRank scores:\")\nprint(result._nodes)", - "metadata": {}, - "execution_count": null, - "outputs": [] - }, - { - "cell_type": "markdown", - "id": "jdzoowghv2q", - "source": "### Example 2: Multi-Stage Analysis with Different Edge Types\n\nThis example shows how to use let bindings to analyze different edge types separately and combine the results:", - "metadata": {} - }, - { - "cell_type": "code", - "id": "40m2wl0yn3m", - "source": "%%time\n\n# Analyze different edge types and combine results\nedge_type_analysis = {\n 'let': {\n # Analyze follow edges\n 'follow_network': {\n 'pipe': [\n {\n 'e_undirected': {\n 'filter': {\n 'eq': [\n {'col': 'type'},\n 'follow'\n ]\n }\n }\n },\n {'n': {}}\n ]\n },\n \n # Compute centrality on follow network\n 'follow_centrality': {\n 'pipe': [\n {'var': 'follow_network'},\n {\n 'call': {\n 'method': 'compute_degree_centrality',\n 'args': [],\n 'kwargs': {}\n }\n }\n ]\n },\n \n # Find mention patterns\n 'mention_edges': {\n 'e_undirected': {\n 'filter': {\n 'eq': [\n {'col': 'type'},\n 'mention'\n ]\n }\n }\n },\n \n # Get nodes that are both highly connected and frequently mentioned\n 'influential_nodes': {\n 'pipe': [\n {'var': 'follow_centrality'},\n {\n 'n': {\n 'filter': {\n 'gte': [\n {'col': 'degree_centrality'},\n 0.5\n ]\n }\n }\n },\n {'var': 'mention_edges'},\n {'n': {}}\n ]\n }\n },\n \n 'in': {'var': 'influential_nodes'}\n}\n\ninfluential_result = g_complex.chain_remote([edge_type_analysis])\n\nprint(f\"Found {len(influential_result._nodes)} influential nodes\")\nprint(f\"Connected by {len(influential_result._edges)} edges\")\nprint(\"\\nInfluential nodes with centrality scores:\")\nprint(influential_result._nodes)", - "metadata": {}, - "execution_count": null, - "outputs": [] - }, - { - "cell_type": "markdown", - "id": "5y02h8p7mkk", - "source": "### Example 3: Conditional Analysis with Let Bindings\n\nThis example demonstrates using let bindings to perform conditional analysis based on graph properties:", - "metadata": {} - }, - { - "cell_type": "code", - "id": "gmy30drvbts", - "source": "%%time\n\n# Complex analysis with multiple algorithms\ncomprehensive_analysis = {\n 'let': {\n # Base graph with all computations\n 'enriched_graph': {\n 'pipe': [\n {'call': {'method': 'compute_pagerank', 'args': [], 'kwargs': {}}},\n {'call': {'method': 'compute_degree_centrality', 'args': [], 'kwargs': {}}},\n # Add edge weight normalization\n {\n 'call': {\n 'method': 'edges',\n 'args': [],\n 'kwargs': {\n 'normalize': {\n 'weight': {\n 'min': 0,\n 'max': 1\n }\n }\n }\n }\n }\n ]\n },\n \n # Find bridge nodes (high betweenness)\n 'bridge_nodes': {\n 'pipe': [\n {'var': 'enriched_graph'},\n {\n 'n': {\n 'filter': {\n 'and': [\n {'gte': [{'col': 'pagerank'}, 0.1]},\n {'lte': [{'col': 'degree_centrality'}, 0.7]}\n ]\n }\n }\n }\n ]\n },\n \n # Find hub nodes (high degree centrality)\n 'hub_nodes': {\n 'pipe': [\n {'var': 'enriched_graph'},\n {\n 'n': {\n 'filter': {\n 'gte': [{'col': 'degree_centrality'}, 0.7]\n }\n }\n }\n ]\n },\n \n # Get connections between bridges and hubs\n 'critical_paths': {\n 'pipe': [\n {'var': 'bridge_nodes'},\n {'e_forward': {'to_nodes': {'var': 'hub_nodes'}}},\n {'n': {}}\n ]\n }\n },\n \n 'in': {'var': 'critical_paths'}\n}\n\n# Execute remotely with GPU acceleration\ncritical_paths_result = g_complex.chain_remote([comprehensive_analysis], engine='cudf')\n\nprint(f\"Critical paths network: {len(critical_paths_result._nodes)} nodes, {len(critical_paths_result._edges)} edges\")\n\n# Check if we got results\nif len(critical_paths_result._nodes) > 0:\n print(\"\\nCritical path nodes:\")\n print(critical_paths_result._nodes)\nelse:\n print(\"\\nNo critical paths found with current thresholds\")", - "metadata": {}, - "execution_count": null, - "outputs": [] - }, - { - "cell_type": "markdown", - "id": "wbt02937wz", - "source": "### Example 4: Visualization-Ready Analysis\n\nThis example shows how to prepare data for visualization by enriching it with multiple metrics and creating a focused subgraph:", - "metadata": {} - }, - { - "cell_type": "code", - "id": "4glzmgi1u3s", - "source": "%%time\n\n# Prepare visualization-ready data with all enrichments\nviz_prep_query = {\n 'let': {\n # Compute all metrics\n 'with_metrics': {\n 'pipe': [\n {'call': {'method': 'compute_pagerank', 'args': [], 'kwargs': {}}},\n {'call': {'method': 'compute_degree_centrality', 'args': [], 'kwargs': {}}},\n # Add node colors based on PageRank\n {\n 'call': {\n 'method': 'nodes',\n 'args': [],\n 'kwargs': {\n 'assign': {\n 'node_color': {\n 'case': [\n {\n 'when': {'gte': [{'col': 'pagerank'}, 0.2]},\n 'then': 65280 # Green for high PageRank\n },\n {\n 'when': {'gte': [{'col': 'pagerank'}, 0.15]},\n 'then': 16776960 # Yellow for medium\n }\n ],\n 'else': 16711680 # Red for low\n },\n 'node_size': {\n 'mul': [\n {'col': 'degree_centrality'},\n 50 # Scale factor\n ]\n }\n }\n }\n }\n }\n ]\n },\n \n # Add edge styling based on type and weight\n 'styled_graph': {\n 'pipe': [\n {'var': 'with_metrics'},\n {\n 'call': {\n 'method': 'edges',\n 'args': [],\n 'kwargs': {\n 'assign': {\n 'edge_color': {\n 'case': [\n {\n 'when': {'eq': [{'col': 'type'}, 'follow']},\n 'then': 255 # Blue for follows\n }\n ],\n 'else': 16711935 # Magenta for mentions\n },\n 'edge_weight': {\n 'col': 'weight'\n }\n }\n }\n }\n }\n ]\n },\n \n # Focus on top nodes and their connections\n 'viz_subgraph': {\n 'pipe': [\n {'var': 'styled_graph'},\n {\n 'n': {\n 'filter': {\n 'or': [\n {'gte': [{'col': 'pagerank'}, 0.15]},\n {'gte': [{'col': 'degree_centrality'}, 0.6]}\n ]\n }\n }\n },\n {'e_undirected': {'hops': 1}},\n {'n': {}}\n ]\n }\n },\n \n 'in': {'var': 'viz_subgraph'}\n}\n\n# Get visualization-ready data\nviz_result = g_complex.chain_remote([viz_prep_query])\n\nprint(f\"Visualization subgraph: {len(viz_result._nodes)} nodes, {len(viz_result._edges)} edges\")\nprint(\"\\nNodes with visualization attributes:\")\nprint(viz_result._nodes)\nprint(\"\\nEdges with styling:\")\nprint(viz_result._edges)\n\n# Ready to visualize\n# viz_result.plot() # Uncomment to create visualization", - "metadata": {}, - "execution_count": null, - "outputs": [] - }, - { - "cell_type": "markdown", - "id": "p1o68dnaemk", - "source": "### Key Benefits of Let Bindings with Remote Calls\n\n1. **Server-Side Orchestration**: All operations happen on the server, minimizing data transfer\n2. **Named Intermediate Results**: Create readable, reusable steps in complex analyses\n3. **GPU Acceleration**: Leverage server GPU for compute-intensive operations like PageRank\n4. **Composability**: Build complex workflows from simple building blocks\n5. **Efficiency**: Avoid redundant computations by reusing named results\n\nWhen working with large graphs, this approach is particularly powerful as it allows you to:\n- Perform multiple analyses without downloading intermediate results\n- Chain together different algorithms and filters\n- Prepare visualization-ready data entirely on the server\n- Return only the final, filtered results you need", - "metadata": {} } ], "metadata": { @@ -1500,4 +1424,4 @@ }, "nbformat": 4, "nbformat_minor": 5 -} \ No newline at end of file +} diff --git a/demos/more_examples/graphistry_features/hop_and_chain_graph_pattern_mining.ipynb b/demos/more_examples/graphistry_features/hop_and_chain_graph_pattern_mining.ipynb index 10a3f88326..e07f836f9d 100644 --- a/demos/more_examples/graphistry_features/hop_and_chain_graph_pattern_mining.ipynb +++ b/demos/more_examples/graphistry_features/hop_and_chain_graph_pattern_mining.ipynb @@ -17,7 +17,7 @@ "This tutorial demonstrates how to use PyGraphistry's `hop()` and `gfql()` methods for graph pattern mining and traversal.\n", "\n", "**Key concepts:**\n", - "- `g.hop()`: Filter by source node → edge → destination node patterns\n", + "- `g.hop()`: Filter by source node \u2192 edge \u2192 destination node patterns\n", "- `g.gfql()`: Chain multiple node and edge filters for complex patterns\n", "- Predicates: Use comparisons, string matching, and other filters\n", "- Result labeling: Name intermediate results for analysis\n", @@ -312,7 +312,7 @@ " \n", " \n", "\n", - "

475 rows × 7 columns

\n", + "

475 rows \u00d7 7 columns

\n", "
\n", "
\n", "\n", @@ -1530,49 +1530,13 @@ ] }, { - "cell_type": "markdown", + "cell_type": "code", "execution_count": null, "metadata": { "id": "w3w4RRYkWXKo" }, "outputs": [], - "source": "## 7. Pattern Reuse with Let Bindings\n\nThe `let` operator allows you to define named graph patterns that can be referenced multiple times in your query. This is particularly useful for:\n- Creating reusable pattern components\n- Building complex patterns from simpler building blocks\n- Avoiding repetition in pattern definitions\n\nLet's explore how to use `let` bindings for finding triangles and other complex patterns." - }, - { - "cell_type": "code", - "source": "# Finding triangles using let bindings\n# Define a reusable pattern for high-influence nodes (top 30% pagerank)\ntop_30_pr = g2._nodes.pagerank.quantile(0.7)\n\n# Find triangles of high-influence members\ng_triangles = g2.gfql([\n {\n 'let': {\n # Define a pattern for high-influence nodes\n 'influential': n({'pagerank': ge(top_30_pr)}),\n # Define a pattern for strong connections\n 'strong_edge': e_undirected({'weight': ge(0.01)})\n }\n },\n # Use the defined patterns to find triangles\n {'pattern': 'influential', 'name': 'node_a'},\n {'pattern': 'strong_edge'},\n {'pattern': 'influential', 'name': 'node_b'},\n {'pattern': 'strong_edge'},\n {'pattern': 'influential', 'name': 'node_c'},\n {'pattern': 'strong_edge'},\n {'pattern': 'influential', 'name': 'node_a'} # Close the triangle\n])\n\nprint(f\"Found {len(g_triangles._nodes)} nodes in triangles\")\nprint(f\"Found {len(g_triangles._edges)} edges in triangles\")\n\n# Visualize the triangles\ng_triangles.encode_point_color('community_infomap', as_categorical=True).plot()", - "metadata": {}, - "execution_count": null, - "outputs": [] - }, - { - "cell_type": "markdown", - "source": "### Finding Community Bridge Patterns with Let\n\nLet's use `let` to define reusable patterns for finding members who bridge different communities:", - "metadata": {} - }, - { - "cell_type": "code", - "source": "# Find members who bridge communities using let bindings\ng_community_bridges = g2.gfql([\n {\n 'let': {\n # Pattern for community 0 members\n 'community_0': n({'community_infomap': 0}),\n # Pattern for community 1 members \n 'community_1': n({'community_infomap': 1}),\n # Pattern for community 2 members\n 'community_2': n({'community_infomap': 2}),\n # Pattern for any edge\n 'any_edge': e_undirected()\n }\n },\n # Find paths from community 0 to community 1 through community 2\n {'pattern': 'community_0', 'name': 'start'},\n {'pattern': 'any_edge'},\n {'pattern': 'community_2', 'name': 'bridge'},\n {'pattern': 'any_edge'},\n {'pattern': 'community_1', 'name': 'end'}\n])\n\nprint(f\"Found {len(g_community_bridges._nodes)} nodes in bridging pattern\")\nbridges = g_community_bridges._nodes[g_community_bridges._nodes.bridge]\nprint(f\"Community 2 members acting as bridges: {list(bridges.title.values)}\")\n\n# Visualize with bridge nodes highlighted\ng_community_bridges.encode_point_color(\n 'bridge',\n as_categorical=True,\n categorical_mapping={\n True: 'red',\n False: 'lightgray'\n }\n).encode_point_size('bridge', categorical_mapping={True: 80, False: 40}).plot()", - "metadata": {}, - "execution_count": null, - "outputs": [] - }, - { - "cell_type": "markdown", - "source": "### Complex Pattern Composition with Let\n\nLet's create more sophisticated patterns by composing smaller patterns:", - "metadata": {} - }, - { - "cell_type": "code", - "source": "# Find star patterns around influential nodes\n# A star pattern is where one central node connects to multiple others\n\ng_star_patterns = g2.gfql([\n {\n 'let': {\n # Very influential nodes (top 10%)\n 'very_influential': n({'pagerank': ge(g2._nodes.pagerank.quantile(0.9))}),\n # Moderately influential nodes (top 50%)\n 'moderately_influential': n({'pagerank': ge(g2._nodes.pagerank.quantile(0.5))}),\n # Strong bidirectional connection\n 'strong_connection': e_undirected({'weight': ge(0.02)})\n }\n },\n # Find star patterns: very influential center connected to multiple moderately influential nodes\n {'pattern': 'very_influential', 'name': 'center'},\n {'pattern': 'strong_connection'},\n {'pattern': 'moderately_influential', 'name': 'spoke1'},\n # Return to center\n e_undirected(),\n {'pattern': 'very_influential', 'name': 'center'},\n {'pattern': 'strong_connection'},\n {'pattern': 'moderately_influential', 'name': 'spoke2'},\n # Return to center again\n e_undirected(),\n {'pattern': 'very_influential', 'name': 'center'},\n {'pattern': 'strong_connection'},\n {'pattern': 'moderately_influential', 'name': 'spoke3'}\n])\n\nprint(f\"Found {len(g_star_patterns._nodes)} nodes in star patterns\")\ncenters = g_star_patterns._nodes[g_star_patterns._nodes.center]\nprint(f\"Central nodes: {list(centers.title.unique())[:5]}...\") # Show first 5\n\n# Visualize with centers highlighted\ng_star_patterns.encode_point_color(\n 'center',\n as_categorical=True,\n categorical_mapping={\n True: 'gold',\n False: 'lightblue'\n }\n).encode_point_size(\n 'center',\n categorical_mapping={True: 100, False: 50}\n).plot()", - "metadata": {}, - "execution_count": null, - "outputs": [] - }, - { - "cell_type": "markdown", - "source": "### Benefits of Let Bindings\n\nThe `let` operator provides several advantages:\n\n1. **Reusability**: Define a pattern once and use it multiple times\n2. **Readability**: Give meaningful names to complex patterns\n3. **Maintainability**: Change pattern definitions in one place\n4. **Composability**: Build complex patterns from simpler components\n\nThis makes it easier to explore and mine complex graph patterns in your data!", - "metadata": {} + "source": [] } ], "metadata": { diff --git a/docs/source/10min.rst b/docs/source/10min.rst index 2bd7c1e802..1108d988f4 100644 --- a/docs/source/10min.rst +++ b/docs/source/10min.rst @@ -229,7 +229,7 @@ Suppose you want to focus on attacks that started with the "MS08067 (NetAPI)" vu .. code-block:: python - g2 = g1.gfql([ + g2 = g1.chain([ n(), e(edge_query="vulnName == 'MS08067 (NetAPI)' & `time(max)` > 1421430000"), n(), @@ -241,48 +241,6 @@ Suppose you want to focus on attacks that started with the "MS08067 (NetAPI)" vu This GFQL query filters the edges based on the vulnerability name and time, then returns the matching nodes and edges for visualization. -Sequencing Programs with Let -~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -For more complex analyses, GFQL's ``let`` feature allows you to create named, reusable graph patterns that can reference each other. This is particularly powerful for multi-step graph algorithms and investigations. - -**Example: PageRank-Guided Exploration** - -Imagine you want to find important nodes using PageRank, then explore everything within 2 hops of the high-scoring nodes: - -.. code-block:: python - - # Run PageRank and explore high-scoring neighborhoods - g_analysis = g1.let({ - # Step 1: Compute PageRank for all nodes - 'ranked': g1.compute_pagerank(columns=['pagerank']), - - # Step 2: Filter to high PageRank nodes (top influencers) - 'influencers': ref('ranked').gfql([ - n(node_query='pagerank > 0.02') - ]), - - # Step 3: Get 2-hop neighborhoods around influencers - 'influence_zone': ref('influencers').gfql([ - n(), - e(hops=2), - n() - ]) - }) - - # Visualize the influence zones with PageRank-based sizing - g_analysis['influence_zone'].encode_point_size('pagerank').plot() - -This example demonstrates how ``let`` enables you to: - -1. **Sequence operations**: Each step builds on previous results -2. **Name intermediate results**: Makes complex queries readable and debuggable -3. **Combine algorithms with traversals**: Mix graph algorithms (PageRank) with pattern matching -4. **Create reusable analysis pipelines**: Save and share investigation patterns - -The ``let`` syntax is especially powerful in remote mode where you can't use Python escape hatches, allowing you to express complex graph programs entirely in GFQL. - - Utilizing Hypergraphs --------------------- diff --git a/docs/source/gfql/about.rst b/docs/source/gfql/about.rst index 4898ad2f85..f9f633ec23 100644 --- a/docs/source/gfql/about.rst +++ b/docs/source/gfql/about.rst @@ -228,51 +228,6 @@ GFQL integrates seamlessly with the PyData ecosystem, allowing you to combine it 8. Combining GFQL with Graph Algorithms ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -GFQL can be combined with graph algorithms in two ways: using Python escape hatches or pure GFQL with `let` bindings. - -**Python Escape Hatch Approach:** - -:: - - # Traditional Python approach - requires local execution - # Step 1: Filter graph - g_filtered = g.gfql([n({'type': 'person'}), e(), n()]) - - # Step 2: Compute PageRank (Python escape) - g_with_pr = g_filtered.compute_pagerank(columns=['pagerank']) - - # Step 3: Filter high PageRank nodes (Python escape) - high_pr_nodes = g_with_pr._nodes[g_with_pr._nodes['pagerank'] > 0.02] - g_high_pr = g_with_pr.nodes(high_pr_nodes) - - # Step 4: Get neighborhoods - g_result = g_high_pr.gfql([n(), e(hops=2), n()]) - -**Pure GFQL Approach with Let:** - -:: - - # Pure GFQL - can run entirely on remote GPU - g_result = g.let({ - # Step 1: Filter to persons - 'persons': n({'type': 'person'}), - - # Step 2: Compute PageRank - 'ranked': ref('persons').call('compute_pagerank', {'columns': ['pagerank']}), - - # Step 3: Filter high PageRank nodes - 'influencers': ref('ranked').gfql([n(node_query='pagerank > 0.02')]), - - # Step 4: Get 2-hop neighborhoods - 'influence_zones': ref('influencers').gfql([n(), e(hops=2), n()]) - })['influence_zones'] - -The pure GFQL approach with `let` is especially powerful for: -- **Remote execution**: Entire computation stays on the GPU server -- **Composability**: Named intermediate results can be reused -- **Readability**: Clear step-by-step logic -- **Performance**: No data movement between steps - **Example: Compute PageRank on the resulting graph** :: @@ -383,33 +338,11 @@ Additional parameters enable controlling options such as the execution `engine` Conclusion and Next Steps ------------------------- -9. Advanced: Let Bindings for Reusable Patterns -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -For complex analysis requiring reusable components, use Let bindings to create DAG patterns: - -**Example: Multi-step investigation with named components** - -:: - - investigation = g.let({ - 'suspects': n({'risk_score': gt(8)}), - 'contacts': ref('suspects').chain([e_undirected(), n()]), - 'evidence': ref('contacts').chain([e_forward({'type': 'transaction'}), n()]) - }) - -**Explanation:** - -- `let()` creates named bindings that can reference each other. -- `ref('suspects')` references the named suspects pattern. -- Enables complex investigations with reusable, composable parts. - Congratulations! You've covered the basics of GFQL in just 10 minutes. You've learned how to: - Query and filter nodes and edges using GFQL. - Chain multiple hops and apply advanced predicates. - Leverage GPU acceleration for high-performance graph querying. -- Create reusable patterns with Let bindings for complex analysis. - Integrate GFQL with graph algorithms and visualization tools. **Next Steps:** diff --git a/docs/source/gfql/overview.rst b/docs/source/gfql/overview.rst index 57daff0300..bbedaf9414 100644 --- a/docs/source/gfql/overview.rst +++ b/docs/source/gfql/overview.rst @@ -64,7 +64,7 @@ Example: Find all nodes where the `type` is `"person"`. from graphistry import n - people_nodes_df = g.gfql([ n({"type": "person"}) ])._nodes + people_nodes_df = g.chain([ n({"type": "person"}) ])._nodes print('Number of person nodes:', len(people_nodes_df)) **Visualize 2-Hop Edge Sequences with an Attribute** @@ -75,7 +75,7 @@ Example: Find 2-hop paths where edges have `"interesting": True`. from graphistry import n, e_forward - g_2_hops = g.gfql([n(), e_forward({"interesting": True}, hops=2) ]) + g_2_hops = g.chain([n(), e_forward({"interesting": True}, hops=2) ]) g_2_hops.plot() **Find Nodes 1-2 Hops Away and Label Each Hop** @@ -86,7 +86,7 @@ Example: Find nodes up to 2 hops away from node `"a"` and label each hop. from graphistry import n, e_undirected - g_2_hops = g.gfql([ + g_2_hops = g.chain([ n({g._node: "a"}), e_undirected(name="hop1"), e_undirected(name="hop2") @@ -106,12 +106,12 @@ Example: Find recent transactions using temporal predicates. import pandas as pd # Find transactions after a specific date - recent = g.gfql([ + recent = g.chain([ n(edge_match={"timestamp": gt(pd.Timestamp("2023-01-01"))}) ]) # Find transactions in a date range during business hours - business_hours_txns = g.gfql([ + business_hours_txns = g.chain([ n(edge_match={ "date": between(date(2023, 6, 1), date(2023, 6, 30)), "time": between(time(9, 0), time(17, 0)) @@ -126,7 +126,7 @@ Example: Find transaction nodes between two kinds of risky nodes. from graphistry import n, e_forward, e_reverse - g_risky = g.gfql([ + g_risky = g.chain([ n({"risk1": True}), e_forward(to_fixed_point=True), n({"type": "transaction"}, name="hit"), @@ -144,7 +144,7 @@ Example: Filter nodes and edges by multiple types. from graphistry import n, e_forward, e_reverse, is_in - g_filtered = g.gfql([ + g_filtered = g.chain([ n({"type": is_in(["person", "company"])}), e_forward({"e_type": is_in(["owns", "reviews"])}, to_fixed_point=True), n({"type": is_in(["transaction", "account"])}, name="hit"), @@ -154,39 +154,6 @@ Example: Filter nodes and edges by multiple types. hits = g_filtered._nodes[ g_filtered._nodes["hit"] == True ] print('Number of filtered hits:', len(hits)) -**Sequence Complex Analysis with Let** - -Example: Use ``let`` to create reusable named patterns for multi-step graph analysis. - -.. code-block:: python - - from graphistry import n, e_forward, ref - - # PageRank-guided analysis: find influencers and their impact zones - analysis = g.let({ - # Compute centrality metrics - 'ranked': g.compute_pagerank(columns=['pagerank']), - - # Find high-influence nodes - 'influencers': ref('ranked').gfql([ - n(node_query='pagerank > 0.01') - ]), - - # Analyze their immediate networks - 'influence_network': ref('influencers').gfql([ - n(), - e_forward(hops=2), - n(name='influenced') - ]) - }) - - # Access results - influential_nodes = analysis['influencers']._nodes - print(f'Found {len(influential_nodes)} influencers') - - # Visualize the influence network - analysis['influence_network'].encode_point_size('pagerank').plot() - Leveraging GPU Acceleration ~~~~~~~~~~~~~~~~~~~~~~~~~~~~ diff --git a/docs/source/gfql/quick.rst b/docs/source/gfql/quick.rst index 487f1dd091..23694998ed 100644 --- a/docs/source/gfql/quick.rst +++ b/docs/source/gfql/quick.rst @@ -127,28 +127,6 @@ Edge Matchers - :class:`e_reverse `: Same as :class:`e_forward `, but traverses in reverse. - :class:`e `: Traverses edges regardless of direction. -Let Bindings (DAG Patterns) ----------------------------- - -- **Basic Let syntax:** - - .. code-block:: python - - g.let({ - 'persons': n({'type': 'person'}), - 'friends': ref('persons').gfql([e_forward({'rel': 'friend'}), n()]) - }) - -- **Complex analysis with reusable components:** - - .. code-block:: python - - g.let({ - 'suspects': n({'risk_score': gt(7)}), - 'contacts': ref('suspects').gfql([e_undirected(), n()]), - 'final': ref('contacts').gfql([n({'active': True})]) - }) - Predicates ----------- @@ -175,7 +153,7 @@ Combined Examples .. code-block:: python - g.gfql([ + g.chain([ n({"type": "person"}), e_forward({"status": "active"}), n({"type": "transaction"}) @@ -185,7 +163,7 @@ Combined Examples .. code-block:: python - g.gfql([ + g.chain([ n({"id": "start_node"}, name="start"), e_forward(name="edge1"), n({"level": 2}, name="middle"), @@ -197,7 +175,7 @@ Combined Examples .. code-block:: python - g.gfql([ + g.chain([ n({"status": "infected"}), e_forward(to_fixed_point=True), n(name="reachable") @@ -207,7 +185,7 @@ Combined Examples .. code-block:: python - g.gfql([ + g.chain([ n({"type": is_in(["server", "database"])}), e_undirected({"protocol": "TCP"}, hops=3), n(query="risk_level >= 8") @@ -217,13 +195,12 @@ Combined Examples .. code-block:: python - g.gfql([ + g.chain([ n(query="age > 30 and country == 'USA'"), e_forward(edge_query="weight > 5"), n(query="status == 'active'") ]) - GPU Acceleration ---------------- @@ -231,7 +208,7 @@ GPU Acceleration .. code-block:: python - g.gfql([...], engine='cudf') + g.chain([...], engine='cudf') - **Example with cuDF DataFrames:** @@ -243,7 +220,7 @@ GPU Acceleration n_gdf = cudf.from_pandas(node_df) g = graphistry.nodes(n_gdf, 'node_id').edges(e_gdf, 'src', 'dst') - g.gfql([...], engine='cudf') + g.chain([...], engine='cudf') Remote Mode ----------- @@ -421,7 +398,7 @@ Examples at a Glance .. code-block:: python - g.gfql([ + g.chain([ n({g._node: "Alice"}), e_undirected(hops=3), n({g._node: "Bob"}) @@ -443,7 +420,7 @@ Examples at a Glance .. code-block:: python - g.gfql([ + g.chain([ n({"community": "A"}), e_undirected(hops=2), n({"community": "B"}, name="bridge_nodes") @@ -453,7 +430,7 @@ Examples at a Glance .. code-block:: python - g.gfql([ + g.chain([ n(query="age >= 18"), e_forward(edge_query="interaction == 'message'"), n(query="location == 'NYC'") diff --git a/docs/source/gfql/remote.rst b/docs/source/gfql/remote.rst index 85b9aa5886..ad3f040cd9 100644 --- a/docs/source/gfql/remote.rst +++ b/docs/source/gfql/remote.rst @@ -208,105 +208,3 @@ Run Python on an existing graph, return JSON obj = g.remote_python_json(first_n_edges_shape) assert obj['num_edges'] == 10 - - -Using Let for Complex Remote Queries ------------------------------------- - -The ``let`` feature is particularly powerful in remote mode where you cannot use Python escape hatches. It allows you to express complex multi-step graph programs entirely in GFQL. - -Basic Let Usage -~~~~~~~~~~~~~~~ - -.. code-block:: python - - from graphistry import n, e_forward, ref - - # Complex analysis with named, reusable patterns - analysis = g1.gfql_remote({ - # Find suspicious accounts - 'suspicious': n({'risk_score': {'$gt': 0.8}}), - - # Get their transaction network - 'tx_network': ref('suspicious').gfql([ - n(), - e_forward({'type': 'transaction'}), - n() - ]), - - # Find high-value transactions in that network - 'high_value': ref('tx_network').gfql([ - e({'amount': {'$gt': 10000}}) - ]) - }) - - # Access individual results - suspicious_accounts = analysis['suspicious'] - high_value_txns = analysis['high_value'] - -PageRank-Guided Remote Analysis -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -Combine graph algorithms with pattern matching in a single remote query: - -.. code-block:: python - - # Run PageRank and explore influential neighborhoods - investigation = g1.gfql_remote({ - # Compute centrality metrics remotely - 'ranked': g1.compute_pagerank(columns=['pagerank']), - - # Find top influencers - 'influencers': ref('ranked').gfql([ - n(node_query='pagerank > 0.02') - ]), - - # Get 2-hop neighborhoods - 'influence_zones': ref('influencers').gfql([ - n(), - e_forward(hops=2), - n(name='influenced') - ]), - - # Find transactions between influencers - 'influencer_txns': ref('influencers').gfql([ - n(), - e_forward({'type': 'transaction'}), - n({'_n': ref('influencers')._nodes.index}) - ]) - }, output='influence_zones') # Return only the influence zones - - # Visualize with PageRank-based sizing - investigation.encode_point_size('pagerank').plot() - -Remote-Only Operations -~~~~~~~~~~~~~~~~~~~~~ - -Some operations are only practical in remote mode due to data size: - -.. code-block:: python - - # Large-scale pattern mining - patterns = g1.gfql_remote({ - # Find all triangles (computationally intensive) - 'triangles': g1.gfql([ - n(name='a'), - e_forward(), - n(name='b'), - e_forward(), - n(name='c'), - e_forward(), - n({'_n': ref('a')._nodes.index}) - ]), - - # Filter to specific triangle types - 'fraud_triangles': ref('triangles').gfql([ - n({'a': True, 'type': 'account'}), - e({'type': 'transaction'}), - n({'b': True, 'type': 'merchant'}), - e({'type': 'payment'}), - n({'c': True, 'type': 'account'}) - ]) - }, engine='cudf') # Force GPU for performance - - print(f"Found {len(patterns['fraud_triangles']._edges)} fraud triangles") diff --git a/docs/source/gfql/spec/cypher_mapping.md b/docs/source/gfql/spec/cypher_mapping.md index c7a0198793..f936358348 100644 --- a/docs/source/gfql/spec/cypher_mapping.md +++ b/docs/source/gfql/spec/cypher_mapping.md @@ -201,84 +201,6 @@ analysis = (trans_df **Note:** Wire protocol returns the filtered graph; aggregations require client-side processing. -## WITH Clause Mapping: Let Bindings - -Cypher's `WITH` clause for intermediate variables maps to GFQL's Let bindings for reusable patterns. - -### Basic WITH Pattern - -**Cypher:** -```cypher -MATCH (u:User)-[:FRIEND]->(f) -WITH u, count(f) as friend_count -MATCH (u)-[:TRANSACTION]->(t:Transaction) -WHERE friend_count > 5 -``` - -**Python:** -```python -g.let({ - 'social_users': n({'type': 'User'}).chain([e_forward({'type': 'FRIEND'}), n()]), - 'high_social': ref('social_users', [n({'friend_count': gt(5)})]), - 'transactions': ref('high_social').chain([e_forward({'type': 'TRANSACTION'}), n({'type': 'Transaction'})]) -}) -``` - -**Wire Protocol:** -```json -{"type": "Let", "bindings": { - "social_users": {"type": "Chain", "chain": [ - {"type": "Node", "filter_dict": {"type": "User"}}, - {"type": "Edge", "direction": "forward", "edge_match": {"type": "FRIEND"}}, - {"type": "Node"} - ]}, - "high_social": {"type": "ChainRef", "ref": "social_users", "chain": [ - {"type": "Node", "filter_dict": {"friend_count": {"type": "GT", "val": 5}}} - ]}, - "transactions": {"type": "ChainRef", "ref": "high_social", "chain": [ - {"type": "Edge", "direction": "forward", "edge_match": {"type": "TRANSACTION"}}, - {"type": "Node", "filter_dict": {"type": "Transaction"}} - ]} -}} -``` - -### Pattern Reuse - -**Cypher:** -```cypher -MATCH (p:Person {risk_score: > 8}) -WITH p as suspects -MATCH (suspects)-[:CONNECTED]-(contacts) -WITH suspects, contacts -MATCH (contacts)-[:TRANSACTION]->(evidence) -``` - -**Python:** -```python -g.let({ - 'suspects': n({'type': 'Person', 'risk_score': gt(8)}), - 'contacts': ref('suspects').chain([e_undirected({'type': 'CONNECTED'}), n()]), - 'evidence': ref('contacts').chain([e_forward({'type': 'TRANSACTION'}), n()]) -}) -``` - -**Wire Protocol:** -```json -{"type": "Let", "bindings": { - "suspects": {"type": "Node", "filter_dict": {"type": "Person", "risk_score": {"type": "GT", "val": 8}}}, - "contacts": {"type": "ChainRef", "ref": "suspects", "chain": [ - {"type": "Edge", "direction": "undirected", "edge_match": {"type": "CONNECTED"}}, - {"type": "Node"} - ]}, - "evidence": {"type": "ChainRef", "ref": "contacts", "chain": [ - {"type": "Edge", "direction": "forward", "edge_match": {"type": "TRANSACTION"}}, - {"type": "Node"} - ]} -}} -``` - -**Note:** GFQL Let bindings provide more flexibility than Cypher WITH - patterns can reference multiple previous bindings and form complex DAG structures. - ## DataFrame Operations Mapping | Cypher Feature | Python DataFrame Operation | Notes | @@ -304,7 +226,8 @@ g.let({ ## Not Supported - `OPTIONAL MATCH` - No equivalent (would need outer joins) - `CREATE`, `DELETE`, `SET` - GFQL is read-only -- Multiple disconnected `MATCH` patterns - Use separate chains or joins +- `WITH` clauses - Requires intermediate variables +- Multiple `MATCH` patterns - Use separate chains or joins ## Best Practices diff --git a/docs/source/gfql/spec/language.md b/docs/source/gfql/spec/language.md index c4ce1551c9..66bf8a5240 100644 --- a/docs/source/gfql/spec/language.md +++ b/docs/source/gfql/spec/language.md @@ -45,11 +45,10 @@ GFQL programs are declarative graph-to-graph transformations: - Enable use cases like search, filter, enrich, and traverse - Express *what* to find (ex: Cypher), not *how* to find it (ex: Gremlin) -#### Chains and DAGs +#### Chains Path pattern expressions for matching graph structures: -- **Chains**: Express linear graph patterns as sequences of node and edge matching operations -- **DAGs**: Use Let bindings to define reusable named operations and complex directed acyclic patterns +- Express graph patterns as sequences of node and edge matching operations - Similar to Cypher patterns but decomposed into composable steps - Define paths through the graph: start nodes → edges → end nodes - Each operation refines the pattern match based on previous results @@ -82,21 +81,13 @@ Type system matching modern data formats: :caption: GFQL Grammar in Extended Backus-Naur Form (* Entry point *) -query ::= chain | let_query +query ::= chain (* Chain - path pattern expression *) chain ::= "[" operation ("," operation)* "]" -(* Let query - DAG pattern at top level *) -let_query ::= "{" binding ("," binding)* "}" - (* Operations *) -operation ::= node_matcher | edge_matcher | let_op | ref_op - -(* Let bindings for DAG patterns *) -let_op ::= "let(" "{" binding ("," binding)* "}" ")" -binding ::= identifier ":" operation -ref_op ::= "ref(" identifier ("," "[" operation ("," operation)* "]")? ")" +operation ::= node_matcher | edge_matcher (* Node Matcher *) node_matcher ::= "n(" node_params? ")" @@ -370,9 +361,9 @@ people_nodes = result._nodes.query("people == True") This pattern is essential for extracting specific subsets from complex graph traversals. -## Call Operations, Let Bindings, and Security +## Call Operations and Security -### Call Operations and Let Bindings +### Call Operations GFQL supports calling Plottable methods through the `call()` operation, providing controlled access to graph transformation and analysis capabilities: diff --git a/docs/source/gfql/spec/python_embedding.md b/docs/source/gfql/spec/python_embedding.md index 45d065eea7..9086c6db10 100644 --- a/docs/source/gfql/spec/python_embedding.md +++ b/docs/source/gfql/spec/python_embedding.md @@ -78,7 +78,7 @@ adults_df = result._nodes[result._nodes['adults']] connection_edges = result._edges[result._edges['connections']] ``` -### Ref (Reference to Named Bindings) +### ChainRef (Reference to Named Bindings) The `ref()` function creates references to named bindings within a Let: @@ -141,7 +141,7 @@ result = g.gfql(let({ })) ``` -## Call Operations and Let Bindings +## Call Operations GFQL supports calling Plottable methods through the `call()` function, providing a safe way to invoke graph transformations and analysis operations. diff --git a/docs/source/gfql/spec/wire_protocol.md b/docs/source/gfql/spec/wire_protocol.md index f5d59bda42..e91d753615 100644 --- a/docs/source/gfql/spec/wire_protocol.md +++ b/docs/source/gfql/spec/wire_protocol.md @@ -43,76 +43,13 @@ All GFQL wire protocol messages are JSON objects with a `type` field that identi ### Type Identification Each object includes a `type` field: -- Operations: `"Node"`, `"Edge"`, `"Chain"`, `"Let"`, `"Ref"`, `"RemoteGraph"`, `"Call"` +- Operations: `"Node"`, `"Edge"`, `"Chain"`, `"Let"`, `"ChainRef"`, `"RemoteGraph"`, `"Call"` - Predicates: `"GT"`, `"LT"`, `"IsIn"`, etc. - Temporal values: `"datetime"`, `"date"`, `"time"` This enables unambiguous deserialization and validation. -## Query Structure - -GFQL queries can be expressed as either Chains (linear patterns) or Let queries (DAG patterns with named bindings). - -### Chain Queries - -Chains represent linear graph patterns as sequences of operations: - -**Python**: -```python -g.gfql([n({"type": "person"}), e_forward(), n({"type": "company"})]) -``` - -**Wire Format**: -```json -{ - "type": "Chain", - "ops": [ - {"type": "Node", "filter_dict": {"type": "person"}}, - {"type": "Edge", "direction": "forward"}, - {"type": "Node", "filter_dict": {"type": "company"}} - ] -} -``` - -### Let Queries (DAG Patterns) - -Let queries enable complex patterns with named bindings and references: - -**Python**: -```python -g.gfql({ - "suspects": n({"risk_score": gt(8)}), - "contacts": ref("suspects").gfql([e(), n()]), - "transactions": ref("contacts").gfql([e_forward({"type": "transaction"})]) -}) -``` - -**Wire Format**: -```json -{ - "type": "Let", - "bindings": { - "suspects": {"type": "Node", "filter_dict": {"risk_score": {"type": "GT", "val": 8}}}, - "contacts": { - "type": "Ref", - "ref": "suspects", - "chain": [ - {"type": "Edge", "direction": "undirected"}, - {"type": "Node"} - ] - }, - "transactions": { - "type": "Ref", - "ref": "contacts", - "chain": [ - {"type": "Edge", "direction": "forward", "filter_dict": {"type": "transaction"}} - ] - } - } -} -``` - ## Operation Serialization ### Node Operation @@ -204,10 +141,10 @@ chain([ ```python ASTLet({ 'persons': n({'type': 'Person'}), - 'adults': ASTRef('persons', [n({'age': ge(18)})]), - 'connections': ASTRef('adults', [ + 'adults': ASTChainRef('persons', [n({'age': ge(18)})]), + 'connections': ASTChainRef('adults', [ e_forward({'type': 'knows'}), - ASTRef('adults') + ASTChainRef('adults') ]) }) ``` @@ -222,7 +159,7 @@ ASTLet({ "filter_dict": {"type": "Person"} }, "adults": { - "type": "Ref", + "type": "ChainRef", "ref": "persons", "chain": [{ "type": "Node", @@ -232,7 +169,7 @@ ASTLet({ }] }, "connections": { - "type": "Ref", + "type": "ChainRef", "ref": "adults", "chain": [ { @@ -241,7 +178,7 @@ ASTLet({ "edge_match": {"type": "knows"} }, { - "type": "Ref", + "type": "ChainRef", "ref": "adults", "chain": [] } @@ -251,11 +188,11 @@ ASTLet({ } ``` -### Ref (Reference to Named Binding) +### ChainRef (Reference to Named Binding) **Python**: ```python -ASTRef('base_pattern', [ +ASTChainRef('base_pattern', [ e_forward({'status': 'active'}), n({'verified': True}) ]) @@ -264,7 +201,7 @@ ASTRef('base_pattern', [ **Wire Format**: ```json { - "type": "Ref", + "type": "ChainRef", "ref": "base_pattern", "chain": [ { @@ -621,11 +558,11 @@ g.chain([ ```python g.gfql(ASTLet({ 'suspicious_ips': n({'risk_score': gt(80)}), - 'lateral_movement': ASTRef('suspicious_ips', [ + 'lateral_movement': ASTChainRef('suspicious_ips', [ e_forward({'type': 'ssh', 'failed_attempts': gt(5)}), n({'type': 'server'}) ]), - 'escalation': ASTRef('lateral_movement', [ + 'escalation': ASTChainRef('lateral_movement', [ e_forward({'type': 'privilege_change'}), n({'admin': True}) ]) @@ -644,7 +581,7 @@ g.gfql(ASTLet({ } }, "lateral_movement": { - "type": "Ref", + "type": "ChainRef", "ref": "suspicious_ips", "chain": [ { @@ -662,7 +599,7 @@ g.gfql(ASTLet({ ] }, "escalation": { - "type": "Ref", + "type": "ChainRef", "ref": "lateral_movement", "chain": [ { @@ -686,7 +623,7 @@ g.gfql(ASTLet({ ```python g.gfql(ASTLet({ 'high_value': n({'amount': gt(100000)}), - 'connected': ASTRef('high_value', [ + 'connected': ASTChainRef('high_value', [ e_forward({'type': 'transfer'}, hops=2) ]), 'analyzed': ASTCall('compute_cugraph', { @@ -708,7 +645,7 @@ g.gfql(ASTLet({ } }, "connected": { - "type": "Ref", + "type": "ChainRef", "ref": "high_value", "chain": [ { @@ -739,7 +676,7 @@ g.gfql(ASTLet({ 4. **Validate before sending**: Use JSON Schema validation 5. **Handle unknown fields**: Ignore unrecognized fields for compatibility 6. **Let bindings**: Define bindings in dependency order (referenced names must be defined first) -7. **Ref validation**: Ensure referenced names exist in the Let binding scope +7. **ChainRef validation**: Ensure referenced names exist in the Let binding scope 8. **RemoteGraph security**: Protect authentication tokens in transit and storage 9. **Call operations**: Only use function names from the safelist 10. **Parameter validation**: Ensure Call parameters match expected types diff --git a/docs/source/gfql/translate.rst b/docs/source/gfql/translate.rst index 912f60234e..86bedd4b16 100644 --- a/docs/source/gfql/translate.rst +++ b/docs/source/gfql/translate.rst @@ -59,11 +59,11 @@ Finding Nodes with Specific Properties from graphistry import n # df[['id', 'type', ...]] - g.gfql([ n({"type": "person"}) ])._nodes + g.chain([ n({"type": "person"}) ])._nodes **Explanation**: -- **GFQL**: `n({"type": "person"})` filters nodes where `type` is `"person"`. `g.gfql([...])` applies this filter to the graph `g`, and `._nodes` retrieves the resulting nodes. The performance is similar to that of Pandas (CPU) or cuDF (GPU). +- **GFQL**: `n({"type": "person"})` filters nodes where `type` is `"person"`. `g.chain([...])` applies this filter to the graph `g`, and `._nodes` retrieves the resulting nodes. The performance is similar to that of Pandas (CPU) or cuDF (GPU). --- @@ -165,7 +165,7 @@ Performing Multi-Hop Traversals from graphistry import n, e_forward # df[['id', ...]] - g.gfql([ + g.chain([ n({g._node: "Alice"}), e_forward(), e_forward(), n(name='m') ])._nodes.query('m') @@ -173,9 +173,6 @@ Performing Multi-Hop Traversals - **GFQL**: Starts at node `"Alice"`, performs two forward hops, and obtains nodes two steps away. Results are in `nodes_df`. Building on the expressive and performance benefits of the previous 1-hop example, it begins adding the parallel path finding benefits of GFQL over Cypher, which benefits both CPU and GPU usage. -.. note:: - For more complex multi-hop patterns with reusable components, see the :ref:`Complex Pattern Reuse and DAG Structures` section below, which demonstrates using ``let`` to create named, composable graph traversals. - --- Filtering Edges and Nodes with Conditions @@ -211,7 +208,7 @@ Filtering Edges and Nodes with Conditions from graphistry import e_forward # df[['src', 'dst', 'weight', ...]] - g.gfql([ e_forward(edge_query='weight > 0.5') ])._edges + g.chain([ e_forward(edge_query='weight > 0.5') ])._edges **Explanation**: @@ -352,7 +349,7 @@ All Paths and Connectivity # g._edges: df[['src', 'dst', ...]] # g._nodes: df[['id', ...]] - g.gfql([ + g.chain([ n({"id": "Alice"}), e_forward( source_node_query='type == "person"', @@ -440,7 +437,7 @@ Time-Windowed Graph Analytics .. code-block:: python past_week = pd.Timestamp.now() - pd.Timedelta(7) - g.gfql([ + g.chain([ n({"id": {"$in": ["Alice", "Bob"]}}), e_forward(edge_query=f'timestamp >= "{past_week}"'), n({"id": {"$in": ["Alice", "Bob"]}}) @@ -491,7 +488,7 @@ Parallel Pathfinding from graphistry import n, e_forward # g._nodes: cudf.DataFrame[['src', 'dst', ...]] - g.gfql([ + g.chain([ n({"id": "Alice"}), e_forward(to_fixed_point=False), n({"id": is_in(["Bob", "Charlie"])}) @@ -530,7 +527,7 @@ GPU Execution from graphistry import n, e_forward # Executing pathfinding queries in parallel - g.gfql([ + g.chain([ n({"id": "Alice"}), e_forward(to_fixed_point=False), n({"id": is_in(["Bob", "Charlie"])}) @@ -546,89 +543,14 @@ This example builds on the previous one, showing how **GFQL** handles parallel e --- -Complex Pattern Reuse and DAG Structures -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -**Objective**: Execute investigations with reusable named patterns and complex DAG dependencies. - -**SQL** - -.. code-block:: sql - - -- SQL requires multiple CTEs or temp tables for complex pattern reuse - WITH suspects AS ( - SELECT id FROM nodes WHERE risk_score > 8 - ), - contacts AS ( - SELECT DISTINCT e.dst as id - FROM suspects s - JOIN edges e ON s.id = e.src - WHERE e.type = 'connected' - ), - evidence AS ( - SELECT DISTINCT e.dst as id - FROM contacts c - JOIN edges e ON c.id = e.src - WHERE e.type = 'transaction' - ) - SELECT * FROM nodes n - WHERE n.id IN (SELECT id FROM evidence); - -**Pandas** -.. code-block:: python - # Pandas requires intermediate variables and merges - suspects = nodes_df[nodes_df['risk_score'] > 8] - - contacts = edges_df[ - (edges_df['src'].isin(suspects['id'])) & - (edges_df['type'] == 'connected') - ]['dst'].unique() - - evidence = edges_df[ - (edges_df['src'].isin(contacts)) & - (edges_df['type'] == 'transaction') - ]['dst'].unique() - - result = nodes_df[nodes_df['id'].isin(evidence)] -**Cypher** -.. code-block:: cypher - // Cypher WITH clauses for intermediate results - MATCH (p:Person) WHERE p.risk_score > 8 - WITH collect(p) as suspects - MATCH (suspects)-[:CONNECTED]-(contacts) - WITH suspects, collect(contacts) as contact_nodes - MATCH (contact_nodes)-[:TRANSACTION]->(evidence) - RETURN evidence; -**GFQL** -.. code-block:: python - from graphistry import n, e_forward, e_undirected, ref, gt - - # Reusable named patterns with DAG dependencies - investigation = g.let({ - 'suspects': n({'risk_score': gt(8)}), - 'contacts': ref('suspects').gfql([e_undirected({'type': 'connected'}), n()]), - 'evidence': ref('contacts').gfql([e_forward({'type': 'transaction'}), n()]) - }) - - # Access any binding results - suspects_df = investigation._nodes[investigation._nodes['suspects']] - evidence_df = investigation._nodes[investigation._nodes['evidence']] - -**Explanation**: - -- **SQL/Pandas**: Require verbose intermediate variables and complex joins for pattern reuse -- **Cypher**: WITH clauses provide some reuse but limited to linear dependencies -- **GFQL**: Let bindings enable true DAG patterns where any binding can reference multiple previous bindings, providing both clarity and performance benefits through query optimization - ---- GFQL Functions and Equivalents ------------------------------ From c7b386aba5d8dff9029d13ccf082ff188692a7ac Mon Sep 17 00:00:00 2001 From: Leo Meyerovich Date: Sun, 27 Jul 2025 19:44:04 -0700 Subject: [PATCH 6/7] docs: Update GFQL API usage patterns MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - Replace compute_pagerank() with call('pagerank') - Replace chain_remote() with gfql_remote() - Replace g.chain() with g.gfql() - Update documentation to reflect unified GFQL API 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude --- docs/source/cheatsheet.md | 2 +- docs/source/gfql/overview.rst | 16 +++++++------- docs/source/gfql/quick.rst | 38 +++++++++++++++++----------------- docs/source/gfql/translate.rst | 16 +++++++------- 4 files changed, 36 insertions(+), 36 deletions(-) diff --git a/docs/source/cheatsheet.md b/docs/source/cheatsheet.md index 98b105ca91..e98f4b64cf 100644 --- a/docs/source/cheatsheet.md +++ b/docs/source/cheatsheet.md @@ -1096,7 +1096,7 @@ from graphistry.compute.chain import Chain pattern = Chain([n(), e(), n()]) pattern_json = pattern.to_json() pattern2 = Chain.from_json(pattern_json) -g.chain(pattern2).plot() +g.gfql(pattern2).plot() ``` Benefit from automatic GPU acceleration by passing in GPU dataframes: diff --git a/docs/source/gfql/overview.rst b/docs/source/gfql/overview.rst index bbedaf9414..a266772962 100644 --- a/docs/source/gfql/overview.rst +++ b/docs/source/gfql/overview.rst @@ -64,7 +64,7 @@ Example: Find all nodes where the `type` is `"person"`. from graphistry import n - people_nodes_df = g.chain([ n({"type": "person"}) ])._nodes + people_nodes_df = g.gfql([ n({"type": "person"}) ])._nodes print('Number of person nodes:', len(people_nodes_df)) **Visualize 2-Hop Edge Sequences with an Attribute** @@ -75,7 +75,7 @@ Example: Find 2-hop paths where edges have `"interesting": True`. from graphistry import n, e_forward - g_2_hops = g.chain([n(), e_forward({"interesting": True}, hops=2) ]) + g_2_hops = g.gfql([n(), e_forward({"interesting": True}, hops=2) ]) g_2_hops.plot() **Find Nodes 1-2 Hops Away and Label Each Hop** @@ -86,7 +86,7 @@ Example: Find nodes up to 2 hops away from node `"a"` and label each hop. from graphistry import n, e_undirected - g_2_hops = g.chain([ + g_2_hops = g.gfql([ n({g._node: "a"}), e_undirected(name="hop1"), e_undirected(name="hop2") @@ -106,12 +106,12 @@ Example: Find recent transactions using temporal predicates. import pandas as pd # Find transactions after a specific date - recent = g.chain([ + recent = g.gfql([ n(edge_match={"timestamp": gt(pd.Timestamp("2023-01-01"))}) ]) # Find transactions in a date range during business hours - business_hours_txns = g.chain([ + business_hours_txns = g.gfql([ n(edge_match={ "date": between(date(2023, 6, 1), date(2023, 6, 30)), "time": between(time(9, 0), time(17, 0)) @@ -126,7 +126,7 @@ Example: Find transaction nodes between two kinds of risky nodes. from graphistry import n, e_forward, e_reverse - g_risky = g.chain([ + g_risky = g.gfql([ n({"risk1": True}), e_forward(to_fixed_point=True), n({"type": "transaction"}, name="hit"), @@ -144,7 +144,7 @@ Example: Filter nodes and edges by multiple types. from graphistry import n, e_forward, e_reverse, is_in - g_filtered = g.chain([ + g_filtered = g.gfql([ n({"type": is_in(["person", "company"])}), e_forward({"e_type": is_in(["owns", "reviews"])}, to_fixed_point=True), n({"type": is_in(["transaction", "account"])}, name="hit"), @@ -203,7 +203,7 @@ Example: Bind to remote data and run queries on remote GPU resources. g = graphistry.bind(dataset_id='my-dataset-id') - nodes_df = g.chain_remote([ n() ])._nodes + nodes_df = g.gfql_remote([ n() ])._nodes **Upload Data and Run GPU Python Remotely** diff --git a/docs/source/gfql/quick.rst b/docs/source/gfql/quick.rst index 23694998ed..acfbda7521 100644 --- a/docs/source/gfql/quick.rst +++ b/docs/source/gfql/quick.rst @@ -12,7 +12,7 @@ Basic Usage .. code-block:: python - g.chain(ops=[...], engine=EngineAbstract.AUTO) + g.gfql(ops=[...], engine=EngineAbstract.AUTO) :meth:`chain ` sequences multiple matchers for more complex patterns of paths and subgraphs @@ -153,7 +153,7 @@ Combined Examples .. code-block:: python - g.chain([ + g.gfql([ n({"type": "person"}), e_forward({"status": "active"}), n({"type": "transaction"}) @@ -163,7 +163,7 @@ Combined Examples .. code-block:: python - g.chain([ + g.gfql([ n({"id": "start_node"}, name="start"), e_forward(name="edge1"), n({"level": 2}, name="middle"), @@ -175,7 +175,7 @@ Combined Examples .. code-block:: python - g.chain([ + g.gfql([ n({"status": "infected"}), e_forward(to_fixed_point=True), n(name="reachable") @@ -185,7 +185,7 @@ Combined Examples .. code-block:: python - g.chain([ + g.gfql([ n({"type": is_in(["server", "database"])}), e_undirected({"protocol": "TCP"}, hops=3), n(query="risk_level >= 8") @@ -195,7 +195,7 @@ Combined Examples .. code-block:: python - g.chain([ + g.gfql([ n(query="age > 30 and country == 'USA'"), e_forward(edge_query="weight > 5"), n(query="status == 'active'") @@ -208,7 +208,7 @@ GPU Acceleration .. code-block:: python - g.chain([...], engine='cudf') + g.gfql([...], engine='cudf') - **Example with cuDF DataFrames:** @@ -220,7 +220,7 @@ GPU Acceleration n_gdf = cudf.from_pandas(node_df) g = graphistry.nodes(n_gdf, 'node_id').edges(e_gdf, 'src', 'dst') - g.chain([...], engine='cudf') + g.gfql([...], engine='cudf') Remote Mode ----------- @@ -231,7 +231,7 @@ Remote Mode g = graphistry.bind(dataset_id='ds-abc-123') - nodes_df = g.chain_remote([n()])._nodes + nodes_df = g.gfql_remote([n()])._nodes - **Upload graph and run GFQL** @@ -239,34 +239,34 @@ Remote Mode g2 = g1.upload() - g3 = g2.chain_remote([n(), e(), n()]) + g3 = g2.gfql_remote([n(), e(), n()]) - **Enforce CPU and GPU mode on remote GFQL** .. code-block:: python - g3a = g2.chain_remote([n(), e(), n()], engine='pandas') - g3b = g2.chain_remote([n(), e(), n()], engine='cudf') + g3a = g2.gfql_remote([n(), e(), n()], engine='pandas') + g3b = g2.gfql_remote([n(), e(), n()], engine='cudf') - **Return only nodes and certain columns** .. code-block:: python cols = ['id', 'name'] - g2b = g1.chain_remote([n(), e(), n()], output_type="edges", edge_col_subset=cols) + g2b = g1.gfql_remote([n(), e(), n()], output_type="edges", edge_col_subset=cols) - **Return only edges and certain columns** .. code-block:: python cols = ['src', 'dst'] - g2b = g1.chain_remote([n(), e(), n()], output_type="edges", edge_col_subset=cols) + g2b = g1.gfql_remote([n(), e(), n()], output_type="edges", edge_col_subset=cols) - **Return only shape metadata** .. code-block:: python - shape_df = g1.chain_remote_shape([n(), e(), n()]) + shape_df = g1.gfql_remote_shape([n(), e(), n()]) - **Run remote Python and get back a graph** @@ -313,7 +313,7 @@ Remote Mode g2 = g1.upload() # ensure method is called "task" and takes a single argument "g" - g3 = g2.chain_remote_python(""" + g3 = g2.gfql_remote_python(""" def task(g): return (g .nodes(g._nodes[:10]) @@ -398,7 +398,7 @@ Examples at a Glance .. code-block:: python - g.chain([ + g.gfql([ n({g._node: "Alice"}), e_undirected(hops=3), n({g._node: "Bob"}) @@ -420,7 +420,7 @@ Examples at a Glance .. code-block:: python - g.chain([ + g.gfql([ n({"community": "A"}), e_undirected(hops=2), n({"community": "B"}, name="bridge_nodes") @@ -430,7 +430,7 @@ Examples at a Glance .. code-block:: python - g.chain([ + g.gfql([ n(query="age >= 18"), e_forward(edge_query="interaction == 'message'"), n(query="location == 'NYC'") diff --git a/docs/source/gfql/translate.rst b/docs/source/gfql/translate.rst index 86bedd4b16..1d45ec3d42 100644 --- a/docs/source/gfql/translate.rst +++ b/docs/source/gfql/translate.rst @@ -59,11 +59,11 @@ Finding Nodes with Specific Properties from graphistry import n # df[['id', 'type', ...]] - g.chain([ n({"type": "person"}) ])._nodes + g.gfql([ n({"type": "person"}) ])._nodes **Explanation**: -- **GFQL**: `n({"type": "person"})` filters nodes where `type` is `"person"`. `g.chain([...])` applies this filter to the graph `g`, and `._nodes` retrieves the resulting nodes. The performance is similar to that of Pandas (CPU) or cuDF (GPU). +- **GFQL**: `n({"type": "person"})` filters nodes where `type` is `"person"`. `g.gfql([...])` applies this filter to the graph `g`, and `._nodes` retrieves the resulting nodes. The performance is similar to that of Pandas (CPU) or cuDF (GPU). --- @@ -165,7 +165,7 @@ Performing Multi-Hop Traversals from graphistry import n, e_forward # df[['id', ...]] - g.chain([ + g.gfql([ n({g._node: "Alice"}), e_forward(), e_forward(), n(name='m') ])._nodes.query('m') @@ -208,7 +208,7 @@ Filtering Edges and Nodes with Conditions from graphistry import e_forward # df[['src', 'dst', 'weight', ...]] - g.chain([ e_forward(edge_query='weight > 0.5') ])._edges + g.gfql([ e_forward(edge_query='weight > 0.5') ])._edges **Explanation**: @@ -349,7 +349,7 @@ All Paths and Connectivity # g._edges: df[['src', 'dst', ...]] # g._nodes: df[['id', ...]] - g.chain([ + g.gfql([ n({"id": "Alice"}), e_forward( source_node_query='type == "person"', @@ -437,7 +437,7 @@ Time-Windowed Graph Analytics .. code-block:: python past_week = pd.Timestamp.now() - pd.Timedelta(7) - g.chain([ + g.gfql([ n({"id": {"$in": ["Alice", "Bob"]}}), e_forward(edge_query=f'timestamp >= "{past_week}"'), n({"id": {"$in": ["Alice", "Bob"]}}) @@ -488,7 +488,7 @@ Parallel Pathfinding from graphistry import n, e_forward # g._nodes: cudf.DataFrame[['src', 'dst', ...]] - g.chain([ + g.gfql([ n({"id": "Alice"}), e_forward(to_fixed_point=False), n({"id": is_in(["Bob", "Charlie"])}) @@ -527,7 +527,7 @@ GPU Execution from graphistry import n, e_forward # Executing pathfinding queries in parallel - g.chain([ + g.gfql([ n({"id": "Alice"}), e_forward(to_fixed_point=False), n({"id": is_in(["Bob", "Charlie"])}) From 4bf37d89d4110123c4d65c890774c796b8fa7e0b Mon Sep 17 00:00:00 2001 From: Leo Meyerovich Date: Sun, 27 Jul 2025 23:11:47 -0700 Subject: [PATCH 7/7] docs: Update GFQL documentation - chain() to gfql() migration and add Built-in Calls reference - Update all g.chain() calls to g.gfql() in documentation - Update chain_remote references to gfql_remote (though most were already done) - Add comprehensive Built-in Calls documentation (builtin_calls.rst) - Update wire_protocol.md and cypher_mapping.md examples - Add builtin_calls to GFQL index These changes align documentation with the deprecated chain() and chain_remote methods. --- docs/source/gfql/builtin_calls.rst | 237 ++++++++++++++++++++++++ docs/source/gfql/index.rst | 1 + docs/source/gfql/spec/cypher_mapping.md | 8 +- docs/source/gfql/spec/wire_protocol.md | 4 +- 4 files changed, 244 insertions(+), 6 deletions(-) create mode 100644 docs/source/gfql/builtin_calls.rst diff --git a/docs/source/gfql/builtin_calls.rst b/docs/source/gfql/builtin_calls.rst new file mode 100644 index 0000000000..ab8c704e4f --- /dev/null +++ b/docs/source/gfql/builtin_calls.rst @@ -0,0 +1,237 @@ +.. _gfql-builtin-calls: + +GFQL Built-in Calls +=================== + +The Call operation in GFQL allows you to invoke methods on the current graph with validated parameters. This provides a safe and extensible way to apply graph algorithms, transformations, and analytics operations within your GFQL queries. + +Basic Usage +----------- + +**Call Syntax** + +.. code-block:: python + + from graphistry import call + + # In a GFQL chain + g.gfql([ + n({"type": "person"}), + call('pagerank'), + n(query='pagerank > 0.5') + ]) + + # In a Let binding + g.gfql(let({ + 'influential': [n(), call('pagerank')], + 'top_nodes': [ref('influential'), n(query='pagerank > 0.5')] + })) + +Available Methods +----------------- + +The following methods are available through the Call operation: + +Graph Algorithms +~~~~~~~~~~~~~~~~ + +**pagerank** + Compute PageRank scores for nodes. + + .. code-block:: python + + call('pagerank') + call('pagerank', {'damping': 0.85, 'iterations': 20}) + +**get_degrees** + Calculate node degrees (in-degree, out-degree, or total). + + .. code-block:: python + + call('get_degrees') + call('get_degrees', {'col': 'total_degree'}) + call('get_degrees', {'col_in': 'in_deg', 'col_out': 'out_deg'}) + +Filtering Operations +~~~~~~~~~~~~~~~~~~~~ + +**filter_nodes_by_dict** + Filter nodes based on attribute values. + + .. code-block:: python + + call('filter_nodes_by_dict', {'filter_dict': {'type': 'user', 'active': True}}) + +**filter_edges_by_dict** + Filter edges based on attribute values. + + .. code-block:: python + + call('filter_edges_by_dict', {'filter_dict': {'weight': gt(0.5)}}) + +Graph Traversal +~~~~~~~~~~~~~~~ + +**hop** + Traverse the graph for N steps from current nodes. + + .. code-block:: python + + call('hop', {'hops': 2}) + call('hop', {'hops': 3, 'direction': 'forward'}) + call('hop', {'to_fixed_point': True, 'direction': 'undirected'}) + +Layout Algorithms +~~~~~~~~~~~~~~~~~ + +**umap** + Apply UMAP dimensionality reduction for graph layout. + + .. code-block:: python + + call('umap') + call('umap', {'n_neighbors': 15, 'min_dist': 0.1}) + +**fa2_layout** + Apply ForceAtlas2 layout algorithm. + + .. code-block:: python + + call('fa2_layout') + call('fa2_layout', {'iterations': 500}) + +Graph Structure +~~~~~~~~~~~~~~~ + +**materialize_nodes** + Generate node table from edges. + + .. code-block:: python + + call('materialize_nodes') + call('materialize_nodes', {'reuse': False}) + +**add_graph** + Combine with another graph. + + .. code-block:: python + + call('add_graph', {'g2': other_graph}) + +**prune_self_edges** + Remove self-referencing edges. + + .. code-block:: python + + call('prune_self_edges') + +Utility Operations +~~~~~~~~~~~~~~~~~~ + +**name** + Tag nodes with a boolean column. + + .. code-block:: python + + call('name', {'name': 'important_nodes'}) + +**sample** + Sample a subset of nodes. + + .. code-block:: python + + call('sample', {'n': 1000}) + +Parameter Validation +-------------------- + +All Call operations have their parameters validated against a safelist to ensure: + +- Type safety: Parameters must be of the correct type +- Required parameters: Missing required parameters will raise an error +- Unknown parameters: Extra parameters not in the safelist will be rejected +- Value constraints: Some parameters have specific allowed values + +Example error handling: + +.. code-block:: python + + # Missing required parameter + call('filter_nodes_by_dict') # Error: Missing 'filter_dict' + + # Wrong parameter type + call('hop', {'hops': 'two'}) # Error: 'hops' must be integer + + # Unknown parameter + call('pagerank', {'unknown_param': 123}) # Error: Unknown parameter + +Integration with Other GFQL Features +------------------------------------ + +Calls can be combined with other GFQL operations: + +**With Predicates** + +.. code-block:: python + + g.gfql([ + n({'type': 'user'}), + call('pagerank'), + n({'pagerank': gt(0.1)}) + ]) + +**With Let Bindings** + +.. code-block:: python + + g.gfql(let({ + 'users': n({'type': 'user'}), + 'ranked': [ref('users'), call('pagerank')], + 'top': [ref('ranked'), n(query='pagerank > 0.5')] + })) + +**With Remote Execution** + +.. code-block:: python + + g.gfql_remote([ + n(), + call('pagerank'), + n(query='pagerank > 0.1') + ]) + +Best Practices +-------------- + +1. **Chain Efficiency**: Place filtering calls early in the chain to reduce data volume +2. **Parameter Reuse**: Store common parameter sets in variables +3. **Error Handling**: Wrap calls in try-except blocks when parameters come from user input +4. **Performance**: Some calls like 'pagerank' are computationally intensive - consider using GPU engine + +GPU Acceleration +---------------- + +Many Call operations support GPU acceleration when using the cuDF engine: + +.. code-block:: python + + # Force GPU execution + g.gfql([ + n(), + call('pagerank'), + n(query='pagerank > 0.1') + ], engine='cudf') + +GPU-accelerated methods include: +- pagerank +- get_degrees +- hop +- filter operations +- most graph algorithms + +See Also +-------- + +- :ref:`gfql-quick` - Quick reference for all GFQL operations +- :ref:`gfql-spec` - Complete GFQL specification +- :ref:`10min-gfql` - Tutorial introduction to GFQL \ No newline at end of file diff --git a/docs/source/gfql/index.rst b/docs/source/gfql/index.rst index e8b26a10db..f4d4827def 100644 --- a/docs/source/gfql/index.rst +++ b/docs/source/gfql/index.rst @@ -19,6 +19,7 @@ See also: translate combo quick + builtin_calls predicates/quick datetime_filtering wire_protocol_examples diff --git a/docs/source/gfql/spec/cypher_mapping.md b/docs/source/gfql/spec/cypher_mapping.md index f936358348..7c0ac3617d 100644 --- a/docs/source/gfql/spec/cypher_mapping.md +++ b/docs/source/gfql/spec/cypher_mapping.md @@ -48,7 +48,7 @@ WHERE p.age > 30 **Python:** ```python -g.chain([ +g.gfql([ n({"type": "Person", "age": gt(30)}, name="p"), e_forward({"type": "FOLLOWS"}, name="r"), n({"type": "Person"}, name="q") @@ -119,7 +119,7 @@ WHERE fof.active = true **Python:** ```python -g.chain([ +g.gfql([ n({"type": "User", "name": "Alice"}), e_forward({"type": "FRIEND"}, hops=2), n({"type": "User", "active": True}, name="fof") @@ -145,7 +145,7 @@ WHERE t.amount > 10000 AND t.date > date('2024-01-01') **Python:** ```python -g.chain([ +g.gfql([ n({"type": "Account"}), e_forward({ "type": "TRANSFER", @@ -183,7 +183,7 @@ LIMIT 10 **Python:** ```python # Step 1: Graph pattern -result = g.chain([ +result = g.gfql([ n({"type": "User"}), e_forward({"type": "TRANSACTION", "date": gt(date(2024,1,1))}, name="trans"), n({"type": "Merchant"}) diff --git a/docs/source/gfql/spec/wire_protocol.md b/docs/source/gfql/spec/wire_protocol.md index e91d753615..cbe8c683ce 100644 --- a/docs/source/gfql/spec/wire_protocol.md +++ b/docs/source/gfql/spec/wire_protocol.md @@ -468,7 +468,7 @@ null // null **Python**: ```python -g.chain([ +g.gfql([ n({"customer_id": "C123"}), e_forward({ "type": "purchase", @@ -511,7 +511,7 @@ g.chain([ **Python**: ```python -g.chain([ +g.gfql([ n({"ip": is_in(["192.168.1.100", "192.168.1.101"])}), e_forward( edge_query="port IN [22, 23, 3389]",