diff --git a/LET_SEQUENCING_DOCS_TASKS.md b/LET_SEQUENCING_DOCS_TASKS.md new file mode 100644 index 0000000000..a366446698 --- /dev/null +++ b/LET_SEQUENCING_DOCS_TASKS.md @@ -0,0 +1,149 @@ +# Let Sequencing Documentation Tasks + +## Raw Task List + +1. **10-minute GFQL Guide** + - Add sequencing GFQL programs section with `let` + - Include PageRank example: compute PageRank, then explore 2 hops from high-scoring nodes + - Location: Find appropriate place for ninja edit + +2. **Overview -> Quick Examples** + - Add similar `let` sequencing example + - Show the power of composing operations + +3. **Remote Mode Documentation** + - Add `let` examples (more important here due to no Python escape hatch) + - Show how to sequence operations in pure GFQL + +4. **Fix chain_let References** + - Find all "chain_let" references in docs + - Update to ".let()" syntax + - Verify correctness across all docs + +5. **Update .chain() Syntax** + - Find all .chain(...) references + - Update to .gfql(Chain(...)) + - OR: If implicit coercion .gfql([...]) => .gfql(Chain([...])) is supported: + - Do code scan to verify + - Add pytest if needed + - Document the coercion clearly + +6. **GFQL Quick Reference** + - Move .let() earlier in the reference + - Place before predicates section + - Show as fundamental composition tool + +7. **GFQL Language Spec** + - Fix top-level to show both chain and let as allowed + - Add chain+let section before Operations section + - Ensure proper hierarchy + +8. **Wire Protocol Documentation** + - Add chain+let section before operations + - Show protocol representation + +9. **hop_and_chain_graph_pattern_mining Notebook** + - Add `let` example + - Validate with docs build that runs notebooks + +10. **call_operations Documentation** + - Update title to include "and Let bindings" + - Show how Call and Let work together + +11. **gfql_remote Notebook** + - Add let+call example + - Show remote execution patterns + +12. **Multi-hop Section Updates** + - Add forward reference to "Complex Pattern Reuse" + - Improve flow between sections + +13. **"Between" Section** + - Has "Complex Pattern Reuse" + - Fix chain_let references + +14. **Combining GFQL Section** + - Start early with Python vs pure GFQL equivalents + - Show the relationship clearly + +## Reordered by PR + +All tasks should be on **PR #708** (top of docs stack): "docs(gfql): Comprehensive Let bindings and Call operations documentation" + +## Granular Implementation Steps + +### Phase 1: Preparation +1. Switch to PR #708 branch +2. Pull latest changes +3. Create working list of all files to modify + +### Phase 2: Find and Fix chain_let References +4. Search for all "chain_let" occurrences in docs +5. List each file and line number +6. Update each to ".let()" syntax +7. Verify context makes sense + +### Phase 3: Update .chain() Syntax +8. Search for all .chain() calls in docs +9. Check if implicit coercion is supported in code +10. If yes: Document coercion behavior +11. If no: Update all to .gfql(Chain(...)) + +### Phase 4: Core Documentation Updates +12. Find 10-minute GFQL guide file +13. Add "Sequencing Programs with Let" section +14. Write PageRank + 2-hop example +15. Find Overview -> Quick Examples +16. Add let sequencing example there +17. Find Remote Mode docs +18. Add comprehensive let examples for remote + +### Phase 5: Reference Documentation +19. Find GFQL Quick Reference +20. Restructure to introduce .let() before predicates +21. Find GFQL Language Spec +22. Update top-level grammar to include let +23. Add chain+let section before Operations +24. Find Wire Protocol docs +25. Add chain+let protocol section before ops + +### Phase 6: Notebook Updates +26. Find hop_and_chain_graph_pattern_mining.ipynb +27. Add let example with pattern reuse +28. Find gfql_remote.ipynb +29. Add let+call remote example +30. Update notebook outputs + +### Phase 7: Cross-references and Titles +31. Find Multi-hop section +32. Add forward reference to Complex Pattern Reuse +33. Find call_operations docs +34. Update title to include "and Let Bindings" +35. Find Combining GFQL section +36. Add early Python vs GFQL comparison + +### Phase 8: Validation +37. Run flake8/ruff linting +38. Run mypy type checking +39. Run docs build locally +40. Run notebook tests +41. Review all changes + +### Phase 9: Commit and Push +42. Stage all changes +43. Create detailed commit message +44. Push to PR #708 + +## Validation Checklist +- [ ] All chain_let replaced with .let() +- [ ] All .chain() updated appropriately +- [ ] PageRank example works in 10-min guide +- [ ] Remote examples are comprehensive +- [ ] Quick Ref has proper ordering +- [ ] Language Spec has let at top level +- [ ] Notebooks execute without errors +- [ ] Cross-references are correct +- [ ] No lint errors +- [ ] No type errors +- [ ] Docs build succeeds +- [ ] Notebook tests pass \ No newline at end of file diff --git a/docs/source/cheatsheet.md b/docs/source/cheatsheet.md index 98b105ca91..e98f4b64cf 100644 --- a/docs/source/cheatsheet.md +++ b/docs/source/cheatsheet.md @@ -1096,7 +1096,7 @@ from graphistry.compute.chain import Chain pattern = Chain([n(), e(), n()]) pattern_json = pattern.to_json() pattern2 = Chain.from_json(pattern_json) -g.chain(pattern2).plot() +g.gfql(pattern2).plot() ``` Benefit from automatic GPU acceleration by passing in GPU dataframes: diff --git a/docs/source/gfql/builtin_calls.rst b/docs/source/gfql/builtin_calls.rst new file mode 100644 index 0000000000..ab8c704e4f --- /dev/null +++ b/docs/source/gfql/builtin_calls.rst @@ -0,0 +1,237 @@ +.. _gfql-builtin-calls: + +GFQL Built-in Calls +=================== + +The Call operation in GFQL allows you to invoke methods on the current graph with validated parameters. This provides a safe and extensible way to apply graph algorithms, transformations, and analytics operations within your GFQL queries. + +Basic Usage +----------- + +**Call Syntax** + +.. code-block:: python + + from graphistry import call + + # In a GFQL chain + g.gfql([ + n({"type": "person"}), + call('pagerank'), + n(query='pagerank > 0.5') + ]) + + # In a Let binding + g.gfql(let({ + 'influential': [n(), call('pagerank')], + 'top_nodes': [ref('influential'), n(query='pagerank > 0.5')] + })) + +Available Methods +----------------- + +The following methods are available through the Call operation: + +Graph Algorithms +~~~~~~~~~~~~~~~~ + +**pagerank** + Compute PageRank scores for nodes. + + .. code-block:: python + + call('pagerank') + call('pagerank', {'damping': 0.85, 'iterations': 20}) + +**get_degrees** + Calculate node degrees (in-degree, out-degree, or total). + + .. code-block:: python + + call('get_degrees') + call('get_degrees', {'col': 'total_degree'}) + call('get_degrees', {'col_in': 'in_deg', 'col_out': 'out_deg'}) + +Filtering Operations +~~~~~~~~~~~~~~~~~~~~ + +**filter_nodes_by_dict** + Filter nodes based on attribute values. + + .. code-block:: python + + call('filter_nodes_by_dict', {'filter_dict': {'type': 'user', 'active': True}}) + +**filter_edges_by_dict** + Filter edges based on attribute values. + + .. code-block:: python + + call('filter_edges_by_dict', {'filter_dict': {'weight': gt(0.5)}}) + +Graph Traversal +~~~~~~~~~~~~~~~ + +**hop** + Traverse the graph for N steps from current nodes. + + .. code-block:: python + + call('hop', {'hops': 2}) + call('hop', {'hops': 3, 'direction': 'forward'}) + call('hop', {'to_fixed_point': True, 'direction': 'undirected'}) + +Layout Algorithms +~~~~~~~~~~~~~~~~~ + +**umap** + Apply UMAP dimensionality reduction for graph layout. + + .. code-block:: python + + call('umap') + call('umap', {'n_neighbors': 15, 'min_dist': 0.1}) + +**fa2_layout** + Apply ForceAtlas2 layout algorithm. + + .. code-block:: python + + call('fa2_layout') + call('fa2_layout', {'iterations': 500}) + +Graph Structure +~~~~~~~~~~~~~~~ + +**materialize_nodes** + Generate node table from edges. + + .. code-block:: python + + call('materialize_nodes') + call('materialize_nodes', {'reuse': False}) + +**add_graph** + Combine with another graph. + + .. code-block:: python + + call('add_graph', {'g2': other_graph}) + +**prune_self_edges** + Remove self-referencing edges. + + .. code-block:: python + + call('prune_self_edges') + +Utility Operations +~~~~~~~~~~~~~~~~~~ + +**name** + Tag nodes with a boolean column. + + .. code-block:: python + + call('name', {'name': 'important_nodes'}) + +**sample** + Sample a subset of nodes. + + .. code-block:: python + + call('sample', {'n': 1000}) + +Parameter Validation +-------------------- + +All Call operations have their parameters validated against a safelist to ensure: + +- Type safety: Parameters must be of the correct type +- Required parameters: Missing required parameters will raise an error +- Unknown parameters: Extra parameters not in the safelist will be rejected +- Value constraints: Some parameters have specific allowed values + +Example error handling: + +.. code-block:: python + + # Missing required parameter + call('filter_nodes_by_dict') # Error: Missing 'filter_dict' + + # Wrong parameter type + call('hop', {'hops': 'two'}) # Error: 'hops' must be integer + + # Unknown parameter + call('pagerank', {'unknown_param': 123}) # Error: Unknown parameter + +Integration with Other GFQL Features +------------------------------------ + +Calls can be combined with other GFQL operations: + +**With Predicates** + +.. code-block:: python + + g.gfql([ + n({'type': 'user'}), + call('pagerank'), + n({'pagerank': gt(0.1)}) + ]) + +**With Let Bindings** + +.. code-block:: python + + g.gfql(let({ + 'users': n({'type': 'user'}), + 'ranked': [ref('users'), call('pagerank')], + 'top': [ref('ranked'), n(query='pagerank > 0.5')] + })) + +**With Remote Execution** + +.. code-block:: python + + g.gfql_remote([ + n(), + call('pagerank'), + n(query='pagerank > 0.1') + ]) + +Best Practices +-------------- + +1. **Chain Efficiency**: Place filtering calls early in the chain to reduce data volume +2. **Parameter Reuse**: Store common parameter sets in variables +3. **Error Handling**: Wrap calls in try-except blocks when parameters come from user input +4. **Performance**: Some calls like 'pagerank' are computationally intensive - consider using GPU engine + +GPU Acceleration +---------------- + +Many Call operations support GPU acceleration when using the cuDF engine: + +.. code-block:: python + + # Force GPU execution + g.gfql([ + n(), + call('pagerank'), + n(query='pagerank > 0.1') + ], engine='cudf') + +GPU-accelerated methods include: +- pagerank +- get_degrees +- hop +- filter operations +- most graph algorithms + +See Also +-------- + +- :ref:`gfql-quick` - Quick reference for all GFQL operations +- :ref:`gfql-spec` - Complete GFQL specification +- :ref:`10min-gfql` - Tutorial introduction to GFQL \ No newline at end of file diff --git a/docs/source/gfql/index.rst b/docs/source/gfql/index.rst index e8b26a10db..f4d4827def 100644 --- a/docs/source/gfql/index.rst +++ b/docs/source/gfql/index.rst @@ -19,6 +19,7 @@ See also: translate combo quick + builtin_calls predicates/quick datetime_filtering wire_protocol_examples diff --git a/docs/source/gfql/overview.rst b/docs/source/gfql/overview.rst index bbedaf9414..a266772962 100644 --- a/docs/source/gfql/overview.rst +++ b/docs/source/gfql/overview.rst @@ -64,7 +64,7 @@ Example: Find all nodes where the `type` is `"person"`. from graphistry import n - people_nodes_df = g.chain([ n({"type": "person"}) ])._nodes + people_nodes_df = g.gfql([ n({"type": "person"}) ])._nodes print('Number of person nodes:', len(people_nodes_df)) **Visualize 2-Hop Edge Sequences with an Attribute** @@ -75,7 +75,7 @@ Example: Find 2-hop paths where edges have `"interesting": True`. from graphistry import n, e_forward - g_2_hops = g.chain([n(), e_forward({"interesting": True}, hops=2) ]) + g_2_hops = g.gfql([n(), e_forward({"interesting": True}, hops=2) ]) g_2_hops.plot() **Find Nodes 1-2 Hops Away and Label Each Hop** @@ -86,7 +86,7 @@ Example: Find nodes up to 2 hops away from node `"a"` and label each hop. from graphistry import n, e_undirected - g_2_hops = g.chain([ + g_2_hops = g.gfql([ n({g._node: "a"}), e_undirected(name="hop1"), e_undirected(name="hop2") @@ -106,12 +106,12 @@ Example: Find recent transactions using temporal predicates. import pandas as pd # Find transactions after a specific date - recent = g.chain([ + recent = g.gfql([ n(edge_match={"timestamp": gt(pd.Timestamp("2023-01-01"))}) ]) # Find transactions in a date range during business hours - business_hours_txns = g.chain([ + business_hours_txns = g.gfql([ n(edge_match={ "date": between(date(2023, 6, 1), date(2023, 6, 30)), "time": between(time(9, 0), time(17, 0)) @@ -126,7 +126,7 @@ Example: Find transaction nodes between two kinds of risky nodes. from graphistry import n, e_forward, e_reverse - g_risky = g.chain([ + g_risky = g.gfql([ n({"risk1": True}), e_forward(to_fixed_point=True), n({"type": "transaction"}, name="hit"), @@ -144,7 +144,7 @@ Example: Filter nodes and edges by multiple types. from graphistry import n, e_forward, e_reverse, is_in - g_filtered = g.chain([ + g_filtered = g.gfql([ n({"type": is_in(["person", "company"])}), e_forward({"e_type": is_in(["owns", "reviews"])}, to_fixed_point=True), n({"type": is_in(["transaction", "account"])}, name="hit"), @@ -203,7 +203,7 @@ Example: Bind to remote data and run queries on remote GPU resources. g = graphistry.bind(dataset_id='my-dataset-id') - nodes_df = g.chain_remote([ n() ])._nodes + nodes_df = g.gfql_remote([ n() ])._nodes **Upload Data and Run GPU Python Remotely** diff --git a/docs/source/gfql/quick.rst b/docs/source/gfql/quick.rst index 23694998ed..acfbda7521 100644 --- a/docs/source/gfql/quick.rst +++ b/docs/source/gfql/quick.rst @@ -12,7 +12,7 @@ Basic Usage .. code-block:: python - g.chain(ops=[...], engine=EngineAbstract.AUTO) + g.gfql(ops=[...], engine=EngineAbstract.AUTO) :meth:`chain ` sequences multiple matchers for more complex patterns of paths and subgraphs @@ -153,7 +153,7 @@ Combined Examples .. code-block:: python - g.chain([ + g.gfql([ n({"type": "person"}), e_forward({"status": "active"}), n({"type": "transaction"}) @@ -163,7 +163,7 @@ Combined Examples .. code-block:: python - g.chain([ + g.gfql([ n({"id": "start_node"}, name="start"), e_forward(name="edge1"), n({"level": 2}, name="middle"), @@ -175,7 +175,7 @@ Combined Examples .. code-block:: python - g.chain([ + g.gfql([ n({"status": "infected"}), e_forward(to_fixed_point=True), n(name="reachable") @@ -185,7 +185,7 @@ Combined Examples .. code-block:: python - g.chain([ + g.gfql([ n({"type": is_in(["server", "database"])}), e_undirected({"protocol": "TCP"}, hops=3), n(query="risk_level >= 8") @@ -195,7 +195,7 @@ Combined Examples .. code-block:: python - g.chain([ + g.gfql([ n(query="age > 30 and country == 'USA'"), e_forward(edge_query="weight > 5"), n(query="status == 'active'") @@ -208,7 +208,7 @@ GPU Acceleration .. code-block:: python - g.chain([...], engine='cudf') + g.gfql([...], engine='cudf') - **Example with cuDF DataFrames:** @@ -220,7 +220,7 @@ GPU Acceleration n_gdf = cudf.from_pandas(node_df) g = graphistry.nodes(n_gdf, 'node_id').edges(e_gdf, 'src', 'dst') - g.chain([...], engine='cudf') + g.gfql([...], engine='cudf') Remote Mode ----------- @@ -231,7 +231,7 @@ Remote Mode g = graphistry.bind(dataset_id='ds-abc-123') - nodes_df = g.chain_remote([n()])._nodes + nodes_df = g.gfql_remote([n()])._nodes - **Upload graph and run GFQL** @@ -239,34 +239,34 @@ Remote Mode g2 = g1.upload() - g3 = g2.chain_remote([n(), e(), n()]) + g3 = g2.gfql_remote([n(), e(), n()]) - **Enforce CPU and GPU mode on remote GFQL** .. code-block:: python - g3a = g2.chain_remote([n(), e(), n()], engine='pandas') - g3b = g2.chain_remote([n(), e(), n()], engine='cudf') + g3a = g2.gfql_remote([n(), e(), n()], engine='pandas') + g3b = g2.gfql_remote([n(), e(), n()], engine='cudf') - **Return only nodes and certain columns** .. code-block:: python cols = ['id', 'name'] - g2b = g1.chain_remote([n(), e(), n()], output_type="edges", edge_col_subset=cols) + g2b = g1.gfql_remote([n(), e(), n()], output_type="edges", edge_col_subset=cols) - **Return only edges and certain columns** .. code-block:: python cols = ['src', 'dst'] - g2b = g1.chain_remote([n(), e(), n()], output_type="edges", edge_col_subset=cols) + g2b = g1.gfql_remote([n(), e(), n()], output_type="edges", edge_col_subset=cols) - **Return only shape metadata** .. code-block:: python - shape_df = g1.chain_remote_shape([n(), e(), n()]) + shape_df = g1.gfql_remote_shape([n(), e(), n()]) - **Run remote Python and get back a graph** @@ -313,7 +313,7 @@ Remote Mode g2 = g1.upload() # ensure method is called "task" and takes a single argument "g" - g3 = g2.chain_remote_python(""" + g3 = g2.gfql_remote_python(""" def task(g): return (g .nodes(g._nodes[:10]) @@ -398,7 +398,7 @@ Examples at a Glance .. code-block:: python - g.chain([ + g.gfql([ n({g._node: "Alice"}), e_undirected(hops=3), n({g._node: "Bob"}) @@ -420,7 +420,7 @@ Examples at a Glance .. code-block:: python - g.chain([ + g.gfql([ n({"community": "A"}), e_undirected(hops=2), n({"community": "B"}, name="bridge_nodes") @@ -430,7 +430,7 @@ Examples at a Glance .. code-block:: python - g.chain([ + g.gfql([ n(query="age >= 18"), e_forward(edge_query="interaction == 'message'"), n(query="location == 'NYC'") diff --git a/docs/source/gfql/spec/cypher_mapping.md b/docs/source/gfql/spec/cypher_mapping.md index f936358348..7c0ac3617d 100644 --- a/docs/source/gfql/spec/cypher_mapping.md +++ b/docs/source/gfql/spec/cypher_mapping.md @@ -48,7 +48,7 @@ WHERE p.age > 30 **Python:** ```python -g.chain([ +g.gfql([ n({"type": "Person", "age": gt(30)}, name="p"), e_forward({"type": "FOLLOWS"}, name="r"), n({"type": "Person"}, name="q") @@ -119,7 +119,7 @@ WHERE fof.active = true **Python:** ```python -g.chain([ +g.gfql([ n({"type": "User", "name": "Alice"}), e_forward({"type": "FRIEND"}, hops=2), n({"type": "User", "active": True}, name="fof") @@ -145,7 +145,7 @@ WHERE t.amount > 10000 AND t.date > date('2024-01-01') **Python:** ```python -g.chain([ +g.gfql([ n({"type": "Account"}), e_forward({ "type": "TRANSFER", @@ -183,7 +183,7 @@ LIMIT 10 **Python:** ```python # Step 1: Graph pattern -result = g.chain([ +result = g.gfql([ n({"type": "User"}), e_forward({"type": "TRANSACTION", "date": gt(date(2024,1,1))}, name="trans"), n({"type": "Merchant"}) diff --git a/docs/source/gfql/spec/language.md b/docs/source/gfql/spec/language.md index 65fae87b4b..66bf8a5240 100644 --- a/docs/source/gfql/spec/language.md +++ b/docs/source/gfql/spec/language.md @@ -361,6 +361,119 @@ people_nodes = result._nodes.query("people == True") This pattern is essential for extracting specific subsets from complex graph traversals. +## Call Operations and Security + +### Call Operations + +GFQL supports calling Plottable methods through the `call()` operation, providing controlled access to graph transformation and analysis capabilities: + +```python +call(function: str, params: dict) -> ASTCall +``` + +Call operations enable: +- Graph algorithms (PageRank, community detection) +- Layout computations (ForceAtlas2, Graphviz) +- Data transformations (filtering, collapsing) +- Visual encodings (color, size, icons) + +### Safelist Architecture + +For security and stability, Call operations are restricted to a predefined safelist of methods. This prevents: +- Arbitrary code execution +- Access to filesystem or network operations +- Modification of global state +- Unsafe graph operations + +#### Safelist Categories + +**Graph Analysis** +- `get_degrees`, `get_indegrees`, `get_outdegrees`: Calculate node degrees +- `compute_cugraph`: Run GPU algorithms (pagerank, louvain, etc.) +- `compute_igraph`: Run CPU algorithms +- `get_topological_levels`: Analyze DAG structure + +**Filtering & Transformation** +- `filter_nodes_by_dict`, `filter_edges_by_dict`: Filter by attributes +- `hop`: Traverse graph with conditions +- `drop_nodes`, `keep_nodes`: Node selection +- `collapse`: Merge nodes by attribute +- `prune_self_edges`: Remove self-loops +- `materialize_nodes`: Generate node table + +**Layout** +- `layout_cugraph`: GPU-accelerated layouts +- `layout_igraph`: CPU-based layouts +- `layout_graphviz`: Graphviz layouts +- `fa2_layout`: ForceAtlas2 layout + +**Visual Encoding** +- `encode_point_color`, `encode_edge_color`: Color mapping +- `encode_point_size`: Size mapping +- `encode_point_icon`: Icon mapping + +**Metadata** +- `name`: Set visualization name +- `description`: Set visualization description + +### Parameter Validation + +Call operations enforce strict parameter validation: + +1. **Type Validation**: Parameters must match expected types + ```python + call('hop', {'hops': 2}) # Valid: integer + call('hop', {'hops': 'two'}) # Error: E201 type mismatch + ``` + +2. **Required Parameters**: Missing required parameters raise errors + ```python + call('filter_nodes_by_dict', {}) # Error: E105 missing filter_dict + ``` + +3. **Unknown Parameters**: Extra parameters are rejected + ```python + call('hop', {'steps': 2}) # Error: E303 unknown parameter + ``` + +### Schema Effects + +Many Call operations modify the graph schema by adding columns: + +```python +# get_degrees adds degree columns +call('get_degrees', { + 'col': 'degree', + 'col_in': 'in_degree', + 'col_out': 'out_degree' +}) +# Schema effect: adds 3 new node columns + +# compute_cugraph adds result column +call('compute_cugraph', { + 'alg': 'pagerank', + 'out_col': 'pr_score' +}) +# Schema effect: adds 'pr_score' node column +``` + +### Security Considerations + +1. **No Direct Code Execution**: Call operations cannot execute arbitrary Python code +2. **No System Access**: Methods cannot access filesystem, network, or system resources +3. **Validated Parameters**: All inputs are type-checked and validated +4. **Deterministic Effects**: Operations have predictable, documented effects +5. **No State Mutation**: Operations return new graphs without modifying originals + +### Error Handling + +Call operations use GFQL's standard error codes: +- **E303**: Function not in safelist +- **E105**: Missing required parameter +- **E201**: Parameter type mismatch +- **E303**: Unknown parameter +- **E301**: Required column not found (runtime) + ## Best Practices 1. **Use specific filters early**: Filter nodes before traversing edges diff --git a/docs/source/gfql/spec/python_embedding.md b/docs/source/gfql/spec/python_embedding.md index cf20db1741..9086c6db10 100644 --- a/docs/source/gfql/spec/python_embedding.md +++ b/docs/source/gfql/spec/python_embedding.md @@ -53,6 +53,268 @@ nodes_df = result._nodes # Filtered nodes DataFrame edges_df = result._edges # Filtered edges DataFrame ``` +## DAG Patterns with Let Bindings + +GFQL supports directed acyclic graph (DAG) patterns using Let bindings, which allow you to define named graph operations that can reference each other. + +### Let Bindings + +```python +from graphistry import let, ref, n, e_forward + +# Define DAG patterns with named bindings +result = g.gfql(let({ + 'persons': n({'type': 'person'}), + 'adults': ref('persons', [n({'age': ge(18)})]), + 'connections': ref('adults', [ + e_forward({'type': 'knows'}), + ref('adults') # Find connections between adults + ]) +})) + +# Access individual binding results +persons_df = result._nodes[result._nodes['persons']] +adults_df = result._nodes[result._nodes['adults']] +connection_edges = result._edges[result._edges['connections']] +``` + +### ChainRef (Reference to Named Bindings) + +The `ref()` function creates references to named bindings within a Let: + +```python +# Basic reference - just the binding result +result = g.gfql(let({ + 'base': n({'status': 'active'}), + 'extended': ref('base') # Just references 'base' +})) + +# Reference with additional operations +result = g.gfql(let({ + 'suspects': n({'risk_score': gt(80)}), + 'lateral_movement': ref('suspects', [ + e_forward({'type': 'ssh', 'failed_attempts': gt(5)}), + n({'type': 'server'}) + ]) +})) +``` + +### Complex DAG Patterns + +```python +# Multi-level analysis pattern +result = g.gfql(let({ + # Find high-value accounts + 'high_value': n({'balance': gt(100000)}), + + # Find transactions from high-value accounts + 'high_value_txns': ref('high_value', [ + e_forward({'type': 'transaction', 'amount': gt(10000)}) + ]), + + # Find recipients of high-value transactions + 'recipients': ref('high_value_txns', [n()]), + + # Find second-hop connections + 'network': ref('recipients', [ + e_forward({'type': 'transaction'}, hops=2) + ]) +})) +``` + +### RemoteGraph (Load Remote Datasets) + +```python +from graphistry import remote_dataset + +# Load a public dataset +remote_g = remote_dataset('public-dataset-id') +result = remote_g.gfql([n({'type': 'user'})]) + +# Load a private dataset with authentication +remote_g = remote_dataset('private-dataset-id', token='auth-token') + +# Use remote dataset in Let bindings +result = g.gfql(let({ + 'remote_data': remote_dataset('dataset-123'), + 'filtered': ref('remote_data', [n({'active': True})]) +})) +``` + +## Call Operations + +GFQL supports calling Plottable methods through the `call()` function, providing a safe way to invoke graph transformations and analysis operations. + +### Basic Call Usage + +```python +from graphistry import call + +# Calculate node degrees +result = g.gfql([ + n({'type': 'person'}), + call('get_degrees', { + 'col': 'centrality', + 'col_in': 'in_centrality', + 'col_out': 'out_centrality' + }) +]) + +# Access degree columns +degree_df = result._nodes[['centrality', 'in_centrality', 'out_centrality']] +``` + +### Graph Analysis Operations + +```python +# PageRank computation +result = g.gfql([ + call('compute_cugraph', { + 'alg': 'pagerank', + 'out_col': 'pagerank_score', + 'params': {'alpha': 0.85} + }) +]) + +# Community detection +result = g.gfql([ + call('compute_cugraph', { + 'alg': 'louvain', + 'out_col': 'community' + }) +]) + +# Topological analysis +result = g.gfql([ + call('get_topological_levels', { + 'level_col': 'topo_level', + 'allow_cycles': True + }) +]) +``` + +### Layout Operations + +```python +# GPU-accelerated layout +result = g.gfql([ + call('layout_cugraph', { + 'layout': 'force_atlas2', + 'params': { + 'iterations': 500, + 'outbound_attraction_distribution': True + } + }) +]) + +# Graphviz layouts +result = g.gfql([ + call('layout_graphviz', { + 'prog': 'dot', + 'directed': True + }) +]) +``` + +### Filtering and Transformation + +```python +# Complex filtering +result = g.gfql([ + call('filter_nodes_by_dict', { + 'filter_dict': {'type': 'server', 'critical': True} + }), + call('hop', { + 'hops': 2, + 'direction': 'forward', + 'edge_match': {'port': 22} + }) +]) + +# Node transformations +result = g.gfql([ + call('collapse', { + 'column': 'department', + 'self_edges': False + }) +]) +``` + +### Visual Encoding + +```python +# Encode visual properties +result = g.gfql([ + call('encode_point_color', { + 'column': 'risk_score', + 'palette': ['green', 'yellow', 'red'], + 'as_continuous': True + }), + call('encode_point_size', { + 'column': 'importance', + 'categorical_mapping': { + 'low': 10, + 'medium': 20, + 'high': 30 + } + }) +]) +``` + +### Call with Let Bindings + +```python +from graphistry import let, ref, call + +# Combine Let bindings with Call operations +result = g.gfql(let({ + 'high_risk': n({'risk_score': gt(80)}), + 'connected': ref('high_risk', [ + e_forward({'type': 'communicates'}) + ]), + 'analyzed': call('compute_cugraph', { + 'alg': 'pagerank', + 'out_col': 'influence' + }) +})) +``` + +### Available Call Methods + +Call operations are restricted to a safelist of Plottable methods: + +- **Graph Analysis**: `get_degrees`, `get_indegrees`, `get_outdegrees`, `compute_cugraph`, `compute_igraph`, `get_topological_levels` +- **Filtering**: `filter_nodes_by_dict`, `filter_edges_by_dict`, `hop`, `drop_nodes`, `keep_nodes` +- **Transformation**: `collapse`, `prune_self_edges`, `materialize_nodes` +- **Layout**: `layout_cugraph`, `layout_igraph`, `layout_graphviz`, `fa2_layout` +- **Visual Encoding**: `encode_point_color`, `encode_edge_color`, `encode_point_size`, `encode_point_icon` +- **Metadata**: `name`, `description` + +### Call Validation + +Call operations are validated at multiple levels: + +1. **Function validation**: Only safelist methods allowed +2. **Parameter validation**: Type checking for all parameters +3. **Schema validation**: Ensures required columns exist + +```python +try: + result = g.gfql([ + call('dangerous_method', {}) # Raises E303: not in safelist + ]) +except GFQLTypeError as e: + print(f"Error: {e.message}") + +# Parameter type validation +try: + result = g.gfql([ + call('hop', {'hops': 'two'}) # Raises E201: wrong type + ]) +except GFQLTypeError as e: + print(f"Error: {e.message}") +``` + ## Engine Selection GFQL supports multiple execution engines: diff --git a/docs/source/gfql/spec/wire_protocol.md b/docs/source/gfql/spec/wire_protocol.md index beacfdf20f..cbe8c683ce 100644 --- a/docs/source/gfql/spec/wire_protocol.md +++ b/docs/source/gfql/spec/wire_protocol.md @@ -43,7 +43,7 @@ All GFQL wire protocol messages are JSON objects with a `type` field that identi ### Type Identification Each object includes a `type` field: -- Operations: `"Node"`, `"Edge"`, `"Chain"` +- Operations: `"Node"`, `"Edge"`, `"Chain"`, `"Let"`, `"ChainRef"`, `"RemoteGraph"`, `"Call"` - Predicates: `"GT"`, `"LT"`, `"IsIn"`, etc. - Temporal values: `"datetime"`, `"date"`, `"time"` @@ -135,6 +135,233 @@ chain([ } ``` +### Let Bindings (DAG Patterns) + +**Python**: +```python +ASTLet({ + 'persons': n({'type': 'Person'}), + 'adults': ASTChainRef('persons', [n({'age': ge(18)})]), + 'connections': ASTChainRef('adults', [ + e_forward({'type': 'knows'}), + ASTChainRef('adults') + ]) +}) +``` + +**Wire Format**: +```json +{ + "type": "Let", + "bindings": { + "persons": { + "type": "Node", + "filter_dict": {"type": "Person"} + }, + "adults": { + "type": "ChainRef", + "ref": "persons", + "chain": [{ + "type": "Node", + "filter_dict": { + "age": {"type": "GE", "val": 18} + } + }] + }, + "connections": { + "type": "ChainRef", + "ref": "adults", + "chain": [ + { + "type": "Edge", + "direction": "forward", + "edge_match": {"type": "knows"} + }, + { + "type": "ChainRef", + "ref": "adults", + "chain": [] + } + ] + } + } +} +``` + +### ChainRef (Reference to Named Binding) + +**Python**: +```python +ASTChainRef('base_pattern', [ + e_forward({'status': 'active'}), + n({'verified': True}) +]) +``` + +**Wire Format**: +```json +{ + "type": "ChainRef", + "ref": "base_pattern", + "chain": [ + { + "type": "Edge", + "direction": "forward", + "edge_match": {"status": "active"} + }, + { + "type": "Node", + "filter_dict": {"verified": true} + } + ] +} +``` + +### RemoteGraph (Load Remote Dataset) + +**Python**: +```python +ASTRemoteGraph('dataset-123', token='auth-token') +``` + +**Wire Format**: +```json +{ + "type": "RemoteGraph", + "dataset_id": "dataset-123", + "token": "auth-token" +} +``` + +Without token (public dataset): +```json +{ + "type": "RemoteGraph", + "dataset_id": "public-dataset-456" +} +``` + +### Call Operation + +**Python**: +```python +ASTCall('get_degrees', { + 'col': 'centrality', + 'col_in': 'in_centrality', + 'col_out': 'out_centrality' +}) +``` + +**Wire Format**: +```json +{ + "type": "Call", + "function": "get_degrees", + "params": { + "col": "centrality", + "col_in": "in_centrality", + "col_out": "out_centrality" + } +} +``` + +#### Call Operation Examples + +**PageRank computation**: +```json +{ + "type": "Call", + "function": "compute_cugraph", + "params": { + "alg": "pagerank", + "out_col": "pagerank_score", + "params": {"alpha": 0.85} + } +} +``` + +**Graph layout**: +```json +{ + "type": "Call", + "function": "layout_cugraph", + "params": { + "layout": "force_atlas2", + "params": { + "iterations": 500, + "outbound_attraction_distribution": true, + "edge_weight_influence": 1.0 + } + } +} +``` + +**Node filtering**: +```json +{ + "type": "Call", + "function": "filter_nodes_by_dict", + "params": { + "filter_dict": { + "type": "person", + "active": true + } + } +} +``` + +**Complex hop traversal**: +```json +{ + "type": "Call", + "function": "hop", + "params": { + "hops": 3, + "direction": "forward", + "edge_match": {"type": "transfer"}, + "destination_node_match": {"account_type": "checking"} + } +} +``` + +#### Available Call Methods + +The following Plottable methods are available through Call operations: + +**Graph Analysis**: +- `get_degrees`: Calculate node degrees +- `get_indegrees`: Calculate in-degrees only +- `get_outdegrees`: Calculate out-degrees only +- `compute_cugraph`: Run GPU algorithms (pagerank, louvain, etc.) +- `compute_igraph`: Run CPU algorithms +- `get_topological_levels`: Analyze DAG structure + +**Filtering & Transformation**: +- `filter_nodes_by_dict`: Filter nodes by attributes +- `filter_edges_by_dict`: Filter edges by attributes +- `hop`: Traverse graph with complex conditions +- `drop_nodes`: Remove specified nodes +- `keep_nodes`: Keep only specified nodes +- `collapse`: Merge nodes by attribute +- `prune_self_edges`: Remove self-loops +- `materialize_nodes`: Generate node table from edges + +**Layout**: +- `layout_cugraph`: GPU-accelerated layouts +- `layout_igraph`: CPU-based layouts +- `layout_graphviz`: Graphviz layouts (dot, neato, etc.) +- `fa2_layout`: ForceAtlas2 layout + +**Visual Encoding**: +- `encode_point_color`: Map node values to colors +- `encode_edge_color`: Map edge values to colors +- `encode_point_size`: Map node values to sizes +- `encode_point_icon`: Map node values to icons + +**Metadata**: +- `name`: Set visualization name +- `description`: Set visualization description + ## Predicate Serialization ### Comparison Predicates @@ -241,7 +468,7 @@ null // null **Python**: ```python -g.chain([ +g.gfql([ n({"customer_id": "C123"}), e_forward({ "type": "purchase", @@ -284,7 +511,7 @@ g.chain([ **Python**: ```python -g.chain([ +g.gfql([ n({"ip": is_in(["192.168.1.100", "192.168.1.101"])}), e_forward( edge_query="port IN [22, 23, 3389]", @@ -325,6 +552,121 @@ g.chain([ } ``` +### Complex DAG Pattern + +**Python**: +```python +g.gfql(ASTLet({ + 'suspicious_ips': n({'risk_score': gt(80)}), + 'lateral_movement': ASTChainRef('suspicious_ips', [ + e_forward({'type': 'ssh', 'failed_attempts': gt(5)}), + n({'type': 'server'}) + ]), + 'escalation': ASTChainRef('lateral_movement', [ + e_forward({'type': 'privilege_change'}), + n({'admin': True}) + ]) +})) +``` + +**Wire Format**: +```json +{ + "type": "Let", + "bindings": { + "suspicious_ips": { + "type": "Node", + "filter_dict": { + "risk_score": {"type": "GT", "val": 80} + } + }, + "lateral_movement": { + "type": "ChainRef", + "ref": "suspicious_ips", + "chain": [ + { + "type": "Edge", + "direction": "forward", + "edge_match": { + "type": "ssh", + "failed_attempts": {"type": "GT", "val": 5} + } + }, + { + "type": "Node", + "filter_dict": {"type": "server"} + } + ] + }, + "escalation": { + "type": "ChainRef", + "ref": "lateral_movement", + "chain": [ + { + "type": "Edge", + "direction": "forward", + "edge_match": {"type": "privilege_change"} + }, + { + "type": "Node", + "filter_dict": {"admin": true} + } + ] + } + } +} +``` + +### DAG Pattern with Call Operations + +**Python**: +```python +g.gfql(ASTLet({ + 'high_value': n({'amount': gt(100000)}), + 'connected': ASTChainRef('high_value', [ + e_forward({'type': 'transfer'}, hops=2) + ]), + 'analyzed': ASTCall('compute_cugraph', { + 'alg': 'pagerank', + 'out_col': 'influence_score' + }) +})) +``` + +**Wire Format**: +```json +{ + "type": "Let", + "bindings": { + "high_value": { + "type": "Node", + "filter_dict": { + "amount": {"type": "GT", "val": 100000} + } + }, + "connected": { + "type": "ChainRef", + "ref": "high_value", + "chain": [ + { + "type": "Edge", + "direction": "forward", + "edge_match": {"type": "transfer"}, + "hops": 2 + } + ] + }, + "analyzed": { + "type": "Call", + "function": "compute_cugraph", + "params": { + "alg": "pagerank", + "out_col": "influence_score" + } + } + } +} +``` ## Best Practices @@ -333,6 +675,12 @@ g.chain([ 3. **Handle timezones consistently**: Include timezone for datetime values when precision matters (defaults to UTC) 4. **Validate before sending**: Use JSON Schema validation 5. **Handle unknown fields**: Ignore unrecognized fields for compatibility +6. **Let bindings**: Define bindings in dependency order (referenced names must be defined first) +7. **ChainRef validation**: Ensure referenced names exist in the Let binding scope +8. **RemoteGraph security**: Protect authentication tokens in transit and storage +9. **Call operations**: Only use function names from the safelist +10. **Parameter validation**: Ensure Call parameters match expected types +11. **Error handling**: Call operations may fail if schema requirements aren't met ## See Also diff --git a/docs/source/gfql/translate.rst b/docs/source/gfql/translate.rst index 86bedd4b16..1d45ec3d42 100644 --- a/docs/source/gfql/translate.rst +++ b/docs/source/gfql/translate.rst @@ -59,11 +59,11 @@ Finding Nodes with Specific Properties from graphistry import n # df[['id', 'type', ...]] - g.chain([ n({"type": "person"}) ])._nodes + g.gfql([ n({"type": "person"}) ])._nodes **Explanation**: -- **GFQL**: `n({"type": "person"})` filters nodes where `type` is `"person"`. `g.chain([...])` applies this filter to the graph `g`, and `._nodes` retrieves the resulting nodes. The performance is similar to that of Pandas (CPU) or cuDF (GPU). +- **GFQL**: `n({"type": "person"})` filters nodes where `type` is `"person"`. `g.gfql([...])` applies this filter to the graph `g`, and `._nodes` retrieves the resulting nodes. The performance is similar to that of Pandas (CPU) or cuDF (GPU). --- @@ -165,7 +165,7 @@ Performing Multi-Hop Traversals from graphistry import n, e_forward # df[['id', ...]] - g.chain([ + g.gfql([ n({g._node: "Alice"}), e_forward(), e_forward(), n(name='m') ])._nodes.query('m') @@ -208,7 +208,7 @@ Filtering Edges and Nodes with Conditions from graphistry import e_forward # df[['src', 'dst', 'weight', ...]] - g.chain([ e_forward(edge_query='weight > 0.5') ])._edges + g.gfql([ e_forward(edge_query='weight > 0.5') ])._edges **Explanation**: @@ -349,7 +349,7 @@ All Paths and Connectivity # g._edges: df[['src', 'dst', ...]] # g._nodes: df[['id', ...]] - g.chain([ + g.gfql([ n({"id": "Alice"}), e_forward( source_node_query='type == "person"', @@ -437,7 +437,7 @@ Time-Windowed Graph Analytics .. code-block:: python past_week = pd.Timestamp.now() - pd.Timedelta(7) - g.chain([ + g.gfql([ n({"id": {"$in": ["Alice", "Bob"]}}), e_forward(edge_query=f'timestamp >= "{past_week}"'), n({"id": {"$in": ["Alice", "Bob"]}}) @@ -488,7 +488,7 @@ Parallel Pathfinding from graphistry import n, e_forward # g._nodes: cudf.DataFrame[['src', 'dst', ...]] - g.chain([ + g.gfql([ n({"id": "Alice"}), e_forward(to_fixed_point=False), n({"id": is_in(["Bob", "Charlie"])}) @@ -527,7 +527,7 @@ GPU Execution from graphistry import n, e_forward # Executing pathfinding queries in parallel - g.chain([ + g.gfql([ n({"id": "Alice"}), e_forward(to_fixed_point=False), n({"id": is_in(["Bob", "Charlie"])})