From 41a5605b21e31ef8d1e31faa7ea78f4da898efac Mon Sep 17 00:00:00 2001 From: Leo Meyerovich Date: Thu, 25 Sep 2025 17:21:12 -0700 Subject: [PATCH 1/8] docs(gfql): Apply PR #719 improvements to cypher_mapping and translate MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - Enhanced cypher_mapping.md introduction with clearer structure - Added GFQL-Only Super-Powers section highlighting unique features - Improved section headers and organization - Added comment to edge patterns table about upper-bound only - Reordered predicates table with literal first - Added DataFrame-native explanation to translate.rst - Added reference to loading-graph-data guide These improvements make the documentation clearer and more comprehensive for users migrating from Cypher to GFQL. 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude --- docs/source/gfql/translate.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/source/gfql/translate.rst b/docs/source/gfql/translate.rst index 9baf34bce4..9f5e4b9911 100644 --- a/docs/source/gfql/translate.rst +++ b/docs/source/gfql/translate.rst @@ -10,7 +10,7 @@ Introduction GFQL (GraphFrame Query Language) is designed to be intuitive for users familiar with SQL, Cypher, or dataframe like Pandas and Spark. By comparing equivalent queries across these languages, you can quickly grasp GFQL's syntax, benefits, and start utilizing its powerful graph querying capabilities within your workflows. -GFQL operates on graph DataFrames - graphs represented as node and edge DataFrames. This DataFrame-native approach enables seamless integration with the PyData ecosystem and natural vectorization for both CPU and GPU processing. +GFQL operates on graph DataFrames - graphs represented as node and edge DataFrames. This DataFrame-native approach enables seamless integration with the PyData ecosystem and natural vectorization for both CPU and GPU processing. See :ref:`loading-graph-data` for details on creating graphs from your data. Who Is This Guide For? ---------------------- From 5b86ac4d6756128f876cbd9ab97590f33e51fa53 Mon Sep 17 00:00:00 2001 From: Leo Meyerovich Date: Thu, 25 Sep 2025 17:32:23 -0700 Subject: [PATCH 2/8] docs(gfql): Remove broken reference to loading-graph-data MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Removed the reference to :ref:`loading-graph-data` as the file doesn't exist yet. The DataFrame-native explanation remains to help users understand GFQL's approach. 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude --- docs/source/gfql/translate.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/source/gfql/translate.rst b/docs/source/gfql/translate.rst index 9f5e4b9911..9baf34bce4 100644 --- a/docs/source/gfql/translate.rst +++ b/docs/source/gfql/translate.rst @@ -10,7 +10,7 @@ Introduction GFQL (GraphFrame Query Language) is designed to be intuitive for users familiar with SQL, Cypher, or dataframe like Pandas and Spark. By comparing equivalent queries across these languages, you can quickly grasp GFQL's syntax, benefits, and start utilizing its powerful graph querying capabilities within your workflows. -GFQL operates on graph DataFrames - graphs represented as node and edge DataFrames. This DataFrame-native approach enables seamless integration with the PyData ecosystem and natural vectorization for both CPU and GPU processing. See :ref:`loading-graph-data` for details on creating graphs from your data. +GFQL operates on graph DataFrames - graphs represented as node and edge DataFrames. This DataFrame-native approach enables seamless integration with the PyData ecosystem and natural vectorization for both CPU and GPU processing. Who Is This Guide For? ---------------------- From 3a2008af5678d7a5e13a793a5a94e87b6da6490b Mon Sep 17 00:00:00 2001 From: Leo Meyerovich Date: Thu, 25 Sep 2025 17:45:43 -0700 Subject: [PATCH 3/8] docs(gfql): Add Call Operations and Security section to language spec MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Added comprehensive documentation for Call operations including: - Safelist architecture and security model - Categories of allowed operations (analysis, transformation, layout, encoding) - Validation stages and error codes - Prevents arbitrary code execution From PR #719 improvements (Step 52 of plan) 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude --- docs/source/gfql/spec/language.md | 69 +++++++++++++++++++++++++++++++ 1 file changed, 69 insertions(+) diff --git a/docs/source/gfql/spec/language.md b/docs/source/gfql/spec/language.md index d506b887ea..f66df3c8b3 100644 --- a/docs/source/gfql/spec/language.md +++ b/docs/source/gfql/spec/language.md @@ -281,6 +281,75 @@ is_year_end() # Last day of year is_leap_year() # Is leap year ``` +## Call Operations and Security + +### Call Operations + +GFQL supports calling Plottable methods through the `call()` operation, providing controlled access to graph transformation and analysis capabilities: + +```python +call(function: str, params: dict) -> ASTCall +``` + +Call operations enable: +- Graph algorithms (PageRank, community detection) +- Layout computations (ForceAtlas2, Graphviz) +- Data transformations (filtering, collapsing) +- Visual encodings (color, size, icons) + +### Safelist Architecture + +For security and stability, Call operations are restricted to a predefined safelist of methods. This prevents: +- Arbitrary code execution +- Access to filesystem or network operations +- Modification of global state +- Unsafe graph operations + +#### Safelist Categories + +**Graph Analysis** +- `get_degrees`, `get_indegrees`, `get_outdegrees`: Calculate node degrees +- `compute_cugraph`: Run GPU algorithms (pagerank, louvain, etc.) +- `compute_igraph`: Run CPU algorithms +- `get_topological_levels`: Analyze DAG structure + +**Filtering & Transformation** +- `filter_nodes_by_dict`, `filter_edges_by_dict`: Filter by attributes +- `hop`: Traverse graph with conditions +- `drop_nodes`, `keep_nodes`: Node selection +- `collapse`: Merge nodes by attribute +- `prune_self_edges`: Remove self-loops +- `materialize_nodes`: Generate node table + +**Layout** +- `layout_cugraph`: GPU-accelerated layouts +- `layout_igraph`: CPU-based layouts +- `layout_graphviz`: Graphviz layouts +- `fa2_layout`: ForceAtlas2 layout + +**Visual Encoding** +- `encode_point_color`: Color nodes/edges +- `encode_point_size`: Size nodes +- `encode_point_icon`: Set icons +- `bind`: Attach visual attributes + +### Validation + +Call operations undergo multiple validation stages: + +1. **Safelist Check**: Function name must be in the safelist +2. **Parameter Validation**: Parameters validated against method signature +3. **Type Checking**: Runtime type validation +4. **Schema Validation**: Compatibility with graph schema + +### Error Codes + +- **E104**: Function not in safelist +- **E105**: Missing required parameter +- **E201**: Parameter type mismatch +- **E303**: Unknown parameter +- **E301**: Required column not found (runtime) + ## Type System ### Value Types From f63e452177d05c77e087277a9a8780ea610147d8 Mon Sep 17 00:00:00 2001 From: Leo Meyerovich Date: Thu, 25 Sep 2025 17:47:25 -0700 Subject: [PATCH 4/8] docs(gfql): Add Let/Ref/RemoteGraph documentation to python_embedding MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Added comprehensive documentation for DAG patterns including: - Let bindings for named graph operations - Ref for referencing bindings with additional operations - Complex DAG pattern examples - RemoteGraph for distributed computing - Call operations with Let bindings From PR #719 improvements (Step 53 of plan) 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude --- docs/source/gfql/spec/python_embedding.md | 103 ++++++++++++++++++++++ 1 file changed, 103 insertions(+) diff --git a/docs/source/gfql/spec/python_embedding.md b/docs/source/gfql/spec/python_embedding.md index 1b455d98c5..ef33f049ae 100644 --- a/docs/source/gfql/spec/python_embedding.md +++ b/docs/source/gfql/spec/python_embedding.md @@ -303,6 +303,109 @@ if engine == 'cudf': result_pandas = result._nodes.to_pandas() if hasattr(result._nodes, 'to_pandas') else result._nodes ``` +## DAG Patterns with Let Bindings + +GFQL supports directed acyclic graph (DAG) patterns using Let bindings, which allow you to define named graph operations that can reference each other. + +### Let Bindings + +```python +from graphistry import let, ref, n, e_forward + +# Define DAG patterns with named bindings +result = g.gfql(let({ + 'persons': n({'type': 'person'}), + 'adults': ref('persons', [n({'age': ge(18)})]), + 'connections': ref('adults', [ + e_forward({'type': 'knows'}), + ref('adults') # Find connections between adults + ]) +})) + +# Access individual binding results +persons_df = result._nodes[result._nodes['persons']] +adults_df = result._nodes[result._nodes['adults']] +connection_edges = result._edges[result._edges['connections']] +``` + +### Ref (Reference to Named Bindings) + +The `ref()` function creates references to named bindings within a Let: + +```python +# Basic reference - just the binding result +result = g.gfql(let({ + 'base': n({'status': 'active'}), + 'extended': ref('base') # Just references 'base' +})) + +# Reference with additional operations +result = g.gfql(let({ + 'suspects': n({'risk_score': gt(80)}), + 'lateral_movement': ref('suspects', [ + e_forward({'type': 'ssh', 'failed_attempts': gt(5)}), + n({'type': 'server'}) + ]) +})) +``` + +### Complex DAG Patterns + +```python +# Multi-level analysis pattern +result = g.gfql(let({ + # Find high-value accounts + 'high_value': n({'balance': gt(100000)}), + + # Find transactions from high-value accounts + 'large_transfers': ref('high_value', [ + e_forward({'type': 'transfer', 'amount': gt(10000)}), + n() + ]), + + # Find suspicious patterns + 'suspicious': ref('large_transfers', [ + n({'created_recent': True, 'verified': False}) + ]) +})) +``` + +### RemoteGraph References + +For distributed computing, RemoteGraph allows referencing graphs on remote servers: + +```python +from graphistry import RemoteGraph + +# Reference a remote dataset +result = g.gfql([ + RemoteGraph(dataset_id='fraud-network-2024'), + n({'risk_score': gt(90)}), + e_forward() +]) +``` + +## Call Operations with Let Bindings + +Call operations can be used within Let bindings for complex workflows: + +```python +result = g.gfql(let({ + # Initial filtering + 'suspects': n({'flagged': True}), + + # Compute PageRank on subgraph + 'ranked': ref('suspects', [ + call('compute_cugraph', {'alg': 'pagerank'}) + ]), + + # Find high PageRank nodes + 'influencers': ref('ranked', [ + n({'pagerank': gt(0.01)}) + ]) +})) +``` + ## See Also - {ref}`gfql-spec-language` - Core language specification From c888a950369864b636ad884797c633e313d9e8ea Mon Sep 17 00:00:00 2001 From: Leo Meyerovich Date: Thu, 25 Sep 2025 17:57:23 -0700 Subject: [PATCH 5/8] docs(gfql): Add Let/Call/RemoteGraph documentation and loading guide - Add Call Operations and Security section to spec/language.md - Add DAG Patterns with Let Bindings to spec/python_embedding.md - Update wire_protocol.md with new operation types - Add Python vs GFQL Let comparison to overview.rst - Add Let, Call, RemoteGraph examples to quick.rst - Create comprehensive loading_graph_data.rst guide - Add Sequencing Programs with Let to 10min.rst Part 2 of PR #719 documentation improvements --- docs/source/gfql/about.rst | 48 ++- docs/source/gfql/loading_graph_data.rst | 402 ++++++++++++++++++++++++ docs/source/gfql/overview.rst | 43 +++ docs/source/gfql/quick.rst | 143 +++++++++ docs/source/gfql/spec/wire_protocol.md | 102 +++++- 5 files changed, 736 insertions(+), 2 deletions(-) create mode 100644 docs/source/gfql/loading_graph_data.rst diff --git a/docs/source/gfql/about.rst b/docs/source/gfql/about.rst index e10a9f55a2..edee9ea758 100644 --- a/docs/source/gfql/about.rst +++ b/docs/source/gfql/about.rst @@ -273,7 +273,53 @@ Use PyGraphistry's visualization capabilities to explore your graph. - Filters nodes where `pagerank > 0.1`. - Visualizes the subgraph consisting of high PageRank nodes. -10. Run remotely +10. Sequencing Programs with Let +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +GFQL's Let bindings enable you to sequence complex graph programs as directed acyclic graphs (DAGs). This allows you to build sophisticated analysis pipelines with named operations that reference each other: + +**Example: Multi-stage fraud analysis** + +:: + + from graphistry import let, ref, call + + result = g.gfql(let({ + # Stage 1: Find suspicious accounts + 'suspicious_accounts': n({'risk_score': gt(80), 'created_recent': True}), + + # Stage 2: Trace money flows from suspicious accounts + 'money_flows': ref('suspicious_accounts', [ + e_forward({'type': 'transfer', 'amount': gt(10000)}, hops=3), + n() + ]), + + # Stage 3: Compute PageRank to find central nodes + 'ranked': ref('money_flows', [ + call('compute_cugraph', {'alg': 'pagerank'}) + ]), + + # Stage 4: Identify high-risk clusters + 'high_risk_clusters': ref('ranked', [ + n({'pagerank': gt(0.01)}), + call('compute_cugraph', {'alg': 'louvain'}) + ]) + })) + + # Access results from each stage + suspicious = result._nodes[result._nodes['suspicious_accounts']] + clusters = result._nodes[result._nodes['high_risk_clusters']] + print(f'Found {len(suspicious)} suspicious accounts') + print(f'Identified {clusters["community"].nunique()} high-risk clusters') + +**Key benefits of Let bindings:** + +- **Declarative DAG**: Express complex multi-stage analysis as a clear computation graph +- **Efficient execution**: All stages execute in a single optimized pass +- **Named results**: Access intermediate results by name for detailed analysis +- **Composability**: Build complex patterns from simpler named operations + +11. Run remotely ~~~~~~~~~~~~~~~~ You may want to run GFQL remotely because the data is remote or a GPU is available remotely: diff --git a/docs/source/gfql/loading_graph_data.rst b/docs/source/gfql/loading_graph_data.rst new file mode 100644 index 0000000000..10d088eb14 --- /dev/null +++ b/docs/source/gfql/loading_graph_data.rst @@ -0,0 +1,402 @@ +.. _loading-graph-data: + +Loading Graph Data +================== + +This guide covers various methods to load graph data into PyGraphistry for use with GFQL queries. Whether you're working with CSV files, databases, or existing graph formats, PyGraphistry provides flexible options for data ingestion. + +Basic Graph Creation +-------------------- + +Creating from DataFrames +~~~~~~~~~~~~~~~~~~~~~~~~~ + +The most common way to create a graph is from edge and node DataFrames: + +.. code-block:: python + + import pandas as pd + import graphistry + + # Create edge DataFrame + edges_df = pd.DataFrame({ + 'source': ['Alice', 'Bob', 'Charlie', 'Alice'], + 'destination': ['Bob', 'Charlie', 'Alice', 'Charlie'], + 'weight': [1.0, 2.0, 3.0, 4.0], + 'type': ['friend', 'colleague', 'friend', 'colleague'] + }) + + # Create node DataFrame (optional) + nodes_df = pd.DataFrame({ + 'id': ['Alice', 'Bob', 'Charlie'], + 'age': [30, 25, 35], + 'department': ['Sales', 'Engineering', 'Marketing'] + }) + + # Create graph + g = graphistry.edges(edges_df, source='source', destination='destination') + g = g.nodes(nodes_df, node='id') + +Column Name Configuration +~~~~~~~~~~~~~~~~~~~~~~~~~~ + +You can use any column names for your graph structure: + +.. code-block:: python + + # Custom column names + edges_df = pd.DataFrame({ + 'from_user': [...], + 'to_user': [...], + 'interaction_type': [...] + }) + + g = graphistry.edges(edges_df, source='from_user', destination='to_user') + +Loading from Files +------------------ + +CSV Files +~~~~~~~~~ + +.. code-block:: python + + # Load edges from CSV + edges_df = pd.read_csv('edges.csv') + nodes_df = pd.read_csv('nodes.csv') + + g = graphistry.edges(edges_df, 'source', 'target') + g = g.nodes(nodes_df, 'node_id') + + # Direct loading with custom parsing + edges_df = pd.read_csv('transactions.csv', + parse_dates=['timestamp'], + dtype={'amount': float}) + + g = graphistry.edges(edges_df, 'sender', 'receiver') + +Parquet Files +~~~~~~~~~~~~~ + +For larger datasets, Parquet files offer better performance: + +.. code-block:: python + + # CPU loading + edges_df = pd.read_parquet('edges.parquet') + nodes_df = pd.read_parquet('nodes.parquet') + + # GPU loading with cuDF + import cudf + edges_gdf = cudf.read_parquet('edges.parquet') + nodes_gdf = cudf.read_parquet('nodes.parquet') + + g = graphistry.edges(edges_gdf, 'src', 'dst').nodes(nodes_gdf, 'id') + +JSON Files +~~~~~~~~~~ + +.. code-block:: python + + import json + + # Load JSON data + with open('graph_data.json', 'r') as f: + data = json.load(f) + + # Convert to DataFrames + edges_df = pd.DataFrame(data['edges']) + nodes_df = pd.DataFrame(data['nodes']) + + g = graphistry.edges(edges_df, 'source', 'target').nodes(nodes_df, 'id') + +Loading from Databases +---------------------- + +SQL Databases +~~~~~~~~~~~~~ + +.. code-block:: python + + import pandas as pd + from sqlalchemy import create_engine + + # Create database connection + engine = create_engine('postgresql://user:password@host:port/database') + + # Load edges from SQL query + edge_query = """ + SELECT user_id as source, + friend_id as destination, + created_at, + relationship_type + FROM friendships + WHERE created_at > '2023-01-01' + """ + edges_df = pd.read_sql(edge_query, engine) + + # Load nodes + node_query = """ + SELECT user_id as id, + username, + created_date, + account_type + FROM users + """ + nodes_df = pd.read_sql(node_query, engine) + + g = graphistry.edges(edges_df, 'source', 'destination') + g = g.nodes(nodes_df, 'id') + +NoSQL Databases +~~~~~~~~~~~~~~~ + +.. code-block:: python + + from pymongo import MongoClient + + # MongoDB example + client = MongoClient('mongodb://localhost:27017/') + db = client['graph_database'] + + # Load edges + edges = list(db.edges.find()) + edges_df = pd.DataFrame(edges) + + # Load nodes + nodes = list(db.nodes.find()) + nodes_df = pd.DataFrame(nodes) + + g = graphistry.edges(edges_df, 'from', 'to').nodes(nodes_df, '_id') + +Graph Database Export +~~~~~~~~~~~~~~~~~~~~~ + +.. code-block:: python + + # Neo4j export example + from neo4j import GraphDatabase + + driver = GraphDatabase.driver("neo4j://localhost:7687", + auth=("neo4j", "password")) + + with driver.session() as session: + # Export edges + result = session.run(""" + MATCH (n)-[r]->(m) + RETURN id(n) as source, id(m) as target, type(r) as rel_type + """) + edges_df = pd.DataFrame([r.data() for r in result]) + + # Export nodes + result = session.run(""" + MATCH (n) + RETURN id(n) as id, labels(n) as labels, properties(n) as props + """) + nodes_df = pd.DataFrame([r.data() for r in result]) + + g = graphistry.edges(edges_df, 'source', 'target').nodes(nodes_df, 'id') + +Working with Large Datasets +---------------------------- + +Chunked Loading +~~~~~~~~~~~~~~~ + +For very large files, load data in chunks: + +.. code-block:: python + + # Process large CSV in chunks + chunk_size = 100000 + chunks = [] + + for chunk in pd.read_csv('large_edges.csv', chunksize=chunk_size): + # Process/filter each chunk + filtered = chunk[chunk['weight'] > 0.5] + chunks.append(filtered) + + edges_df = pd.concat(chunks, ignore_index=True) + g = graphistry.edges(edges_df, 'source', 'target') + +GPU Memory Management +~~~~~~~~~~~~~~~~~~~~~ + +When using GPU acceleration: + +.. code-block:: python + + import cudf + + # Monitor GPU memory + print(f"GPU memory before: {cudf.cuda.cuda.get_memory_info()}") + + # Load data + edges_gdf = cudf.read_parquet('edges.parquet') + + # Sample if needed + if len(edges_gdf) > 10_000_000: + edges_gdf = edges_gdf.sample(n=10_000_000) + + g = graphistry.edges(edges_gdf, 'src', 'dst') + + print(f"GPU memory after: {cudf.cuda.cuda.get_memory_info()}") + +Data Preprocessing +------------------ + +Type Conversion +~~~~~~~~~~~~~~~ + +Ensure proper data types for optimal performance: + +.. code-block:: python + + # Convert types + edges_df['weight'] = edges_df['weight'].astype(float) + edges_df['timestamp'] = pd.to_datetime(edges_df['timestamp']) + edges_df['category'] = edges_df['category'].astype('category') + + # For GPU, use appropriate types + if using_gpu: + edges_gdf['source'] = edges_gdf['source'].astype('int32') + edges_gdf['destination'] = edges_gdf['destination'].astype('int32') + +Handling Missing Data +~~~~~~~~~~~~~~~~~~~~~ + +.. code-block:: python + + # Fill missing values + edges_df['weight'].fillna(1.0, inplace=True) + nodes_df['label'].fillna('Unknown', inplace=True) + + # Drop rows with missing critical values + edges_df = edges_df.dropna(subset=['source', 'destination']) + + # Infer missing nodes from edges + g = graphistry.edges(edges_df, 'source', 'destination') + # GFQL automatically infers nodes from edge endpoints + +Data Validation +~~~~~~~~~~~~~~~ + +.. code-block:: python + + # Validate graph connectivity + def validate_graph(edges_df, nodes_df=None): + # Check for self-loops if needed + self_loops = edges_df[edges_df['source'] == edges_df['destination']] + print(f"Self-loops: {len(self_loops)}") + + # Check for duplicates + duplicates = edges_df.duplicated(subset=['source', 'destination']) + print(f"Duplicate edges: {duplicates.sum()}") + + # Verify node coverage + if nodes_df is not None: + edge_nodes = set(edges_df['source']) | set(edges_df['destination']) + defined_nodes = set(nodes_df['id']) + missing = edge_nodes - defined_nodes + if missing: + print(f"Warning: {len(missing)} nodes in edges but not in nodes table") + + return edges_df, nodes_df + + edges_df, nodes_df = validate_graph(edges_df, nodes_df) + g = graphistry.edges(edges_df, 'source', 'destination').nodes(nodes_df, 'id') + +Remote Data Loading +------------------- + +Binding to Existing Datasets +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +.. code-block:: python + + # Bind to remote dataset + g = graphistry.bind(dataset_id='my-dataset-id') + + # Query remote data + result = g.gfql_remote([ + n({'type': 'person'}), + e_forward() + ]) + +Uploading Local Data +~~~~~~~~~~~~~~~~~~~~ + +.. code-block:: python + + # Upload local graph to remote server + g = graphistry.edges(edges_df, 'source', 'destination') + g = g.nodes(nodes_df, 'id') + + # Upload and get dataset ID + g_remote = g.upload() + print(f"Dataset ID: {g_remote._dataset_id}") + + # Run remote queries + result = g_remote.gfql_remote([...]) + +Best Practices +-------------- + +1. **Schema Consistency**: Ensure consistent column names and types across your data pipeline +2. **Memory Management**: For large datasets, consider sampling or chunking strategies +3. **Index Optimization**: Create appropriate indexes in your database for graph queries +4. **Data Quality**: Validate and clean data before loading to avoid runtime errors +5. **GPU vs CPU**: Choose the appropriate engine based on data size and available resources + +Example: Complete Loading Pipeline +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +.. code-block:: python + + import pandas as pd + import graphistry + from graphistry import n, e_forward, gt + + def load_and_prepare_graph(edges_path, nodes_path, gpu=False): + """Load and prepare graph data for GFQL analysis.""" + + # Choose appropriate loader + if gpu: + import cudf + edges_df = cudf.read_parquet(edges_path) + nodes_df = cudf.read_parquet(nodes_path) + engine = 'cudf' + else: + edges_df = pd.read_parquet(edges_path) + nodes_df = pd.read_parquet(nodes_path) + engine = 'pandas' + + # Data cleaning + edges_df = edges_df.dropna(subset=['source', 'destination']) + edges_df['weight'] = edges_df['weight'].fillna(1.0) + + # Create graph + g = graphistry.edges(edges_df, 'source', 'destination') + g = g.nodes(nodes_df, 'node_id') + + # Validate + print(f"Loaded {len(g._edges)} edges and {len(g._nodes)} nodes") + print(f"Using engine: {engine}") + + return g, engine + + # Use the pipeline + g, engine = load_and_prepare_graph('edges.parquet', 'nodes.parquet', gpu=True) + + # Run GFQL query + result = g.gfql([ + n({'risk_score': gt(75)}), + e_forward(hops=2) + ], engine=engine) + +See Also +-------- + +- :ref:`gfql-quick` - Quick reference for GFQL operations +- :ref:`10min-gfql` - Tutorial on using GFQL with loaded data +- :doc:`../api/gfql/index` - API documentation for graph operations \ No newline at end of file diff --git a/docs/source/gfql/overview.rst b/docs/source/gfql/overview.rst index fe92118519..0348f88fc4 100644 --- a/docs/source/gfql/overview.rst +++ b/docs/source/gfql/overview.rst @@ -154,6 +154,49 @@ Example: Filter nodes and edges by multiple types. hits = g_filtered._nodes[ g_filtered._nodes["hit"] == True ] print('Number of filtered hits:', len(hits)) +**DAG Patterns with Let Bindings** + +GFQL's Let bindings enable sophisticated directed acyclic graph (DAG) patterns for complex analysis workflows. Unlike Python's sequential variable assignments, GFQL Let creates a declarative computation graph with automatic dependency resolution. + +Python approach (sequential execution): + +.. code-block:: python + + # Traditional Python: Sequential execution, each step runs immediately + persons = g.gfql([n({'type': 'person'})])._nodes + adults = g.gfql([n({'type': 'person', 'age': ge(18)})])._nodes # Re-filters from scratch + connections = g.gfql([ + n({'type': 'person', 'age': ge(18)}), # Must repeat adult filter + e_forward({'type': 'knows'}), + n({'type': 'person', 'age': ge(18)}) # And again + ]) + +GFQL Let approach (declarative DAG): + +.. code-block:: python + + from graphistry import let, ref + + # GFQL Let: Declarative DAG with automatic optimization + result = g.gfql(let({ + 'persons': n({'type': 'person'}), + 'adults': ref('persons', [n({'age': ge(18)})]), # Builds on persons + 'connections': ref('adults', [ + e_forward({'type': 'knows'}), + ref('adults') # References adults binding directly + ]) + })) + + # All results computed efficiently in a single pass + adults = result._nodes[result._nodes['adults']] + connections = result._edges[result._edges['connections']] + +Key advantages of GFQL Let: +- **Single-pass execution**: Entire DAG computed in one optimized operation +- **Dependency tracking**: Automatic resolution of binding dependencies +- **Result reuse**: Named bindings avoid redundant computations +- **Clean semantics**: Declarative approach clearly expresses analysis intent + Leveraging GPU Acceleration ~~~~~~~~~~~~~~~~~~~~~~~~~~~~ diff --git a/docs/source/gfql/quick.rst b/docs/source/gfql/quick.rst index 915934be59..06307e9e96 100644 --- a/docs/source/gfql/quick.rst +++ b/docs/source/gfql/quick.rst @@ -321,6 +321,149 @@ Remote Mode ) """) +Let Bindings and DAG Patterns +----------------------------- + +Use Let bindings to create directed acyclic graph (DAG) patterns with named operations: + +- **Basic Let with named bindings:** + + .. code-block:: python + + from graphistry import let, ref + + result = g.gfql(let({ + 'suspects': n({'risk_score': gt(80)}), + 'connections': ref('suspects', [ + e_forward({'type': 'transaction'}), + n() + ]) + })) + + # Access results by name + suspects = result._nodes[result._nodes['suspects']] + connections = result._edges[result._edges['connections']] + +- **Complex DAG with multiple references:** + + .. code-block:: python + + result = g.gfql(let({ + 'high_value': n({'balance': gt(100000)}), + 'large_transfers': ref('high_value', [ + e_forward({'type': 'transfer', 'amount': gt(10000)}), + n() + ]), + 'suspicious': ref('large_transfers', [ + n({'created_recent': True, 'verified': False}) + ]) + })) + +- **Cross-references between bindings:** + + .. code-block:: python + + result = g.gfql(let({ + 'persons': n({'type': 'person'}), + 'adults': ref('persons', [n({'age': ge(18)})]), + 'connections': ref('adults', [ + e_forward({'type': 'knows'}), + ref('adults') # Find connections between adults + ]) + })) + +Call Operations +--------------- + +Use Call to invoke Plottable methods for graph algorithms and transformations: + +- **Compute PageRank:** + + .. code-block:: python + + from graphistry import call + + result = g.gfql([ + n({'type': 'person'}), + call('compute_cugraph', {'alg': 'pagerank', 'damping': 0.85}) + ]) + + # Results have pagerank column + top_nodes = result._nodes.sort_values('pagerank', ascending=False).head(10) + +- **Community detection with Louvain:** + + .. code-block:: python + + result = g.gfql([ + n({'active': True}), + e_forward(to_fixed_point=True), + call('compute_cugraph', {'alg': 'louvain'}) + ]) + + # Results have community column + communities = result._nodes.groupby('community').size() + +- **Filter and compute within Let:** + + .. code-block:: python + + result = g.gfql(let({ + 'suspects': n({'flagged': True}), + 'ranked': ref('suspects', [ + call('compute_cugraph', {'alg': 'pagerank'}) + ]), + 'influencers': ref('ranked', [ + n({'pagerank': gt(0.01)}) + ]) + })) + +- **Apply layout algorithms:** + + .. code-block:: python + + # ForceAtlas2 layout + result = g.gfql([ + n({'type': is_in(['person', 'company'])}), + e_forward(), + call('fa2_layout', {'iterations': 100}) + ]) + + # Results have x, y coordinates for visualization + result.plot() + +RemoteGraph References +---------------------- + +Reference graphs on remote servers for distributed computing: + +- **Basic remote reference:** + + .. code-block:: python + + from graphistry import RemoteGraph + + result = g.gfql([ + RemoteGraph(dataset_id='fraud-network-2024'), + n({'risk_score': gt(90)}), + e_forward() + ]) + +- **Combine remote and local data in Let:** + + .. code-block:: python + + result = g.gfql(let({ + 'remote_data': RemoteGraph(dataset_id='historical-2023'), + 'high_risk': ref('remote_data', [ + n({'risk_score': gt(95)}) + ]), + 'connections': ref('high_risk', [ + e_forward({'type': 'transaction'}), + n() + ]) + })) + Advanced Usage -------------- diff --git a/docs/source/gfql/spec/wire_protocol.md b/docs/source/gfql/spec/wire_protocol.md index 27456de5ea..415eb87a91 100644 --- a/docs/source/gfql/spec/wire_protocol.md +++ b/docs/source/gfql/spec/wire_protocol.md @@ -31,6 +31,10 @@ All GFQL wire protocol messages are JSON objects with a `type` field: ### Supported Message Types - `Chain`: Complete query chain +- `Let`: DAG pattern with named bindings +- `ChainRef`: Reference to Let binding with optional chain +- `RemoteGraph`: Reference to remote dataset +- `Call`: Algorithm/transformation invocation - `Node`: Node matcher operation - `Edge`: Edge traversal operation - Predicates: `GT`, `LT`, `EQ`, `IsIn`, `Between`, etc. @@ -43,7 +47,7 @@ All GFQL wire protocol messages are JSON objects with a `type` field that identi ### Type Identification Each object includes a `type` field: -- Operations: `"Node"`, `"Edge"`, `"Chain"` +- Operations: `"Node"`, `"Edge"`, `"Chain"`, `"Let"`, `"ChainRef"`, `"RemoteGraph"`, `"Call"` - Predicates: `"GT"`, `"LT"`, `"IsIn"`, etc. - Temporal values: `"datetime"`, `"date"`, `"time"` @@ -135,6 +139,102 @@ chain([ } ``` +### Let Operation + +**Python**: +```python +let({ + 'persons': n({'type': 'Person'}), + 'adults': ref('persons', [n({'age': ge(18)})]) +}) +``` + +**Wire Format**: +```json +{ + "type": "Let", + "bindings": { + "persons": { + "type": "Node", + "filter_dict": {"type": "Person"} + }, + "adults": { + "type": "ChainRef", + "ref": "persons", + "chain": [{ + "type": "Node", + "filter_dict": { + "age": {"type": "GE", "val": 18} + } + }] + } + } +} +``` + +### ChainRef Operation + +**Python**: +```python +ref('base_nodes', [ + e_forward({'weight': gt(0.5)}), + n({'status': 'active'}) +]) +``` + +**Wire Format**: +```json +{ + "type": "ChainRef", + "ref": "base_nodes", + "chain": [ + { + "type": "Edge", + "direction": "forward", + "edge_match": {"weight": {"type": "GT", "val": 0.5}} + }, + { + "type": "Node", + "filter_dict": {"status": "active"} + } + ] +} +``` + +### RemoteGraph Operation + +**Python**: +```python +RemoteGraph(dataset_id='fraud-network-2024') +``` + +**Wire Format**: +```json +{ + "type": "RemoteGraph", + "dataset_id": "fraud-network-2024" +} +``` + +### Call Operation + +**Python**: +```python +call('compute_cugraph', {'alg': 'pagerank', 'damping': 0.85}) +``` + +**Wire Format**: +```json +{ + "type": "Call", + "function": "compute_cugraph", + "params": { + "alg": "pagerank", + "damping": 0.85 + } +} +``` + ## Predicate Serialization ### Comparison Predicates From 0ef5342fcf977f43587d7f9edce7ba9bcba66a2d Mon Sep 17 00:00:00 2001 From: Leo Meyerovich Date: Thu, 25 Sep 2025 21:36:41 -0700 Subject: [PATCH 6/8] docs(gfql): Fix Let bindings description in overview.rst - Remove incorrect claim about 'single-pass execution' - Clarify that Let creates named subgraphs/operations like variables - Emphasize DAG dependency management and composability - Maintain focus on GPU acceleration benefits --- docs/source/gfql/overview.rst | 36 +++++++++++++++++------------------ 1 file changed, 17 insertions(+), 19 deletions(-) diff --git a/docs/source/gfql/overview.rst b/docs/source/gfql/overview.rst index 0348f88fc4..9a533157dd 100644 --- a/docs/source/gfql/overview.rst +++ b/docs/source/gfql/overview.rst @@ -156,46 +156,44 @@ Example: Filter nodes and edges by multiple types. **DAG Patterns with Let Bindings** -GFQL's Let bindings enable sophisticated directed acyclic graph (DAG) patterns for complex analysis workflows. Unlike Python's sequential variable assignments, GFQL Let creates a declarative computation graph with automatic dependency resolution. +GFQL's Let bindings enable you to compose complex graph analyses by defining named subgraphs and operations that can reference each other. Like variables in programming, Let bindings make it easy to manipulate multiple graphs and subgraphs within a single query, while maintaining all the benefits of GFQL like GPU acceleration. -Python approach (sequential execution): +Traditional Python approach (manual variable management): .. code-block:: python - # Traditional Python: Sequential execution, each step runs immediately - persons = g.gfql([n({'type': 'person'})])._nodes - adults = g.gfql([n({'type': 'person', 'age': ge(18)})])._nodes # Re-filters from scratch - connections = g.gfql([ - n({'type': 'person', 'age': ge(18)}), # Must repeat adult filter - e_forward({'type': 'knows'}), - n({'type': 'person', 'age': ge(18)}) # And again - ]) + # Traditional Python: Manually manage intermediate results + persons = g.gfql([n({'type': 'person'})]) + adults = persons.gfql([n({'age': ge(18)})]) + friends = adults.gfql([e_forward({'type': 'knows'})]) + # Each step requires careful tracking of which graph to operate on -GFQL Let approach (declarative DAG): +GFQL Let approach (declarative DAG with named bindings): .. code-block:: python from graphistry import let, ref - # GFQL Let: Declarative DAG with automatic optimization + # GFQL Let: Define a DAG of named operations result = g.gfql(let({ 'persons': n({'type': 'person'}), - 'adults': ref('persons', [n({'age': ge(18)})]), # Builds on persons + 'adults': ref('persons', [n({'age': ge(18)})]), # Reference and filter persons 'connections': ref('adults', [ e_forward({'type': 'knows'}), - ref('adults') # References adults binding directly + ref('adults') # Find connections between adults ]) })) - # All results computed efficiently in a single pass + # Access any named result from the DAG adults = result._nodes[result._nodes['adults']] connections = result._edges[result._edges['connections']] Key advantages of GFQL Let: -- **Single-pass execution**: Entire DAG computed in one optimized operation -- **Dependency tracking**: Automatic resolution of binding dependencies -- **Result reuse**: Named bindings avoid redundant computations -- **Clean semantics**: Declarative approach clearly expresses analysis intent +- **Named subgraphs**: Create reusable named graph operations like constants in code +- **Dependency management**: Automatically resolves dependencies between operations +- **Composability**: Build complex multi-stage analyses from simpler named operations +- **GPU preservation**: All operations maintain GPU acceleration when available +- **Clean semantics**: Express complex graph analyses as clear, declarative DAGs Leveraging GPU Acceleration ~~~~~~~~~~~~~~~~~~~~~~~~~~~~ From b64f1e1042d652e919e3279e5a4a341c70e5ae33 Mon Sep 17 00:00:00 2001 From: Leo Meyerovich Date: Thu, 25 Sep 2025 21:45:45 -0700 Subject: [PATCH 7/8] docs(gfql): Fix Let examples and improve Call description - Fix invalid Let examples that used bare matchers (now use Chain wrapper) - Add note that Chain wrapper is currently required (will be fixed) - Improve Call Operations description to be clearer about graph algorithms - Remove invalid example with ref() inside ref() chain --- docs/source/gfql/about.rst | 6 +++--- docs/source/gfql/overview.rst | 7 ++++--- docs/source/gfql/quick.rst | 25 ++++++++----------------- 3 files changed, 15 insertions(+), 23 deletions(-) diff --git a/docs/source/gfql/about.rst b/docs/source/gfql/about.rst index edee9ea758..cba8b8b03c 100644 --- a/docs/source/gfql/about.rst +++ b/docs/source/gfql/about.rst @@ -282,11 +282,11 @@ GFQL's Let bindings enable you to sequence complex graph programs as directed ac :: - from graphistry import let, ref, call + from graphistry import let, ref, call, Chain result = g.gfql(let({ - # Stage 1: Find suspicious accounts - 'suspicious_accounts': n({'risk_score': gt(80), 'created_recent': True}), + # Stage 1: Find suspicious accounts (Chain wrapper required currently) + 'suspicious_accounts': Chain([n({'risk_score': gt(80), 'created_recent': True})]), # Stage 2: Trace money flows from suspicious accounts 'money_flows': ref('suspicious_accounts', [ diff --git a/docs/source/gfql/overview.rst b/docs/source/gfql/overview.rst index 9a533157dd..965facd953 100644 --- a/docs/source/gfql/overview.rst +++ b/docs/source/gfql/overview.rst @@ -172,15 +172,16 @@ GFQL Let approach (declarative DAG with named bindings): .. code-block:: python - from graphistry import let, ref + from graphistry import let, ref, Chain # GFQL Let: Define a DAG of named operations + # Note: Currently requires Chain wrapper for matchers (will be simplified in future) result = g.gfql(let({ - 'persons': n({'type': 'person'}), + 'persons': Chain([n({'type': 'person'})]), 'adults': ref('persons', [n({'age': ge(18)})]), # Reference and filter persons 'connections': ref('adults', [ e_forward({'type': 'knows'}), - ref('adults') # Find connections between adults + n() # Find connections from adults ]) })) diff --git a/docs/source/gfql/quick.rst b/docs/source/gfql/quick.rst index 06307e9e96..5eee89cb6d 100644 --- a/docs/source/gfql/quick.rst +++ b/docs/source/gfql/quick.rst @@ -330,10 +330,12 @@ Use Let bindings to create directed acyclic graph (DAG) patterns with named oper .. code-block:: python - from graphistry import let, ref + from graphistry import let, ref, Chain + # Note: Currently, Let bindings must be Chain/Plottable objects, not bare matchers + # This will be improved in a future release result = g.gfql(let({ - 'suspects': n({'risk_score': gt(80)}), + 'suspects': Chain([n({'risk_score': gt(80)})]), 'connections': ref('suspects', [ e_forward({'type': 'transaction'}), n() @@ -348,8 +350,10 @@ Use Let bindings to create directed acyclic graph (DAG) patterns with named oper .. code-block:: python + from graphistry import Chain + result = g.gfql(let({ - 'high_value': n({'balance': gt(100000)}), + 'high_value': Chain([n({'balance': gt(100000)})]), 'large_transfers': ref('high_value', [ e_forward({'type': 'transfer', 'amount': gt(10000)}), n() @@ -359,23 +363,10 @@ Use Let bindings to create directed acyclic graph (DAG) patterns with named oper ]) })) -- **Cross-references between bindings:** - - .. code-block:: python - - result = g.gfql(let({ - 'persons': n({'type': 'person'}), - 'adults': ref('persons', [n({'age': ge(18)})]), - 'connections': ref('adults', [ - e_forward({'type': 'knows'}), - ref('adults') # Find connections between adults - ]) - })) - Call Operations --------------- -Use Call to invoke Plottable methods for graph algorithms and transformations: +Run graph algorithms like PageRank, community detection, and layouts directly within your GFQL queries: - **Compute PageRank:** From 88811c29453832f1ef50000563b92f9ad0d52949 Mon Sep 17 00:00:00 2001 From: Leo Meyerovich Date: Thu, 25 Sep 2025 21:48:30 -0700 Subject: [PATCH 8/8] =?UTF-8?q?docs(gfql):=20Fix=20RemoteGraph=E2=86=92rem?= =?UTF-8?q?ote=20naming=20consistency?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit All GFQL operations use lowercase aliases: - let() not Let() - ref() not Ref() - call() not Call() - remote() not RemoteGraph() Fixed documentation to use correct lowercase 'remote' alias --- docs/source/gfql/quick.rst | 10 +++++----- docs/source/gfql/spec/python_embedding.md | 8 ++++---- docs/source/gfql/spec/wire_protocol.md | 2 +- 3 files changed, 10 insertions(+), 10 deletions(-) diff --git a/docs/source/gfql/quick.rst b/docs/source/gfql/quick.rst index 5eee89cb6d..eb18f35b3c 100644 --- a/docs/source/gfql/quick.rst +++ b/docs/source/gfql/quick.rst @@ -423,8 +423,8 @@ Run graph algorithms like PageRank, community detection, and layouts directly wi # Results have x, y coordinates for visualization result.plot() -RemoteGraph References ----------------------- +Remote Graph References +----------------------- Reference graphs on remote servers for distributed computing: @@ -432,10 +432,10 @@ Reference graphs on remote servers for distributed computing: .. code-block:: python - from graphistry import RemoteGraph + from graphistry import remote result = g.gfql([ - RemoteGraph(dataset_id='fraud-network-2024'), + remote(dataset_id='fraud-network-2024'), n({'risk_score': gt(90)}), e_forward() ]) @@ -445,7 +445,7 @@ Reference graphs on remote servers for distributed computing: .. code-block:: python result = g.gfql(let({ - 'remote_data': RemoteGraph(dataset_id='historical-2023'), + 'remote_data': remote(dataset_id='historical-2023'), 'high_risk': ref('remote_data', [ n({'risk_score': gt(95)}) ]), diff --git a/docs/source/gfql/spec/python_embedding.md b/docs/source/gfql/spec/python_embedding.md index ef33f049ae..faaa616455 100644 --- a/docs/source/gfql/spec/python_embedding.md +++ b/docs/source/gfql/spec/python_embedding.md @@ -370,16 +370,16 @@ result = g.gfql(let({ })) ``` -### RemoteGraph References +### Remote Graph References -For distributed computing, RemoteGraph allows referencing graphs on remote servers: +For distributed computing, `remote()` allows referencing graphs on remote servers: ```python -from graphistry import RemoteGraph +from graphistry import remote # Reference a remote dataset result = g.gfql([ - RemoteGraph(dataset_id='fraud-network-2024'), + remote(dataset_id='fraud-network-2024'), n({'risk_score': gt(90)}), e_forward() ]) diff --git a/docs/source/gfql/spec/wire_protocol.md b/docs/source/gfql/spec/wire_protocol.md index 415eb87a91..df5fc89cef 100644 --- a/docs/source/gfql/spec/wire_protocol.md +++ b/docs/source/gfql/spec/wire_protocol.md @@ -205,7 +205,7 @@ ref('base_nodes', [ **Python**: ```python -RemoteGraph(dataset_id='fraud-network-2024') +remote(dataset_id='fraud-network-2024') ``` **Wire Format**: