From 1aa7b309d0cf53b3c12449a1b581a830c8e8b0ac Mon Sep 17 00:00:00 2001 From: Leo Meyerovich Date: Tue, 15 Jul 2025 02:01:08 -0700 Subject: [PATCH 1/8] docs(gfql): add specification documentation MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - Add complete GFQL specification documentation - language.md: Core language specification with grammar and operations - wire_protocol.md: JSON serialization format for client-server communication - cypher_mapping.md: Cypher to GFQL translation with Python and wire protocol - python_embedding.md: Python-specific implementation details - index.md: Specification overview and navigation - Update main gfql/index.rst to include Developer Resources section with spec link - Add ai_code_notes/gfql/README.md with GFQL quick reference for AI assistants This establishes the documentation foundation for GFQL specifications, supporting both human developers and AI-assisted code generation. ๐Ÿค– Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude --- ai_code_notes/gfql/README.md | 211 ++++++++++ docs/source/gfql/index.rst | 7 + docs/source/gfql/spec/cypher_mapping.md | 258 ++++++++++++ docs/source/gfql/spec/index.md | 23 ++ docs/source/gfql/spec/language.md | 365 +++++++++++++++++ docs/source/gfql/spec/python_embedding.md | 195 ++++++++++ docs/source/gfql/spec/wire_protocol.md | 455 ++++++++++++++++++++++ 7 files changed, 1514 insertions(+) create mode 100644 ai_code_notes/gfql/README.md create mode 100644 docs/source/gfql/spec/cypher_mapping.md create mode 100644 docs/source/gfql/spec/index.md create mode 100644 docs/source/gfql/spec/language.md create mode 100644 docs/source/gfql/spec/python_embedding.md create mode 100644 docs/source/gfql/spec/wire_protocol.md diff --git a/ai_code_notes/gfql/README.md b/ai_code_notes/gfql/README.md new file mode 100644 index 0000000000..16d84b3c0e --- /dev/null +++ b/ai_code_notes/gfql/README.md @@ -0,0 +1,211 @@ +# GFQL AI Assistant Guide + +Guide for AI assistants working with GFQL (Graph Frame Query Language) in PyGraphistry. + +## ๐ŸŽฏ Quick Reference + +### Essential GFQL Operations +```python +# Node matching +n() # All nodes +n({"type": "person"}) # Filter by property +n({"age": gt(30)}) # With predicate +n(name="result") # Named results + +# Edge traversal +e_forward() # Forward direction +e_reverse() # Reverse direction +e() or e_undirected() # Both directions +e_forward(hops=2) # Multi-hop +e_forward(to_fixed_point=True) # All reachable + +# Chaining +g.chain([n(), e_forward(), n()]) # Pattern matching +``` + +### Key Predicates +- Comparison: `gt()`, `lt()`, `ge()`, `le()`, `eq()`, `ne()` +- Membership: `is_in([...])` +- Range: `between(lower, upper)` +- String: `contains()`, `startswith()`, `endswith()` +- Null: `is_null()`, `not_null()` +- Temporal: `is_month_start()`, `is_year_end()`, etc. + +### Performance Tips +- Filter early in the chain +- Use specific hop counts vs `to_fixed_point` +- Prefer `filter_dict` over `query` strings +- Use appropriate engine: `pandas` (CPU) or `cudf` (GPU) + +## ๐Ÿ“‹ When to Use GFQL + +### Use GFQL When +- Performing graph traversals or path queries +- Finding patterns in connected data +- Need efficient multi-hop operations +- Working with node/edge dataframes + +### Use Pandas/Aggregations When +- Need sorting (`sort_values()`) +- Need limiting (`head()`, `tail()`) +- Aggregating results (`groupby()`, `count()`) +- Complex transformations + +## ๐Ÿš€ Common Patterns + +### User 360 Query +```python +# Customer's recent activity +g.chain([ + n({"customer_id": "C123"}), + e_forward({ + "type": is_in(["purchase", "view", "support"]), + "timestamp": gt(pd.Timestamp.now() - pd.Timedelta(days=30)) + }) +]) +``` + +### Cyber Security Pattern +```python +# Lateral movement detection +g.chain([ + n({"status": "compromised"}), + e_forward({"type": "login", "success": True}, hops=3), + n({"criticality": "high"}, name="at_risk") +]) +``` + +### Business Intelligence +```python +# Cross-sell opportunities +g.chain([ + n({"product_id": "P123"}), + e_reverse({"type": "purchased"}), + n({"type": "customer"}), + e_forward({"type": "purchased"}), + n({"product_id": ne("P123")}, name="also_bought") +]) +``` + +## ๐Ÿ”ง Code Style Guidelines + +### Preferred Style +```python +# โœ… Good - Clean, code-golfed chains +g.chain([n({"type": "user"}), e({"active": True}), n()]) + +# โŒ Avoid - Overly verbose +result = g.chain([ + n(filter_dict={"type": "user"}), + e_forward(edge_match={"active": True}, hops=1), + n(filter_dict={}) +]) +``` + +### Naming Conventions +- Use descriptive names for `name` parameters +- Keep filter keys consistent with dataframe columns +- Use snake_case for all identifiers + +## ๐Ÿ› Common Errors and Fixes + +### Schema Errors +```python +# โŒ Wrong - Column doesn't exist +n({"username": "Alice"}) + +# โœ… Fix - Use correct column name +n({"name": "Alice"}) +``` + +### Type Errors +```python +# โŒ Wrong - String predicate on number +n({"age": contains("30")}) + +# โœ… Fix - Use numeric predicate +n({"age": gt(30)}) +``` + +### Temporal Errors +```python +# โŒ Wrong - Raw string for datetime +n({"created": gt("2024-01-01")}) + +# โœ… Fix - Use proper datetime +n({"created": gt(pd.Timestamp("2024-01-01"))}) +``` + +## ๐Ÿ“ Natural Language to GFQL + +### Translation Patterns +- "recent" โ†’ `gt(pd.Timestamp.now() - pd.Timedelta(days=N))` +- "between X and Y" โ†’ `between(X, Y)` +- "any of" โ†’ `is_in([...])` +- "connected to" โ†’ `e()` or `e_undirected()` +- "from X to Y" โ†’ X with `e_forward()` to Y +- "within N hops" โ†’ `hops=N` + +### Example Translations + +**NL**: "Find all employees who report to managers in NYC" +```python +g.chain([ + n({"type": "employee"}), + e_forward({"type": "reports_to"}), + n({"type": "manager", "office": "NYC"}) +]) +``` + +**NL**: "Show me high-value customers from last week" +```python +g.chain([ + n({"customer_tier": "high_value"}), + e_forward({ + "type": "purchase", + "date": gt(pd.Timestamp.now() - pd.Timedelta(days=7)) + }) +]) +``` + +## ๐Ÿ”„ Cypher to GFQL + +### Basic Mappings +| Cypher | GFQL | +|--------|------| +| `(n)` | `n()` | +| `(n:Label)` | `n({"type": "Label"})` | +| `-[r]->` | `e_forward()` | +| `<-[r]-` | `e_reverse()` | +| `-[r*2]-` | `e_forward(hops=2)` | +| `WHERE n.prop = val` | `n({"prop": val})` | + +### Unsupported in GFQL +- `OPTIONAL MATCH` - Handle nulls in post-processing +- `WITH` clauses - Use intermediate chains +- `ORDER BY/LIMIT` - Use pandas after +- `CREATE/DELETE` - GFQL is read-only + +## ๐Ÿงช Validation Checklist + +Before generating GFQL: +1. โœ“ Check column names exist in schema +2. โœ“ Verify predicate types match column types +3. โœ“ Ensure temporal values use proper types +4. โœ“ Validate operation names (n, e_forward, etc.) +5. โœ“ Check chain structure is valid + +## ๐Ÿ“š Additional Resources + +- Full specifications in: `AI_PROGRESS/gfql_llm_specs/` + - `gfql_language_spec.md` - Complete language specification + - `gfql_wire_protocol_spec.md` - JSON wire format + - `cypher_to_gfql_mapping_spec.md` - Cypher translation + +## ๐ŸŽฏ Key Takeaways + +1. **GFQL is functional**: Chain operations, don't mutate +2. **Filter early**: Put selective conditions first +3. **Think patterns**: Focus on graph patterns, not procedures +4. **Post-process**: Use pandas for sorting/aggregating +5. **Code golf**: Keep queries concise and elegant \ No newline at end of file diff --git a/docs/source/gfql/index.rst b/docs/source/gfql/index.rst index fd21b5f2ba..7d3bba504f 100644 --- a/docs/source/gfql/index.rst +++ b/docs/source/gfql/index.rst @@ -10,6 +10,7 @@ See also: .. toctree:: :maxdepth: 1 + :caption: User Guide about overview @@ -21,3 +22,9 @@ See also: predicates/quick datetime_filtering wire_protocol_examples + +.. toctree:: + :maxdepth: 2 + :caption: Developer Resources + + spec/index diff --git a/docs/source/gfql/spec/cypher_mapping.md b/docs/source/gfql/spec/cypher_mapping.md new file mode 100644 index 0000000000..d511d22e81 --- /dev/null +++ b/docs/source/gfql/spec/cypher_mapping.md @@ -0,0 +1,258 @@ +(gfql-spec-cypher-mapping)= + +# Cypher to GFQL Python & Wire Protocol Mapping + +## Introduction + +This specification shows how to translate Cypher queries to both GFQL Python code and Wire Protocol JSON, enabling: +- Migration from Cypher-based systems +- Two-stage LLM synthesis: Text โ†’ Cypher โ†’ GFQL +- Language-agnostic API integration +- Secure query generation without code execution + +## Conceptual Framework + +### Translation Scenarios + +When translating from Cypher, you'll encounter three scenarios: + +**1. Direct Translation** - Most pattern matching maps cleanly to pure GFQL +**2. Hybrid Approach** - Post-processing operations (RETURN clauses) use dataframes +**3. GFQL Advantages** - Some capabilities go beyond what Cypher offers + +### What Translates Directly +- Graph patterns: `(a)-[r]->(b)` โ†’ chain operations +- Property filters: WHERE clauses embed into operations +- Path traversals: Variable-length paths use `hops` parameter +- Pattern composition: Multiple patterns become sequential operations + +### What Requires DataFrames +- Aggregations: COUNT, SUM, AVG โ†’ pandas operations +- Projections: RETURN specific columns โ†’ DataFrame selection +- Sorting/limiting: ORDER BY, LIMIT โ†’ DataFrame methods +- Joins: Multiple disconnected patterns โ†’ pandas merge + +### GFQL Advantages Beyond Cypher +- **Rich edge properties**: Query edges as first-class entities +- **Dataframe-native**: Zero-cost transitions between graph and tabular operations +- **GPU acceleration**: Massively parallel execution on NVIDIA hardware +- **Heterogeneous graphs**: No schema constraints on types or properties + +## Quick Example + +**Cypher:** +```cypher +MATCH (p:Person)-[r:FOLLOWS]->(q:Person) +WHERE p.age > 30 +``` + +**Python:** +```python +g.chain([ + n({"type": "Person", "age": gt(30)}, name="p"), + e_forward({"type": "FOLLOWS"}, name="r"), + n({"type": "Person"}, name="q") +]) +``` + +**Wire Protocol:** +```json +{"type": "Chain", "chain": [ + {"type": "Node", "filter_dict": {"type": "Person", "age": {"type": "GT", "val": 30}}, "name": "p"}, + {"type": "Edge", "direction": "forward", "edge_match": {"type": "FOLLOWS"}, "name": "r"}, + {"type": "Node", "filter_dict": {"type": "Person"}, "name": "q"} +]} + +## Pattern Translations + +### Node Patterns + +| Cypher | Python | Wire Protocol | +|--------|--------|---------------| +| `(n)` | `n()` | `{"type": "Node"}` | +| `(n:Label)` | `n({"type": "Label"})` | `{"type": "Node", "filter_dict": {"type": "Label"}}` | +| `(n {prop: val})` | `n({"prop": val})` | `{"type": "Node", "filter_dict": {"prop": val}}` | +| `(n:Person) WHERE n.age > 30` | `n({"type": "Person", "age": gt(30)})` | `{"type": "Node", "filter_dict": {"type": "Person", "age": {"type": "GT", "val": 30}}}` | + +### Edge Patterns + +| Cypher | Python | Wire Protocol (compact) | +|--------|--------|-------------------------| +| `-[]->` | `e_forward()` | `{"type": "Edge", "direction": "forward"}` | +| `-[r:KNOWS]->` | `e_forward({"type": "KNOWS"}, name="r")` | `{"type": "Edge", "direction": "forward", "edge_match": {"type": "KNOWS"}, "name": "r"}` | +| `<-[r]-` | `e_reverse(name="r")` | `{"type": "Edge", "direction": "reverse", "name": "r"}` | +| `-[r]-` | `e(name="r")` | `{"type": "Edge", "direction": "undirected", "name": "r"}` | +| `-[*2]->` | `e_forward(hops=2)` | `{"type": "Edge", "direction": "forward", "hops": 2}` | +| `-[*1..3]->` | `e_forward(hops=3)` | `{"type": "Edge", "direction": "forward", "hops": 3}` | +| `-[*]->` | `e_forward(to_fixed_point=True)` | `{"type": "Edge", "direction": "forward", "to_fixed_point": true}` | +| `-[r:BOUGHT {amount: gt(100)}]->` | `e_forward({"type": "BOUGHT", "amount": gt(100)}, name="r")` | `{"type": "Edge", "direction": "forward", "edge_match": {"type": "BOUGHT", "amount": {"type": "GT", "val": 100}}, "name": "r"}` | + +### Predicates + +| Cypher | Python | Wire Protocol | +|--------|--------|---------------| +| `n.age > 30` | `gt(30)` | `{"type": "GT", "val": 30}` | +| `n.age >= 50` | `ge(50)` | `{"type": "GE", "val": 50}` | +| `n.age < 100` | `lt(100)` | `{"type": "LT", "val": 100}` | +| `n.age <= 50` | `le(50)` | `{"type": "LE", "val": 50}` | +| `n.status = 'active'` | `"active"` | `"active"` | +| `n.status <> 'deleted'` | `ne("deleted")` | `{"type": "NE", "val": "deleted"}` | +| `n.id IN [1,2,3]` | `is_in([1,2,3])` | `{"type": "IsIn", "options": [1,2,3]}` | +| `n.score BETWEEN 0 AND 100` | `between(0, 100)` | `{"type": "Between", "lower": 0, "upper": 100}` | +| `n.name =~ '^A.*'` | `match("^A.*")` | `{"type": "Match", "pattern": "^A.*"}` | +| `n.text CONTAINS 'search'` | `contains("search")` | `{"type": "Contains", "pattern": "search"}` | +| `n.name STARTS WITH 'Dr'` | `startswith("Dr")` | `{"type": "Startswith", "pattern": "Dr"}` | +| `n.email ENDS WITH '.com'` | `endswith(".com")` | `{"type": "Endswith", "pattern": ".com"}` | +| `n.val IS NULL` | `is_null()` | `{"type": "IsNull"}` | +| `n.val IS NOT NULL` | `not_null()` | `{"type": "NotNull"}` | + +## Complete Examples + +### Friend of Friend + +**Cypher:** +```cypher +MATCH (u:User {name: 'Alice'})-[:FRIEND*2]->(fof:User) +WHERE fof.active = true +``` + +**Python:** +```python +g.chain([ + n({"type": "User", "name": "Alice"}), + e_forward({"type": "FRIEND"}, hops=2), + n({"type": "User", "active": True}, name="fof") +]) +``` + +**Wire Protocol:** +```json +{"type": "Chain", "chain": [ + {"type": "Node", "filter_dict": {"type": "User", "name": "Alice"}}, + {"type": "Edge", "direction": "forward", "edge_match": {"type": "FRIEND"}, "hops": 2}, + {"type": "Node", "filter_dict": {"type": "User", "active": true}, "name": "fof"} +]} + +### Fraud Detection + +**Cypher:** +```cypher +MATCH (a:Account)-[t:TRANSFER]->(b:Account) +WHERE t.amount > 10000 AND t.date > date('2024-01-01') +``` + +**Python:** +```python +g.chain([ + n({"type": "Account"}), + e_forward({ + "type": "TRANSFER", + "amount": gt(10000), + "date": gt(date(2024,1,1)) + }, name="t"), + n({"type": "Account"}) +]) +``` + +**Wire Protocol:** +```json +{"type": "Chain", "chain": [ + {"type": "Node", "filter_dict": {"type": "Account"}}, + {"type": "Edge", "direction": "forward", "edge_match": { + "type": "TRANSFER", + "amount": {"type": "GT", "val": 10000}, + "date": {"type": "GT", "val": {"type": "date", "value": "2024-01-01"}} + }, "name": "t"}, + {"type": "Node", "filter_dict": {"type": "Account"}} +]} +``` + +### Complex Aggregation Example + +**Cypher:** +```cypher +MATCH (u:User)-[t:TRANSACTION]->(m:Merchant) +WHERE t.date > date('2024-01-01') +RETURN m.category, count(*) as cnt, sum(t.amount) as total +ORDER BY total DESC +LIMIT 10 +``` + +**Python:** +```python +# Step 1: Graph pattern +result = g.chain([ + n({"type": "User"}), + e_forward({"type": "TRANSACTION", "date": gt(date(2024,1,1))}, name="trans"), + n({"type": "Merchant"}) +]) + +# Step 2: DataFrame operations +trans_df = result._edges[result._edges["trans"]] +merchant_df = result._nodes +analysis = (trans_df + .merge(merchant_df, left_on=g._destination, right_on=g._node) + .groupby('category') + .agg(cnt=('amount', 'count'), total=('amount', 'sum')) + .nlargest(10, 'total')) +``` + +**Note:** Wire protocol returns the filtered graph; aggregations require client-side processing. + +## DataFrame Operations Mapping + +| Cypher Feature | Python DataFrame Operation | Notes | +|----------------|---------------------------|--------| +| `RETURN a, b, c` | `df[['a', 'b', 'c']]` | Column selection | +| `RETURN DISTINCT` | `df.drop_duplicates()` | Remove duplicates | +| `ORDER BY x DESC` | `df.sort_values('x', ascending=False)` | Sort results | +| `LIMIT 10` | `df.head(10)` | Limit rows | +| `count(*)` | `len(df)` or `df.groupby(...).size()` | Count rows | +| `sum(n.val)` | `df['val'].sum()` or `df.groupby(...).agg(sum)` | Aggregation | +| `collect(n.x)` | `df.groupby(...).agg(list)` | Collect to list | +| Named patterns | `df[df['pattern_name']]` | Boolean column filtering | + +## Key Differences + +| Feature | Python | Wire Protocol | +|---------|--------|---------------| +| **Temporal values** | `pd.Timestamp()`, `date()` | `{"type": "date", "value": "..."}` | +| **Direct equality** | `"active"` | `"active"` (same) | +| **Comparisons** | `gt(30)` | `{"type": "GT", "val": 30}` | +| **Collections** | `is_in([...])` | `{"type": "IsIn", "options": [...]}` | + +## Not Supported +- `OPTIONAL MATCH` - No equivalent (would need outer joins) +- `CREATE`, `DELETE`, `SET` - GFQL is read-only +- `WITH` clauses - Requires intermediate variables +- Multiple `MATCH` patterns - Use separate chains or joins + +## Best Practices + +1. **Direct Translation First**: Try pure GFQL before adding DataFrame operations +2. **Use Named Patterns**: Label important results with `name=` for easy access +3. **Filter Early**: Apply selective node filters before traversing edges +4. **Type Consistency**: Ensure wire protocol types match expected column types +5. **Validate JSON**: Test wire protocol against schema before sending + +## LLM Integration Guide + +When building translators: + +``` +Given Cypher: {cypher_query} + +Generate both: +1. Python: Human-readable GFQL code +2. Wire Protocol: JSON for API calls + +Rules: +- (n:Label) โ†’ Python: n({"type": "Label"}) โ†’ JSON: {"type": "Node", "filter_dict": {"type": "Label"}} +- WHERE โ†’ Embed as predicates in both formats +- Aggregations โ†’ Note as requiring DataFrame post-processing +``` + +## See Also +- {ref}`gfql-spec-wire-protocol` - Full wire protocol specification +- {ref}`gfql-spec-language` - Language specification +- {ref}`gfql-spec-python-embedding` - Python implementation details \ No newline at end of file diff --git a/docs/source/gfql/spec/index.md b/docs/source/gfql/spec/index.md new file mode 100644 index 0000000000..4e1c3f6b41 --- /dev/null +++ b/docs/source/gfql/spec/index.md @@ -0,0 +1,23 @@ +(gfql-specifications)= + +# GFQL Specifications + +This section contains formal specifications for GFQL (Graph Frame Query Language), designed to support both human understanding and automated tooling, including LLM-based code synthesis. + +```{toctree} +:maxdepth: 1 + +language +python_embedding +wire_protocol +cypher_mapping +``` + +## Overview + +- {ref}`gfql-spec-language` - Complete formal grammar, operations, predicates, and type system +- {ref}`gfql-spec-python-embedding` - Python-specific implementation with pandas/cuDF +- {ref}`gfql-spec-wire-protocol` - JSON serialization format for client-server communication +- {ref}`gfql-spec-cypher-mapping` - Cypher to GFQL translations with both Python and wire protocol + +These specifications are optimized for text-to-GFQL synthesis, Cypher-to-GFQL pipelines, query validation, and schema-aware code generation. \ No newline at end of file diff --git a/docs/source/gfql/spec/language.md b/docs/source/gfql/spec/language.md new file mode 100644 index 0000000000..a75c08ff44 --- /dev/null +++ b/docs/source/gfql/spec/language.md @@ -0,0 +1,365 @@ +(gfql-spec-language)= + +# GFQL Language Specification + +## Introduction + +GFQL (Graph Frame Query Language) is a DataFrame-native graph query language designed for expressing graph patterns and traversals on tabular data. It operates on node and edge DataFrames, providing a functional, composable approach to graph querying with native GPU acceleration support. + +### Design Principles +- **Dataframe-native**: Type-safe functional bulk operations over dataframe libraries like pandas, cuDF +- **Declarative**: Focus on what to retrieve, and give the engine freedom to optimize how +- **Accessible**: Designed for both human readability and machine generation, and building on intuitions from popular tabular and graph systems +- **Performance-oriented**: Vectorized operations by default, including GPU acceleration +- **Embeddable**: Similar to DuckDB, can be embedded in different languages, and initially focused on Python data ecosystem +- **Computer-tier**: Decoupling from storage enables flexible execution - embedded locally or via remote acceleration servers + +### Language Forms + +GFQL exists in three complementary forms: + +1. **Core Language**: Abstract graph pattern matching language defined by this specification +2. **Embedded DSL**: Host language implementations (currently Python with pandas/cuDF) +3. **Wire Protocol**: JSON serialization for client-server communication (see Wire Protocol spec) + +This specification focuses on the core language concepts. Examples use Python syntax for concreteness, but the patterns apply to any embedding. + +## Language Overview + +### Core Concepts + +1. **Graph Model**: Graphs consist of node and edge dataframes + - Edges: DataFrame with source and destination columns + - Nodes: DataFrame with unique identifier column + - Column names are user-defined globals for the graph: + - Node ID attribute: `g._node` (e.g., "node_id", "id") + - Edge source attribute: `g._source` (e.g., "source", "from") + - Edge destination attribute: `g._destination` (e.g., "destination", "to") + - GFQL infers nodes from edge references when only edges are provided + +2. **GFQL Programs**: GFQL programs are declarative graph-to-graph transformations + - Enable use cases like search, filter, enrich, and traverse + - Express *what* to find (ex: Cypher), not *how* to find it (ex: Gremlin) + +3. **Chains**: Path pattern expressions for matching graph structures + - Express graph patterns as sequences of node and edge matching operations + - Similar to Cypher patterns but decomposed into composable steps + - Define paths through the graph: start nodes โ†’ edges โ†’ end nodes + - Each operation refines the pattern match based on previous results + +4. **Operations**: Act on graph entities (nodes and edges) + - Node matchers: Filter and select nodes + - Edge matchers: Traverse relationships + - Operations work on the graph structure itself + +5. **Predicates**: Act on attributes of nodes and edges + - Filter based on property values + - Comparison, membership, string matching, temporal checks + - Composable within operations to build complex conditions + +6. **Values**: Type system matching modern data formats + - Scalars: numbers, strings, booleans, null + - Temporal: ISO datetimes, dates, times with timezone support + - Collections: lists for membership tests + - Compatible with JSON, Arrow, and DataFrame type systems + +## Formal Grammar + +```{code-block} ebnf +:caption: GFQL Grammar in Extended Backus-Naur Form + +(* Entry point *) +query ::= chain + +(* Chain - path pattern expression *) +chain ::= "[" operation ("," operation)* "]" + +(* Operations *) +operation ::= node_matcher | edge_matcher + +(* Node Matcher *) +node_matcher ::= "n(" node_params? ")" +node_params ::= filter_dict ("," name_param)? ("," query_param)? + | name_param ("," query_param)? + | query_param + +(* Edge Matchers *) +edge_matcher ::= edge_forward | edge_reverse | edge_undirected +edge_forward ::= "e_forward(" edge_params? ")" +edge_reverse ::= "e_reverse(" edge_params? ")" +edge_undirected ::= ("e" | "e_undirected") "(" edge_params? ")" + +(* Parameters *) +edge_params ::= edge_match_params ("," hop_params)? ("," node_filter_params)? ("," name_param)? + +filter_dict ::= "{" (property_filter ("," property_filter)*)? "}" +property_filter ::= string ":" (value | predicate) + +hop_params ::= "hops=" integer | "to_fixed_point=True" +node_filter_params ::= source_filter ("," dest_filter)? +source_filter ::= "source_node_match=" filter_dict | "source_node_query=" string +dest_filter ::= "destination_node_match=" filter_dict | "destination_node_query=" string + +name_param ::= "name=" string +query_param ::= "query=" string +edge_query_param ::= "edge_query=" string +edge_match_params ::= filter_dict | edge_query_param + +(* Predicates *) +predicate ::= comparison | membership | range | null_check | string_pred | temporal_pred + +comparison ::= ("gt" | "lt" | "ge" | "le" | "eq" | "ne") "(" value ")" +membership ::= "is_in(" "[" value ("," value)* "]" ")" +range ::= "between(" value "," value ("," "inclusive=" boolean)? ")" +null_check ::= "is_null()" | "not_null()" | "is_na()" | "not_na()" +string_pred ::= string_match | string_check +string_match ::= "contains(" string ("," "case=" boolean)? ("," "regex=" boolean)? ")" + | "match(" string ("," "case=" boolean)? ")" + | ("startswith" | "endswith") "(" string ")" +string_check ::= ("isalpha" | "isnumeric" | "isdigit" | "isalnum" + | "isupper" | "islower") "()" +temporal_pred ::= temporal_check "()" +temporal_check ::= "is_month_start" | "is_month_end" | "is_quarter_start" + | "is_quarter_end" | "is_year_start" | "is_year_end" | "is_leap_year" + +(* Values *) +value ::= scalar | temporal_value | collection +scalar ::= number | string | boolean | null +temporal_value ::= datetime_value | date_value | time_value +datetime_value ::= "pd.Timestamp(" string ("," "tz=" string)? ")" + | "datetime(" datetime_args ")" +date_value ::= "date(" date_args ")" +time_value ::= "time(" time_args ")" +collection ::= "[" (value ("," value)*)? "]" + +(* Primitives *) +string ::= '"' [^"]* '"' | "'" [^']* "'" +number ::= integer | float +integer ::= ["-"]? [0-9]+ +float ::= ["-"]? [0-9]+ "." [0-9]+ +boolean ::= "True" | "False" +null ::= "None" +datetime_args ::= integer ("," integer)* +date_args ::= integer "," integer "," integer +time_args ::= integer "," integer ("," integer)? +``` + +## Operations + +### Node Matcher: `n()` + +Filters nodes based on attributes. + +**Syntax**: `n(filter_dict?, name?, query?)` + +**Parameters**: +- `filter_dict`: Dictionary of attribute filters +- `name`: Optional string label for results +- `query`: Pandas query string expression + +**Examples**: +```python +n() # All nodes +n({"type": "person"}) # Nodes where type='person' +n({"age": gt(30)}) # Nodes where age > 30 +n(name="important") # Label matching nodes +n(query="age > 30 and status == 'active'") # Query string +``` + +### Edge Matchers + +#### Forward Traversal: `e_forward()` + +Traverses edges in forward direction (source โ†’ destination). + +**Syntax**: `e_forward(edge_match?, hops?, to_fixed_point?, source_node_match?, destination_node_match?, name?)` + +**Parameters**: +- `edge_match`: Edge attribute filters +- `hops`: Number of hops (default: 1) +- `to_fixed_point`: Continue until no new nodes (default: False) +- `source_node_match`: Filters for source nodes +- `destination_node_match`: Filters for destination nodes +- `name`: Optional label + +**Examples**: +```python +e_forward() # One hop forward +e_forward(hops=2) # Two hops forward +e_forward(to_fixed_point=True) # All reachable nodes +e_forward({"type": "follows"}) # Only 'follows' edges +e_forward(source_node_match={"active": True}) # From active nodes +``` + +#### Reverse Traversal: `e_reverse()` + +Traverses edges in reverse direction (destination โ†’ source). + +**Syntax**: Same as `e_forward()` + +#### Undirected Traversal: `e()` or `e_undirected()` + +Traverses edges in both directions. + +**Syntax**: Same as `e_forward()` + +## Predicates + +### Comparison Predicates + +```python +gt(value) # Greater than +lt(value) # Less than +ge(value) # Greater than or equal +le(value) # Less than or equal +eq(value) # Equal +ne(value) # Not equal +``` + +### Membership Predicate + +```python +is_in([value1, value2, ...]) # Value in list +``` + +### Range Predicate + +```python +between(lower, upper, inclusive=True) # Value in range +``` + +### String Predicates + +Pattern matching predicates: +```python +contains(pat, case=True, regex=True) # Contains pattern (substring or regex) +startswith(prefix) # Starts with prefix (case-sensitive) +endswith(suffix) # Ends with suffix (case-sensitive) +match(pat, case=True) # Matches regex from start of string +``` + +String type checking predicates: +```python +isalpha() # Alphabetic characters only +isnumeric() # Numeric characters only +isdigit() # Digits only +isalnum() # Alphanumeric +isupper() # All uppercase +islower() # All lowercase +``` + +### Null Predicates + +```python +is_null() # Is null/None +not_null() # Is not null/None +is_na() # Is NaN (numeric) +not_na() # Is not NaN +``` + +### Temporal Predicates + +```python +is_month_start() # First day of month +is_month_end() # Last day of month +is_quarter_start() # First day of quarter +is_quarter_end() # Last day of quarter +is_year_start() # First day of year +is_year_end() # Last day of year +is_leap_year() # Is leap year +``` + +## Type System + +### Value Types + +1. **Scalars** + - `number`: int, float + - `string`: Text values + - `boolean`: True/False + - `null`: None + +2. **Temporal Types** + - `datetime`: Timestamp with optional timezone + - `date`: Calendar date + - `time`: Time of day + +3. **Collections** + - `list`: Ordered sequence of values + +### Type Coercion + +GFQL performs automatic type coercion: +- Python datetime โ†’ pandas Timestamp +- Numeric types โ†’ appropriate precision +- Collections โ†’ lists for `is_in()` + +## Execution Model + +### Declarative Pattern Matching + +GFQL follows a declarative execution model similar to Neo4j's Cypher: + +1. **Pattern Declaration**: Chains express path patterns in the graph + - Users declare graph patterns as sequences of node and edge constraints + - Patterns specify *what* paths to match, not *how* to find them + - The engine optimizes pattern matching based on data characteristics + +2. **Set-Based Operations**: All operations work on sets of entities + - No explicit iteration or traversal order + - Results include all matching patterns in the graph + - Current GFQL engines use a novel bulk-oriented execution model that is asymptotically faster than traditional iterative approaches used for Cypher, but this is not a requirement of the language itself + +3. **Lazy Evaluation**: Chains define pattern transformations without immediate execution + - Allows engines to optimize path finding and pattern matching strategies +\ +### Result Access + +Query execution returns filtered node and edge datasets. In the Python embedding: + +```python +result = g.chain([...]) +nodes_df = result._nodes # Filtered nodes +edges_df = result._edges # Filtered edges +``` + +### Named Results + +Operations with `name` parameter add boolean columns to mark matched entities: + +```python +result = g.chain([ + n({"type": "person"}, name="people"), + e_forward(name="connections"), + n({"active": True}, name="active_targets") +]) + +# Access all matched nodes and edges: +all_nodes = result._nodes +all_edges = result._edges + +# Access specific matched nodes/edges using pandas filtering: +people_nodes = result._nodes[result._nodes["people"]] +connection_edges = result._edges[result._edges["connections"]] +active_nodes = result._nodes[result._nodes["active_targets"]] + +# Or using standard pandas query syntax: +people_nodes = result._nodes.query("people == True") +``` + +This pattern is essential for extracting specific subsets from complex graph traversals. + +## Best Practices + +1. **Use specific filters early**: Filter nodes before traversing edges +2. **Limit hops**: Use reasonable hop limits to avoid explosion +3. **Name important results**: Use `name` parameter for analysis +4. **Prefer filter_dict**: More efficient than query strings +5. **Use appropriate predicates**: Match predicate to column type + +## See Also + +- {ref}`gfql-spec-python-embedding` - Python implementation details +- {ref}`gfql-spec-wire-protocol` - JSON serialization format +- {ref}`gfql-spec-cypher-mapping` - Cypher to GFQL translation with wire protocol +- [GFQL Quick Reference](../quick.rst) - Comprehensive examples and usage patterns \ No newline at end of file diff --git a/docs/source/gfql/spec/python_embedding.md b/docs/source/gfql/spec/python_embedding.md new file mode 100644 index 0000000000..27be847e4b --- /dev/null +++ b/docs/source/gfql/spec/python_embedding.md @@ -0,0 +1,195 @@ +(gfql-spec-python-embedding)= + +# GFQL Python Embedding + +This document describes the Python-specific implementation of GFQL using pandas and cuDF dataframes. + +## Graph Construction + +In Python, graphs are created with user-defined column names: + +```python +import graphistry +assert 'src_col' in df.columns and 'dst_col' in df.columns +g = graphistry.edges(df, source='src_col', destination='dst_col') + +# Optional; GFQL infers node existence when only edges are provided +assert 'node_col' in df.columns +g2 = graphistry.nodes(df, node='node_col') +``` + +### Schema Access + +The graph schema is accessible via attributes: +- `g._node`: Node ID column name +- `g._source`: Edge source column name +- `g._destination`: Edge destination column name + +Graph nodes can be generically accessed using these attributes: +- `g._nodes`: Node DataFrame +- `g._nodes[g._node]`: Node ID column +- `g._nodes[[attr for attr in g._nodes.columns if attr != g._node]]`: All other node attributes + +Graph edges can be accessed similarly: +- `g._edges`: Edge DataFrame +- `g._edges[g._source]`: Edge source column +- `g._edges[g._destination]`: Edge destination column +- `g._edges[[attr for attr in g._edges.columns if attr not in [g._source, g._destination]]]`: All other edge attributes + +## Query Execution + +```python +from graphistry import n, e_forward + +# Execute a chain +result = g.chain([ + n({"type": "person"}), + e_forward(), + n() +]) + +# Access results +nodes_df = result._nodes # Filtered nodes DataFrame +edges_df = result._edges # Filtered edges DataFrame +``` + +## Engine Selection + +GFQL supports multiple execution engines: + +- **pandas**: CPU execution (default) +- **cudf**: GPU acceleration +- **auto**: Automatic selection based on data type + +```python +# Force specific engine +g.chain([...], engine='cudf') # GPU execution +g.chain([...], engine='pandas') # CPU execution +g.chain([...], engine='auto') # Auto-select +``` + +## Python-Specific Values + +### Temporal Values + +```python +import pandas as pd + +# Timestamps +pd.Timestamp('2023-01-01') +pd.Timestamp.now() + +# Time deltas +pd.Timedelta(days=30) +pd.Timedelta(hours=24) +``` + +### DataFrame Operations + +Results can be further processed using standard pandas operations: + +```python +# Using boolean columns from named operations +people_nodes = result._nodes[result._nodes["people"]] + +# Using pandas query +active_nodes = result._nodes.query("active == True") + +# Standard pandas operations +result._nodes.groupby('type').size() +``` + +## Common Errors and Validation + +### Type Mismatches + +```python +# Wrong - String predicate on numeric column +n({"age": contains("3")}) + +# Correct - Use numeric predicate +n({"age": gt(30)}) + +# Wrong - String comparison on datetime +n({"created": gt("2024-01-01")}) + +# Correct - Use proper datetime type +n({"created": gt(pd.Timestamp("2024-01-01"))}) +``` + +### Schema Validation + +```python +# Check available columns before querying +print(g._nodes.columns) # ['id', 'type', 'name'] + +# Wrong - Column doesn't exist +g.chain([n({"username": "Alice"})]) # KeyError + +# Correct - Use existing column +g.chain([n({"name": "Alice"})]) +``` + +### Unsupported Operations + +```python +# Wrong - Can't aggregate in chain +# g.chain([n(), e(), count()]) + +# Correct - Aggregate after chain +result = g.chain([n(), e()]) +count = len(result._edges) + +# Wrong - OPTIONAL MATCH not supported +# No direct GFQL equivalent + +# Correct - Handle optionality in post-processing +result = g.chain([n(), e_forward()]) +# Check for nodes without edges +nodes_with_edges = result._nodes[result._nodes[g._node].isin(result._edges[g._source])] +``` + +## Best Practices + +### Query Construction +```python +# Good: Build queries programmatically +node_filters = {"type": "User"} +if min_age: + node_filters["age"] = gt(min_age) +g.chain([n(node_filters)]) + +# Avoid: Hardcoded query strings +g.chain([n(query=f"type == 'User' and age > {min_age}")]) # SQL injection risk +``` + +### Memory Efficiency +```python +# Good: Filter early and use named results +result = g.chain([ + n({"active": True}, name="active_users"), # Filter first + e_forward({"recent": True}) +]) +# Only access what you need +active_users = result._nodes[result._nodes["active_users"]] + +# Avoid: Loading everything then filtering +all_nodes = g._nodes +active = all_nodes[all_nodes["active"] == True] # Loads entire graph +``` + +### GPU Best Practices +```python +# Check GPU memory before large operations +if engine == 'cudf': + import cudf + print(f"GPU memory: {cudf.cuda.cuda.get_memory_info()}") + +# Convert results back to pandas if needed for compatibility +result_pandas = result._nodes.to_pandas() if hasattr(result._nodes, 'to_pandas') else result._nodes +``` + +## See Also + +- {ref}`gfql-spec-language` - Core language specification +- [GFQL Quick Reference](../quick.rst) - Python API examples \ No newline at end of file diff --git a/docs/source/gfql/spec/wire_protocol.md b/docs/source/gfql/spec/wire_protocol.md new file mode 100644 index 0000000000..ca8cdc0654 --- /dev/null +++ b/docs/source/gfql/spec/wire_protocol.md @@ -0,0 +1,455 @@ +(gfql-spec-wire-protocol)= + +# GFQL Wire Protocol Specification + +## Introduction + +The GFQL Wire Protocol defines the JSON serialization format for GFQL queries, enabling: +- Client-server communication +- Query persistence and storage +- Cross-language interoperability between Python, JavaScript, and other clients +- Configuration-driven query generation + +### Design Principles +- **Type Safety**: Tagged dictionaries preserve type information +- **Self-Describing**: Each object includes type metadata +- **Extensible**: Schema supports future additions +- **Round-Trip Safe**: Lossless serialization/deserialization + +## Protocol Overview + +### Message Structure + +All GFQL wire protocol messages are JSON objects with a `type` field: + +```json +{ + "type": "MessageType", + ...additional fields... +} +``` + +### Supported Message Types +- `Chain`: Complete query chain +- `Node`: Node matcher operation +- `Edge`: Edge traversal operation +- Predicates: `GT`, `LT`, `EQ`, `IsIn`, `Between`, etc. +- Temporal values: `datetime`, `date`, `time` + +## JSON Schema + +```{code-block} json +:caption: Complete JSON Schema for GFQL Wire Protocol + +{ + "$schema": "http://json-schema.org/draft-07/schema#", + "$id": "https://graphistry.com/schemas/gfql/wire-protocol.json", + + "definitions": { + "Chain": { + "type": "object", + "properties": { + "type": {"const": "Chain"}, + "chain": { + "type": "array", + "items": {"$ref": "#/definitions/Operation"}, + "minItems": 1 + } + }, + "required": ["type", "chain"], + "additionalProperties": false + }, + + "Operation": { + "oneOf": [ + {"$ref": "#/definitions/NodeOperation"}, + {"$ref": "#/definitions/EdgeOperation"} + ] + }, + + "NodeOperation": { + "type": "object", + "properties": { + "type": {"const": "Node"}, + "filter_dict": {"$ref": "#/definitions/FilterDict"}, + "query": {"type": "string"}, + "name": {"type": "string"} + }, + "required": ["type"], + "additionalProperties": false + }, + + "EdgeOperation": { + "type": "object", + "properties": { + "type": {"const": "Edge"}, + "direction": { + "enum": ["forward", "reverse", "undirected"] + }, + "edge_match": {"$ref": "#/definitions/FilterDict"}, + "edge_query": {"type": "string"}, + "hops": { + "type": "integer", + "minimum": 1, + "default": 1 + }, + "to_fixed_point": { + "type": "boolean", + "default": false + }, + "source_node_match": {"$ref": "#/definitions/FilterDict"}, + "source_node_query": {"type": "string"}, + "destination_node_match": {"$ref": "#/definitions/FilterDict"}, + "destination_node_query": {"type": "string"}, + "name": {"type": "string"} + }, + "required": ["type", "direction"], + "additionalProperties": false + } + } +} +``` + +## Operation Serialization + +### Node Operation + +**Python**: +```python +n({"type": "person", "age": gt(30)}, name="adults") +``` + +**Wire Format**: +```json +{ + "type": "Node", + "filter_dict": { + "type": "person", + "age": { + "type": "GT", + "val": 30 + } + }, + "name": "adults" +} +``` + +### Edge Operation + +**Python**: +```python +e_forward( + {"type": "transaction"}, + hops=2, + source_node_match={"active": True}, + name="txns" +) +``` + +**Wire Format**: +```json +{ + "type": "Edge", + "direction": "forward", + "edge_match": { + "type": "transaction" + }, + "hops": 2, + "source_node_match": { + "active": true + }, + "name": "txns" +} +``` + +### Chain + +**Python**: +```python +chain([ + n({"id": "Alice"}), + e_forward({"type": "friend"}), + n({"status": "active"}) +]) +``` + +**Wire Format**: +```json +{ + "type": "Chain", + "chain": [ + { + "type": "Node", + "filter_dict": {"id": "Alice"} + }, + { + "type": "Edge", + "direction": "forward", + "edge_match": {"type": "friend"} + }, + { + "type": "Node", + "filter_dict": {"status": "active"} + } + ] +} +``` + +## Predicate Serialization + +### Comparison Predicates + +```json +{"type": "GT", "val": 100} +{"type": "LT", "val": 50.5} +{"type": "GE", "val": "2024-01-01"} +{"type": "LE", "val": true} +{"type": "EQ", "val": "active"} +{"type": "NE", "val": null} +``` + +### Between Predicate + +```json +{ + "type": "Between", + "lower": 10, + "upper": 20, + "inclusive": true +} +``` + +### IsIn Predicate + +```json +{ + "type": "IsIn", + "options": ["A", "B", "C"] +} +``` + +### String Predicates + +```json +{"type": "Contains", "pattern": "search"} +{"type": "Startswith", "pattern": "prefix"} +{"type": "Endswith", "pattern": "suffix"} +{"type": "Match", "pattern": "^[A-Z]+\\d+$"} +``` + +### Null Predicates + +```json +{"type": "IsNull"} +{"type": "NotNull"} +{"type": "IsNA"} +{"type": "NotNA"} +``` + +### Temporal Check Predicates + +```json +{"type": "IsMonthStart"} +{"type": "IsYearEnd"} +{"type": "IsLeapYear"} +``` + +## Type Serialization + +### Scalar Types + +```json +"hello world" // string +42 // integer +3.14159 // float +true // boolean +null // null +``` + +### Temporal Types + +#### DateTime +```json +{ + "type": "datetime", + "value": "2024-01-15T10:30:00", + "timezone": "America/New_York" // Optional, defaults to "UTC" +} +``` + +#### Date +```json +{ + "type": "date", + "value": "2024-01-15" +} +``` + +#### Time +```json +{ + "type": "time", + "value": "14:30:00.123456" +} +``` + +**Note**: The `timezone` field is optional for DateTime values and defaults to "UTC" if omitted. This ensures consistent behavior across systems while allowing explicit timezone specification when needed. + +## Examples + +### User 360 Query + +**Python**: +```python +g.chain([ + n({"customer_id": "C123"}), + e_forward({ + "type": "purchase", + "timestamp": gt(pd.Timestamp("2024-01-01")) + }) +]) +``` + +**Wire Format**: +```json +{ + "type": "Chain", + "chain": [ + { + "type": "Node", + "filter_dict": { + "customer_id": "C123" + } + }, + { + "type": "Edge", + "direction": "forward", + "edge_match": { + "type": "purchase", + "timestamp": { + "type": "GT", + "val": { + "type": "datetime", + "value": "2024-01-01T00:00:00", + "timezone": "UTC" + } + } + } + } + ] +} +``` + +### Cyber Security Pattern + +**Python**: +```python +g.chain([ + n({"ip": is_in(["192.168.1.100", "192.168.1.101"])}), + e_forward( + edge_query="port IN [22, 23, 3389]", + to_fixed_point=True + ), + n({"type": "server", "critical": True}) +]) +``` + +**Wire Format**: +```json +{ + "type": "Chain", + "chain": [ + { + "type": "Node", + "filter_dict": { + "ip": { + "type": "IsIn", + "options": ["192.168.1.100", "192.168.1.101"] + } + } + }, + { + "type": "Edge", + "direction": "forward", + "edge_query": "port IN [22, 23, 3389]", + "to_fixed_point": true + }, + { + "type": "Node", + "filter_dict": { + "type": "server", + "critical": true + } + } + ] +} +``` + +## Error Handling + +### Error Response Format + +```json +{ + "type": "Error", + "error_type": "ValidationError", + "message": "Invalid predicate type 'GT' for string column 'name'", + "details": { + "column": "name", + "predicate": "GT", + "expected_types": ["string"], + "actual_type": "comparison" + } +} +``` + +### Error Types + +1. **ValidationError**: Schema validation failed +2. **ParseError**: JSON parsing failed +3. **TypeError**: Type mismatch +4. **SemanticError**: Logical inconsistency +5. **UnsupportedError**: Feature not supported + +## Protocol Extensions + +### Future Considerations + +The protocol is designed to support future extensions: + +1. **Versioning**: Add `"version"` field for protocol versioning +2. **Metadata**: Add `"metadata": {}` for additional context +3. **Streaming**: Support for partial results +4. **Transactions**: Batch multiple operations +5. **Optimization Hints**: Engine-specific parameters + +### Custom Predicates + +New predicates can be added by extending the schema: + +```json +{ + "definitions": { + "CustomPredicate": { + "type": "object", + "properties": { + "type": {"const": "CustomPredicateName"}, + ...custom fields... + } + } + } +} +``` + +## Best Practices + +1. **Always include type fields**: Every object must have a `type` +2. **Use ISO formats**: Dates and times in ISO 8601 +3. **Handle timezones consistently**: Include timezone for datetime values when precision matters (defaults to UTC) +4. **Validate before sending**: Use JSON Schema validation +5. **Handle unknown fields**: Ignore unrecognized fields for compatibility + +## See Also + +- {ref}`gfql-spec-language` - Language specification +- {ref}`gfql-spec-cypher-mapping` - Cypher to GFQL translation with wire protocol examples \ No newline at end of file From 6fd06a6c185722760cce6d6aa2847248c833d98a Mon Sep 17 00:00:00 2001 From: Leo Meyerovich Date: Tue, 15 Jul 2025 02:13:27 -0700 Subject: [PATCH 2/8] docs(gfql): use headers for Core Concepts to enable TOC navigation - Convert Core Concepts from numbered list to headers (h4) - This allows each concept to appear in the table of contents - Makes it easier to navigate directly to specific concepts like Graph Model, Chains, Operations, etc. --- docs/source/gfql/spec/language.md | 80 ++++++++++++++++++------------- 1 file changed, 46 insertions(+), 34 deletions(-) diff --git a/docs/source/gfql/spec/language.md b/docs/source/gfql/spec/language.md index a75c08ff44..3d0c444af5 100644 --- a/docs/source/gfql/spec/language.md +++ b/docs/source/gfql/spec/language.md @@ -28,40 +28,52 @@ This specification focuses on the core language concepts. Examples use Python sy ### Core Concepts -1. **Graph Model**: Graphs consist of node and edge dataframes - - Edges: DataFrame with source and destination columns - - Nodes: DataFrame with unique identifier column - - Column names are user-defined globals for the graph: - - Node ID attribute: `g._node` (e.g., "node_id", "id") - - Edge source attribute: `g._source` (e.g., "source", "from") - - Edge destination attribute: `g._destination` (e.g., "destination", "to") - - GFQL infers nodes from edge references when only edges are provided - -2. **GFQL Programs**: GFQL programs are declarative graph-to-graph transformations - - Enable use cases like search, filter, enrich, and traverse - - Express *what* to find (ex: Cypher), not *how* to find it (ex: Gremlin) - -3. **Chains**: Path pattern expressions for matching graph structures - - Express graph patterns as sequences of node and edge matching operations - - Similar to Cypher patterns but decomposed into composable steps - - Define paths through the graph: start nodes โ†’ edges โ†’ end nodes - - Each operation refines the pattern match based on previous results - -4. **Operations**: Act on graph entities (nodes and edges) - - Node matchers: Filter and select nodes - - Edge matchers: Traverse relationships - - Operations work on the graph structure itself - -5. **Predicates**: Act on attributes of nodes and edges - - Filter based on property values - - Comparison, membership, string matching, temporal checks - - Composable within operations to build complex conditions - -6. **Values**: Type system matching modern data formats - - Scalars: numbers, strings, booleans, null - - Temporal: ISO datetimes, dates, times with timezone support - - Collections: lists for membership tests - - Compatible with JSON, Arrow, and DataFrame type systems +#### Graph Model + +Graphs consist of node and edge dataframes: +- Edges: DataFrame with source and destination columns +- Nodes: DataFrame with unique identifier column +- Column names are user-defined globals for the graph: + - Node ID attribute: `g._node` (e.g., "node_id", "id") + - Edge source attribute: `g._source` (e.g., "source", "from") + - Edge destination attribute: `g._destination` (e.g., "destination", "to") +- GFQL infers nodes from edge references when only edges are provided + +#### GFQL Programs + +GFQL programs are declarative graph-to-graph transformations: +- Enable use cases like search, filter, enrich, and traverse +- Express *what* to find (ex: Cypher), not *how* to find it (ex: Gremlin) + +#### Chains + +Path pattern expressions for matching graph structures: +- Express graph patterns as sequences of node and edge matching operations +- Similar to Cypher patterns but decomposed into composable steps +- Define paths through the graph: start nodes โ†’ edges โ†’ end nodes +- Each operation refines the pattern match based on previous results + +#### Operations + +Act on graph entities (nodes and edges): +- Node matchers: Filter and select nodes +- Edge matchers: Traverse relationships +- Operations work on the graph structure itself + +#### Predicates + +Act on attributes of nodes and edges: +- Filter based on property values +- Comparison, membership, string matching, temporal checks +- Composable within operations to build complex conditions + +#### Values + +Type system matching modern data formats: +- Scalars: numbers, strings, booleans, null +- Temporal: ISO datetimes, dates, times with timezone support +- Collections: lists for membership tests +- Compatible with JSON, Arrow, and DataFrame type systems ## Formal Grammar From 71f36d216034af27b514a6c14c561f2ec0a0fa68 Mon Sep 17 00:00:00 2001 From: Leo Meyerovich Date: Tue, 15 Jul 2025 02:40:03 -0700 Subject: [PATCH 3/8] docs(gfql): remove incomplete JSON Schema from wire protocol spec The JSON Schema was incomplete (missing FilterDict and predicate definitions) and not used by the actual implementation. Removing it to avoid confusion. --- docs/source/gfql/spec/wire_protocol.md | 82 ++++---------------------- 1 file changed, 11 insertions(+), 71 deletions(-) diff --git a/docs/source/gfql/spec/wire_protocol.md b/docs/source/gfql/spec/wire_protocol.md index ca8cdc0654..8e00f4b85c 100644 --- a/docs/source/gfql/spec/wire_protocol.md +++ b/docs/source/gfql/spec/wire_protocol.md @@ -36,79 +36,19 @@ All GFQL wire protocol messages are JSON objects with a `type` field: - Predicates: `GT`, `LT`, `EQ`, `IsIn`, `Between`, etc. - Temporal values: `datetime`, `date`, `time` -## JSON Schema +## Message Structure -```{code-block} json -:caption: Complete JSON Schema for GFQL Wire Protocol +All GFQL wire protocol messages are JSON objects with a `type` field that identifies the message type. The protocol uses discriminated unions for polymorphic types. + +### Type Identification + +Each object includes a `type` field: +- Operations: `"Node"`, `"Edge"`, `"Chain"` +- Predicates: `"GT"`, `"LT"`, `"IsIn"`, etc. +- Temporal values: `"datetime"`, `"date"`, `"time"` + +This enables unambiguous deserialization and validation. -{ - "$schema": "http://json-schema.org/draft-07/schema#", - "$id": "https://graphistry.com/schemas/gfql/wire-protocol.json", - - "definitions": { - "Chain": { - "type": "object", - "properties": { - "type": {"const": "Chain"}, - "chain": { - "type": "array", - "items": {"$ref": "#/definitions/Operation"}, - "minItems": 1 - } - }, - "required": ["type", "chain"], - "additionalProperties": false - }, - - "Operation": { - "oneOf": [ - {"$ref": "#/definitions/NodeOperation"}, - {"$ref": "#/definitions/EdgeOperation"} - ] - }, - - "NodeOperation": { - "type": "object", - "properties": { - "type": {"const": "Node"}, - "filter_dict": {"$ref": "#/definitions/FilterDict"}, - "query": {"type": "string"}, - "name": {"type": "string"} - }, - "required": ["type"], - "additionalProperties": false - }, - - "EdgeOperation": { - "type": "object", - "properties": { - "type": {"const": "Edge"}, - "direction": { - "enum": ["forward", "reverse", "undirected"] - }, - "edge_match": {"$ref": "#/definitions/FilterDict"}, - "edge_query": {"type": "string"}, - "hops": { - "type": "integer", - "minimum": 1, - "default": 1 - }, - "to_fixed_point": { - "type": "boolean", - "default": false - }, - "source_node_match": {"$ref": "#/definitions/FilterDict"}, - "source_node_query": {"type": "string"}, - "destination_node_match": {"$ref": "#/definitions/FilterDict"}, - "destination_node_query": {"type": "string"}, - "name": {"type": "string"} - }, - "required": ["type", "direction"], - "additionalProperties": false - } - } -} -``` ## Operation Serialization From f8f5c206649481f24f89601823b67ece344f278b Mon Sep 17 00:00:00 2001 From: Leo Meyerovich Date: Tue, 15 Jul 2025 02:48:00 -0700 Subject: [PATCH 4/8] docs(gfql): remove Protocol Extensions section from wire protocol spec Remove speculative content about future extensions to keep documentation focused on current implementation. --- docs/source/gfql/spec/wire_protocol.md | 30 -------------------------- 1 file changed, 30 deletions(-) diff --git a/docs/source/gfql/spec/wire_protocol.md b/docs/source/gfql/spec/wire_protocol.md index 8e00f4b85c..c79f09484e 100644 --- a/docs/source/gfql/spec/wire_protocol.md +++ b/docs/source/gfql/spec/wire_protocol.md @@ -351,36 +351,6 @@ g.chain([ 4. **SemanticError**: Logical inconsistency 5. **UnsupportedError**: Feature not supported -## Protocol Extensions - -### Future Considerations - -The protocol is designed to support future extensions: - -1. **Versioning**: Add `"version"` field for protocol versioning -2. **Metadata**: Add `"metadata": {}` for additional context -3. **Streaming**: Support for partial results -4. **Transactions**: Batch multiple operations -5. **Optimization Hints**: Engine-specific parameters - -### Custom Predicates - -New predicates can be added by extending the schema: - -```json -{ - "definitions": { - "CustomPredicate": { - "type": "object", - "properties": { - "type": {"const": "CustomPredicateName"}, - ...custom fields... - } - } - } -} -``` - ## Best Practices 1. **Always include type fields**: Every object must have a `type` From a7f461c9f8e262dc3eb014f5ddf54f9c1cbf5282 Mon Sep 17 00:00:00 2001 From: Leo Meyerovich Date: Tue, 15 Jul 2025 02:55:35 -0700 Subject: [PATCH 5/8] docs(gfql): remove incorrect Error Handling section from wire protocol The documented error response format does not match the implementation. The actual implementation uses HTTP status codes for remote errors and Python exceptions for local validation, not structured JSON error objects. --- docs/source/gfql/spec/wire_protocol.md | 25 ------------------------- 1 file changed, 25 deletions(-) diff --git a/docs/source/gfql/spec/wire_protocol.md b/docs/source/gfql/spec/wire_protocol.md index c79f09484e..beacfdf20f 100644 --- a/docs/source/gfql/spec/wire_protocol.md +++ b/docs/source/gfql/spec/wire_protocol.md @@ -325,31 +325,6 @@ g.chain([ } ``` -## Error Handling - -### Error Response Format - -```json -{ - "type": "Error", - "error_type": "ValidationError", - "message": "Invalid predicate type 'GT' for string column 'name'", - "details": { - "column": "name", - "predicate": "GT", - "expected_types": ["string"], - "actual_type": "comparison" - } -} -``` - -### Error Types - -1. **ValidationError**: Schema validation failed -2. **ParseError**: JSON parsing failed -3. **TypeError**: Type mismatch -4. **SemanticError**: Logical inconsistency -5. **UnsupportedError**: Feature not supported ## Best Practices From b3d0f67883dd8e4ea64312a54964fb30023fb0b5 Mon Sep 17 00:00:00 2001 From: Leo Meyerovich Date: Tue, 15 Jul 2025 02:58:03 -0700 Subject: [PATCH 6/8] docs(gfql): fix missing closing backticks in cypher_mapping.md Add missing closing triple backticks for JSON code block before Pattern Translations section to fix HTML rendering. --- docs/source/gfql/spec/cypher_mapping.md | 1 + 1 file changed, 1 insertion(+) diff --git a/docs/source/gfql/spec/cypher_mapping.md b/docs/source/gfql/spec/cypher_mapping.md index d511d22e81..7228719133 100644 --- a/docs/source/gfql/spec/cypher_mapping.md +++ b/docs/source/gfql/spec/cypher_mapping.md @@ -62,6 +62,7 @@ g.chain([ {"type": "Edge", "direction": "forward", "edge_match": {"type": "FOLLOWS"}, "name": "r"}, {"type": "Node", "filter_dict": {"type": "Person"}, "name": "q"} ]} +``` ## Pattern Translations From 15102206bef7234d04062bc298a0824e9948512e Mon Sep 17 00:00:00 2001 From: Leo Meyerovich Date: Tue, 15 Jul 2025 03:11:17 -0700 Subject: [PATCH 7/8] docs(gfql): fix code block formatting in cypher_mapping.md Ensure all JSON code blocks have proper closing backticks to prevent markdown rendering issues. --- docs/source/gfql/spec/cypher_mapping.md | 1 + 1 file changed, 1 insertion(+) diff --git a/docs/source/gfql/spec/cypher_mapping.md b/docs/source/gfql/spec/cypher_mapping.md index 7228719133..f936358348 100644 --- a/docs/source/gfql/spec/cypher_mapping.md +++ b/docs/source/gfql/spec/cypher_mapping.md @@ -133,6 +133,7 @@ g.chain([ {"type": "Edge", "direction": "forward", "edge_match": {"type": "FRIEND"}, "hops": 2}, {"type": "Node", "filter_dict": {"type": "User", "active": true}, "name": "fof"} ]} +``` ### Fraud Detection From 9b93facbf76b7c28e0adc2fbbe3dd34e00b223b8 Mon Sep 17 00:00:00 2001 From: Leo Meyerovich Date: Tue, 15 Jul 2025 03:25:18 -0700 Subject: [PATCH 8/8] docs: Add GFQL specification documentation to changelog MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - Added entry for PR #698 in Dev section - Listed key documentation improvements and fixes ๐Ÿค– Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude --- CHANGELOG.md | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index 7293251bc0..59d9c0faad 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -9,6 +9,13 @@ This project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.htm ### Docs * Update copyright year from 2024 to 2025 in documentation and LICENSE.txt +* GFQL: Add comprehensive specification documentation (#698) + * Core language specification with formal grammar, operations, predicates, and type system + * Cypher to GFQL translation guide with Python and wire protocol examples + * Python embedding guide with pandas/cuDF integration details + * Wire protocol JSON format for client-server communication + * Fix terminology: clarify g._node (node ID column) vs g._nodes (DataFrame) + * Emphasize GFQL's declarative nature for graph-to-graph transformations ## [0.39.1 - 2025-07-07]