diff --git a/CHANGELOG.md b/CHANGELOG.md index 7293251bc0..59d9c0faad 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -9,6 +9,13 @@ This project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.htm ### Docs * Update copyright year from 2024 to 2025 in documentation and LICENSE.txt +* GFQL: Add comprehensive specification documentation (#698) + * Core language specification with formal grammar, operations, predicates, and type system + * Cypher to GFQL translation guide with Python and wire protocol examples + * Python embedding guide with pandas/cuDF integration details + * Wire protocol JSON format for client-server communication + * Fix terminology: clarify g._node (node ID column) vs g._nodes (DataFrame) + * Emphasize GFQL's declarative nature for graph-to-graph transformations ## [0.39.1 - 2025-07-07] diff --git a/ai_code_notes/gfql/README.md b/ai_code_notes/gfql/README.md new file mode 100644 index 0000000000..16d84b3c0e --- /dev/null +++ b/ai_code_notes/gfql/README.md @@ -0,0 +1,211 @@ +# GFQL AI Assistant Guide + +Guide for AI assistants working with GFQL (Graph Frame Query Language) in PyGraphistry. + +## ๐ŸŽฏ Quick Reference + +### Essential GFQL Operations +```python +# Node matching +n() # All nodes +n({"type": "person"}) # Filter by property +n({"age": gt(30)}) # With predicate +n(name="result") # Named results + +# Edge traversal +e_forward() # Forward direction +e_reverse() # Reverse direction +e() or e_undirected() # Both directions +e_forward(hops=2) # Multi-hop +e_forward(to_fixed_point=True) # All reachable + +# Chaining +g.chain([n(), e_forward(), n()]) # Pattern matching +``` + +### Key Predicates +- Comparison: `gt()`, `lt()`, `ge()`, `le()`, `eq()`, `ne()` +- Membership: `is_in([...])` +- Range: `between(lower, upper)` +- String: `contains()`, `startswith()`, `endswith()` +- Null: `is_null()`, `not_null()` +- Temporal: `is_month_start()`, `is_year_end()`, etc. + +### Performance Tips +- Filter early in the chain +- Use specific hop counts vs `to_fixed_point` +- Prefer `filter_dict` over `query` strings +- Use appropriate engine: `pandas` (CPU) or `cudf` (GPU) + +## ๐Ÿ“‹ When to Use GFQL + +### Use GFQL When +- Performing graph traversals or path queries +- Finding patterns in connected data +- Need efficient multi-hop operations +- Working with node/edge dataframes + +### Use Pandas/Aggregations When +- Need sorting (`sort_values()`) +- Need limiting (`head()`, `tail()`) +- Aggregating results (`groupby()`, `count()`) +- Complex transformations + +## ๐Ÿš€ Common Patterns + +### User 360 Query +```python +# Customer's recent activity +g.chain([ + n({"customer_id": "C123"}), + e_forward({ + "type": is_in(["purchase", "view", "support"]), + "timestamp": gt(pd.Timestamp.now() - pd.Timedelta(days=30)) + }) +]) +``` + +### Cyber Security Pattern +```python +# Lateral movement detection +g.chain([ + n({"status": "compromised"}), + e_forward({"type": "login", "success": True}, hops=3), + n({"criticality": "high"}, name="at_risk") +]) +``` + +### Business Intelligence +```python +# Cross-sell opportunities +g.chain([ + n({"product_id": "P123"}), + e_reverse({"type": "purchased"}), + n({"type": "customer"}), + e_forward({"type": "purchased"}), + n({"product_id": ne("P123")}, name="also_bought") +]) +``` + +## ๐Ÿ”ง Code Style Guidelines + +### Preferred Style +```python +# โœ… Good - Clean, code-golfed chains +g.chain([n({"type": "user"}), e({"active": True}), n()]) + +# โŒ Avoid - Overly verbose +result = g.chain([ + n(filter_dict={"type": "user"}), + e_forward(edge_match={"active": True}, hops=1), + n(filter_dict={}) +]) +``` + +### Naming Conventions +- Use descriptive names for `name` parameters +- Keep filter keys consistent with dataframe columns +- Use snake_case for all identifiers + +## ๐Ÿ› Common Errors and Fixes + +### Schema Errors +```python +# โŒ Wrong - Column doesn't exist +n({"username": "Alice"}) + +# โœ… Fix - Use correct column name +n({"name": "Alice"}) +``` + +### Type Errors +```python +# โŒ Wrong - String predicate on number +n({"age": contains("30")}) + +# โœ… Fix - Use numeric predicate +n({"age": gt(30)}) +``` + +### Temporal Errors +```python +# โŒ Wrong - Raw string for datetime +n({"created": gt("2024-01-01")}) + +# โœ… Fix - Use proper datetime +n({"created": gt(pd.Timestamp("2024-01-01"))}) +``` + +## ๐Ÿ“ Natural Language to GFQL + +### Translation Patterns +- "recent" โ†’ `gt(pd.Timestamp.now() - pd.Timedelta(days=N))` +- "between X and Y" โ†’ `between(X, Y)` +- "any of" โ†’ `is_in([...])` +- "connected to" โ†’ `e()` or `e_undirected()` +- "from X to Y" โ†’ X with `e_forward()` to Y +- "within N hops" โ†’ `hops=N` + +### Example Translations + +**NL**: "Find all employees who report to managers in NYC" +```python +g.chain([ + n({"type": "employee"}), + e_forward({"type": "reports_to"}), + n({"type": "manager", "office": "NYC"}) +]) +``` + +**NL**: "Show me high-value customers from last week" +```python +g.chain([ + n({"customer_tier": "high_value"}), + e_forward({ + "type": "purchase", + "date": gt(pd.Timestamp.now() - pd.Timedelta(days=7)) + }) +]) +``` + +## ๐Ÿ”„ Cypher to GFQL + +### Basic Mappings +| Cypher | GFQL | +|--------|------| +| `(n)` | `n()` | +| `(n:Label)` | `n({"type": "Label"})` | +| `-[r]->` | `e_forward()` | +| `<-[r]-` | `e_reverse()` | +| `-[r*2]-` | `e_forward(hops=2)` | +| `WHERE n.prop = val` | `n({"prop": val})` | + +### Unsupported in GFQL +- `OPTIONAL MATCH` - Handle nulls in post-processing +- `WITH` clauses - Use intermediate chains +- `ORDER BY/LIMIT` - Use pandas after +- `CREATE/DELETE` - GFQL is read-only + +## ๐Ÿงช Validation Checklist + +Before generating GFQL: +1. โœ“ Check column names exist in schema +2. โœ“ Verify predicate types match column types +3. โœ“ Ensure temporal values use proper types +4. โœ“ Validate operation names (n, e_forward, etc.) +5. โœ“ Check chain structure is valid + +## ๐Ÿ“š Additional Resources + +- Full specifications in: `AI_PROGRESS/gfql_llm_specs/` + - `gfql_language_spec.md` - Complete language specification + - `gfql_wire_protocol_spec.md` - JSON wire format + - `cypher_to_gfql_mapping_spec.md` - Cypher translation + +## ๐ŸŽฏ Key Takeaways + +1. **GFQL is functional**: Chain operations, don't mutate +2. **Filter early**: Put selective conditions first +3. **Think patterns**: Focus on graph patterns, not procedures +4. **Post-process**: Use pandas for sorting/aggregating +5. **Code golf**: Keep queries concise and elegant \ No newline at end of file diff --git a/docs/source/gfql/index.rst b/docs/source/gfql/index.rst index fd21b5f2ba..7d3bba504f 100644 --- a/docs/source/gfql/index.rst +++ b/docs/source/gfql/index.rst @@ -10,6 +10,7 @@ See also: .. toctree:: :maxdepth: 1 + :caption: User Guide about overview @@ -21,3 +22,9 @@ See also: predicates/quick datetime_filtering wire_protocol_examples + +.. toctree:: + :maxdepth: 2 + :caption: Developer Resources + + spec/index diff --git a/docs/source/gfql/spec/cypher_mapping.md b/docs/source/gfql/spec/cypher_mapping.md new file mode 100644 index 0000000000..f936358348 --- /dev/null +++ b/docs/source/gfql/spec/cypher_mapping.md @@ -0,0 +1,260 @@ +(gfql-spec-cypher-mapping)= + +# Cypher to GFQL Python & Wire Protocol Mapping + +## Introduction + +This specification shows how to translate Cypher queries to both GFQL Python code and Wire Protocol JSON, enabling: +- Migration from Cypher-based systems +- Two-stage LLM synthesis: Text โ†’ Cypher โ†’ GFQL +- Language-agnostic API integration +- Secure query generation without code execution + +## Conceptual Framework + +### Translation Scenarios + +When translating from Cypher, you'll encounter three scenarios: + +**1. Direct Translation** - Most pattern matching maps cleanly to pure GFQL +**2. Hybrid Approach** - Post-processing operations (RETURN clauses) use dataframes +**3. GFQL Advantages** - Some capabilities go beyond what Cypher offers + +### What Translates Directly +- Graph patterns: `(a)-[r]->(b)` โ†’ chain operations +- Property filters: WHERE clauses embed into operations +- Path traversals: Variable-length paths use `hops` parameter +- Pattern composition: Multiple patterns become sequential operations + +### What Requires DataFrames +- Aggregations: COUNT, SUM, AVG โ†’ pandas operations +- Projections: RETURN specific columns โ†’ DataFrame selection +- Sorting/limiting: ORDER BY, LIMIT โ†’ DataFrame methods +- Joins: Multiple disconnected patterns โ†’ pandas merge + +### GFQL Advantages Beyond Cypher +- **Rich edge properties**: Query edges as first-class entities +- **Dataframe-native**: Zero-cost transitions between graph and tabular operations +- **GPU acceleration**: Massively parallel execution on NVIDIA hardware +- **Heterogeneous graphs**: No schema constraints on types or properties + +## Quick Example + +**Cypher:** +```cypher +MATCH (p:Person)-[r:FOLLOWS]->(q:Person) +WHERE p.age > 30 +``` + +**Python:** +```python +g.chain([ + n({"type": "Person", "age": gt(30)}, name="p"), + e_forward({"type": "FOLLOWS"}, name="r"), + n({"type": "Person"}, name="q") +]) +``` + +**Wire Protocol:** +```json +{"type": "Chain", "chain": [ + {"type": "Node", "filter_dict": {"type": "Person", "age": {"type": "GT", "val": 30}}, "name": "p"}, + {"type": "Edge", "direction": "forward", "edge_match": {"type": "FOLLOWS"}, "name": "r"}, + {"type": "Node", "filter_dict": {"type": "Person"}, "name": "q"} +]} +``` + +## Pattern Translations + +### Node Patterns + +| Cypher | Python | Wire Protocol | +|--------|--------|---------------| +| `(n)` | `n()` | `{"type": "Node"}` | +| `(n:Label)` | `n({"type": "Label"})` | `{"type": "Node", "filter_dict": {"type": "Label"}}` | +| `(n {prop: val})` | `n({"prop": val})` | `{"type": "Node", "filter_dict": {"prop": val}}` | +| `(n:Person) WHERE n.age > 30` | `n({"type": "Person", "age": gt(30)})` | `{"type": "Node", "filter_dict": {"type": "Person", "age": {"type": "GT", "val": 30}}}` | + +### Edge Patterns + +| Cypher | Python | Wire Protocol (compact) | +|--------|--------|-------------------------| +| `-[]->` | `e_forward()` | `{"type": "Edge", "direction": "forward"}` | +| `-[r:KNOWS]->` | `e_forward({"type": "KNOWS"}, name="r")` | `{"type": "Edge", "direction": "forward", "edge_match": {"type": "KNOWS"}, "name": "r"}` | +| `<-[r]-` | `e_reverse(name="r")` | `{"type": "Edge", "direction": "reverse", "name": "r"}` | +| `-[r]-` | `e(name="r")` | `{"type": "Edge", "direction": "undirected", "name": "r"}` | +| `-[*2]->` | `e_forward(hops=2)` | `{"type": "Edge", "direction": "forward", "hops": 2}` | +| `-[*1..3]->` | `e_forward(hops=3)` | `{"type": "Edge", "direction": "forward", "hops": 3}` | +| `-[*]->` | `e_forward(to_fixed_point=True)` | `{"type": "Edge", "direction": "forward", "to_fixed_point": true}` | +| `-[r:BOUGHT {amount: gt(100)}]->` | `e_forward({"type": "BOUGHT", "amount": gt(100)}, name="r")` | `{"type": "Edge", "direction": "forward", "edge_match": {"type": "BOUGHT", "amount": {"type": "GT", "val": 100}}, "name": "r"}` | + +### Predicates + +| Cypher | Python | Wire Protocol | +|--------|--------|---------------| +| `n.age > 30` | `gt(30)` | `{"type": "GT", "val": 30}` | +| `n.age >= 50` | `ge(50)` | `{"type": "GE", "val": 50}` | +| `n.age < 100` | `lt(100)` | `{"type": "LT", "val": 100}` | +| `n.age <= 50` | `le(50)` | `{"type": "LE", "val": 50}` | +| `n.status = 'active'` | `"active"` | `"active"` | +| `n.status <> 'deleted'` | `ne("deleted")` | `{"type": "NE", "val": "deleted"}` | +| `n.id IN [1,2,3]` | `is_in([1,2,3])` | `{"type": "IsIn", "options": [1,2,3]}` | +| `n.score BETWEEN 0 AND 100` | `between(0, 100)` | `{"type": "Between", "lower": 0, "upper": 100}` | +| `n.name =~ '^A.*'` | `match("^A.*")` | `{"type": "Match", "pattern": "^A.*"}` | +| `n.text CONTAINS 'search'` | `contains("search")` | `{"type": "Contains", "pattern": "search"}` | +| `n.name STARTS WITH 'Dr'` | `startswith("Dr")` | `{"type": "Startswith", "pattern": "Dr"}` | +| `n.email ENDS WITH '.com'` | `endswith(".com")` | `{"type": "Endswith", "pattern": ".com"}` | +| `n.val IS NULL` | `is_null()` | `{"type": "IsNull"}` | +| `n.val IS NOT NULL` | `not_null()` | `{"type": "NotNull"}` | + +## Complete Examples + +### Friend of Friend + +**Cypher:** +```cypher +MATCH (u:User {name: 'Alice'})-[:FRIEND*2]->(fof:User) +WHERE fof.active = true +``` + +**Python:** +```python +g.chain([ + n({"type": "User", "name": "Alice"}), + e_forward({"type": "FRIEND"}, hops=2), + n({"type": "User", "active": True}, name="fof") +]) +``` + +**Wire Protocol:** +```json +{"type": "Chain", "chain": [ + {"type": "Node", "filter_dict": {"type": "User", "name": "Alice"}}, + {"type": "Edge", "direction": "forward", "edge_match": {"type": "FRIEND"}, "hops": 2}, + {"type": "Node", "filter_dict": {"type": "User", "active": true}, "name": "fof"} +]} +``` + +### Fraud Detection + +**Cypher:** +```cypher +MATCH (a:Account)-[t:TRANSFER]->(b:Account) +WHERE t.amount > 10000 AND t.date > date('2024-01-01') +``` + +**Python:** +```python +g.chain([ + n({"type": "Account"}), + e_forward({ + "type": "TRANSFER", + "amount": gt(10000), + "date": gt(date(2024,1,1)) + }, name="t"), + n({"type": "Account"}) +]) +``` + +**Wire Protocol:** +```json +{"type": "Chain", "chain": [ + {"type": "Node", "filter_dict": {"type": "Account"}}, + {"type": "Edge", "direction": "forward", "edge_match": { + "type": "TRANSFER", + "amount": {"type": "GT", "val": 10000}, + "date": {"type": "GT", "val": {"type": "date", "value": "2024-01-01"}} + }, "name": "t"}, + {"type": "Node", "filter_dict": {"type": "Account"}} +]} +``` + +### Complex Aggregation Example + +**Cypher:** +```cypher +MATCH (u:User)-[t:TRANSACTION]->(m:Merchant) +WHERE t.date > date('2024-01-01') +RETURN m.category, count(*) as cnt, sum(t.amount) as total +ORDER BY total DESC +LIMIT 10 +``` + +**Python:** +```python +# Step 1: Graph pattern +result = g.chain([ + n({"type": "User"}), + e_forward({"type": "TRANSACTION", "date": gt(date(2024,1,1))}, name="trans"), + n({"type": "Merchant"}) +]) + +# Step 2: DataFrame operations +trans_df = result._edges[result._edges["trans"]] +merchant_df = result._nodes +analysis = (trans_df + .merge(merchant_df, left_on=g._destination, right_on=g._node) + .groupby('category') + .agg(cnt=('amount', 'count'), total=('amount', 'sum')) + .nlargest(10, 'total')) +``` + +**Note:** Wire protocol returns the filtered graph; aggregations require client-side processing. + +## DataFrame Operations Mapping + +| Cypher Feature | Python DataFrame Operation | Notes | +|----------------|---------------------------|--------| +| `RETURN a, b, c` | `df[['a', 'b', 'c']]` | Column selection | +| `RETURN DISTINCT` | `df.drop_duplicates()` | Remove duplicates | +| `ORDER BY x DESC` | `df.sort_values('x', ascending=False)` | Sort results | +| `LIMIT 10` | `df.head(10)` | Limit rows | +| `count(*)` | `len(df)` or `df.groupby(...).size()` | Count rows | +| `sum(n.val)` | `df['val'].sum()` or `df.groupby(...).agg(sum)` | Aggregation | +| `collect(n.x)` | `df.groupby(...).agg(list)` | Collect to list | +| Named patterns | `df[df['pattern_name']]` | Boolean column filtering | + +## Key Differences + +| Feature | Python | Wire Protocol | +|---------|--------|---------------| +| **Temporal values** | `pd.Timestamp()`, `date()` | `{"type": "date", "value": "..."}` | +| **Direct equality** | `"active"` | `"active"` (same) | +| **Comparisons** | `gt(30)` | `{"type": "GT", "val": 30}` | +| **Collections** | `is_in([...])` | `{"type": "IsIn", "options": [...]}` | + +## Not Supported +- `OPTIONAL MATCH` - No equivalent (would need outer joins) +- `CREATE`, `DELETE`, `SET` - GFQL is read-only +- `WITH` clauses - Requires intermediate variables +- Multiple `MATCH` patterns - Use separate chains or joins + +## Best Practices + +1. **Direct Translation First**: Try pure GFQL before adding DataFrame operations +2. **Use Named Patterns**: Label important results with `name=` for easy access +3. **Filter Early**: Apply selective node filters before traversing edges +4. **Type Consistency**: Ensure wire protocol types match expected column types +5. **Validate JSON**: Test wire protocol against schema before sending + +## LLM Integration Guide + +When building translators: + +``` +Given Cypher: {cypher_query} + +Generate both: +1. Python: Human-readable GFQL code +2. Wire Protocol: JSON for API calls + +Rules: +- (n:Label) โ†’ Python: n({"type": "Label"}) โ†’ JSON: {"type": "Node", "filter_dict": {"type": "Label"}} +- WHERE โ†’ Embed as predicates in both formats +- Aggregations โ†’ Note as requiring DataFrame post-processing +``` + +## See Also +- {ref}`gfql-spec-wire-protocol` - Full wire protocol specification +- {ref}`gfql-spec-language` - Language specification +- {ref}`gfql-spec-python-embedding` - Python implementation details \ No newline at end of file diff --git a/docs/source/gfql/spec/index.md b/docs/source/gfql/spec/index.md new file mode 100644 index 0000000000..4e1c3f6b41 --- /dev/null +++ b/docs/source/gfql/spec/index.md @@ -0,0 +1,23 @@ +(gfql-specifications)= + +# GFQL Specifications + +This section contains formal specifications for GFQL (Graph Frame Query Language), designed to support both human understanding and automated tooling, including LLM-based code synthesis. + +```{toctree} +:maxdepth: 1 + +language +python_embedding +wire_protocol +cypher_mapping +``` + +## Overview + +- {ref}`gfql-spec-language` - Complete formal grammar, operations, predicates, and type system +- {ref}`gfql-spec-python-embedding` - Python-specific implementation with pandas/cuDF +- {ref}`gfql-spec-wire-protocol` - JSON serialization format for client-server communication +- {ref}`gfql-spec-cypher-mapping` - Cypher to GFQL translations with both Python and wire protocol + +These specifications are optimized for text-to-GFQL synthesis, Cypher-to-GFQL pipelines, query validation, and schema-aware code generation. \ No newline at end of file diff --git a/docs/source/gfql/spec/language.md b/docs/source/gfql/spec/language.md new file mode 100644 index 0000000000..3d0c444af5 --- /dev/null +++ b/docs/source/gfql/spec/language.md @@ -0,0 +1,377 @@ +(gfql-spec-language)= + +# GFQL Language Specification + +## Introduction + +GFQL (Graph Frame Query Language) is a DataFrame-native graph query language designed for expressing graph patterns and traversals on tabular data. It operates on node and edge DataFrames, providing a functional, composable approach to graph querying with native GPU acceleration support. + +### Design Principles +- **Dataframe-native**: Type-safe functional bulk operations over dataframe libraries like pandas, cuDF +- **Declarative**: Focus on what to retrieve, and give the engine freedom to optimize how +- **Accessible**: Designed for both human readability and machine generation, and building on intuitions from popular tabular and graph systems +- **Performance-oriented**: Vectorized operations by default, including GPU acceleration +- **Embeddable**: Similar to DuckDB, can be embedded in different languages, and initially focused on Python data ecosystem +- **Computer-tier**: Decoupling from storage enables flexible execution - embedded locally or via remote acceleration servers + +### Language Forms + +GFQL exists in three complementary forms: + +1. **Core Language**: Abstract graph pattern matching language defined by this specification +2. **Embedded DSL**: Host language implementations (currently Python with pandas/cuDF) +3. **Wire Protocol**: JSON serialization for client-server communication (see Wire Protocol spec) + +This specification focuses on the core language concepts. Examples use Python syntax for concreteness, but the patterns apply to any embedding. + +## Language Overview + +### Core Concepts + +#### Graph Model + +Graphs consist of node and edge dataframes: +- Edges: DataFrame with source and destination columns +- Nodes: DataFrame with unique identifier column +- Column names are user-defined globals for the graph: + - Node ID attribute: `g._node` (e.g., "node_id", "id") + - Edge source attribute: `g._source` (e.g., "source", "from") + - Edge destination attribute: `g._destination` (e.g., "destination", "to") +- GFQL infers nodes from edge references when only edges are provided + +#### GFQL Programs + +GFQL programs are declarative graph-to-graph transformations: +- Enable use cases like search, filter, enrich, and traverse +- Express *what* to find (ex: Cypher), not *how* to find it (ex: Gremlin) + +#### Chains + +Path pattern expressions for matching graph structures: +- Express graph patterns as sequences of node and edge matching operations +- Similar to Cypher patterns but decomposed into composable steps +- Define paths through the graph: start nodes โ†’ edges โ†’ end nodes +- Each operation refines the pattern match based on previous results + +#### Operations + +Act on graph entities (nodes and edges): +- Node matchers: Filter and select nodes +- Edge matchers: Traverse relationships +- Operations work on the graph structure itself + +#### Predicates + +Act on attributes of nodes and edges: +- Filter based on property values +- Comparison, membership, string matching, temporal checks +- Composable within operations to build complex conditions + +#### Values + +Type system matching modern data formats: +- Scalars: numbers, strings, booleans, null +- Temporal: ISO datetimes, dates, times with timezone support +- Collections: lists for membership tests +- Compatible with JSON, Arrow, and DataFrame type systems + +## Formal Grammar + +```{code-block} ebnf +:caption: GFQL Grammar in Extended Backus-Naur Form + +(* Entry point *) +query ::= chain + +(* Chain - path pattern expression *) +chain ::= "[" operation ("," operation)* "]" + +(* Operations *) +operation ::= node_matcher | edge_matcher + +(* Node Matcher *) +node_matcher ::= "n(" node_params? ")" +node_params ::= filter_dict ("," name_param)? ("," query_param)? + | name_param ("," query_param)? + | query_param + +(* Edge Matchers *) +edge_matcher ::= edge_forward | edge_reverse | edge_undirected +edge_forward ::= "e_forward(" edge_params? ")" +edge_reverse ::= "e_reverse(" edge_params? ")" +edge_undirected ::= ("e" | "e_undirected") "(" edge_params? ")" + +(* Parameters *) +edge_params ::= edge_match_params ("," hop_params)? ("," node_filter_params)? ("," name_param)? + +filter_dict ::= "{" (property_filter ("," property_filter)*)? "}" +property_filter ::= string ":" (value | predicate) + +hop_params ::= "hops=" integer | "to_fixed_point=True" +node_filter_params ::= source_filter ("," dest_filter)? +source_filter ::= "source_node_match=" filter_dict | "source_node_query=" string +dest_filter ::= "destination_node_match=" filter_dict | "destination_node_query=" string + +name_param ::= "name=" string +query_param ::= "query=" string +edge_query_param ::= "edge_query=" string +edge_match_params ::= filter_dict | edge_query_param + +(* Predicates *) +predicate ::= comparison | membership | range | null_check | string_pred | temporal_pred + +comparison ::= ("gt" | "lt" | "ge" | "le" | "eq" | "ne") "(" value ")" +membership ::= "is_in(" "[" value ("," value)* "]" ")" +range ::= "between(" value "," value ("," "inclusive=" boolean)? ")" +null_check ::= "is_null()" | "not_null()" | "is_na()" | "not_na()" +string_pred ::= string_match | string_check +string_match ::= "contains(" string ("," "case=" boolean)? ("," "regex=" boolean)? ")" + | "match(" string ("," "case=" boolean)? ")" + | ("startswith" | "endswith") "(" string ")" +string_check ::= ("isalpha" | "isnumeric" | "isdigit" | "isalnum" + | "isupper" | "islower") "()" +temporal_pred ::= temporal_check "()" +temporal_check ::= "is_month_start" | "is_month_end" | "is_quarter_start" + | "is_quarter_end" | "is_year_start" | "is_year_end" | "is_leap_year" + +(* Values *) +value ::= scalar | temporal_value | collection +scalar ::= number | string | boolean | null +temporal_value ::= datetime_value | date_value | time_value +datetime_value ::= "pd.Timestamp(" string ("," "tz=" string)? ")" + | "datetime(" datetime_args ")" +date_value ::= "date(" date_args ")" +time_value ::= "time(" time_args ")" +collection ::= "[" (value ("," value)*)? "]" + +(* Primitives *) +string ::= '"' [^"]* '"' | "'" [^']* "'" +number ::= integer | float +integer ::= ["-"]? [0-9]+ +float ::= ["-"]? [0-9]+ "." [0-9]+ +boolean ::= "True" | "False" +null ::= "None" +datetime_args ::= integer ("," integer)* +date_args ::= integer "," integer "," integer +time_args ::= integer "," integer ("," integer)? +``` + +## Operations + +### Node Matcher: `n()` + +Filters nodes based on attributes. + +**Syntax**: `n(filter_dict?, name?, query?)` + +**Parameters**: +- `filter_dict`: Dictionary of attribute filters +- `name`: Optional string label for results +- `query`: Pandas query string expression + +**Examples**: +```python +n() # All nodes +n({"type": "person"}) # Nodes where type='person' +n({"age": gt(30)}) # Nodes where age > 30 +n(name="important") # Label matching nodes +n(query="age > 30 and status == 'active'") # Query string +``` + +### Edge Matchers + +#### Forward Traversal: `e_forward()` + +Traverses edges in forward direction (source โ†’ destination). + +**Syntax**: `e_forward(edge_match?, hops?, to_fixed_point?, source_node_match?, destination_node_match?, name?)` + +**Parameters**: +- `edge_match`: Edge attribute filters +- `hops`: Number of hops (default: 1) +- `to_fixed_point`: Continue until no new nodes (default: False) +- `source_node_match`: Filters for source nodes +- `destination_node_match`: Filters for destination nodes +- `name`: Optional label + +**Examples**: +```python +e_forward() # One hop forward +e_forward(hops=2) # Two hops forward +e_forward(to_fixed_point=True) # All reachable nodes +e_forward({"type": "follows"}) # Only 'follows' edges +e_forward(source_node_match={"active": True}) # From active nodes +``` + +#### Reverse Traversal: `e_reverse()` + +Traverses edges in reverse direction (destination โ†’ source). + +**Syntax**: Same as `e_forward()` + +#### Undirected Traversal: `e()` or `e_undirected()` + +Traverses edges in both directions. + +**Syntax**: Same as `e_forward()` + +## Predicates + +### Comparison Predicates + +```python +gt(value) # Greater than +lt(value) # Less than +ge(value) # Greater than or equal +le(value) # Less than or equal +eq(value) # Equal +ne(value) # Not equal +``` + +### Membership Predicate + +```python +is_in([value1, value2, ...]) # Value in list +``` + +### Range Predicate + +```python +between(lower, upper, inclusive=True) # Value in range +``` + +### String Predicates + +Pattern matching predicates: +```python +contains(pat, case=True, regex=True) # Contains pattern (substring or regex) +startswith(prefix) # Starts with prefix (case-sensitive) +endswith(suffix) # Ends with suffix (case-sensitive) +match(pat, case=True) # Matches regex from start of string +``` + +String type checking predicates: +```python +isalpha() # Alphabetic characters only +isnumeric() # Numeric characters only +isdigit() # Digits only +isalnum() # Alphanumeric +isupper() # All uppercase +islower() # All lowercase +``` + +### Null Predicates + +```python +is_null() # Is null/None +not_null() # Is not null/None +is_na() # Is NaN (numeric) +not_na() # Is not NaN +``` + +### Temporal Predicates + +```python +is_month_start() # First day of month +is_month_end() # Last day of month +is_quarter_start() # First day of quarter +is_quarter_end() # Last day of quarter +is_year_start() # First day of year +is_year_end() # Last day of year +is_leap_year() # Is leap year +``` + +## Type System + +### Value Types + +1. **Scalars** + - `number`: int, float + - `string`: Text values + - `boolean`: True/False + - `null`: None + +2. **Temporal Types** + - `datetime`: Timestamp with optional timezone + - `date`: Calendar date + - `time`: Time of day + +3. **Collections** + - `list`: Ordered sequence of values + +### Type Coercion + +GFQL performs automatic type coercion: +- Python datetime โ†’ pandas Timestamp +- Numeric types โ†’ appropriate precision +- Collections โ†’ lists for `is_in()` + +## Execution Model + +### Declarative Pattern Matching + +GFQL follows a declarative execution model similar to Neo4j's Cypher: + +1. **Pattern Declaration**: Chains express path patterns in the graph + - Users declare graph patterns as sequences of node and edge constraints + - Patterns specify *what* paths to match, not *how* to find them + - The engine optimizes pattern matching based on data characteristics + +2. **Set-Based Operations**: All operations work on sets of entities + - No explicit iteration or traversal order + - Results include all matching patterns in the graph + - Current GFQL engines use a novel bulk-oriented execution model that is asymptotically faster than traditional iterative approaches used for Cypher, but this is not a requirement of the language itself + +3. **Lazy Evaluation**: Chains define pattern transformations without immediate execution + - Allows engines to optimize path finding and pattern matching strategies +\ +### Result Access + +Query execution returns filtered node and edge datasets. In the Python embedding: + +```python +result = g.chain([...]) +nodes_df = result._nodes # Filtered nodes +edges_df = result._edges # Filtered edges +``` + +### Named Results + +Operations with `name` parameter add boolean columns to mark matched entities: + +```python +result = g.chain([ + n({"type": "person"}, name="people"), + e_forward(name="connections"), + n({"active": True}, name="active_targets") +]) + +# Access all matched nodes and edges: +all_nodes = result._nodes +all_edges = result._edges + +# Access specific matched nodes/edges using pandas filtering: +people_nodes = result._nodes[result._nodes["people"]] +connection_edges = result._edges[result._edges["connections"]] +active_nodes = result._nodes[result._nodes["active_targets"]] + +# Or using standard pandas query syntax: +people_nodes = result._nodes.query("people == True") +``` + +This pattern is essential for extracting specific subsets from complex graph traversals. + +## Best Practices + +1. **Use specific filters early**: Filter nodes before traversing edges +2. **Limit hops**: Use reasonable hop limits to avoid explosion +3. **Name important results**: Use `name` parameter for analysis +4. **Prefer filter_dict**: More efficient than query strings +5. **Use appropriate predicates**: Match predicate to column type + +## See Also + +- {ref}`gfql-spec-python-embedding` - Python implementation details +- {ref}`gfql-spec-wire-protocol` - JSON serialization format +- {ref}`gfql-spec-cypher-mapping` - Cypher to GFQL translation with wire protocol +- [GFQL Quick Reference](../quick.rst) - Comprehensive examples and usage patterns \ No newline at end of file diff --git a/docs/source/gfql/spec/python_embedding.md b/docs/source/gfql/spec/python_embedding.md new file mode 100644 index 0000000000..27be847e4b --- /dev/null +++ b/docs/source/gfql/spec/python_embedding.md @@ -0,0 +1,195 @@ +(gfql-spec-python-embedding)= + +# GFQL Python Embedding + +This document describes the Python-specific implementation of GFQL using pandas and cuDF dataframes. + +## Graph Construction + +In Python, graphs are created with user-defined column names: + +```python +import graphistry +assert 'src_col' in df.columns and 'dst_col' in df.columns +g = graphistry.edges(df, source='src_col', destination='dst_col') + +# Optional; GFQL infers node existence when only edges are provided +assert 'node_col' in df.columns +g2 = graphistry.nodes(df, node='node_col') +``` + +### Schema Access + +The graph schema is accessible via attributes: +- `g._node`: Node ID column name +- `g._source`: Edge source column name +- `g._destination`: Edge destination column name + +Graph nodes can be generically accessed using these attributes: +- `g._nodes`: Node DataFrame +- `g._nodes[g._node]`: Node ID column +- `g._nodes[[attr for attr in g._nodes.columns if attr != g._node]]`: All other node attributes + +Graph edges can be accessed similarly: +- `g._edges`: Edge DataFrame +- `g._edges[g._source]`: Edge source column +- `g._edges[g._destination]`: Edge destination column +- `g._edges[[attr for attr in g._edges.columns if attr not in [g._source, g._destination]]]`: All other edge attributes + +## Query Execution + +```python +from graphistry import n, e_forward + +# Execute a chain +result = g.chain([ + n({"type": "person"}), + e_forward(), + n() +]) + +# Access results +nodes_df = result._nodes # Filtered nodes DataFrame +edges_df = result._edges # Filtered edges DataFrame +``` + +## Engine Selection + +GFQL supports multiple execution engines: + +- **pandas**: CPU execution (default) +- **cudf**: GPU acceleration +- **auto**: Automatic selection based on data type + +```python +# Force specific engine +g.chain([...], engine='cudf') # GPU execution +g.chain([...], engine='pandas') # CPU execution +g.chain([...], engine='auto') # Auto-select +``` + +## Python-Specific Values + +### Temporal Values + +```python +import pandas as pd + +# Timestamps +pd.Timestamp('2023-01-01') +pd.Timestamp.now() + +# Time deltas +pd.Timedelta(days=30) +pd.Timedelta(hours=24) +``` + +### DataFrame Operations + +Results can be further processed using standard pandas operations: + +```python +# Using boolean columns from named operations +people_nodes = result._nodes[result._nodes["people"]] + +# Using pandas query +active_nodes = result._nodes.query("active == True") + +# Standard pandas operations +result._nodes.groupby('type').size() +``` + +## Common Errors and Validation + +### Type Mismatches + +```python +# Wrong - String predicate on numeric column +n({"age": contains("3")}) + +# Correct - Use numeric predicate +n({"age": gt(30)}) + +# Wrong - String comparison on datetime +n({"created": gt("2024-01-01")}) + +# Correct - Use proper datetime type +n({"created": gt(pd.Timestamp("2024-01-01"))}) +``` + +### Schema Validation + +```python +# Check available columns before querying +print(g._nodes.columns) # ['id', 'type', 'name'] + +# Wrong - Column doesn't exist +g.chain([n({"username": "Alice"})]) # KeyError + +# Correct - Use existing column +g.chain([n({"name": "Alice"})]) +``` + +### Unsupported Operations + +```python +# Wrong - Can't aggregate in chain +# g.chain([n(), e(), count()]) + +# Correct - Aggregate after chain +result = g.chain([n(), e()]) +count = len(result._edges) + +# Wrong - OPTIONAL MATCH not supported +# No direct GFQL equivalent + +# Correct - Handle optionality in post-processing +result = g.chain([n(), e_forward()]) +# Check for nodes without edges +nodes_with_edges = result._nodes[result._nodes[g._node].isin(result._edges[g._source])] +``` + +## Best Practices + +### Query Construction +```python +# Good: Build queries programmatically +node_filters = {"type": "User"} +if min_age: + node_filters["age"] = gt(min_age) +g.chain([n(node_filters)]) + +# Avoid: Hardcoded query strings +g.chain([n(query=f"type == 'User' and age > {min_age}")]) # SQL injection risk +``` + +### Memory Efficiency +```python +# Good: Filter early and use named results +result = g.chain([ + n({"active": True}, name="active_users"), # Filter first + e_forward({"recent": True}) +]) +# Only access what you need +active_users = result._nodes[result._nodes["active_users"]] + +# Avoid: Loading everything then filtering +all_nodes = g._nodes +active = all_nodes[all_nodes["active"] == True] # Loads entire graph +``` + +### GPU Best Practices +```python +# Check GPU memory before large operations +if engine == 'cudf': + import cudf + print(f"GPU memory: {cudf.cuda.cuda.get_memory_info()}") + +# Convert results back to pandas if needed for compatibility +result_pandas = result._nodes.to_pandas() if hasattr(result._nodes, 'to_pandas') else result._nodes +``` + +## See Also + +- {ref}`gfql-spec-language` - Core language specification +- [GFQL Quick Reference](../quick.rst) - Python API examples \ No newline at end of file diff --git a/docs/source/gfql/spec/wire_protocol.md b/docs/source/gfql/spec/wire_protocol.md new file mode 100644 index 0000000000..beacfdf20f --- /dev/null +++ b/docs/source/gfql/spec/wire_protocol.md @@ -0,0 +1,340 @@ +(gfql-spec-wire-protocol)= + +# GFQL Wire Protocol Specification + +## Introduction + +The GFQL Wire Protocol defines the JSON serialization format for GFQL queries, enabling: +- Client-server communication +- Query persistence and storage +- Cross-language interoperability between Python, JavaScript, and other clients +- Configuration-driven query generation + +### Design Principles +- **Type Safety**: Tagged dictionaries preserve type information +- **Self-Describing**: Each object includes type metadata +- **Extensible**: Schema supports future additions +- **Round-Trip Safe**: Lossless serialization/deserialization + +## Protocol Overview + +### Message Structure + +All GFQL wire protocol messages are JSON objects with a `type` field: + +```json +{ + "type": "MessageType", + ...additional fields... +} +``` + +### Supported Message Types +- `Chain`: Complete query chain +- `Node`: Node matcher operation +- `Edge`: Edge traversal operation +- Predicates: `GT`, `LT`, `EQ`, `IsIn`, `Between`, etc. +- Temporal values: `datetime`, `date`, `time` + +## Message Structure + +All GFQL wire protocol messages are JSON objects with a `type` field that identifies the message type. The protocol uses discriminated unions for polymorphic types. + +### Type Identification + +Each object includes a `type` field: +- Operations: `"Node"`, `"Edge"`, `"Chain"` +- Predicates: `"GT"`, `"LT"`, `"IsIn"`, etc. +- Temporal values: `"datetime"`, `"date"`, `"time"` + +This enables unambiguous deserialization and validation. + + +## Operation Serialization + +### Node Operation + +**Python**: +```python +n({"type": "person", "age": gt(30)}, name="adults") +``` + +**Wire Format**: +```json +{ + "type": "Node", + "filter_dict": { + "type": "person", + "age": { + "type": "GT", + "val": 30 + } + }, + "name": "adults" +} +``` + +### Edge Operation + +**Python**: +```python +e_forward( + {"type": "transaction"}, + hops=2, + source_node_match={"active": True}, + name="txns" +) +``` + +**Wire Format**: +```json +{ + "type": "Edge", + "direction": "forward", + "edge_match": { + "type": "transaction" + }, + "hops": 2, + "source_node_match": { + "active": true + }, + "name": "txns" +} +``` + +### Chain + +**Python**: +```python +chain([ + n({"id": "Alice"}), + e_forward({"type": "friend"}), + n({"status": "active"}) +]) +``` + +**Wire Format**: +```json +{ + "type": "Chain", + "chain": [ + { + "type": "Node", + "filter_dict": {"id": "Alice"} + }, + { + "type": "Edge", + "direction": "forward", + "edge_match": {"type": "friend"} + }, + { + "type": "Node", + "filter_dict": {"status": "active"} + } + ] +} +``` + +## Predicate Serialization + +### Comparison Predicates + +```json +{"type": "GT", "val": 100} +{"type": "LT", "val": 50.5} +{"type": "GE", "val": "2024-01-01"} +{"type": "LE", "val": true} +{"type": "EQ", "val": "active"} +{"type": "NE", "val": null} +``` + +### Between Predicate + +```json +{ + "type": "Between", + "lower": 10, + "upper": 20, + "inclusive": true +} +``` + +### IsIn Predicate + +```json +{ + "type": "IsIn", + "options": ["A", "B", "C"] +} +``` + +### String Predicates + +```json +{"type": "Contains", "pattern": "search"} +{"type": "Startswith", "pattern": "prefix"} +{"type": "Endswith", "pattern": "suffix"} +{"type": "Match", "pattern": "^[A-Z]+\\d+$"} +``` + +### Null Predicates + +```json +{"type": "IsNull"} +{"type": "NotNull"} +{"type": "IsNA"} +{"type": "NotNA"} +``` + +### Temporal Check Predicates + +```json +{"type": "IsMonthStart"} +{"type": "IsYearEnd"} +{"type": "IsLeapYear"} +``` + +## Type Serialization + +### Scalar Types + +```json +"hello world" // string +42 // integer +3.14159 // float +true // boolean +null // null +``` + +### Temporal Types + +#### DateTime +```json +{ + "type": "datetime", + "value": "2024-01-15T10:30:00", + "timezone": "America/New_York" // Optional, defaults to "UTC" +} +``` + +#### Date +```json +{ + "type": "date", + "value": "2024-01-15" +} +``` + +#### Time +```json +{ + "type": "time", + "value": "14:30:00.123456" +} +``` + +**Note**: The `timezone` field is optional for DateTime values and defaults to "UTC" if omitted. This ensures consistent behavior across systems while allowing explicit timezone specification when needed. + +## Examples + +### User 360 Query + +**Python**: +```python +g.chain([ + n({"customer_id": "C123"}), + e_forward({ + "type": "purchase", + "timestamp": gt(pd.Timestamp("2024-01-01")) + }) +]) +``` + +**Wire Format**: +```json +{ + "type": "Chain", + "chain": [ + { + "type": "Node", + "filter_dict": { + "customer_id": "C123" + } + }, + { + "type": "Edge", + "direction": "forward", + "edge_match": { + "type": "purchase", + "timestamp": { + "type": "GT", + "val": { + "type": "datetime", + "value": "2024-01-01T00:00:00", + "timezone": "UTC" + } + } + } + } + ] +} +``` + +### Cyber Security Pattern + +**Python**: +```python +g.chain([ + n({"ip": is_in(["192.168.1.100", "192.168.1.101"])}), + e_forward( + edge_query="port IN [22, 23, 3389]", + to_fixed_point=True + ), + n({"type": "server", "critical": True}) +]) +``` + +**Wire Format**: +```json +{ + "type": "Chain", + "chain": [ + { + "type": "Node", + "filter_dict": { + "ip": { + "type": "IsIn", + "options": ["192.168.1.100", "192.168.1.101"] + } + } + }, + { + "type": "Edge", + "direction": "forward", + "edge_query": "port IN [22, 23, 3389]", + "to_fixed_point": true + }, + { + "type": "Node", + "filter_dict": { + "type": "server", + "critical": true + } + } + ] +} +``` + + +## Best Practices + +1. **Always include type fields**: Every object must have a `type` +2. **Use ISO formats**: Dates and times in ISO 8601 +3. **Handle timezones consistently**: Include timezone for datetime values when precision matters (defaults to UTC) +4. **Validate before sending**: Use JSON Schema validation +5. **Handle unknown fields**: Ignore unrecognized fields for compatibility + +## See Also + +- {ref}`gfql-spec-language` - Language specification +- {ref}`gfql-spec-cypher-mapping` - Cypher to GFQL translation with wire protocol examples \ No newline at end of file