feat: Add comprehensive GFQL LLM specifications and validation framework

ai_code_notes/README.md

@@ -136,6 +136,35 @@ When adding a new guide:
 - **USER_TESTING_PLAYBOOK.md** [TODO]: AI-driven testing workflows
 - **Load when**: Starting new tasks, creating commits, fixing code quality issues
 
+## 📚 Documentation Building
+
+### Quick Commands
+```bash
+# Build HTML docs only (fastest)
+cd docs && ./html.sh
+
+# Full CI-like build with Docker
+cd docs && ./ci.sh
+
+# Build without notebook validation
+cd docs && VALIDATE_NOTEBOOK_EXECUTION=0 ./ci.sh
+
+# Build specific format
+cd docs && DOCS_FORMAT=html ./ci.sh
+```
+
+### Architecture
+- **Sphinx**: Main doc generator with MyST for Markdown support
+- **nbsphinx**: Converts notebooks to docs (execution disabled by default)
+- **Docker**: Consistent build environment using `sphinxdoc/sphinx:8.0.2`
+- **Validation**: Strict mode (`-W`), notebook structure checks, optional execution
+
+### Key Paths
+- `docs/source/`: Sphinx source files (.rst, .md)
+- `demos/`: Notebooks mounted to docs build
+- `docs/docker/build-docs.sh`: CI validation script
+- `docs/test_notebooks/`: Test-specific notebooks
+
 ## 🧪 Testing Quick Reference
 
 ### Docker Commands (Recommended)

ai_code_notes/gfql/README.md

@@ -0,0 +1,211 @@
+# GFQL AI Assistant Guide
+
+Guide for AI assistants working with GFQL (Graph Frame Query Language) in PyGraphistry.
+
+## 🎯 Quick Reference
+
+### Essential GFQL Operations
+```python
+# Node matching
+n()                                    # All nodes
+n({"type": "person"})                 # Filter by property
+n({"age": gt(30)})                    # With predicate
+n(name="result")                      # Named results
+
+# Edge traversal
+e_forward()                           # Forward direction
+e_reverse()                           # Reverse direction
+e() or e_undirected()                # Both directions
+e_forward(hops=2)                     # Multi-hop
+e_forward(to_fixed_point=True)        # All reachable
+
+# Chaining
+g.chain([n(), e_forward(), n()])      # Pattern matching
+```
+
+### Key Predicates
+- Comparison: `gt()`, `lt()`, `ge()`, `le()`, `eq()`, `ne()`
+- Membership: `is_in([...])`
+- Range: `between(lower, upper)`
+- String: `contains()`, `startswith()`, `endswith()`
+- Null: `is_null()`, `not_null()`
+- Temporal: `is_month_start()`, `is_year_end()`, etc.
+
+### Performance Tips
+- Filter early in the chain
+- Use specific hop counts vs `to_fixed_point`
+- Prefer `filter_dict` over `query` strings
+- Use appropriate engine: `pandas` (CPU) or `cudf` (GPU)
+
+## 📋 When to Use GFQL
+
+### Use GFQL When
+- Performing graph traversals or path queries
+- Finding patterns in connected data
+- Need efficient multi-hop operations
+- Working with node/edge dataframes
+
+### Use Pandas/Aggregations When
+- Need sorting (`sort_values()`)
+- Need limiting (`head()`, `tail()`)
+- Aggregating results (`groupby()`, `count()`)
+- Complex transformations
+
+## 🚀 Common Patterns
+
+### User 360 Query
+```python
+# Customer's recent activity
+g.chain([
+    n({"customer_id": "C123"}),
+    e_forward({
+        "type": is_in(["purchase", "view", "support"]),
+        "timestamp": gt(pd.Timestamp.now() - pd.Timedelta(days=30))
+    })
+])
+```
+
+### Cyber Security Pattern
+```python
+# Lateral movement detection
+g.chain([
+    n({"status": "compromised"}),
+    e_forward({"type": "login", "success": True}, hops=3),
+    n({"criticality": "high"}, name="at_risk")
+])
+```
+
+### Business Intelligence
+```python
+# Cross-sell opportunities
+g.chain([
+    n({"product_id": "P123"}),
+    e_reverse({"type": "purchased"}),
+    n({"type": "customer"}),
+    e_forward({"type": "purchased"}),
+    n({"product_id": ne("P123")}, name="also_bought")
+])
+```
+
+## 🔧 Code Style Guidelines
+
+### Preferred Style
+```python
+# ✅ Good - Clean, code-golfed chains
+g.chain([n({"type": "user"}), e({"active": True}), n()])
+
+# ❌ Avoid - Overly verbose
+result = g.chain([
+    n(filter_dict={"type": "user"}),
+    e_forward(edge_match={"active": True}, hops=1),
+    n(filter_dict={})
+])
+```
+
+### Naming Conventions
+- Use descriptive names for `name` parameters
+- Keep filter keys consistent with dataframe columns
+- Use snake_case for all identifiers
+
+## 🐛 Common Errors and Fixes
+
+### Schema Errors
+```python
+# ❌ Wrong - Column doesn't exist
+n({"username": "Alice"})
+
+# ✅ Fix - Use correct column name
+n({"name": "Alice"})
+```
+
+### Type Errors
+```python
+# ❌ Wrong - String predicate on number
+n({"age": contains("30")})
+
+# ✅ Fix - Use numeric predicate
+n({"age": gt(30)})
+```
+
+### Temporal Errors
+```python
+# ❌ Wrong - Raw string for datetime
+n({"created": gt("2024-01-01")})
+
+# ✅ Fix - Use proper datetime
+n({"created": gt(pd.Timestamp("2024-01-01"))})
+```
+
+## 📝 Natural Language to GFQL
+
+### Translation Patterns
+- "recent" → `gt(pd.Timestamp.now() - pd.Timedelta(days=N))`
+- "between X and Y" → `between(X, Y)`
+- "any of" → `is_in([...])`
+- "connected to" → `e()` or `e_undirected()`
+- "from X to Y" → X with `e_forward()` to Y
+- "within N hops" → `hops=N`
+
+### Example Translations
+
+**NL**: "Find all employees who report to managers in NYC"
+```python
+g.chain([
+    n({"type": "employee"}),
+    e_forward({"type": "reports_to"}),
+    n({"type": "manager", "office": "NYC"})
+])
+```
+
+**NL**: "Show me high-value customers from last week"
+```python
+g.chain([
+    n({"customer_tier": "high_value"}),
+    e_forward({
+        "type": "purchase",
+        "date": gt(pd.Timestamp.now() - pd.Timedelta(days=7))
+    })
+])
+```
+
+## 🔄 Cypher to GFQL
+
+### Basic Mappings
+| Cypher | GFQL |
+|--------|------|
+| `(n)` | `n()` |
+| `(n:Label)` | `n({"type": "Label"})` |
+| `-[r]->` | `e_forward()` |
+| `<-[r]-` | `e_reverse()` |
+| `-[r*2]-` | `e_forward(hops=2)` |
+| `WHERE n.prop = val` | `n({"prop": val})` |
+
+### Unsupported in GFQL
+- `OPTIONAL MATCH` - Handle nulls in post-processing
+- `WITH` clauses - Use intermediate chains
+- `ORDER BY/LIMIT` - Use pandas after
+- `CREATE/DELETE` - GFQL is read-only
+
+## 🧪 Validation Checklist
+
+Before generating GFQL:
+1. ✓ Check column names exist in schema
+2. ✓ Verify predicate types match column types
+3. ✓ Ensure temporal values use proper types
+4. ✓ Validate operation names (n, e_forward, etc.)
+5. ✓ Check chain structure is valid
+
+## 📚 Additional Resources
+
+- Full specifications in: `AI_PROGRESS/gfql_llm_specs/`
+  - `gfql_language_spec.md` - Complete language specification
+  - `gfql_wire_protocol_spec.md` - JSON wire format
+  - `cypher_to_gfql_mapping_spec.md` - Cypher translation
+
+## 🎯 Key Takeaways
+
+1. **GFQL is functional**: Chain operations, don't mutate
+2. **Filter early**: Put selective conditions first
+3. **Think patterns**: Focus on graph patterns, not procedures
+4. **Post-process**: Use pandas for sorting/aggregating
+5. **Code golf**: Keep queries concise and elegant

demos/demos_databases_apis/hypernetx/hypernetx.ipynb

@@ -6,22 +6,7 @@
     "colab_type": "text",
     "id": "bdxNPN0PhlsS"
    },
-   "source": [
-    "# HyperNetX + Graphistry = 💪💪💪\n",
-    "\n",
-    "You can quickly explore HyperNetX graphs using Graphistry through the below sample class.\n",
-    "\n",
-    "PNNL's [HyperNetX](https://github.com/pnnl/HyperNetX) is a new Python library for manipulating hypergraphs: graphs where an edge may connect any number of nodes. The below helper class converts HyperNetX graphs into 2-edge graphs (node/edge property tables as Panda dataframes ) in two different modes:\n",
-    "\n",
-    "* `hypernetx_to_graphistry_bipartite(hnx_graph)`: \n",
-    "    * Turn every hyperedge and hypernode into nodes. They form a bipartite graph: whenever a hyperedge includes a hypernode, create an edge from the hyperedge's node to the hypernode's node.\n",
-    "    * ex: Hyperedge `0: ['a', 'b', 'c']` => edge `(0, 'a')`, edge `(0, 'b')`, edge `(0, 'c')`\n",
-    "* `hypernetx_to_graphistry_nodes(hnx_graph)`:\n",
-    "    * Turn every hypernode into a node, and whenever two hypernodes share the same hyperedge, create an edge between their corresponding nodes\n",
-    "    * To emphasize that edges are undirect, the library sets the edge curvature to 0 (straight)\n",
-    "    * ex: Hyperedge `0: ['a', 'b', 'c']` => edge `('a', 'b')`, edge `('a', 'c')`, edge `('b', 'c')`\n",
-    "\n"
-   ]
+   "source": "# HyperNetX + Graphistry = [POWER]\n\nYou can quickly explore HyperNetX graphs using Graphistry through the below sample class.\n\nPNNL's [HyperNetX](https://github.com/pnnl/HyperNetX) is a new Python library for manipulating hypergraphs: graphs where an edge may connect any number of nodes. The below helper class converts HyperNetX graphs into 2-edge graphs (node/edge property tables as Panda dataframes ) in two different modes:\n\n* `hypernetx_to_graphistry_bipartite(hnx_graph)`: \n    * Turn every hyperedge and hypernode into nodes. They form a bipartite graph: whenever a hyperedge includes a hypernode, create an edge from the hyperedge's node to the hypernode's node.\n    * ex: Hyperedge `0: ['a', 'b', 'c']` => edge `(0, 'a')`, edge `(0, 'b')`, edge `(0, 'c')`\n* `hypernetx_to_graphistry_nodes(hnx_graph)`:\n    * Turn every hypernode into a node, and whenever two hypernodes share the same hyperedge, create an edge between their corresponding nodes\n    * To emphasize that edges are undirect, the library sets the edge curvature to 0 (straight)\n    * ex: Hyperedge `0: ['a', 'b', 'c']` => edge `('a', 'b')`, edge `('a', 'c')`, edge `('b', 'c')`"
   },
   {
    "cell_type": "markdown",
@@ -720,4 +705,4 @@
  },
  "nbformat": 4,
  "nbformat_minor": 1
-}
+}

demos/demos_databases_apis/memgraph/visualizing_iam_dataset.ipynb

@@ -289,7 +289,7 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "![Screenshot](https://github.com/karmenrabar/pygraphistry_images/blob/main/memgraphlab.png?raw=true)"
+    "![Memgraph Lab Schema](https://github.com/karmenrabar/pygraphistry_images/blob/main/memgraphlab.png?raw=true)"
    ]
   },
   {
@@ -339,15 +339,13 @@
   {
    "cell_type": "markdown",
    "metadata": {},
-   "source": [
-    "Screenshot:"
-   ]
+   "source": "Screenshot - All Access View:"
   },
   {
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "![Screenshot](https://github.com/karmenrabar/pygraphistry_images/blob/main/allaccess.png?raw=true)"
+    "![All Access View](https://github.com/karmenrabar/pygraphistry_images/blob/main/allaccess.png?raw=true)"
    ]
   },
   {
@@ -399,15 +397,13 @@
   {
    "cell_type": "markdown",
    "metadata": {},
-   "source": [
-    "Screenshot:\n"
-   ]
+   "source": "Screenshot - Carl's Direct Access:"
   },
   {
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "![Screenshot](https://github.com/karmenrabar/pygraphistry_images/blob/main/access3.png?raw=true)"
+    "![Carl Direct Access](https://github.com/karmenrabar/pygraphistry_images/blob/main/access3.png?raw=true)"
    ]
   },
   {
@@ -461,15 +457,13 @@
   {
    "cell_type": "markdown",
    "metadata": {},
-   "source": [
-    "Screenshot:"
-   ]
+   "source": "Screenshot - Carl's All File Access:"
   },
   {
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "![Screenshot](https://github.com/karmenrabar/pygraphistry_images/blob/main/access2.png?raw=true)"
+    "![Carl All File Access](https://github.com/karmenrabar/pygraphistry_images/blob/main/access2.png?raw=true)"
    ]
   },
   {
@@ -523,15 +517,13 @@
   {
    "cell_type": "markdown",
    "metadata": {},
-   "source": [
-    "Screenshot:"
-   ]
+   "source": "Screenshot - All Persons File Access:"
   },
   {
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "![Screenshot](https://github.com/karmenrabar/pygraphistry_images/blob/main/access.png?raw=true)"
+    "![All Persons File Access](https://github.com/karmenrabar/pygraphistry_images/blob/main/access.png?raw=true)"
    ]
   },
   {
@@ -578,4 +570,4 @@
  },
  "nbformat": 4,
  "nbformat_minor": 2
-}
+}