feat(gfql): Complete GFQL implementation with AST, Let, and Call operations

.github/workflows/ci.yml

@@ -389,11 +389,6 @@ jobs:
       run: |
         ./bin/build.sh
 
-    - name: Validate py.typed in wheel
-      run: |
-        unzip -l dist/graphistry*.whl | grep -q "graphistry/py.typed" || (echo "ERROR: py.typed marker missing from wheel - users won't get type information" && exit 1)
-        echo "✅ py.typed marker confirmed in wheel distribution"
-
 
   test-docs:
     needs: [changes, python-lint-types]

.gitignore

@@ -94,3 +94,4 @@ docs/source/demos
 # AI assistant working directories
 AI_PROGRESS/
 PLAN.md
+test_env/

CHANGELOG.md

@@ -7,14 +7,17 @@ This project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.htm
 
 ## Dev
 
-## [0.41.0 - 2025-07-26]
-
 ### Added
-* Typing: Add PEP 561 type distribution support (#714)
-  * Add py.typed marker file to enable type checking with mypy, pyright, and PyCharm
-  * Configure MANIFEST.in and setup.cfg to include py.typed in source and wheel distributions
-  * Add CI validation to prevent regressions where py.typed might be accidentally removed
-  * Enables accurate type information and autocompletion for PyGraphistry APIs
+* GFQL: Add comprehensive validation framework with detailed error reporting
+  * Built-in validation: `Chain()` constructor validates syntax automatically
+  * Schema validation: `validate_chain_schema()` validates queries against DataFrame schemas
+  * Pre-execution validation: `g.chain(ops, validate_schema=True)` catches errors before execution
+  * Structured error types: `GFQLValidationError`, `GFQLSyntaxError`, `GFQLTypeError`, `GFQLSchemaError`
+  * Error codes (E1xx syntax, E2xx type, E3xx schema) for programmatic error handling
+  * Collect-all mode: `validate(collect_all=True)` returns all errors instead of fail-fast
+  * JSON validation: `Chain.from_json()` validates during parsing for safe LLM integration
+  * Helpful error suggestions for common mistakes
+  * Example notebook: `demos/gfql/gfql_validation_fundamentals.ipynb`
 
 ### Fixed
 * Docs: Fix notebook validation error in hop_and_chain_graph_pattern_mining.ipynb by adding missing 'outputs' field to code cell

MANIFEST.in

@@ -1,3 +1,2 @@
 include versioneer.py
 include graphistry/_version.py
-include graphistry/py.typed

demos/gfql/gfql_remote.ipynb

@@ -1401,6 +1401,82 @@
    "metadata": {},
    "outputs": [],
    "source": []
+  },
+  {
+   "cell_type": "markdown",
+   "id": "fs1pabrqfaj",
+   "source": "## Combining Let Bindings with Call Operations\n\nLet bindings in GFQL allow you to create named intermediate results and compose complex operations. When combined with call operations in remote mode, you can orchestrate sophisticated graph analyses entirely on the server, minimizing data transfer and leveraging server-side GPU acceleration.",
+   "metadata": {}
+  },
+  {
+   "cell_type": "markdown",
+   "id": "bs7rghntlp",
+   "source": "### Example 1: PageRank Analysis with Filtering\n\nThis example demonstrates using let bindings to:\n1. Compute PageRank scores\n2. Filter high-value nodes\n3. Extract subgraphs around important nodes\n4. Return results for visualization",
+   "metadata": {}
+  },
+  {
+   "cell_type": "code",
+   "id": "wurwk0xplp",
+   "source": "# Create a more complex graph for demonstration\ncomplex_edges = pd.DataFrame({\n    's': ['a', 'b', 'c', 'd', 'e', 'f', 'a', 'b', 'c', 'd'],\n    'd': ['b', 'c', 'd', 'e', 'f', 'a', 'c', 'd', 'e', 'f'],\n    'weight': [1, 2, 1, 3, 1, 2, 1, 2, 1, 1],\n    'type': ['follow', 'mention', 'follow', 'follow', 'mention', 'follow', 'mention', 'follow', 'follow', 'mention']\n})\n\ng_complex = graphistry.edges(complex_edges, 's', 'd').upload()\nprint(f\"Uploaded graph with {len(complex_edges)} edges\")",
+   "metadata": {},
+   "execution_count": null,
+   "outputs": []
+  },
+  {
+   "cell_type": "code",
+   "id": "mwr3948llv",
+   "source": "%%time\n\n# Define a complex query using let bindings with call operations\npagerank_analysis_query = {\n    'let': {\n        # Step 1: Compute PageRank scores\n        'with_pagerank': {\n            'call': {\n                'method': 'compute_pagerank',\n                'args': [],\n                'kwargs': {}\n            }\n        },\n        \n        # Step 2: Filter nodes with high PageRank scores\n        'important_nodes': {\n            'pipe': [\n                {'var': 'with_pagerank'},\n                {\n                    'n': {\n                        'filter': {\n                            'gte': [\n                                {'col': 'pagerank'},\n                                0.15  # Threshold for \"important\" nodes\n                            ]\n                        }\n                    }\n                }\n            ]\n        },\n        \n        # Step 3: Get 1-hop neighborhoods of important nodes\n        'important_neighborhoods': {\n            'pipe': [\n                {'var': 'important_nodes'},\n                {'e_undirected': {'hops': 1}},\n                {'n': {}}\n            ]\n        }\n    },\n    \n    # Return the final result\n    'in': {'var': 'important_neighborhoods'}\n}\n\n# Execute the query remotely\nresult = g_complex.chain_remote([pagerank_analysis_query])\n\nprint(f\"Result has {len(result._nodes)} nodes and {len(result._edges)} edges\")\nprint(\"\\nNodes with PageRank scores:\")\nprint(result._nodes)",
+   "metadata": {},
+   "execution_count": null,
+   "outputs": []
+  },
+  {
+   "cell_type": "markdown",
+   "id": "jdzoowghv2q",
+   "source": "### Example 2: Multi-Stage Analysis with Different Edge Types\n\nThis example shows how to use let bindings to analyze different edge types separately and combine the results:",
+   "metadata": {}
+  },
+  {
+   "cell_type": "code",
+   "id": "40m2wl0yn3m",
+   "source": "%%time\n\n# Analyze different edge types and combine results\nedge_type_analysis = {\n    'let': {\n        # Analyze follow edges\n        'follow_network': {\n            'pipe': [\n                {\n                    'e_undirected': {\n                        'filter': {\n                            'eq': [\n                                {'col': 'type'},\n                                'follow'\n                            ]\n                        }\n                    }\n                },\n                {'n': {}}\n            ]\n        },\n        \n        # Compute centrality on follow network\n        'follow_centrality': {\n            'pipe': [\n                {'var': 'follow_network'},\n                {\n                    'call': {\n                        'method': 'compute_degree_centrality',\n                        'args': [],\n                        'kwargs': {}\n                    }\n                }\n            ]\n        },\n        \n        # Find mention patterns\n        'mention_edges': {\n            'e_undirected': {\n                'filter': {\n                    'eq': [\n                        {'col': 'type'},\n                        'mention'\n                    ]\n                }\n            }\n        },\n        \n        # Get nodes that are both highly connected and frequently mentioned\n        'influential_nodes': {\n            'pipe': [\n                {'var': 'follow_centrality'},\n                {\n                    'n': {\n                        'filter': {\n                            'gte': [\n                                {'col': 'degree_centrality'},\n                                0.5\n                            ]\n                        }\n                    }\n                },\n                {'var': 'mention_edges'},\n                {'n': {}}\n            ]\n        }\n    },\n    \n    'in': {'var': 'influential_nodes'}\n}\n\ninfluential_result = g_complex.chain_remote([edge_type_analysis])\n\nprint(f\"Found {len(influential_result._nodes)} influential nodes\")\nprint(f\"Connected by {len(influential_result._edges)} edges\")\nprint(\"\\nInfluential nodes with centrality scores:\")\nprint(influential_result._nodes)",
+   "metadata": {},
+   "execution_count": null,
+   "outputs": []
+  },
+  {
+   "cell_type": "markdown",
+   "id": "5y02h8p7mkk",
+   "source": "### Example 3: Conditional Analysis with Let Bindings\n\nThis example demonstrates using let bindings to perform conditional analysis based on graph properties:",
+   "metadata": {}
+  },
+  {
+   "cell_type": "code",
+   "id": "gmy30drvbts",
+   "source": "%%time\n\n# Complex analysis with multiple algorithms\ncomprehensive_analysis = {\n    'let': {\n        # Base graph with all computations\n        'enriched_graph': {\n            'pipe': [\n                {'call': {'method': 'compute_pagerank', 'args': [], 'kwargs': {}}},\n                {'call': {'method': 'compute_degree_centrality', 'args': [], 'kwargs': {}}},\n                # Add edge weight normalization\n                {\n                    'call': {\n                        'method': 'edges',\n                        'args': [],\n                        'kwargs': {\n                            'normalize': {\n                                'weight': {\n                                    'min': 0,\n                                    'max': 1\n                                }\n                            }\n                        }\n                    }\n                }\n            ]\n        },\n        \n        # Find bridge nodes (high betweenness)\n        'bridge_nodes': {\n            'pipe': [\n                {'var': 'enriched_graph'},\n                {\n                    'n': {\n                        'filter': {\n                            'and': [\n                                {'gte': [{'col': 'pagerank'}, 0.1]},\n                                {'lte': [{'col': 'degree_centrality'}, 0.7]}\n                            ]\n                        }\n                    }\n                }\n            ]\n        },\n        \n        # Find hub nodes (high degree centrality)\n        'hub_nodes': {\n            'pipe': [\n                {'var': 'enriched_graph'},\n                {\n                    'n': {\n                        'filter': {\n                            'gte': [{'col': 'degree_centrality'}, 0.7]\n                        }\n                    }\n                }\n            ]\n        },\n        \n        # Get connections between bridges and hubs\n        'critical_paths': {\n            'pipe': [\n                {'var': 'bridge_nodes'},\n                {'e_forward': {'to_nodes': {'var': 'hub_nodes'}}},\n                {'n': {}}\n            ]\n        }\n    },\n    \n    'in': {'var': 'critical_paths'}\n}\n\n# Execute remotely with GPU acceleration\ncritical_paths_result = g_complex.chain_remote([comprehensive_analysis], engine='cudf')\n\nprint(f\"Critical paths network: {len(critical_paths_result._nodes)} nodes, {len(critical_paths_result._edges)} edges\")\n\n# Check if we got results\nif len(critical_paths_result._nodes) > 0:\n    print(\"\\nCritical path nodes:\")\n    print(critical_paths_result._nodes)\nelse:\n    print(\"\\nNo critical paths found with current thresholds\")",
+   "metadata": {},
+   "execution_count": null,
+   "outputs": []
+  },
+  {
+   "cell_type": "markdown",
+   "id": "wbt02937wz",
+   "source": "### Example 4: Visualization-Ready Analysis\n\nThis example shows how to prepare data for visualization by enriching it with multiple metrics and creating a focused subgraph:",
+   "metadata": {}
+  },
+  {
+   "cell_type": "code",
+   "id": "4glzmgi1u3s",
+   "source": "%%time\n\n# Prepare visualization-ready data with all enrichments\nviz_prep_query = {\n    'let': {\n        # Compute all metrics\n        'with_metrics': {\n            'pipe': [\n                {'call': {'method': 'compute_pagerank', 'args': [], 'kwargs': {}}},\n                {'call': {'method': 'compute_degree_centrality', 'args': [], 'kwargs': {}}},\n                # Add node colors based on PageRank\n                {\n                    'call': {\n                        'method': 'nodes',\n                        'args': [],\n                        'kwargs': {\n                            'assign': {\n                                'node_color': {\n                                    'case': [\n                                        {\n                                            'when': {'gte': [{'col': 'pagerank'}, 0.2]},\n                                            'then': 65280  # Green for high PageRank\n                                        },\n                                        {\n                                            'when': {'gte': [{'col': 'pagerank'}, 0.15]},\n                                            'then': 16776960  # Yellow for medium\n                                        }\n                                    ],\n                                    'else': 16711680  # Red for low\n                                },\n                                'node_size': {\n                                    'mul': [\n                                        {'col': 'degree_centrality'},\n                                        50  # Scale factor\n                                    ]\n                                }\n                            }\n                        }\n                    }\n                }\n            ]\n        },\n        \n        # Add edge styling based on type and weight\n        'styled_graph': {\n            'pipe': [\n                {'var': 'with_metrics'},\n                {\n                    'call': {\n                        'method': 'edges',\n                        'args': [],\n                        'kwargs': {\n                            'assign': {\n                                'edge_color': {\n                                    'case': [\n                                        {\n                                            'when': {'eq': [{'col': 'type'}, 'follow']},\n                                            'then': 255  # Blue for follows\n                                        }\n                                    ],\n                                    'else': 16711935  # Magenta for mentions\n                                },\n                                'edge_weight': {\n                                    'col': 'weight'\n                                }\n                            }\n                        }\n                    }\n                }\n            ]\n        },\n        \n        # Focus on top nodes and their connections\n        'viz_subgraph': {\n            'pipe': [\n                {'var': 'styled_graph'},\n                {\n                    'n': {\n                        'filter': {\n                            'or': [\n                                {'gte': [{'col': 'pagerank'}, 0.15]},\n                                {'gte': [{'col': 'degree_centrality'}, 0.6]}\n                            ]\n                        }\n                    }\n                },\n                {'e_undirected': {'hops': 1}},\n                {'n': {}}\n            ]\n        }\n    },\n    \n    'in': {'var': 'viz_subgraph'}\n}\n\n# Get visualization-ready data\nviz_result = g_complex.chain_remote([viz_prep_query])\n\nprint(f\"Visualization subgraph: {len(viz_result._nodes)} nodes, {len(viz_result._edges)} edges\")\nprint(\"\\nNodes with visualization attributes:\")\nprint(viz_result._nodes)\nprint(\"\\nEdges with styling:\")\nprint(viz_result._edges)\n\n# Ready to visualize\n# viz_result.plot()  # Uncomment to create visualization",
+   "metadata": {},
+   "execution_count": null,
+   "outputs": []
+  },
+  {
+   "cell_type": "markdown",
+   "id": "p1o68dnaemk",
+   "source": "### Key Benefits of Let Bindings with Remote Calls\n\n1. **Server-Side Orchestration**: All operations happen on the server, minimizing data transfer\n2. **Named Intermediate Results**: Create readable, reusable steps in complex analyses\n3. **GPU Acceleration**: Leverage server GPU for compute-intensive operations like PageRank\n4. **Composability**: Build complex workflows from simple building blocks\n5. **Efficiency**: Avoid redundant computations by reusing named results\n\nWhen working with large graphs, this approach is particularly powerful as it allows you to:\n- Perform multiple analyses without downloading intermediate results\n- Chain together different algorithms and filters\n- Prepare visualization-ready data entirely on the server\n- Return only the final, filtered results you need",
+   "metadata": {}
   }
  ],
  "metadata": {
@@ -1424,4 +1500,4 @@
  },
  "nbformat": 4,
  "nbformat_minor": 5
-}
+}