docs(gfql): Consolidate wire protocol documentation by lmeyerov · Pull Request #709 · graphistry/pygraphistry

lmeyerov · 2025-07-23T02:50:26Z

Summary

Reorganizes GFQL code into dedicated directory structure and updates import paths for better modularity.

Stack Position: #3 - Builds on PR #706-707's implementations and prepares for the rename in PR #713.

Stack Order

The complete GFQL stack (merge in this order):

PR feat(gfql): PR 1.2 - Basic Working DAG execution #706: Core AST classes (ASTLet, ASTRemoteGraph, ASTChainRef)
PR feat(gfql): PR 1.3 - Call Operations for safe method execution #707: ASTCall implementation with validation
PR docs(gfql): Consolidate wire protocol documentation #709: This PR - GFQL directory restructuring and import path updates
PR refactor: rename chain_dag to chain_let for semantic clarity #713: Rename chain_dag → chain_let for semantic consistency
PR docs(gfql): Comprehensive Let bindings and Call operations documentation #708: Comprehensive documentation (rebased onto refactor: rename chain_dag to chain_let for semantic clarity #713)

Directory Restructuring

New GFQL Module Structure

graphistry/compute/gfql/
├── __init__.py                 # Public API exports
├── validate/
│   ├── __init__.py            # Validation framework exports  
│   ├── validate.py            # Core validation logic
│   └── schema.py              # Schema validation utilities
├── call_executor.py           # Call execution logic
├── call_safelist.py           # Method safelist configuration
└── chain_let.py               # Let binding execution (renamed from chain_dag)

Import Path Updates

Updated all imports from graphistry.compute.validate → graphistry.compute.gfql.validate
Updated call executor imports to use new gfql paths
Updated test imports to match new structure
Maintained backward compatibility where possible

File Migrations

graphistry/compute/validate.py → graphistry/compute/gfql/validate/validate.py
graphistry/compute/call_executor.py → graphistry/compute/gfql/call_executor.py
graphistry/compute/call_safelist.py → graphistry/compute/gfql/call_safelist.py
Enhanced modular organization for better maintenance

API Updates

GFQL Module Exports

Clean public API through graphistry.compute.gfql module
Validation functions exported for external use
Schema classes available for type checking
Call execution utilities accessible

Test Updates

Updated test imports to new module structure
Fixed integration test paths
Updated README references for testing
Maintained test coverage across refactoring

Mypy Fixes

Fixed type annotation issues discovered during import restructuring
Added proper casts for JSON deserialization in ASTLet.from_json
Enhanced type safety across the GFQL module
Resolved circular import issues with new structure

Benefits

Modularity: Clear separation of GFQL functionality
Maintainability: Logical organization of related code
Extensibility: Easy to add new GFQL features
Testing: Cleaner test organization and imports

This restructuring prepares the codebase for the semantic rename in PR #713 and provides a clean foundation for future GFQL enhancements.

🤖 Generated with Claude Code

Co-Authored-By: Claude noreply@anthropic.com

- Keep chain_dag import reference in ComputeMixin.py (rename not applied yet) - Add missing AST imports in validate_schema.py after rebase - Ensure compatibility with master's validation framework 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com>

- Update import from .gfql to .gfql_unified in ComputeMixin.py - Update import path for call_safelist in validate_schema.py - Update import paths in chain.py and test files - Fix call_executor to not pass engine param to get_degrees These changes fix the lint errors in PR #709 caused by the directory restructuring where Call operations were moved to the gfql/ subdirectory.

- Add ASTCall node type for safe method execution - Create safelist framework with method allowlists - Implement parameter validation for exposed methods - Add call execution support in chain_dag - Include core graph methods (get_degrees, filter_by_dict, etc) 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com>

- Add 'call' alias for ASTCall to make the API more accessible - Export 'call' from compute module - Follows the pattern of other AST aliases (n, e, dag, ref, remote) This completes the user-friendly aliasing for all public AST classes. 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com>

- Expand safelist with more Plottable methods (transformations, visual encoding, metadata) - Add schema_effects metadata to track columns added/required by each method - Implement _validate_call_op in validate_schema.py - Hook Call validation into existing schema validation protocol - Add comprehensive tests for Call schema validation - Support dry-run validation with empty DataFrames Methods now track: - adds_node_cols: Columns added to nodes - adds_edge_cols: Columns added to edges - requires_node_cols: Node columns that must exist - requires_edge_cols: Edge columns that must exist 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com>

…tions - Add detailed docstrings to all functions in call_safelist.py - Add comprehensive docstrings to call_executor.py - Add docstrings to ASTCall class and its methods in ast.py - Add docstrings to _validate_call_op in validate_schema.py - Create comprehensive GPU test coverage in test_call_operations_gpu.py - Fix failing test that incorrectly used Call operations in chains - Fix all linting issues (W292, E501) The documentation now clearly explains: - Purpose and behavior of each function - Parameter types and validation rules - Return values and exceptions - Schema effects for static analysis 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com>

- Move call_executor.py and call_safelist.py to compute/gfql/ for better organization - Update all imports to use new location - All tests pass

…int error

- Move call_executor.py and call_safelist.py back to compute directory - Update all imports to use original paths - Remove empty gfql subdirectory that conflicted with gfql.py - Fix ASTLet alias and type annotations - Fix lint issues (newlines, comparison operators)

- Update test files to use ASTLet instead of deprecated ASTQueryDAG - Add missing Any import to ast.py for ASTCall type hints - Fixes import errors after rebasing onto PR #706 changes This completes the rebase integration with the ASTQueryDAG → ASTLet rename. 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com>

Add comprehensive validation framework for GFQL queries including syntax and schema validation. ## Python Code - `graphistry/compute/gfql/validate.py` - Core validation module with syntax and schema validators - `graphistry/compute/gfql/exceptions.py` - GFQLValidationError exception class - `graphistry/compute/chain_validate.py` - Chain function with validation support - `graphistry/validate/` - General validation utilities - Tests for all validation functionality ## Documentation - `docs/source/gfql/validation/` - Comprehensive validation guide - fundamentals.rst - Basic validation concepts and examples - advanced.rst - Complex query validation patterns - llm.rst - LLM integration patterns - production.rst - Production deployment patterns - API documentation for validation modules - Updated references in spec and main docs ## Notebook - `demos/gfql/gfql_validation_fundamentals.ipynb` - Interactive tutorial This provides a complete framework for validating GFQL queries at both syntax and schema levels, with helpful error messages to guide users. 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com>

- Added structured error system with E1xx-E3xx error codes - Updated all AST classes to use new validation pattern - Fixed ASTPredicate validate() method conflicts - Added schema validation to filter_by_dict - Created pre-execution validation capability - Updated all documentation and created migration guide Co-authored-by: Claude <noreply@anthropic.com>

- Remove trailing whitespace - Fix unused imports - Add newlines at end of files - Fix monkey-patch syntax for validate_schema - Format code with black Co-authored-by: Claude <noreply@anthropic.com>

- Fix E402 by moving imports before deprecation warning in validate.py - Add missing newlines at end of test files Co-authored-by: Claude <noreply@anthropic.com>

- Add gfql_remote() as the preferred remote execution method - Add gfql_remote_shape() for remote shape queries - Deprecate chain_remote() in favor of gfql_remote() - Deprecate chain_remote_shape() in favor of gfql_remote_shape() - Add deprecation warnings pointing users to new methods This provides a unified gfql() API for both local and remote execution, supporting both simple chains and complex DAG patterns with Let bindings. 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com>

- Move call_executor.py and call_safelist.py to compute/gfql/ for better organization - Update all imports to use new location - All tests pass

- Add JSON serialization format for ASTCall with function/params - Add examples for get_degrees, compute_cugraph, layout methods - Add comprehensive list of safelist methods by category - Add DAG pattern example with Call operations - Update supported message types list - Add Call-specific best practices 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com>

- Add comprehensive Call operations section with examples - Document all available Call methods by category - Add Call validation and error handling examples - Show integration with Let bindings - Update all .chain() references to .gfql() - Update validation examples to use GFQL terminology 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com>

…e spec - Add comprehensive Call operations section - Document safelist architecture and security model - List all available Call methods by category - Document parameter validation rules - Explain schema effects of Call operations - Add security considerations section - Update .chain() references to .gfql() 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com>

- Replace .chain() with .gfql() in all GFQL documentation - Replace chain_remote() with gfql_remote() - Replace chain_remote_shape() with gfql_remote_shape() - Replace chain_remote_python() with gfql_remote_python() - Update documentation in: - about.rst (12 instances) - combo.rst (1 instance) - datetime_filtering.md (13 instances) - remote.rst (13 instances) - wire_protocol_examples.md (2 instances) - validation/fundamentals.rst (9 instances) - validation/production.rst (4 instances) - predicates/quick.rst (4 instances)

- Created comprehensive tutorial for ASTCall/call operations - Demonstrates user-defined function invocation in queries - Shows data transformation, enrichment, and analysis pipelines - Includes security considerations and safelist concepts - Combines Call with Let bindings for complex workflows - Added to docs/source/notebooks/gfql.rst toctree 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com>

- Move validate_schema method to Chain class to avoid circular import - Fix Chain validation to properly raise GFQLTypeError/GFQLSyntaxError - Update _validate_fields to raise proper exceptions instead of asserts - Override validate() to collect all errors when collect_all=True - Update from_json to raise GFQLSyntaxError instead of asserts - Pass validate parameter through to ASTObject_from_json All tests now pass with proper error handling and no circular imports.

- Remove string quotes from List['ASTSerializable'] return type annotation - Add proper type casting in validate_schema.py for Chain handling

- Merge temporal predicate examples from wire_protocol_examples.md into spec/wire_protocol.md - Expand existing Temporal Types section with comprehensive examples - Add DateTime, Date, and Time comparison examples - Include timezone-aware DateTime handling - Add round-trip serialization examples - Include TypeScript interfaces for frontend developers - Remove redundant wire_protocol_examples.md file - Update gfql/index.rst to remove deleted file reference This consolidation provides a single source of truth for wire protocol documentation and improves discoverability by keeping all related content in the specification document. 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com>

- Keep chain_dag import reference in ComputeMixin.py (rename not applied yet) - Add missing AST imports in validate_schema.py after rebase - Ensure compatibility with master's validation framework 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com>

- Update import from .gfql to .gfql_unified in ComputeMixin.py - Update import path for call_safelist in validate_schema.py - Update import paths in chain.py and test files - Fix call_executor to not pass engine param to get_degrees These changes fix the lint errors in PR #709 caused by the directory restructuring where Call operations were moved to the gfql/ subdirectory.

Fix mypy type error where from_json returns Union type but ASTLet expects Dict[str, ASTObject]. Added explicit cast to satisfy type checker. 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com>

Updates PR #709 to be compatible with the ASTChainRef → ASTRef refactoring completed in PR #707. This ensures the PR stack remains consistent. Changes: - Rename ASTChainRef class to ASTRef in ast.py - Update JSON type from 'ChainRef' to 'Ref' - Update all references in chain_dag.py and validate_schema.py - Fix import paths for call_executor and call_safelist - Support both 'ChainRef' and 'Ref' JSON types for backward compatibility 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com>

- Update all test files to use ASTRef instead of ASTChainRef - Fix test assertion to expect 'Ref missing ref' error message - Update validate_schema.py imports and type annotations - All references now consistently use ASTRef for cleaner terminology 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com>

lmeyerov · 2025-07-28T07:42:22Z

Split and superseded by PR #715 (code changes) and PR #716 (documentation changes) for cleaner separation of concerns.

lmeyerov force-pushed the feature/gfql-pr1-3-call-operations branch 4 times, most recently from 73a9375 to 7025999 Compare July 24, 2025 11:20

lmeyerov force-pushed the feature/gfql-pr4-consolidate-wire-protocol branch from a9eb057 to 765fede Compare July 24, 2025 11:28

lmeyerov force-pushed the feature/gfql-pr1-3-call-operations branch from 7025999 to 60c8cd2 Compare July 25, 2025 00:03

lmeyerov force-pushed the feature/gfql-pr4-consolidate-wire-protocol branch from ce104b0 to 476ab46 Compare July 25, 2025 01:59

lmeyerov force-pushed the feature/gfql-pr4-consolidate-wire-protocol branch from daf65b9 to 09e729d Compare July 25, 2025 03:47

lmeyerov force-pushed the feature/gfql-pr4-consolidate-wire-protocol branch from 09e729d to ce1b8e5 Compare July 25, 2025 04:03

lmeyerov mentioned this pull request Jul 25, 2025

refactor: rename chain_dag to chain_let for semantic clarity #713

Closed

lmeyerov and others added 10 commits July 25, 2025 21:51

refactor: move Call operation files to compute/gfql/

e831abf

- Move call_executor.py and call_safelist.py to compute/gfql/ for better organization - Update all imports to use new location - All tests pass

fix: add missing newline to test_call_operations_gpu.py to fix W292 l…

59af1e0

…int error

fix(lint): add 'call' to __all__ exports after rebase

8eb8bee

fix: update validate_schema import paths in test files

f7bcbe1

lmeyerov force-pushed the feature/gfql-pr1-3-call-operations branch from fdd16e6 to f7bcbe1 Compare July 26, 2025 04:53

lmeyerov and others added 4 commits July 25, 2025 21:57

fix: clean up linting and type issues in GFQL validation

6764a1e

- Remove trailing whitespace - Fix unused imports - Add newlines at end of files - Fix monkey-patch syntax for validate_schema - Format code with black Co-authored-by: Claude <noreply@anthropic.com>

fix: resolve remaining linting issues

bf4c9a9

- Fix E402 by moving imports before deprecation warning in validate.py - Add missing newlines at end of test files Co-authored-by: Claude <noreply@anthropic.com>

lmeyerov and others added 16 commits July 25, 2025 21:57

fix: add type annotation to validate_schema.py to fix mypy error

ecb797f

refactor: move Call operation files to compute/gfql/

6aac130

- Move call_executor.py and call_safelist.py to compute/gfql/ for better organization - Update all imports to use new location - All tests pass

fix: apply Chain class type signature fixes from base branch

524e69b

fix: resolve type annotation and mypy issues in chain validation

0aaed95

- Remove string quotes from List['ASTSerializable'] return type annotation - Add proper type casting in validate_schema.py for Chain handling

chore: remove backup files

2b4049a

fix: remove duplicate Literal import causing F811 lint error

a4c95bc

lmeyerov force-pushed the feature/gfql-pr4-consolidate-wire-protocol branch from 63cf6a0 to 75efcf7 Compare July 26, 2025 04:57

This was referenced Jul 26, 2025

docs(gfql): Comprehensive Let bindings and Call operations documentation #708

Closed

feat(gfql): PR 1.2 - Basic Working DAG execution #706

Closed

feat(gfql): PR 1.3 - Call Operations for safe method execution #707

Closed

lmeyerov force-pushed the feature/gfql-pr1-3-call-operations branch from f7bcbe1 to 2a57411 Compare July 26, 2025 08:29

lmeyerov and others added 2 commits July 26, 2025 01:36

This was referenced Jul 28, 2025

feat(gfql): Consolidated GFQL core implementation - AST nodes, Let bindings, Call operations #715

Closed

docs(gfql): Comprehensive GFQL documentation updates #716

Closed

lmeyerov closed this Jul 28, 2025

This was referenced Jul 28, 2025

feat(gfql): Complete GFQL implementation with AST, Let, and Call operations #717

Closed

docs(gfql): Comprehensive GFQL documentation with Let, Call, and advanced patterns #718

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

docs(gfql): Consolidate wire protocol documentation#709

docs(gfql): Consolidate wire protocol documentation#709
lmeyerov wants to merge 100 commits into
feature/gfql-pr1-3-call-operationsfrom
feature/gfql-pr4-consolidate-wire-protocol

lmeyerov commented Jul 23, 2025 •

edited

Loading

Uh oh!

lmeyerov commented Jul 28, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Conversation

lmeyerov commented Jul 23, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Stack Order

Directory Restructuring

New GFQL Module Structure

Import Path Updates

File Migrations

API Updates

GFQL Module Exports

Test Updates

Mypy Fixes

Benefits

Uh oh!

lmeyerov commented Jul 28, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

lmeyerov commented Jul 23, 2025 •

edited

Loading