Engine runtime separation

_workdir/progress-2026-01-22-013.md

@@ -0,0 +1,111 @@
+---
+date: 2026-01-22
+session: engine-runtime-separation-spec
+objective: Create RFC-0102 and update implementation guide for engine-runtime separation
+status: completed
+---
+
+## Objective
+
+1. Create RFC-0102 as the authoritative specification for execution engine architecture
+2. Update `specs/8_bridge_distributed_impl.md` as implementation-specific guide
+
+## Completed
+
+### Part 1: Initial Specification Update
+1. **Reviewed existing specifications**
+   - Read `specs/7_local_engine_milestone.md` for current local engine architecture
+   - Read `specs/8_bridge_distributed_impl.md` for existing distributed bridge design
+   - Reviewed `specs/rfc-0010.md` for distributed execution model
+   - Reviewed `specs/rfc-namings.md` for naming conventions
+   - Reviewed `specs/rfc-0100.md` and `specs/rfc-0008.md` for RFC format
+
+### Part 2: Created RFC-0102
+2. **Created `specs/rfc-0102.md`** - Execution Engine Architecture:
+   - Standalone authoritative RFC (no references to intermediate files)
+   - Defines three-crate architecture: grism-engine, grism-local, grism-ray
+   - Specifies physical operator model and contracts
+   - Defines expression evaluation architecture
+   - Documents local execution model (pull-based pipeline)
+   - Documents distributed execution model (stage-based with Exchange)
+   - Covers failure/recovery semantics
+   - Includes API compatibility section
+   - References other RFCs appropriately
+
+### Part 3: Updated Implementation Guide  
+3. **Rewrote `specs/8_bridge_distributed_impl.md`** as implementation guide:
+   - References RFC-0102 for authoritative design
+   - Documents current code status and what works today
+   - Provides detailed 6-phase migration plan
+   - Includes concrete code examples and patterns
+   - Lists specific files to create, modify, and move
+   - Contains comprehensive implementation checklist
+
+### Part 4: Updated RFC Index
+4. **Updated `specs/rfc-index.md`**:
+   - Added RFC-0102 to API & Interfaces section
+   - Updated dependency graph
+   - Added RFC-0102 to "RFC by Layer" section
+
+## Files Changed
+
+| File | Description |
+|------|-------------|
+| `specs/rfc-0102.md` | NEW - Execution Engine Architecture RFC |
+| `specs/8_bridge_distributed_impl.md` | Rewritten as implementation guide |
+| `specs/rfc-index.md` | Updated to include RFC-0102 |
+
+## Tests
+
+```
+make test
+All 335 tests passed (90 grism-core, 97 grism-engine, 91 grism-logical, 46 grism-optimizer, 5 grism-storage, plus doc tests)
+```
+
+## Lint
+
+```
+make lint
+0 warnings, 0 errors - PASSED
+```
+
+## Notes
+
+### RFC-0102 Structure
+
+The new RFC covers:
+- **Section 4**: Crate architecture with responsibilities
+- **Section 5**: Common engine layer (operators, expressions, schemas)
+- **Section 6**: Local runtime (LocalExecutor, pull-based model)
+- **Section 7**: Ray runtime (ExchangeExec, stages, DistributedPlanner)
+- **Section 8**: Blocking operators in distributed context
+- **Section 9**: Graph operators in distributed context
+- **Section 10-11**: Failure recovery and explainability
+
+### Implementation Guide Updates
+
+The updated implementation guide now includes:
+- Current code status table (what works, what's partial, what's not started)
+- 6-phase migration plan with concrete tasks
+- Code snippets for new traits and types
+- File changes summary (new, modified, moved)
+- Implementation checklist for tracking progress
+
+### Key Guarantees
+
+From RFC-0102:
+1. Semantic Equivalence: Local and Ray execution produce identical results
+2. Operator Transparency: Operators unaware of execution context
+3. Explicit Distribution: All data movement visible in plans
+4. Hypergraph Correctness: Expand operators preserve adjacency semantics
+5. Explainability: Both runtimes support plan explanation and metrics
+
+## Next Steps
+
+Implementation phases as defined in `8_bridge_distributed_impl.md`:
+1. Phase 1: Extract common traits from grism-engine
+2. Phase 2: Create grism-local crate
+3. Phase 3: Rename grism-distributed to grism-ray
+4. Phase 4: Implement Exchange operator
+5. Phase 5: Implement stage splitting
+6. Phase 6: Implement DistributedPlanner