Skip to content

MIGRATION.md

Latest commit

 

History

History
347 lines (255 loc) · 8.91 KB

MIGRATION.md

File metadata and controls

347 lines (255 loc) · 8.91 KB
Raw

Migration from Pathway to Native Architecture

Executive Summary

FlashPoint v2.0 represents a complete architecture migration from Pathway-based real-time processing to a native Python async stack. This migration was completed in March 2026 to achieve:

✅ Full control over the data pipeline
✅ Persistent storage with PostgreSQL + TimescaleDB
✅ Horizontal scalability with Celery workers
✅ Better debugging and error handling
✅ Flexibility to add new data sources easily

Status: ✅ COMPLETE (All 8 tasks finished)

What Changed

Before (Pathway Architecture)

Data Sources → Pathway Engine (port 8011) → FastAPI Proxy → Frontend
                    ↓
              In-Memory KNN Index

Problems:

  • Black-box processing pipeline
  • Limited debugging capabilities
  • Single-process bottleneck
  • In-memory buffer (lost on restart)
  • Port 8011 dependency
  • Hard to add new sources

After (Native Architecture)

Data Sources → Celery Workers → Processing → PostgreSQL/Qdrant/Redis → FastAPI → Frontend

Benefits:

  • Transparent pipeline with full control
  • Distributed workers for parallelism
  • Persistent storage (survives restarts)
  • Real-time Redis pub/sub + SSE
  • Easy to extend and debug

Migration Tasks

✅ Task 1: Infrastructure Setup

Completed: Docker Compose with 3 services

services:
  postgres:     # PostgreSQL 15 + TimescaleDB
  redis:        # Redis 7
  qdrant:       # Qdrant vector database

Scripts Created:

  • docker-compose.yml - Infrastructure definition
  • start.sh - One-command startup
  • stop.sh - Graceful shutdown

✅ Task 2: Celery Workers

Completed: 6 workers for data ingestion

Worker Schedule Purpose
rss_worker.py Every 5 min Fetch 18 RSS feeds
reddit_worker.py Every 1 min Poll Reddit JSON API
news_worker.py Every 10 min GNews API integration
telegram_worker.py Real-time Telethon streaming (25 channels)
conflict_worker.py Every 12 hours CFR conflict scraping
commodity_worker.py Every 3 hours Price fetching

Configuration: backend/config/celery_config.py

✅ Task 3: Processing Pipeline

Completed: backend/workers/tasks/processor.py

Pipeline Stages:

  1. Deduplication: SHA256 hash + Redis (24h TTL)
  2. Embedding: sentence-transformers (all-MiniLM-L6-v2, 384-dim)
  3. Storage: PostgreSQL (structured) + Qdrant (vectors)
  4. Broadcasting: Redis pub/sub → SSE clients

✅ Task 4: LangChain RAG

Completed: backend/services/rag_service.py

Replaced:

  • ❌ Pathway query service (port 8011)
  • ❌ Pathway RAG implementation

New Implementation:

  • ✅ LangChain RetrievalQA chain
  • ✅ Qdrant as retriever (top-10 docs, 0.5 threshold)
  • ✅ OpenRouter LLM (Llama 3.3 70B)
  • ✅ Custom geopolitical analysis prompt
  • ✅ Streaming support for real-time chat

✅ Task 5: API Modernization

Completed: Complete rewrite of backend/api.py

Old Endpoints (removed):

  • ❌ /v1/stream (POST) - Pathway event ingestion
  • ❌ /v1/feed/stream - Pathway-backed SSE

New Endpoints:

  • ✅ GET /api/events/recent - PostgreSQL query for initial load
  • ✅ GET /api/events/stream - Redis pub/sub SSE stream
  • ✅ POST /v1/chat - LangChain RAG with streaming
  • ✅ GET /v1/generate_report - SITREP from PostgreSQL
  • ✅ GET /api/commodities/latest - Cached prices
  • ✅ GET /api/conflicts/all - CFR data

✅ Task 6: Cleanup

Deleted Files:

backend/pipeline.py          # Pathway RAG engine
backend/data_registry.py     # Pathway data loading
backend/query_service.py     # Pathway query endpoint
backend/connectors/          # Old connector files
backend/rag_pipeline.py      # Obsolete
backend/stream_writer.py     # Obsolete

✅ Task 7: Backend Reorganization

New Structure:

backend/
├── api.py                    # FastAPI routes (cleaned)
├── main.py                   # Entry point
├── init_infra.py            # Database initialization
├── models/
│   ├── database.py          # SQLAlchemy models
│   └── redis_client.py      # Redis utilities
├── services/
│   ├── rag_service.py       # LangChain RAG ⭐
│   ├── report_service.py
│   ├── commodity_service.py
│   ├── conflict_service.py
│   └── geo_extractor.py
├── workers/tasks/
│   ├── rss_worker.py
│   ├── reddit_worker.py
│   ├── news_worker.py
│   ├── telegram_worker.py
│   ├── conflict_worker.py
│   ├── commodity_worker.py
│   └── processor.py
└── config/
    ├── celery_config.py
    └── auth_telegram.py

Import Updates: 9 files updated with new module paths

✅ Task 8: Frontend Modularization

Before: 592-line monolithic app.js

After: 7 modular ES6 files

frontend/web/
├── app.js (28 lines)         # Main entry point
└── js/
    ├── utils.js (81 lines)   # Shared utilities
    ├── feed.js (165 lines)   # SSE event stream
    ├── map.js (127 lines)    # Leaflet integration
    ├── chat.js (168 lines)   # RAG streaming chat
    ├── commodities.js (100)  # Price widget
    ├── conflicts.js (36)     # Conflict markers
    └── reports.js (111)      # SITREP generation

Benefits:

  • Proper separation of concerns
  • ES6 imports/exports (no global scope pollution)
  • Easy to test and maintain
  • Better code navigation

Breaking Changes

For Developers

  1. No Pathway dependency - Remove from requirements.txt
  2. New environment variables - See .env.example
  3. Database required - PostgreSQL + TimescaleDB
  4. Redis required - For caching and pub/sub
  5. Qdrant required - For vector storage

For Users

  1. New startup process - Use ./start.sh instead of multiple commands
  2. Port 8011 no longer used - Everything on port 8000
  3. Persistent data - Events survive restarts
  4. Faster initial load - PostgreSQL query instead of in-memory buffer

Performance Comparison

Metric Pathway (Before) Native (After)
Startup Time ~30 seconds ~10 seconds
Event Loss on Restart Yes (in-memory) No (PostgreSQL)
Parallelism Single process Multiple workers
Debugging Limited logs Full traceability
Scalability Vertical only Horizontal + Vertical
Source Addition Code changes JSON config

Migration Timeline

Total Duration: ~3 days
Completed: March 7, 2026

Day 1:

  • Infrastructure setup (Docker Compose)
  • Database models (SQLAlchemy)
  • Redis client utilities

Day 2:

  • 6 Celery workers implemented
  • Processing pipeline with embeddings
  • LangChain RAG service

Day 3:

  • API rewrite with SSE
  • Backend reorganization
  • Frontend modularization
  • Documentation

Rollback Plan (If Needed)

The Pathway-based code is preserved in git history:

# View old architecture
git log --all --grep="Pathway"

# Checkout old version (if needed)
git checkout <commit-hash>

Validation

All components tested and verified:

✅ Infrastructure - Docker services healthy
✅ Database - Tables created, hypertables configured
✅ Workers - All 6 workers registered in Celery
✅ RAG - LangChain + Qdrant querying works
✅ API - All endpoints responding correctly
✅ Frontend - ES6 modules loading, SSE connecting
✅ Integration - End-to-end data flow working

Post-Migration Checklist

  • All Pathway files deleted
  • Dependencies updated in requirements.txt
  • Environment variables documented
  • Docker Compose configured
  • Database initialization script created
  • Startup script tested
  • Documentation updated
  • Frontend modernized
  • Code reorganized
  • Import paths fixed

Lessons Learned

What Worked Well

  1. Incremental Migration - Built new system alongside old one
  2. Docker First - Infrastructure as code prevented config drift
  3. Modular Workers - Easy to test and debug individually
  4. ES6 Modules - Clean frontend without build complexity

What Could Be Improved

  1. Testing - Add unit tests for workers
  2. Monitoring - Prometheus metrics not yet implemented
  3. Documentation - Could use more API examples

Future Improvements

  • Add Prometheus exporters
  • Implement Grafana dashboards
  • Add unit tests for workers
  • Set up CI/CD pipeline
  • Add integration tests
  • Implement backup automation
  • Add load testing

Support

For migration issues:

  1. Check docs/ARCHITECTURE.md for system design
  2. Review docs/DEVELOPMENT.md for dev setup
  3. Open GitHub issue with logs

Migration completed successfully on March 7, 2026
All 8 tasks: ✅ COMPLETE