docs/development-plan.mdx at main · toobutta/docs

title	Development Plan
description	Comprehensive 90-day development, architecture, and implementation plan for MVP

Executive Summary

This document outlines the complete development plan for the Evoteli MVP, covering methodology, architecture, implementation phases, and success criteria for a 90-day delivery.

Target: Production-ready MVP with 10 pilot sites and 1,000+ analyzed parcels Team: 4.5 FTE (Platform, ML, Backend, Frontend, DevOps) Budget: $8,555 (3 months including hardware) Methodology: Agile with 2-week sprints

Development Methodology

Agile Framework

Sprint Duration: 2 weeks Total Sprints: 6 sprints (12 weeks) Sprint Structure:

Day 1: Sprint planning (4 hours)
Days 2-9: Development with daily standups (15 min)
Day 10: Sprint review & retrospective (2 hours)

Ceremonies:

Daily Standup: 9:00 AM, 15 minutes (What did you do? What will you do? Blockers?)
Sprint Planning: Define sprint goals, break down user stories, estimate effort
Sprint Review: Demo working software to stakeholders
Retrospective: What went well? What to improve? Action items

Definition of Done

A task is "Done" when:

✅ Code is written and reviewed
✅ Unit tests pass (>80% coverage)
✅ Integration tests pass
✅ Documentation is updated
✅ Deployed to staging environment
✅ Product owner approves

Architecture Design Principles

1. Modular & Decoupled

Why: Enable independent development and deployment of components

┌─────────────┐     ┌─────────────┐     ┌─────────────┐
│   Frontend  │────▶│  API Gateway│────▶│  Services   │
│   (React)   │     │  (FastAPI)  │     │  (Micro)    │
└─────────────┘     └─────────────┘     └─────────────┘
                            │
                            ▼
                    ┌─────────────┐
                    │  Databases  │
                    │  Queues     │
                    └─────────────┘

Implementation:

Each service has its own database (no shared DB)
Communication via REST APIs or message queues
Services can be deployed independently

2. API-First Design

Why: Enable multiple clients (web, mobile, partners) and parallel development

Process:

Define OpenAPI spec first
Generate client SDKs
Frontend and backend teams work in parallel
Mock servers for frontend development

3. Data-Driven Architecture

Why: Optimize for read-heavy workloads (1000:1 read:write ratio)

Strategy:

Write path: Edge → Kafka → Flink → ClickHouse (optimized for ingestion)
Read path: API → Redis cache → ClickHouse (optimized for queries)
Cache first: 95% cache hit rate target

4. Privacy by Design

Why: Legal compliance and user trust

Implementation:

Redaction at edge (before network transmission)
Aggregation by default (no individual tracking)
Retention policies enforced at database level (TTL)
Opt-out registry with immediate effect

5. Observability from Day 1

Why: Catch issues before users do

Stack:

Metrics: Prometheus (every component exports metrics)
Logs: OpenSearch (structured JSON logs)
Traces: Jaeger (distributed tracing across services)
Dashboards: Superset (business metrics)

System Architecture

High-Level Architecture

┌──────────────────────────────────────────────────────────────────┐
│                         USER LAYER                                │
│  ┌──────────────┐                              ┌──────────────┐  │
│  │  Web App     │                              │  Mobile App  │  │
│  │  (React)     │                              │  (Future)    │  │
│  └──────┬───────┘                              └──────┬───────┘  │
└─────────┼──────────────────────────────────────────────┼──────────┘
          │                                              │
          │              HTTPS (TLS 1.3)                 │
          ▼                                              ▼
┌──────────────────────────────────────────────────────────────────┐
│                      API GATEWAY LAYER                            │
│  ┌────────────────────────────────────────────────────────────┐  │
│  │  NGINX (Load Balancer)                                     │  │
│  │  - Rate limiting                                           │  │
│  │  - TLS termination                                         │  │
│  │  - Request routing                                         │  │
│  └────────────────────┬───────────────────────────────────────┘  │
└───────────────────────┼──────────────────────────────────────────┘
                        │
          ┌─────────────┼─────────────┐
          │             │             │
          ▼             ▼             ▼
┌─────────────┐ ┌─────────────┐ ┌─────────────┐
│   FastAPI   │ │   FastAPI   │ │   FastAPI   │
│  Instance 1 │ │  Instance 2 │ │  Instance 3 │
└──────┬──────┘ └──────┬──────┘ └──────┬──────┘
       │               │               │
       └───────────────┼───────────────┘
                       │
          ┌────────────┼────────────┐
          │            │            │
          ▼            ▼            ▼
┌─────────────┐ ┌─────────────┐ ┌─────────────┐
│   Valkey    │ │ ClickHouse  │ │   PostGIS   │
│   (Cache)   │ │(Time-Series)│ │  (Spatial)  │
└─────────────┘ └─────────────┘ └─────────────┘

Edge Processing Architecture

┌─────────────────────────────────────────────────────────────────┐
│                      EDGE DEVICE (Jetson Orin)                   │
│                                                                   │
│  ┌─────────────┐        ┌─────────────┐        ┌─────────────┐ │
│  │   Camera    │───────▶│  PP-YOLOE   │───────▶│ ByteTrack   │ │
│  │  (RTSP)     │        │  Detector   │        │  Tracker    │ │
│  └─────────────┘        └─────────────┘        └──────┬──────┘ │
│                                                        │         │
│                                                        ▼         │
│                                                 ┌─────────────┐ │
│                                                 │ DeepPrivacy │ │
│                                                 │  Redaction  │ │
│                                                 └──────┬──────┘ │
│                                                        │         │
│                                                        ▼         │
│                                                 ┌─────────────┐ │
│                                                 │    MQTT     │ │
│                                                 │  Publisher  │ │
│                                                 └──────┬──────┘ │
└────────────────────────────────────────────────────────┼────────┘
                                                         │
                                                         │ WiFi/LTE
                                                         ▼
┌─────────────────────────────────────────────────────────────────┐
│                       CLOUD INGESTION                            │
│  ┌─────────────┐        ┌─────────────┐        ┌─────────────┐ │
│  │    MQTT     │───────▶│    Kafka    │───────▶│    Flink    │ │
│  │   Broker    │        │   Topics    │        │  Processing │ │
│  └─────────────┘        └─────────────┘        └──────┬──────┘ │
│                                                        │         │
│                                                        ▼         │
│                                                 ┌─────────────┐ │
│                                                 │ ClickHouse  │ │
│                                                 │  Insertion  │ │
│                                                 └─────────────┘ │
└─────────────────────────────────────────────────────────────────┘

Batch Processing Architecture

┌─────────────────────────────────────────────────────────────────┐
│                    BATCH ORCHESTRATION (Airflow)                 │
│                                                                   │
│  ┌──────────────────────────────────────────────────────────┐   │
│  │                      DAG: weekly_imagery                  │   │
│  │                                                            │   │
│  │  1. fetch_sentinel2 ──▶ 2. segment_roofs ──▶             │   │
│  │                                                            │   │
│  │  3. analyze_solar ────▶ 4. compute_metrics ──▶           │   │
│  │                                                            │   │
│  │  5. update_postgis ───▶ 6. update_clickhouse             │   │
│  └──────────────────────────────────────────────────────────┘   │
└─────────────────────────────────────────────────────────────────┘
         │              │              │              │
         ▼              ▼              ▼              ▼
┌─────────────┐ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐
│  Sentinel2  │ │  PP-YOLOE   │ │   Earth     │ │  PostGIS    │
│    API      │ │ Segmentation│ │   Engine    │ │  Update     │
└─────────────┘ └─────────────┘ └─────────────┘ └─────────────┘

Data Architecture

Database Selection Matrix

Use Case	Database	Why
Time-series signals	ClickHouse	100x faster than Postgres for time-series, column-oriented compression
Geospatial data	PostGIS	Industry standard, spatial indexes, geometry operations
Cache	Valkey	Sub-millisecond latency, Redis-compatible
Object storage	SeaweedFS	S3-compatible, 10x faster for small files
Search/logs	OpenSearch	Full-text search, 75% faster than Loki

ClickHouse Schema Design

-- Primary time-series table
CREATE TABLE signals_timeseries (
    site_id LowCardinality(String),
    polygon_id Nullable(String),
    ts DateTime64(3, 'UTC'),
    metric LowCardinality(String),
    value Float64,
    unit LowCardinality(String),
    method Enum8(
        'edge_cv' = 1,
        'sat_change' = 2,
        'aerial_oblique' = 3,
        'earth_engine' = 4,
        'fused' = 5
    ),
    quality_score Float32,
    provenance String  -- JSON
) ENGINE = MergeTree()
PARTITION BY toYYYYMM(ts)
ORDER BY (site_id, metric, ts)
SETTINGS index_granularity = 8192;

-- 15-minute rollup (materialized view)
CREATE MATERIALIZED VIEW signals_15m_mv
ENGINE = AggregatingMergeTree()
PARTITION BY toYYYYMM(ts)
ORDER BY (site_id, metric, ts)
AS SELECT
    site_id,
    polygon_id,
    toStartOfFifteenMinutes(ts) AS ts,
    metric,
    avgState(value) AS value_avg,
    medianState(value) AS value_median,
    sumState(value) AS value_sum,
    maxState(value) AS value_max,
    avgState(quality_score) AS quality_avg,
    count() AS sample_count
FROM signals_timeseries
GROUP BY site_id, polygon_id, ts, metric;

-- Query 15m rollup
SELECT
    ts,
    metric,
    avgMerge(value_avg) AS avg_value,
    medianMerge(value_median) AS median_value
FROM signals_15m_mv
WHERE site_id = 'ATL-CTF-001'
  AND metric = 'occupancy'
  AND ts >= now() - INTERVAL 24 HOUR
GROUP BY ts, metric
ORDER BY ts;

PostGIS Schema Design

-- Sites table
CREATE TABLE sites (
    site_id VARCHAR(50) PRIMARY KEY,
    name VARCHAR(200) NOT NULL,
    brand VARCHAR(100),
    geometry GEOMETRY(Polygon, 4326) NOT NULL,
    address JSONB,
    metadata JSONB,
    created_at TIMESTAMPTZ DEFAULT NOW(),
    updated_at TIMESTAMPTZ DEFAULT NOW()
);

CREATE INDEX sites_geom_idx ON sites USING GIST (geometry);
CREATE INDEX sites_brand_idx ON sites (brand);

-- Parcels table
CREATE TABLE parcels (
    parcel_id VARCHAR(50) PRIMARY KEY,
    geometry GEOMETRY(Polygon, 4326) NOT NULL,
    area_sqft FLOAT,
    property_type VARCHAR(50),
    metadata JSONB,
    created_at TIMESTAMPTZ DEFAULT NOW()
);

CREATE INDEX parcels_geom_idx ON parcels USING GIST (geometry);
CREATE INDEX parcels_type_idx ON parcels (property_type);

-- Model inferences (roof segmentation, etc.)
CREATE TABLE model_inferences (
    inference_id SERIAL PRIMARY KEY,
    parcel_id VARCHAR(50) REFERENCES parcels(parcel_id),
    model_name VARCHAR(100) NOT NULL,
    model_version VARCHAR(50) NOT NULL,
    inference_date TIMESTAMPTZ NOT NULL,
    result JSONB NOT NULL,  -- Segmentation masks, bboxes, scores
    artifact_url TEXT,
    created_at TIMESTAMPTZ DEFAULT NOW()
);

CREATE INDEX model_inferences_parcel_idx
    ON model_inferences (parcel_id, inference_date DESC);

-- Territories (drawn by users)
CREATE TABLE territories (
    territory_id SERIAL PRIMARY KEY,
    user_id VARCHAR(50) NOT NULL,
    name VARCHAR(200) NOT NULL,
    geometry GEOMETRY(Polygon, 4326) NOT NULL,
    metadata JSONB,
    created_at TIMESTAMPTZ DEFAULT NOW()
);

CREATE INDEX territories_geom_idx ON territories USING GIST (geometry);
CREATE INDEX territories_user_idx ON territories (user_id);

API Design

RESTful Endpoints

# Signals API
POST   /v1/signals:query          # Query time-series metrics
GET    /v1/signals/metrics         # List available metrics
GET    /v1/signals/quality         # Quality audit report

# Sites API
GET    /v1/sites                   # List sites
POST   /v1/sites                   # Create site
GET    /v1/sites/{site_id}         # Get site details
PATCH  /v1/sites/{site_id}         # Update site
DELETE /v1/sites/{site_id}         # Delete site

# Parcels API
GET    /v1/parcels                 # List parcels (with bbox filter)
POST   /v1/parcels                 # Create parcel
GET    /v1/parcels/{parcel_id}     # Get parcel details
GET    /v1/parcels/{parcel_id}/analysis  # RoofIQ, SolarFit, etc.

# Alerts API
POST   /v1/alerts                  # Create alert
GET    /v1/alerts                  # List alerts
GET    /v1/alerts/{alert_id}       # Get alert
PATCH  /v1/alerts/{alert_id}       # Update alert
DELETE /v1/alerts/{alert_id}       # Delete alert

# Audiences API
POST   /v1/audiences/export        # Create export job
GET    /v1/audiences/export/{job_id}  # Check export status

# Earth Engine API
POST   /v1/earthengine/task        # Trigger analysis
GET    /v1/earthengine/task/{job_id}  # Check task status

# Territories API
POST   /v1/territories             # Create territory
GET    /v1/territories             # List territories
DELETE /v1/territories/{id}        # Delete territory

# Provenance API
GET    /v1/provenance/{metric_id}  # Get metric provenance
POST   /v1/provenance:batch        # Batch provenance lookup

OpenAPI Specification Structure

openapi: 3.0.3
info:
  title: Evoteli API
  version: 1.0.0
  description: |
    Decision-grade intelligence from computer vision, satellite imagery,
    and location data.

servers:
  - url: https://api.evoteli.com/v1
    description: Production
  - url: https://staging-api.evoteli.com/v1
    description: Staging

security:
  - OAuth2: [signals:read, signals:write]

components:
  securitySchemes:
    OAuth2:
      type: oauth2
      flows:
        clientCredentials:
          tokenUrl: https://auth.evoteli.com/oauth/token
          scopes:
            signals:read: Read time-series signals
            signals:write: Write custom signals
            alerts:read: Read alerts
            alerts:write: Create and manage alerts
            audiences:write: Export audiences

  schemas:
    Signal:
      type: object
      required: [ts, metric, value, unit, method, quality_score]
      properties:
        ts:
          type: string
          format: date-time
          example: "2025-11-06T14:30:00Z"
        metric:
          type: string
          example: "occupancy"
        value:
          type: number
          format: double
          example: 18.4
        unit:
          type: string
          example: "count"
        method:
          type: string
          enum: [edge_cv, sat_change, aerial_oblique, earth_engine, fused]
        quality_score:
          type: number
          format: float
          minimum: 0
          maximum: 1
          example: 0.82
        provenance:
          $ref: '#/components/schemas/Provenance'

paths:
  /signals:query:
    post:
      summary: Query time-series signals
      operationId: querySignals
      security:
        - OAuth2: [signals:read]
      requestBody:
        required: true
        content:
          application/json:
            schema:
              $ref: '#/components/schemas/SignalQuery'
      responses:
        '200':
          description: Successful query
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/SignalQueryResponse'

Implementation Phases

Sprint 1-2: Foundation (Weeks 1-4)

Goal: Infrastructure + basic API

Tasks:

Infrastructure Setup (Week 1)
- [ ] Provision cloud resources (AWS/GCP)
- [ ] Deploy ClickHouse cluster (3 nodes)
- [ ] Deploy PostGIS database
- [ ] Deploy Valkey cache
- [ ] Set up VPC and security groups
API Scaffolding (Week 1-2)
- [ ] Create FastAPI project structure
- [ ] Implement OAuth 2.0 with Keycloak
- [ ] Create OpenAPI spec
- [ ] Set up CI/CD pipeline (GitHub Actions)
Database Schemas (Week 2)
- [ ] Create ClickHouse tables and views
- [ ] Create PostGIS tables
- [ ] Write migration scripts
- [ ] Seed test data
Basic Endpoints (Week 3-4)
- [ ] Implement /v1/sites CRUD
- [ ] Implement /v1/signals:query
- [ ] Add Valkey caching layer
- [ ] Write integration tests

Deliverables:

✅ Working API with 2 endpoints
✅ Database schemas deployed
✅ CI/CD pipeline running
✅ Postman collection for testing

Sprint 3-4: Edge + Stream (Weeks 5-8)

Goal: Real-time camera ingestion

Tasks:

Edge Device Setup (Week 5)
- [ ] Flash Jetson Orin with Ubuntu 22.04
- [ ] Install PP-YOLOE model
- [ ] Implement ByteTrack tracking
- [ ] Test on sample video
Privacy Redaction (Week 5)
- [ ] Integrate DeepPrivacy2
- [ ] Benchmark redaction speed
- [ ] Validate blur accuracy (>95%)
MQTT Pipeline (Week 6)
- [ ] Deploy Eclipse Mosquitto broker
- [ ] Implement MQTT publisher on Jetson
- [ ] Create MQTT → Kafka bridge
- [ ] Test end-to-end latency (<10s)
Stream Processing (Week 7-8)
- [ ] Deploy Kafka cluster (3 brokers)
- [ ] Implement Flink job for aggregation
- [ ] Create 5m/15m/1h rollup logic
- [ ] Insert into ClickHouse

Deliverables:

✅ 1 Jetson device ingesting live camera
✅ Occupancy and queue_len metrics in ClickHouse
✅ End-to-end latency <10 seconds
✅ Redaction working (no faces/plates)

Sprint 5-6: Batch + Frontend (Weeks 9-12)

Goal: Satellite analysis + map interface

Tasks:

Satellite Imagery (Week 9)
- [ ] Integrate Sentinel-2 API
- [ ] Download sample tiles (100 parcels)
- [ ] Implement roof segmentation (SAM or U-Net)
- [ ] Store results in PostGIS
Solar Analysis (Week 9-10)
- [ ] Integrate Google Earth Engine
- [ ] Compute insolation for parcels
- [ ] Calculate solar_score
- [ ] Store in ClickHouse
Map Interface (Week 10-11)
- [ ] Create React app with Vite
- [ ] Integrate MapLibre GL
- [ ] Add Protomaps tiles
- [ ] Implement parcel layer
Advanced Features (Week 11-12)
- [ ] Add Mapbox GL Draw (territories)
- [ ] Implement advanced search (Cmd+K)
- [ ] Create property details modal
- [ ] Add demographics overlay (Census API)

Deliverables:

✅ 1,000 parcels analyzed (RoofIQ + SolarFit)
✅ Interactive map with territory drawing
✅ Advanced search working
✅ Property details modal

Testing Strategy

Unit Tests

Coverage Target: 80%

Framework: pytest (Python), Jest (TypeScript)

# Example: test_signals_query.py
import pytest
from app.services.signals import SignalService

def test_query_signals_with_rollup():
    service = SignalService()
    result = service.query(
        site_id="ATL-CTF-001",
        metrics=["occupancy"],
        time_from="2025-11-06T00:00:00Z",
        time_to="2025-11-06T01:00:00Z",
        rollup="15m"
    )

    assert len(result.rows) == 4  # 4 x 15-min windows in 1 hour
    assert result.rows[0].metric == "occupancy"
    assert 0 <= result.rows[0].quality_score <= 1

Integration Tests

Framework: pytest + Docker Compose

# Example: test_edge_to_clickhouse.py
import pytest
from mqtt_publisher import publish_detection
from clickhouse_client import query_signals

@pytest.mark.integration
def test_edge_to_clickhouse_pipeline():
    # Publish detection event
    publish_detection(
        site_id="TEST-001",
        detections=[{"class": "car", "confidence": 0.9}]
    )

    # Wait for pipeline (max 15 seconds)
    import time
    time.sleep(15)

    # Query ClickHouse
    result = query_signals(
        site_id="TEST-001",
        metric="occupancy",
        time_from="now() - 1 minute"
    )

    assert len(result) > 0
    assert result[0]["value"] == 1  # 1 car detected

End-to-End Tests

Framework: Playwright (frontend), pytest (API)

// Example: e2e/test_map_interface.spec.ts
import { test, expect } from '@playwright/test';

test('user can draw territory and save', async ({ page }) => {
  // Navigate to map
  await page.goto('http://localhost:3000');

  // Wait for map to load
  await page.waitForSelector('.maplibregl-canvas');

  // Click polygon draw button
  await page.click('[data-testid="draw-polygon-btn"]');

  // Draw polygon (click 4 points)
  const map = page.locator('.maplibregl-canvas');
  await map.click({ position: { x: 100, y: 100 } });
  await map.click({ position: { x: 200, y: 100 } });
  await map.click({ position: { x: 200, y: 200 } });
  await map.click({ position: { x: 100, y: 200 } });
  await map.click({ position: { x: 100, y: 100 } }); // Close

  // Save territory
  await page.fill('[data-testid="territory-name"]', 'Atlanta North');
  await page.click('[data-testid="save-territory-btn"]');

  // Verify saved
  await expect(page.locator('.toast-success')).toBeVisible();
});

Performance Requirements

API Performance

Metric	Target	Measurement
Query latency (cached)	p95 < 800ms	Prometheus histogram
Query latency (cold)	p95 < 2.5s	Prometheus histogram
Throughput	1,000 req/sec	Load test with k6
Error rate	< 0.1%	Prometheus counter

Edge Processing

Metric	Target	Measurement
Inference FPS	> 20 FPS	Jetson logs
Detection accuracy	> 85% AP	Manual validation
End-to-end latency	< 10 seconds	Kafka lag monitor

Data Quality

Metric	Target	Measurement
Average quality_score	≥ 0.75	ClickHouse query
Uptime	> 99%	Prometheus uptime
Data completeness	> 95%	Great Expectations

Success Criteria (90-Day Exit)

Technical

✅ 10 pilot sites ingesting camera streams
✅ Occupancy accuracy within ±8% vs. manual counts
✅ RoofIQ geometry error <5% (n≥50 parcels)
✅ API p95 latency <800ms (cached)
✅ Average quality_score ≥0.7
✅ 99% uptime over last 2 weeks
✅ Zero critical security findings

Business

✅ 2 pilot customers trained and using API
✅ 1 documented case study (queue management or solar leads)
✅ Pricing model validated ($2k-$5k/mo Starter tier)

Legal/Compliance

✅ 100% permissive licenses (no AGPL/GPL)
✅ Privacy redaction >95% accuracy
✅ Provenance on 100% of metrics
✅ DPIA template completed

Risk Mitigation

Risk	Likelihood	Impact	Mitigation
Jetson supply delays	Medium	High	Order devices Week 0; fallback to CPU
Model accuracy below target	Low	High	Use pre-trained COCO weights; fine-tune if needed
ClickHouse performance issues	Low	Medium	Start with 3-node cluster; use rollups
Satellite imagery license delays	Medium	Medium	Start with free Sentinel-2; upgrade to commercial later
Privacy redaction failures	Low	High	Test with 1000+ frames; achieve >95% accuracy

Next Steps

Review the full system architecture Detailed task breakdown with dependencies OpenAPI spec and endpoint documentation 100% permissively-licensed tech stack

FilesExpand file tree

development-plan.mdx

Latest commit

History

development-plan.mdx

File metadata and controls

Executive Summary

Development Methodology

Agile Framework

Definition of Done

Architecture Design Principles

1. Modular & Decoupled

2. API-First Design

3. Data-Driven Architecture

4. Privacy by Design

5. Observability from Day 1

System Architecture

High-Level Architecture

Edge Processing Architecture

Batch Processing Architecture

Data Architecture

Database Selection Matrix

ClickHouse Schema Design

PostGIS Schema Design

API Design

RESTful Endpoints

OpenAPI Specification Structure

Implementation Phases

Sprint 1-2: Foundation (Weeks 1-4)

Sprint 3-4: Edge + Stream (Weeks 5-8)

Sprint 5-6: Batch + Frontend (Weeks 9-12)

Testing Strategy

Unit Tests

Integration Tests

End-to-End Tests

Performance Requirements

API Performance

Edge Processing

Data Quality

Success Criteria (90-Day Exit)

Technical

Business

Legal/Compliance

Risk Mitigation

Next Steps