Skip to content

tech-stack.mdx

Latest commit

 

History

History
960 lines (738 loc) · 23.9 KB

tech-stack.mdx

File metadata and controls

960 lines (738 loc) · 23.9 KB
Raw
title Technology Stack
description Cost-optimized, performant open-source technology stack for the Evoteli

Overview

This document defines the production technology stack for the Evoteli MVP, optimized for:

  • Cost efficiency: 97%+ savings vs. commercial alternatives
  • Performance: Sub-10s latency, 10k+ events/sec throughput
  • Scalability: Start small, scale to enterprise
  • Open-source first: Minimize vendor lock-in

Total MVP Cost (3 months): ~$6,000 (vs. $220,000+ commercial)

Frontend: Map Interface

Primary Map Library: MapLibre GL JS (Open Source)

Why MapLibre:

  • ✅ Free & Open Source (BSD 3-Clause)
  • ✅ Google Maps alternative with identical API
  • ✅ Vector tiles support (smooth zoom, client-side styling)
  • ✅ 3D terrain & buildings support
  • ✅ Mobile-friendly (React Native support)

Installation:

npm install maplibre-gl

Basic Setup:

import maplibregl from 'maplibre-gl';
import 'maplibre-gl/dist/maplibre-gl.css';

const map = new maplibregl.Map({
  container: 'map',
  style: 'https://demotiles.maplibre.org/style.json', // Free tiles
  center: [-84.3880, 33.7490], // Atlanta
  zoom: 12
});

Cost: $0/month (self-hosted tiles)

Base Map Tiles: OpenStreetMap (OSM) via Protomaps

Option 1: Protomaps (Recommended)

  • Pre-built vector tiles from OSM data
  • Self-hosted (S3/GCS bucket)
  • Global coverage in single ~100GB file
  • Cost: $0.23/GB storage + $0.09/GB transfer = ~$50/month

Option 2: OpenMapTiles

  • Vector tiles from OSM
  • Self-hosted or cloud-hosted
  • Cost: $0 (self-hosted) or $49/month (cloud)

Option 3: Maptiler (Commercial fallback)

  • 100,000 free map loads/month
  • $99/month for 500k loads
  • Use case: MVP testing before self-hosting

Overlay Layers

Parcel Boundaries: PostGIS + GeoJSON

map.addSource('parcels', {
  type: 'geojson',
  data: '/api/parcels?bbox=-84.4,-84.3,33.7,33.8'
});

map.addLayer({
  id: 'parcels-fill',
  type: 'fill',
  source: 'parcels',
  paint: {
    'fill-color': '#088',
    'fill-opacity': 0.4
  }
});

Heat Maps: Deck.gl

import { HeatmapLayer } from '@deck.gl/aggregation-layers';

const layer = new HeatmapLayer({
  id: 'heatmap',
  data: occupancyData,
  getPosition: d => [d.lon, d.lat],
  getWeight: d => d.occupancy,
  radiusPixels: 30
});

Territory Mapping Tool

Draw & Edit Polygons: Mapbox GL Draw (Open Source)

npm install @mapbox/mapbox-gl-draw
import MapboxDraw from '@mapbox/mapbox-gl-draw';

const draw = new MapboxDraw({
  displayControlsDefault: false,
  controls: {
    polygon: true,
    trash: true
  }
});

map.addControl(draw);

// Save drawn territories
map.on('draw.create', (e) => {
  const territory = e.features[0];
  fetch('/api/territories', {
    method: 'POST',
    body: JSON.stringify(territory)
  });
});

Cost: $0 (MIT license)

Frontend: UI Framework

React + TypeScript + Vite

Why React:

  • ✅ Largest ecosystem for map/geo libraries
  • ✅ Component reusability
  • ✅ MapLibre GL has official React wrapper
  • ✅ Deck.gl has React bindings

Alternative: Svelte (lighter, faster, but smaller ecosystem)

Setup:

npm create vite@latest geointel-ui -- --template react-ts
cd geointel-ui
npm install maplibre-gl @mapbox/mapbox-gl-draw deck.gl

UI Component Library: shadcn/ui (Open Source)

Why shadcn/ui:

  • ✅ Free (copy-paste components, no bundle bloat)
  • ✅ Radix UI primitives (accessible, unstyled)
  • ✅ Tailwind CSS styling
  • ✅ Customizable (own your code)

Components Needed:

  • Command - Advanced search (Cmd+K style)
  • Dialog - Property details modal
  • DropdownMenu - Filters and actions
  • Table - Results list
  • Tabs - Product switcher (LotWatch/HomeScope)

Cost: $0

State Management: Zustand (Lightweight)

npm install zustand
import create from 'zustand';

interface MapState {
  selectedParcels: string[];
  filters: Record<string, any>;
  addParcel: (id: string) => void;
}

const useMapStore = create<MapState>((set) => ({
  selectedParcels: [],
  filters: {},
  addParcel: (id) => set((state) => ({
    selectedParcels: [...state.selectedParcels, id]
  }))
}));

Alternative: Jotai (atom-based, more granular)

Backend: API Gateway & Services

FastAPI (Python 3.11+)

Why FastAPI:

  • ✅ Fastest Python framework (async/await native)
  • ✅ Auto-generated OpenAPI docs
  • ✅ Type safety with Pydantic
  • ✅ Easy ML model integration (PyTorch, ONNX)

Setup:

pip install fastapi[all] uvicorn[standard]

Example Endpoint:

from fastapi import FastAPI
from pydantic import BaseModel

app = FastAPI()

class SignalQuery(BaseModel):
    site_id: str
    metrics: list[str]
    time_from: str
    time_to: str

@app.post("/v1/signals:query")
async def query_signals(query: SignalQuery):
    # Query ClickHouse
    results = await db.query(
        "SELECT * FROM signals WHERE site_id = ?",
        query.site_id
    )
    return {"rows": results}

Performance: 10k+ req/sec (with Uvicorn)

Alternative: Go with Gin framework (faster, but less ML integration)

Database Layer

Time-Series: ClickHouse (Open Source)

Why ClickHouse:

  • ✅ 100x faster than PostgreSQL for time-series
  • ✅ Column-oriented (compress to 10% of row-store)
  • ✅ Real-time aggregations (sub-second on billions of rows)
  • ✅ Free (Apache 2.0)

Deployment:

docker run -d \
  --name clickhouse \
  -p 8123:8123 -p 9000:9000 \
  -v clickhouse-data:/var/lib/clickhouse \
  clickhouse/clickhouse-server

Schema:

CREATE TABLE signals (
  site_id String,
  ts DateTime64(3),
  metric LowCardinality(String),
  value Float64,
  quality_score Float32
) ENGINE = MergeTree()
PARTITION BY toYYYYMM(ts)
ORDER BY (site_id, metric, ts);

Cost (Cloud):

  • ClickHouse Cloud: $0.36/hour (~$260/month for 1TB RAM node)
  • Self-hosted (AWS): r6i.2xlarge = $0.504/hour (~$365/month)

Alternative: TimescaleDB (PostgreSQL extension, easier but slower)

Geospatial: PostGIS (PostgreSQL + PostGIS Extension)

Why PostGIS:

  • ✅ Industry standard for geo queries
  • ✅ Spatial indexes (GiST, BRIN)
  • ✅ Geometry operations (intersect, buffer, union)
  • ✅ Free (PostgreSQL license)

Deployment:

docker run -d \
  --name postgis \
  -e POSTGRES_PASSWORD=mysecret \
  -p 5432:5432 \
  postgis/postgis:15-3.3

Example Query:

-- Find parcels within 5 miles of site
SELECT p.parcel_id, p.address
FROM parcels p
WHERE ST_DWithin(
  p.geometry,
  ST_SetSRID(ST_MakePoint(-84.3880, 33.7490), 4326)::geography,
  8046.72  -- 5 miles in meters
);

Cost (Cloud):

  • AWS RDS: db.t3.medium = $0.068/hour (~$50/month)
  • Supabase (managed): Free tier (500MB) or $25/month (8GB)

Cache: Redis (In-Memory)

Why Redis:

  • ✅ Sub-millisecond latency
  • ✅ TTL support (auto-expire cached queries)
  • ✅ Pub/Sub for real-time updates
  • ✅ Free (BSD license)

Deployment:

docker run -d \
  --name redis \
  -p 6379:6379 \
  redis:7-alpine

Usage:

import redis

r = redis.Redis(host='localhost', port=6379)

# Cache signal query (5-minute TTL)
cache_key = f"signals:{site_id}:{metric}:{time_range}"
cached = r.get(cache_key)

if cached:
    return json.loads(cached)
else:
    result = query_database()
    r.setex(cache_key, 300, json.dumps(result))
    return result

Cost (Cloud):

  • Upstash: Free tier (10k requests/day) or $0.2/100k requests
  • AWS ElastiCache: cache.t3.micro = $0.017/hour (~$12/month)

Object Storage: MinIO (S3-Compatible)

Why MinIO:

  • ✅ S3-compatible API (drop-in replacement)
  • ✅ Self-hosted (no egress fees)
  • ✅ Free (AGPL license, Apache for enterprise)

Deployment:

docker run -d \
  --name minio \
  -p 9000:9000 -p 9001:9001 \
  -v minio-data:/data \
  minio/minio server /data --console-address ":9001"

Alternative: Use AWS S3 ($0.023/GB + $0.09/GB egress) or Cloudflare R2 ($0.015/GB, no egress)

Cost (Self-hosted): $0 + storage hardware

Computer Vision: Edge Processing

Detection Model: YOLO11 (Ultralytics)

Why YOLO11:

  • ✅ AGPL-3.0 license (free for open-source projects)
  • ✅ State-of-art accuracy (90%+ mAP on COCO)
  • ✅ Fast inference (50+ FPS on Jetson Orin)
  • ✅ Pre-trained models (vehicle detection ready)

Installation:

pip install ultralytics

Inference:

from ultralytics import YOLO

model = YOLO('yolo11n.pt')  # Nano model
results = model('frame.jpg')

vehicles = [r for r in results[0].boxes if r.cls in [2, 3, 5, 7]]  # car, motorcycle, bus, truck
print(f"Detected {len(vehicles)} vehicles")

Alternative: YOLOv8 (slightly older, same license)

Tracking: ByteTrack (MIT License)

pip install bytetrack
from bytetrack import BYTETracker

tracker = BYTETracker(track_thresh=0.5)
tracks = tracker.update(detections)

Privacy Redaction: DeepPrivacy2 (MIT License)

Face/Plate Anonymization:

pip install deep-privacy
from deep_privacy import DeepPrivacy

anonymizer = DeepPrivacy()
anonymized_frame = anonymizer.anonymize(frame)

Alternative: Simple blur with OpenCV (faster, less accurate)

Edge Hardware: NVIDIA Jetson Orin Nano ($499)

Specs:

  • 6-core ARM CPU
  • 1024 CUDA cores
  • 8GB RAM
  • 40 TOPS AI performance

Alternative: Google Coral TPU Dev Board ($150, but only 4 TOPS)

Software Stack:

# JetPack 6.0 (Ubuntu 22.04 + CUDA 12)
sudo apt install nvidia-jetpack
pip3 install ultralytics opencv-python paho-mqtt

Satellite & Aerial Imagery

Free/Open-Source Options

1. Sentinel-2 (ESA - Free)

  • Resolution: 10-20m
  • Cadence: 5-10 days
  • Coverage: Global
  • API: Copernicus Data Space (free, requires registration)
  • Cost: $0
from sentinelsat import SentinelAPI

api = SentinelAPI('username', 'password')
products = api.query(
    area='POLYGON((-84.4 33.7, ...))',
    date=('20250101', '20250131'),
    platformname='Sentinel-2',
    cloudcoverpercentage=(0, 10)
)

2. Google Earth Engine (Free for Research/Non-Commercial)

  • Resolution: Varies (Landsat: 30m, Sentinel: 10m)
  • Cadence: Daily composites available
  • Cost: $0 (non-commercial) or $0.01-$0.10/image (commercial)
import ee

ee.Initialize()

# Get insolation data
insolation = ee.Image('NASA/NREL/NSRDB/v1').select('clearsky_ghi')
mean_insolation = insolation.reduceRegion(
    reducer=ee.Reducer.mean(),
    geometry=parcel_geometry,
    scale=1000
).getInfo()

3. NASA APIs (Free)

  • Earth API: Landsat 8 imagery
  • POWER API: Solar radiation data
  • GIBS: Global Imagery Browse Services
import requests

# NASA Earth API
url = f"https://api.nasa.gov/planetary/earth/imagery"
params = {
    'lon': -84.3880,
    'lat': 33.7490,
    'date': '2025-01-01',
    'api_key': 'DEMO_KEY'
}
response = requests.get(url, params=params)

Cost: $0 (rate limited to 1000 requests/hour)

Commercial Fallback (If Needed)

Planet Labs:

  • Resolution: 3-5m
  • Cadence: Daily
  • Cost: $10,000/year (education) or $30,000+/year (commercial)

Maxar/Nearmap:

  • Resolution: 30-50cm
  • Cost: $50,000+/year

Recommendation: Start with Sentinel-2 + Earth Engine (free), upgrade to Planet only if needed.

Batch Processing & Orchestration

Workflow: Apache Airflow (Free)

Why Airflow:

  • ✅ Industry standard for data pipelines
  • ✅ DAG-based (visualize dependencies)
  • ✅ Extensive integrations (ClickHouse, PostGIS, APIs)
  • ✅ Free (Apache 2.0)

Setup:

pip install apache-airflow
airflow standalone

Example DAG:

from airflow import DAG
from airflow.operators.python import PythonOperator

dag = DAG('homescope_weekly', schedule='@weekly')

def fetch_satellite():
    # Download Sentinel-2 tiles
    pass

def analyze_roofs():
    # Run SAM segmentation
    pass

fetch = PythonOperator(task_id='fetch', python_callable=fetch_satellite, dag=dag)
analyze = PythonOperator(task_id='analyze', python_callable=analyze_roofs, dag=dag)

fetch >> analyze

Alternative: Dagster (modern, better UI, but less mature)

Cost: $0 (self-hosted) or $100/month (Astronomer Cloud)

Stream Processing

Messaging: Apache Kafka (Free)

Why Kafka:

  • ✅ High throughput (millions of events/sec)
  • ✅ Durable (disk-backed, replicated)
  • ✅ Industry standard
  • ✅ Free (Apache 2.0)

Setup (Docker Compose):

services:
  kafka:
    image: confluentinc/cp-kafka:7.5.0
    ports:
      - "9092:9092"
    environment:
      KAFKA_BROKER_ID: 1
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181

Alternative: RedPanda (Kafka-compatible, C++, 10x faster, easier to manage)

docker run -d --name redpanda \
  -p 9092:9092 \
  vectorized/redpanda:latest \
  redpanda start --smp 1

Cost (Cloud):

  • Confluent Cloud: $1/hour (~$730/month)
  • AWS MSK: $0.21/hour (~$150/month)
  • Self-hosted: $0 + compute

Stream Processing: Apache Flink (Free)

Why Flink:

  • ✅ True streaming (not micro-batch like Spark)
  • ✅ Low latency (<100ms)
  • ✅ Stateful processing (windows, joins)
  • ✅ Free (Apache 2.0)

Alternative: Apache Beam (unified batch + stream, runs on Flink/Spark/Dataflow)

Recommendation: Use Beam with FlinkRunner (flexibility + performance)

import apache_beam as beam

with beam.Pipeline() as p:
    (p
     | beam.io.ReadFromKafka(...)
     | beam.WindowInto(beam.window.FixedWindows(60))  # 1-min windows
     | beam.CombinePerKey(beam.combiners.MeanCombineFn())
     | beam.io.WriteToClickHouse(...))

Cost: $0 (self-hosted)

Model Serving

ONNX Runtime (Free)

Why ONNX:

  • ✅ Framework agnostic (PyTorch, TF, etc. → ONNX)
  • ✅ Optimized inference (2-5x faster than native)
  • ✅ Cross-platform (CPU, GPU, edge)
  • ✅ Free (MIT license)

Convert YOLO to ONNX:

from ultralytics import YOLO

model = YOLO('yolo11n.pt')
model.export(format='onnx', opset=14)

Inference:

import onnxruntime as ort

session = ort.InferenceSession('yolo11n.onnx')
outputs = session.run(None, {'images': input_tensor})

Alternative: NVIDIA Triton Inference Server (more features, overkill for MVP)

Authentication

Keycloak (Open Source)

Why Keycloak:

  • ✅ OAuth 2.0 / OIDC standard
  • ✅ SSO support (Google, GitHub, SAML)
  • ✅ Multi-tenant
  • ✅ Free (Apache 2.0)

Setup:

docker run -d \
  --name keycloak \
  -p 8080:8080 \
  -e KEYCLOAK_ADMIN=admin \
  -e KEYCLOAK_ADMIN_PASSWORD=admin \
  quay.io/keycloak/keycloak:23.0 start-dev

Alternative: Auth0 (free tier: 7k MAUs) or Supabase Auth (free tier: 50k MAUs)

Cost: $0 (self-hosted) or $0 (Auth0/Supabase free tier)

Observability

Metrics: Prometheus + Grafana (Free)

Prometheus:

docker run -d \
  --name prometheus \
  -p 9090:9090 \
  -v prometheus.yml:/etc/prometheus/prometheus.yml \
  prom/prometheus

Grafana:

docker run -d \
  --name grafana \
  -p 3000:3000 \
  grafana/grafana

Cost: $0 (self-hosted) or $50/month (Grafana Cloud)

Tracing: Jaeger (Free)

docker run -d \
  --name jaeger \
  -p 16686:16686 \
  jaegertracing/all-in-one

Logging: Loki + Promtail (Free)

Loki is like Prometheus but for logs (from Grafana Labs).

Cost: $0

Demographics & Market Data APIs (Free)

U.S. Census Bureau API (Free, No Limits)

import requests

url = "https://api.census.gov/data/2021/acs/acs5"
params = {
    'get': 'B01001_001E,B19013_001E',  # Population, Median Income
    'for': 'tract:*',
    'in': 'state:13 county:121'  # Fulton County, GA
}
response = requests.get(url, params=params).json()

Data Available:

  • Population, age, race
  • Income, poverty
  • Housing units, occupancy
  • Education, employment

Cost: $0, unlimited

OpenStreetMap Overpass API (Free)

import requests

query = """
[out:json];
area["name"="Atlanta"]->.a;
(
  node(area.a)["amenity"="restaurant"];
  way(area.a)["amenity"="restaurant"];
);
out center;
"""

url = "https://overpass-api.de/api/interpreter"
response = requests.post(url, data={'data': query})
restaurants = response.json()

Data: POIs (restaurants, schools, parks, etc.)

Open-Meteo Weather API (Free)

import requests

url = "https://api.open-meteo.com/v1/forecast"
params = {
    'latitude': 33.7490,
    'longitude': -84.3880,
    'hourly': 'temperature_2m,precipitation',
    'timezone': 'America/New_York'
}
response = requests.get(url, params=params).json()

Cost: $0, 10k requests/day (upgrade to $50/month for 5M)

Cost Breakdown (MVP - 3 Months)

Compute (Self-Hosted on AWS/GCP/Hetzner)

Component Instance Type Monthly Cost
ClickHouse c6i.2xlarge (8 vCPU, 16GB) $244
PostGIS db.t3.medium (2 vCPU, 4GB) $50
FastAPI (2 nodes) t3.medium (2 vCPU, 4GB) × 2 $120
Kafka/RedPanda c6i.large (2 vCPU, 4GB) $61
Flink c6i.xlarge (4 vCPU, 8GB) $122
Redis cache.t3.micro $12
Airflow t3.medium $60
Total Compute $669/month

Storage

Component Usage Monthly Cost
ClickHouse data (compressed) 500GB $11.50 (S3)
PostGIS backup 50GB $1.15 (S3)
Imagery tiles (Protomaps) 100GB $2.30 (S3)
Object storage (MinIO/S3) 200GB $4.60 (S3)
Total Storage $19.55/month

APIs (Free Tier)

API Usage Cost
Sentinel-2 (ESA) Unlimited $0
Earth Engine 1000 images/month $0
Census.gov Unlimited $0
Open-Meteo 10k requests/day $0
OpenStreetMap Unlimited $0
Geocode.xyz Unlimited (throttled) $0
Total APIs $0/month

Edge Devices (One-Time)

Device Quantity Total
Jetson Orin Nano 10 $4,990
Cameras (IP cameras) 10 $1,000
Mounts/enclosures 10 $500
Total Hardware $6,490 (one-time)

Total MVP Cost (3 Months)

  • Monthly recurring: $688.55/month × 3 = $2,065
  • One-time hardware: $6,490
  • Grand total: $8,555 (3 months)

vs. Commercial Stack: $220,000+ (Planet + Google Maps + Snowflake)

Savings: 97%

Deployment Architecture

┌─────────────────────────────────────────────────────────────────┐
│                         FRONTEND (React)                        │
│  MapLibre GL + Deck.gl + Mapbox Draw + shadcn/ui              │
│  Hosted: Vercel/Netlify (Free tier) or Cloudflare Pages       │
└────────────────────────────┬────────────────────────────────────┘
                             │ HTTPS
┌────────────────────────────▼────────────────────────────────────┐
│                   API GATEWAY (FastAPI)                         │
│  OAuth 2.0 (Keycloak) + Rate Limiting + Caching (Redis)       │
│  Hosted: AWS ECS/Fargate or Fly.io                            │
└──────┬──────────────────────┬──────────────────────┬───────────┘
       │                      │                      │
       ▼                      ▼                      ▼
┌────────────┐      ┌──────────────────┐      ┌──────────────┐
│ ClickHouse │      │     PostGIS      │      │    Redis     │
│ (Signals)  │      │   (Parcels)      │      │   (Cache)    │
└────────────┘      └──────────────────┘      └──────────────┘

┌─────────────────────────────────────────────────────────────────┐
│                      EDGE LAYER (Jetson)                        │
│  YOLO11 + ByteTrack + DeepPrivacy → MQTT → Kafka             │
└────────────────────────────┬────────────────────────────────────┘
                             │
                             ▼
                    ┌────────────────┐
                    │ Kafka/RedPanda │
                    └────────┬───────┘
                             │
                             ▼
                    ┌────────────────┐
                    │ Flink/Beam     │
                    │ (Aggregations) │
                    └────────┬───────┘
                             │
                             ▼
                    ┌────────────────┐
                    │  ClickHouse    │
                    └────────────────┘

┌─────────────────────────────────────────────────────────────────┐
│                   BATCH LAYER (Airflow)                         │
│  Sentinel-2 → SAM Segmentation → PostGIS                       │
│  Earth Engine → Insolation/NDVI → ClickHouse                   │
└─────────────────────────────────────────────────────────────────┘

Implementation Priority

Week 1-2: Infrastructure Foundation

  1. ✅ Deploy ClickHouse (Docker)
  2. ✅ Deploy PostGIS (Docker)
  3. ✅ Deploy Redis (Docker)
  4. ✅ Set up Keycloak (OAuth)
  5. ✅ FastAPI skeleton + OpenAPI docs

Week 3-4: Map Interface

  1. ✅ MapLibre GL + Protomaps tiles
  2. ✅ Mapbox Draw (territory mapping)
  3. ✅ Deck.gl heat maps
  4. ✅ shadcn/ui filters & modals

Week 5-6: Edge Processing

  1. ✅ Flash Jetson with YOLO11
  2. ✅ ByteTrack integration
  3. ✅ DeepPrivacy redaction
  4. ✅ MQTT → Kafka pipeline

Week 7-8: Batch Processing

  1. ✅ Sentinel-2 API integration
  2. ✅ SAM roof segmentation
  3. ✅ Earth Engine insolation
  4. ✅ Airflow DAGs

Week 9-10: APIs & Integrations

  1. ✅ Census.gov demographics
  2. ✅ Open-Meteo weather
  3. ✅ OSM Overpass POI data
  4. ✅ Geocode.xyz reverse geocoding

Week 11-12: Polish & Deploy

  1. ✅ Grafana dashboards
  2. ✅ Quality assurance
  3. ✅ Load testing
  4. ✅ Documentation

Next Steps

Detailed map UI/UX specifications Step-by-step integration for 65+ public APIs Deploy the full stack on AWS/GCP/Hetzner Reduce monthly costs by 50%+ with caching