Skip to content

observability-hub

Directory actions

More options

Directory actions

More options

Latest commit

 

History

History

observability-hub

README.md

GraphLens Observability Hub

Purpose: Distributed telemetry aggregation system for multi-machine lens validation monitoring

Status: Phase 3 Complete ✅ (HMAC auth, SQLite WAL, WebSocket reconnect, Bun hub, Vue dashboard, Claude Code hook)

Overview

GraphLens Observability Hub collects telemetry from multiple Soulfield OS instances and provides real-time monitoring through WebSocket-powered dashboards. The system uses HMAC-SHA256 authentication to prevent telemetry spoofing and stores 30 days of history in a SQLite database with WAL mode for concurrent writes.

Architecture:

┌─────────────────┐
│ Machine 1       │──┐
│ Claude Code     │  │  HMAC-signed
│ Hook            │  │  POST /telemetry
└─────────────────┘  │
                     │
┌─────────────────┐  │   ┌──────────────────┐
│ Machine 2       │──┼──▶│ Observability    │
│ Claude Code     │  │   │ Hub (Bun)        │
│ Hook            │  │   │ :3000            │
└─────────────────┘  │   └──────────────────┘
                     │            │
┌─────────────────┐  │            │ WebSocket
│ Machine 3       │──┘            │ broadcast
│ Claude Code     │               │
│ Hook            │               ▼
└─────────────────┘      ┌─────────────────┐
                         │ Vue Dashboard   │
                         │ (Live metrics)  │
                         └─────────────────┘

Key Features:

  • HMAC Authentication: Prevents telemetry spoofing with 5-minute replay window
  • SQLite WAL Mode: Concurrent writes from multiple sources
  • WebSocket Broadcasting: Real-time updates to connected dashboards
  • Auto-Reconnect: Exponential backoff with event buffering
  • Rate Limiting: 100 requests/minute per IP
  • 30-Day Retention: Automatic cleanup via maintenance script
  • Cross-Platform: Works with Bun (native WebSocket) and Node.js (ws package)

Installation

Prerequisites

# Required: Node.js ≥20 or Bun ≥1.0
node --version  # v20.0.0 or higher
# OR
bun --version   # 1.0.0 or higher

# Required: better-sqlite3 (for Node.js)
npm install better-sqlite3

# Required: ws package (for Node.js, not needed for Bun)
npm install ws

# Optional: Bun (for native WebSocket support)
curl -fsSL https://bun.sh/install | bash

Hub Installation

# 1. Navigate to observability hub directory
cd tools/observability-hub

# 2. Generate authentication secret (64 hex chars)
node -e "console.log(require('crypto').randomBytes(32).toString('hex'))"
# Example output: 42e17aed4830bcb165d7ad5dcfe7fd3c18924ee4463455e83ed74b170bc96c86

# 3. Add to environment (.env or export)
export GRAPHLENS_AUTH_SECRET="42e17aed4830bcb165d7ad5dcfe7fd3c18924ee4463455e83ed74b170bc96c86"

# 4. Start hub (Bun for best performance)
bun run index.js
# OR with Node.js
node index.js

# 5. Verify health
curl http://localhost:3000/health

Hook Installation (Per Machine)

# 1. Enable GraphLens distributed mode
export GRAPHLENS_DISTRIBUTED=1
export GRAPHLENS_HUB_URL="http://your-hub-server:3000"
export GRAPHLENS_AUTH_SECRET="<same-secret-as-hub>"

# 2. Add to .env file
cat >> .env << 'EOF'
# GraphLens Distributed Observability
GRAPHLENS_DISTRIBUTED=1
GRAPHLENS_HUB_URL=http://your-hub-server:3000
GRAPHLENS_AUTH_SECRET=42e17aed4830bcb165d7ad5dcfe7fd3c18924ee4463455e83ed74b170bc96c86
EOF

# 3. Verify hook sends telemetry
curl -X POST http://localhost:8790/chat \
  -H 'Content-Type: application/json' \
  -d '{"text":"@marketing test"}'

# 4. Check hub received event
curl http://your-hub-server:3000/telemetry/query?hours=1

Configuration

Environment Variables

Hub Server:

Variable Required Default Description
GRAPHLENS_AUTH_SECRET ✅ Yes None Shared secret for HMAC auth (min 32 chars)
GRAPHLENS_HUB_PORT No 3000 HTTP/WebSocket server port

Client Machines (Hooks):

Variable Required Default Description
GRAPHLENS_DISTRIBUTED ✅ Yes 0 Enable distributed mode (1=on, 0=off)
GRAPHLENS_HUB_URL ✅ Yes None Hub server URL (http://host:port)
GRAPHLENS_AUTH_SECRET ✅ Yes None Shared secret matching hub
ENABLE_GRAPHLENS_CONTENT No false Include full sentence content in telemetry
ENABLE_GRAPHLENS_ALERTS No false Enable alert generation

Legacy (Phase 1/2 flags):

Variable Default Description
GRAPHLENS_TELEMETRY_INTERVAL 15 Sampling interval (seconds)
GRAPHLENS_ALERT_P95_THRESHOLD 80 P95 latency alert threshold (ms)
GRAPHLENS_ALERT_PASS_RATE_THRESHOLD 10 Pass rate alert threshold (%)

Generating Secrets

# Generate 32-byte (64 hex chars) secret
node -e "console.log(require('crypto').randomBytes(32).toString('hex'))"

# Using backend utility
node -e "console.log(require('./backend/services/hmac-auth.cjs').generateSecret())"

Security Notes:

  • Minimum 32 characters required
  • Use different secrets for dev/staging/prod
  • Rotate secrets quarterly
  • Never commit secrets to git (add to .gitignore)

Usage

Starting the Hub

With Bun (Recommended):

cd tools/observability-hub
export GRAPHLENS_AUTH_SECRET="your-secret-here"
bun run index.js

With Node.js:

cd tools/observability-hub
export GRAPHLENS_AUTH_SECRET="your-secret-here"
node index.js

Expected Output:

[Hub] Using Bun native WebSocket
[Hub] Database initialized

🚀 GraphLens Telemetry Hub running on port 3000
   POST /telemetry - Submit telemetry (requires HMAC signature)
   GET /telemetry/query?hours=24 - Query historical data
   GET /telemetry/stats?hours=24 - Get statistics
   GET /health - Health check
   WS /ws - WebSocket for live updates

   Database: /home/user/soulfield/tools/observability-hub/telemetry.db
   Auth: HMAC-SHA256 (5-minute replay window)

API Endpoints

GET /health

Health check with database and telemetry stats (no authentication required).

Example:

curl http://localhost:3000/health

Response:

{
  "status": "ok",
  "database": {
    "size_mb": "2.45",
    "row_count": 1523,
    "journal_mode": "wal"
  },
  "telemetry": {
    "total_count": 1523,
    "pass_rate": "82.5",
    "p95_execution_ms": 23
  },
  "websocket": {
    "connected_clients": 2
  },
  "uptime": 86400
}

POST /telemetry

Submit signed telemetry event (requires HMAC signature).

Example:

# Generate signature
PAYLOAD='{"timestamp":"2025-10-31T12:00:00Z","agent_id":"marketing","quality_score":85,"overall_pass":true}'
SECRET="your-secret-here"

# Sign with Node.js
SIGNATURE=$(node -e "
  const crypto = require('crypto');
  const payload = $PAYLOAD;
  const canonical = JSON.stringify(payload, Object.keys(payload).sort());
  const hmac = crypto.createHmac('sha256', '$SECRET');
  hmac.update(canonical);
  console.log(hmac.digest('hex'));
")

# Submit telemetry
curl -X POST http://localhost:3000/telemetry \
  -H "Content-Type: application/json" \
  -H "X-GraphLens-Signature: $SIGNATURE" \
  -d "$PAYLOAD"

Response:

{"success":true,"id":1524}

GET /telemetry/query

Query historical telemetry data.

Query Parameters:

  • hours - Time range in hours (default: 24)
  • limit - Max rows to return (default: 1000)

Example:

# Last 3 hours, max 100 rows
curl 'http://localhost:3000/telemetry/query?hours=3&limit=100'

Response:

[
  {
    "id": 1524,
    "timestamp": "2025-10-31T12:00:00Z",
    "machine_id": "dev-laptop",
    "agent_id": "marketing",
    "quality_score": 85,
    "gap_count": 2,
    "community_count": 5,
    "execution_ms": 23,
    "overall_pass": 1,
    "bottleneck_type": null,
    "raw_json": "{...}",
    "created_at": 1730376000
  }
]

GET /telemetry/stats

Get aggregated statistics and breakdowns.

Query Parameters:

  • hours - Time range in hours (default: 24)
  • groupBy - Group by field: agent_id or machine_id (default: agent_id)

Example:

curl 'http://localhost:3000/telemetry/stats?hours=24&groupBy=agent_id'

Response:

{
  "stats": {
    "total_count": 1523,
    "pass_count": 1256,
    "avg_execution_ms": 18.5,
    "max_execution_ms": 67,
    "machine_count": 3,
    "agent_count": 5,
    "pass_rate": "82.5",
    "p95_execution_ms": 23
  },
  "breakdown": [
    {
      "group_key": "marketing",
      "count": 523,
      "pass_count": 450,
      "avg_execution_ms": 19.2
    },
    {
      "group_key": "finance",
      "count": 412,
      "pass_count": 380,
      "avg_execution_ms": 17.8
    }
  ]
}

WebSocket Connection

Connect to live telemetry stream:

const ws = new WebSocket('ws://localhost:3000/ws');

ws.onopen = () => {
  console.log('Connected to GraphLens Hub');
};

ws.onmessage = (event) => {
  const data = JSON.parse(event.data);

  if (data.type === 'telemetry') {
    console.log('New telemetry event:', data.data);
  }
};

// Server sends 'ping' every 30s, respond with 'pong'
ws.on('message', (data) => {
  if (data === 'ping') {
    ws.send('pong');
  }
});

Using Auto-Reconnect Client:

const { ReconnectingWebSocket } = require('./client/reconnect.js');

const ws = new ReconnectingWebSocket('ws://localhost:3000/ws', {
  maxRetries: Infinity,
  retryDelays: [1000, 2000, 4000, 8000, 16000, 30000],
  heartbeatInterval: 30000,
  heartbeatTimeout: 90000
});

ws.onmessage = (event) => {
  const data = JSON.parse(event.data);
  console.log('Telemetry:', data);
};

ws.onstatechange = (newState, oldState) => {
  console.log(`WebSocket state: ${oldState} → ${newState}`);
};

Database Maintenance

Run manual cleanup:

cd tools/observability-hub
node maintenance.js

Schedule via cron (runs daily at 3 AM):

crontab -e

# Add this line:
0 3 * * * cd /path/to/soulfield/tools/observability-hub && node maintenance.js >> logs/cron.log 2>&1

Maintenance tasks:

  1. Database integrity check (PRAGMA integrity_check)
  2. Delete telemetry older than 30 days
  3. Incremental vacuum (reclaim space)
  4. Log database size and statistics

Production Deployment

Systemd Service (Linux)

Create service file:

sudo nano /etc/systemd/system/graphlens-hub.service

Service configuration:

[Unit]
Description=GraphLens Telemetry Hub
After=network.target

[Service]
Type=simple
User=soulfield
WorkingDirectory=/home/soulfield/soulfield/tools/observability-hub
Environment="GRAPHLENS_AUTH_SECRET=your-secret-here"
Environment="GRAPHLENS_HUB_PORT=3000"
Environment="NODE_ENV=production"
ExecStart=/usr/local/bin/bun run index.js
Restart=always
RestartSec=10
StandardOutput=journal
StandardError=journal
SyslogIdentifier=graphlens-hub

[Install]
WantedBy=multi-user.target

Enable and start service:

# Reload systemd
sudo systemctl daemon-reload

# Enable auto-start on boot
sudo systemctl enable graphlens-hub

# Start service
sudo systemctl start graphlens-hub

# Check status
sudo systemctl status graphlens-hub

# View logs
sudo journalctl -u graphlens-hub -f

Reverse Proxy (Caddy)

HTTPS with automatic SSL:

# /etc/caddy/Caddyfile

graphlens.yourdomain.com {
    reverse_proxy localhost:3000

    # WebSocket support
    @websockets {
        header Connection *Upgrade*
        header Upgrade websocket
    }
    reverse_proxy @websockets localhost:3000

    # Security headers
    header {
        Strict-Transport-Security "max-age=31536000;"
        X-Content-Type-Options "nosniff"
        X-Frame-Options "DENY"
        Referrer-Policy "no-referrer-when-downgrade"
    }

    # Rate limiting (optional)
    rate_limit {
        zone graphlens {
            key {remote_host}
            events 100
            window 1m
        }
    }

    # Access log
    log {
        output file /var/log/caddy/graphlens-access.log
    }
}

Reload Caddy:

sudo systemctl reload caddy

Nginx Alternative

# /etc/nginx/sites-available/graphlens

upstream graphlens_backend {
    server 127.0.0.1:3000;
}

server {
    listen 80;
    server_name graphlens.yourdomain.com;

    # Redirect to HTTPS
    return 301 https://$server_name$request_uri;
}

server {
    listen 443 ssl http2;
    server_name graphlens.yourdomain.com;

    # SSL certificates (use certbot)
    ssl_certificate /etc/letsencrypt/live/graphlens.yourdomain.com/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/graphlens.yourdomain.com/privkey.pem;

    # WebSocket support
    location /ws {
        proxy_pass http://graphlens_backend;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_read_timeout 86400;
    }

    # HTTP endpoints
    location / {
        proxy_pass http://graphlens_backend;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
    }

    # Security headers
    add_header Strict-Transport-Security "max-age=31536000; includeSubDomains" always;
    add_header X-Content-Type-Options "nosniff" always;
    add_header X-Frame-Options "DENY" always;

    # Rate limiting
    limit_req_zone $binary_remote_addr zone=graphlens:10m rate=100r/m;
    limit_req zone=graphlens burst=20 nodelay;
}

Enable and reload:

sudo ln -s /etc/nginx/sites-available/graphlens /etc/nginx/sites-enabled/
sudo nginx -t
sudo systemctl reload nginx

Firewall Configuration

# Allow hub port (if not using reverse proxy)
sudo ufw allow 3000/tcp

# Or allow only from specific machines
sudo ufw allow from 192.168.1.0/24 to any port 3000

# Allow HTTPS (if using reverse proxy)
sudo ufw allow 443/tcp

# Reload firewall
sudo ufw reload

Troubleshooting

Authentication Failures

Symptom: 401 MISSING_SIGNATURE or SIGNATURE_MISMATCH

Cause: HMAC secret mismatch between client and hub

Fix:

# 1. Check hub secret
sudo journalctl -u graphlens-hub -n 100 | grep AUTH

# 2. Check client .env file
cat .env | grep GRAPHLENS_AUTH_SECRET

# 3. Ensure secrets match exactly
# Hub:   GRAPHLENS_AUTH_SECRET=abc123...
# Client: GRAPHLENS_AUTH_SECRET=abc123... (same)

# 4. Regenerate and update both sides
NEW_SECRET=$(node -e "console.log(require('crypto').randomBytes(32).toString('hex'))")
echo "GRAPHLENS_AUTH_SECRET=$NEW_SECRET"

# Update .env files and restart services

Connection Refused

Symptom: ECONNREFUSED when posting telemetry

Cause: Hub not running or firewall blocking

Fix:

# 1. Check hub is running
sudo systemctl status graphlens-hub
# OR
curl http://localhost:3000/health

# 2. Check firewall rules
sudo ufw status | grep 3000

# 3. Check port binding
sudo netstat -tlnp | grep 3000

# 4. Restart hub if needed
sudo systemctl restart graphlens-hub

Database Locked

Symptom: Error: database is locked

Cause: WAL mode not enabled or too many concurrent writes

Fix:

# 1. Check WAL mode
sqlite3 tools/observability-hub/telemetry.db "PRAGMA journal_mode;"
# Should output: wal

# 2. Enable WAL mode if disabled
sqlite3 tools/observability-hub/telemetry.db "PRAGMA journal_mode=WAL;"

# 3. Check for stale WAL files
ls -la tools/observability-hub/*.db*
# If -wal or -shm files are huge, run maintenance:
node tools/observability-hub/maintenance.js

# 4. Increase SQLite timeout (in db.js)
# this.db.pragma('busy_timeout = 5000'); // 5 seconds

WebSocket Disconnected

Symptom: Dashboard shows "disconnected" or frequent reconnects

Cause: Firewall blocking WebSocket or heartbeat timeout

Fix:

# 1. Check WebSocket handshake
wscat -c ws://localhost:3000/ws
# Should connect and show "ping" every 30s

# 2. Check reverse proxy WebSocket config (if using Caddy/Nginx)
# Ensure Upgrade header is proxied

# 3. Adjust heartbeat timeouts (client/reconnect.js)
# heartbeatInterval: 30000,  // Send ping every 30s
# heartbeatTimeout: 90000,   // Close after 90s no pong

# 4. Check for NAT/firewall timeout
# Some routers close idle connections after 60s
# Reduce heartbeatInterval to 20000 if needed

High Memory Usage

Symptom: Hub using >500MB RAM

Cause: Large database or too many WebSocket clients

Fix:

# 1. Check database size
du -h tools/observability-hub/telemetry.db
# Should be <100MB for 30 days

# 2. Run maintenance cleanup
node tools/observability-hub/maintenance.js

# 3. Reduce retention period (in maintenance.js)
# db.cleanup(7); // Keep only 7 days instead of 30

# 4. Limit WebSocket clients
# Add connection limit in index.js:
# if (wsClients.size >= 10) {
#   ws.close();
#   return;
# }

Rate Limit Exceeded

Symptom: 429 RATE_LIMIT_EXCEEDED responses

Cause: Client sending >100 requests/minute

Fix:

# 1. Check request rate from client
grep "POST /telemetry" /var/log/caddy/graphlens-access.log | \
  awk '{print $1}' | sort | uniq -c | sort -rn

# 2. Adjust rate limit in index.js
# const RATE_LIMIT_MAX = 200; // Increase from 100

# 3. Or batch telemetry events client-side
# Send every 60s instead of real-time

# 4. Whitelist specific IPs (add to index.js)
# const WHITELIST = ['192.168.1.100'];
# if (WHITELIST.includes(clientIp)) {
#   // Skip rate limit
# }

Development

Running Tests

# Start hub in test mode
cd tools/observability-hub
GRAPHLENS_AUTH_SECRET="test-secret-32-chars-minimum123" node index.js &

# Run integration tests
./test-hub.sh

# Stop hub
pkill -f "node index.js"

Manual Telemetry Submission

# Generate valid signature
SECRET="test-secret-32-chars-minimum123"
PAYLOAD='{"timestamp":"'$(date -u +%Y-%m-%dT%H:%M:%SZ)'","agent_id":"test","overall_pass":true}'

SIGNATURE=$(node -e "
  const crypto = require('crypto');
  const payload = $PAYLOAD;
  const canonical = JSON.stringify(payload, Object.keys(payload).sort());
  const hmac = crypto.createHmac('sha256', '$SECRET');
  hmac.update(canonical);
  console.log(hmac.digest('hex'));
")

# Submit telemetry
curl -X POST http://localhost:3000/telemetry \
  -H "Content-Type: application/json" \
  -H "X-GraphLens-Signature: $SIGNATURE" \
  -d "$PAYLOAD"

Database Inspection

# Open database shell
sqlite3 tools/observability-hub/telemetry.db

# View schema
.schema

# Count rows
SELECT COUNT(*) FROM telemetry;

# Recent events
SELECT * FROM telemetry ORDER BY timestamp DESC LIMIT 10;

# Stats by agent
SELECT agent_id, COUNT(*), AVG(execution_ms)
FROM telemetry
GROUP BY agent_id;

# Exit
.quit

Architecture Details

HMAC Authentication Flow

Client                          Hub
  │                              │
  ├─── Generate payload ─────────┤
  │    {timestamp, agent_id, ...}│
  │                              │
  ├─── Sign with HMAC ───────────┤
  │    HMAC-SHA256(payload, secret)
  │                              │
  ├─── POST /telemetry ──────────▶
  │    X-GraphLens-Signature: abc123...
  │                              │
  │                         Validate ────┐
  │                         signature    │
  │                         timestamp    │
  │                         (5 min TTL)  │
  │                              ◀───────┘
  │                              │
  │◀─── 200 OK {id: 1524} ───────┤
  │                              │
  │                         Store in DB
  │                         Broadcast WS

SQLite WAL Mode

Write-Ahead Logging (WAL):

  • Writes go to separate -wal file
  • Readers access main database concurrently
  • Periodic checkpoints merge -wal → main DB
  • ~10x faster than DELETE mode for concurrent writes

Files:

  • telemetry.db - Main database
  • telemetry.db-wal - Write-ahead log (pending commits)
  • telemetry.db-shm - Shared memory index

Checkpoint Strategy:

  • Auto-checkpoint when -wal reaches 1000 pages (~4MB)
  • Manual checkpoint: PRAGMA wal_checkpoint(TRUNCATE);
  • Runs during maintenance.js cleanup

WebSocket Heartbeat

Purpose: Detect stale connections (NAT timeout, firewall, crash)

Flow:

Server                  Client
  │                       │
  ├─── ping ─────────────▶│
  │                       │
  │◀──── pong ────────────┤
  │                       │
  (wait 30s)              │
  │                       │
  ├─── ping ─────────────▶│
  │                       │
  │       (no response)   │
  │                       │
  (90s timeout)           │
  │                       │
  └─── close ───────────▶ │
                          │
                     reconnect

Tuning:

  • heartbeatInterval: 30000 - Send ping every 30s
  • heartbeatTimeout: 90000 - Close after 90s no pong
  • Reduce interval for unstable networks (20s)
  • Increase timeout for high-latency connections (120s)

Related Documentation

Last Updated: 2025-10-31 Phase: 3 (Production-ready distributed observability) Maintainer: Sentinel (via Claude Code CLI)