title	Synapse — SRE Incident Co-Pilot

Synapse

Agentic SRE incident co-pilot. Fully offline. No API keys.

Synapse investigates production incidents autonomously. When an alert fires, it queries your logs, searches a runbook knowledge base using hybrid vector + full-text retrieval, runs diagnostic commands in an idempotent, guardrailed tool-use loop, and returns a structured resolution — root cause, exact commands, prevention, and MTTR — then stays available for multi-turn follow-up.

Every tool call the agent makes passes through an in-process guardrail gateway first. Destructive commands (drops, deletes, force flags, scale-to-zero) never execute automatically — they pause the loop and wait for a human to approve, force a dry run, or deny them. See Guardrails below.

Everything runs in Docker on your machine. Zero data egress.

---

Quickstart

git clone https://github.com/joshuabvarghesearghese/synapse && cd synapse
./start.sh

Opens at http://localhost:8501

First run pulls two models (~2–5 GB total depending on model size, cached permanently in a Docker volume):

• qwen2.5-coder:3b — inference + tool-use (default; set OLLAMA_MODEL=qwen2.5-coder:7b for stronger reasoning if you have the RAM)
• nomic-embed-text — 768-dim embeddings

make up       # start everything
make status   # health check all services
make logs     # follow app logs
make shell    # psql into the database
make down     # stop (data preserved)
make reset    # wipe all volumes, start fresh

Running on Hugging Face Spaces

This repo is also set up to run as a single-container HF Space (Postgres + Ollama + Streamlit baked into one image — see Dockerfile / start_hf.sh). The README.md frontmatter at the top of this file (sdk: docker, app_port: 7860) is what tells HF Spaces to build and run the Dockerfile at all; if it's missing or edited out, the Space silently falls back to a default template and none of Postgres/Ollama/Streamlit start correctly. Don't remove it.

Two things worth knowing about the HF deployment specifically:

• Storage is ephemeral by default. Every restart/sleep-wake re-runs start_hf.sh, which re-bootstraps Postgres from init.sql and re-seeds the KB from kb/*.md. That's intentional for a stateless demo; add an HF persistent storage volume if you want data (incident history, custom runbooks) to survive restarts.
• External managed Postgres works too. Set POSTGRES_HOST / POSTGRES_PORT / POSTGRES_USER / POSTGRES_PASSWORD / POSTGRES_DB / POSTGRES_SSLMODE as Space secrets to point at Neon/Supabase/RDS instead of the baked-in container instance — just remember managed providers generally require POSTGRES_SSLMODE=require, not the container default of disable.

If the sidebar ever shows "Postgres unreachable", it now shows the actual driver error underneath (auth failure, TLS required, connection refused, etc.) instead of a generic message — that's almost always enough to tell you which of the above is the cause.

---

How it works

Alert fires
    │
    ▼
Synapse Agent Loop  (qwen2.5-coder:7b + tool-use, up to 8 steps)
    ├─→ query_logs(service, level)      → recent errors from Postgres
    ├─→ search_kb(query)                → hybrid pgvector + FTS → RRF fusion
    ├─→ run_diagnostic(kubectl/redis/aws) → simulate against live env
    └─→ run_diagnostic(verify fix)
    │
    ▼
Structured resolution: root cause · runbook · prevention · MTTR
    │
    ▼
Multi-turn follow-up conversation

---

Guardrails — human-in-the-loop, MCP-Gateway-style

This is the same problem an MCP Gateway solves for LLM tool use in production: a model deciding to call a tool is not the same thing as that call being safe to run, and the only place that distinction can be reliably enforced is in the infrastructure the call passes through — not in the prompt, and not in the model's own judgment.

Synapse implements that gateway in miniature (app/core/guardrails.py):

• Classification. Every proposed tool call is inspected before execution. Read-only tools (query_logs, search_kb) are always safe. run_diagnostic commands are pattern-matched against a destructive-operation list: DROP/TRUNCATE/DELETE FROM, kubectl delete, Redis FLUSHALL/FLUSHDB, nodetool decommission, RDS instance deletion, --force flags, scale-to-zero, rm -rf, etcd member removal, Kafka topic deletion.
• Interception. A destructive call is never executed automatically. The agent loop pauses (AgentSession.pending) and the UI renders an approval card with three explicit choices: approve & run, dry-run only (preview the effect with zero side effects), or deny.
• Audit trail. Every decision — including auto-approved safe calls — is written to guardrail_audit_log in Postgres, independent of the browser session, so there's a durable record of what the agent tried to do and what was actually allowed.
• Idempotency. Tool calls are cached by (name, canonical arguments) for the life of the investigation. A Streamlit rerun, a slow human clicking "approve," or the model asking for the same evidence twice all replay the cached result instead of re-executing — so a destructive action that already ran can never silently run a second time just because the page re-rendered.
• Fail-safe tool execution. Every tool call is wrapped so a failure (DB down, embedding service unreachable) degrades to an error string the agent can reason about, instead of an unhandled exception taking down the whole investigation — and the whole Streamlit page with it.

Toggle it off with GUARDRAILS_ENABLED=false (not recommended outside local testing) and tune the step budget with AGENT_MAX_STEPS (default 8).

---

---

Demo scenarios

Scenario	Severity	Service	What the agent finds
CPU runaway — nginx fork storm	P1	nginx	worker_processes=512 in ConfigMap; rolling restart + HPA
DB replica lag — checkout 500s	P1	postgres	Aurora failover lag; promotes replica, rotates secret
Memory leak — auth-svc OOM	P2	auth-svc	Unbounded JWTCache; sets eviction policy + restarts
Bad deploy — ConfigMap missing key	P2	checkout	DB_REPLICA_HOST missing from v2.4.1; rollback to v2.4.0
Redis cluster fail — cascade timeout	P2	redis	Disk-full node; sentinel failover + circuit breaker reset

---

Stack

Layer	Technology	Notes
LLM	Ollama qwen2.5-coder:3b (default)	Runs comfortably on CPU-only Spaces; swap in 7b/14b for stronger reasoning
Embeddings	nomic-embed-text (768-dim)	Runs in Ollama, no extra process
Vector search	PostgreSQL + pgvector HNSW	m=16, ef_construction=64
Full-text search	Postgres tsvector + GIN	Auto-maintained by trigger
Search fusion	Reciprocal Rank Fusion (RRF)	Outperforms either signal alone
Agent	Ollama tool-use loop	Up to 8 steps, idempotent, full audit trail
Guardrails	In-process MCP-Gateway-style policy layer	Blocks destructive calls pending human approval
Conversation	Multi-turn with full context	Incident + resolution in system prompt
KB watcher	watchdog file observer	Drop .md files → indexed in seconds
UI	Streamlit	Custom dark-mode terminal aesthetic
Infra	Docker Compose	Postgres 16, Ollama, Streamlit

---

Adding your own runbooks

Drop .md files into kb/ — they are embedded and indexed automatically (no restart):

## INC-2025-0099 · P1 · redis · cache, eviction

**Title:** Redis maxmemory hit — cache miss storm

**Resolution:**
Root cause: maxmemory-policy was set to noeviction. Under memory pressure,
all writes returned OOM errors, cascading to checkout timeouts.

Steps:
1. `redis-cli CONFIG SET maxmemory-policy allkeys-lru`
2. `redis-cli BGREWRITEAOF`
3. Verify hit rate recovers: `redis-cli INFO stats | grep keyspace`

MTTR: 4 minutes.
Prevention: Set maxmemory-policy in redis.conf before deployment.

---

Changing the model

Set OLLAMA_MODEL in .env (copy from .env.example):

OLLAMA_MODEL=qwen2.5-coder:14b   # better reasoning, needs ~10 GB RAM
OLLAMA_MODEL=mistral-small        # alternative, strong structured output
OLLAMA_MODEL=llama3.1:8b          # general purpose, good tool-use

---

Configuration

All settings via environment variables. Copy .env.example to .env:

cp .env.example .env

Variable	Default	Description
OLLAMA_MODEL	qwen2.5-coder:3b	Inference model
EMBED_MODEL	nomic-embed-text	Embedding model (must produce 768-dim vectors)
POSTGRES_HOST	localhost	Postgres host — point at an external managed instance if desired
POSTGRES_PORT	5432	Postgres port
POSTGRES_USER	ops	Database user
POSTGRES_PASSWORD	ops	Database password
POSTGRES_DB	synapse	Database name
POSTGRES_SSLMODE	disable	Set to require for managed providers (Neon/Supabase/RDS)
GUARDRAILS_ENABLED	true	Require human approval for destructive tool calls
AGENT_MAX_STEPS	8	Tool-use loop step budget
APP_PORT	8501	Streamlit port

GPU (NVIDIA): uncomment the deploy block in docker-compose.yml.

---

Architecture

See ARCHITECTURE.md for a full technical deep-dive: design decisions, the hybrid search implementation, why HNSW over IVFFlat, RRF math, and future directions.

---

License

MIT