Feature/phase2 mssql cache 20260721 1313

README.md

@@ -1,20 +1,15 @@
-# Zero-Shot SDD Harness for Building Agents — Hermes Native
+# CrimAnalyze — UP Police Data Analyst Agent
 
-Give it a one-line idea. Walk away with a working, tested, phased agent.
+Give it multiple CSV datasets and an investigator question. Walk away with a grounded insight summary and a chart spec.
 
-A lean, **Hermes-native** harness for building agentic software **spec-first**. One person
-with an idea and one API key can drive a real, production-shaped agent into existence — and
-a senior engineer opening the result finds a conventional, reviewable stack, not generated
-mush.
+A **data analyst agent** built for the UP Police on the **zero-shot harness** (FastAPI + LangGraph + SQLite, provider-agnostic LLM). Phase 1 supports **multi-file CSV upload** and **natural-language Q&A**; Phase 2 adds a **read-only MsSQL federation layer** with query caching to reduce live-DB load and improve latency.
 
 ---
 
 ## The Spirit
 
-Six convictions the whole repo is built around:
-
-1. **Spec is the source of truth.** Written before the code, always. When spec and code disagree, the spec wins and the code is fixed (`/zero-shot-sync`).
-2. **Built for two audiences at once.** A non-coder drives it with a single sentence; a senior engineer inherits a clean FastAPI + LangGraph stack they can read, review, and own.
+1. **Spec is the source of truth.** Written before the code, always. When spec and code disagree, the spec wins (`/zero-shot-sync`).
+2. **Built for two audiences at once.** A police analyst drives it with files + one question; a senior engineer inherits a clean FastAPI + LangGraph stack they can read, review, and own.
 3. **Lean harness, not a framework.** `harness/` is engineering *mindfulness* — rules and patterns that keep every session consistent. The product runtime stays provider-agnostic.
 4. **Smallest first-time-right win, phase by phase.** Each phase ships the smallest increment a human can actually test, and it must work the *first* time they test it.
 5. **A human gates every phase.** Autonomous *within* a phase; stops at each boundary for you to test the increment.
@@ -24,175 +19,88 @@ Six convictions the whole repo is built around:
 
 ## What This Is
 
-- A working **baseline agent** in `src/` — FastAPI + LangGraph + SQLite, provider-agnostic
-  LLM (**Anthropic, Gemini, or OpenRouter** — pure httpx, no SDKs), `transform_text` as the
-  capability slot, structured logging, graceful error paths. **Tests pass out of the box.**
-- A **zero-build static frontend** in `frontend/public/` served by the backend at `/app` —
-  no npm, no bundler, nothing to break at clone time.
-- A **spec template** in `spec/` — roadmap, architecture, capabilities, data, api, ui, and
-  the agent graph.
+- A working **data analyst agent** in `src/` — FastAPI + LangGraph + SQLite, provider-agnostic
+  LLM (`analyze_data` capability), structured logging, graceful error paths. **Tests pass out of the box.**
+- A **zero-build static frontend** in `frontend/public/` served by the backend at `/app` — multi-file
+  CSV upload + question form.
+- A **filled spec** in `spec/` — roadmap, architecture, capabilities, data, api, ui, and
+  the agent graph for multi-CSV analysis.
+- A **Phase 2 plan** for MsSQL federation + query cache.
 - Three **skills**: `/zero-shot-build`, `/zero-shot-fix`, `/zero-shot-sync`.
-- Three **dual-mode specialist roles** (`harness/agents/`): spec-writer, code-generator,
-  qa-auditor — delegated when the runtime allows, executed inline by the root session
-  otherwise.
-- **A human testing gate between phases** — you click a live URL; you never run a terminal
-  command to test.
-
-## The Hermes-Native Architecture
-
-The original Claude-Code harness delegated the whole build to an orchestrator sub-agent
-that fanned out workers. Hermes caps delegation depth (`max_spawn_depth=1`), so that design
-silently degrades. This port makes the constraint the architecture:
-
-```
-YOU (idea, .env key, clicking the app)
- │
-ROOT SESSION — the orchestrator. Owns: human channel (clarify), git/PR, server lifecycle.
- │
- ├─ spec-writer      ─ full spec + phased plan, self-reviewed        (delegate or inline)
- ├─ code-generator   ─ one slice + tests, per slice                  (parallel or inline)
- └─ qa-auditor       ─ read-only review + runs the REAL gates        (delegate or inline)
- │
-per phase:  implement → run real gate → READ output → fix → re-run
-            → boot on the documented command → live smoke → commit+push
-            → HUMAN GATE: one live URL + multi-select checklist
-```
-
-Three properties make it robust on Hermes:
-
-- **Inline fallback everywhere.** Delegation is an optimization, never a dependency — the
-  root executes any role file as a checklist when workers can't spawn.
-- **Trust-but-verify handbacks.** Workers routinely return at "95% done"; the root checks
-  the files, re-runs the gate, and finishes remainders itself.
-- **The gate owns the run.** Only the root launches servers (workers' processes die on
-  return), with the pinned interpreter on a verified free port — you get ONE live URL that
-  has already been smoke-tested.
 
 ---
 
 ## How to Use This
 
 ```bash
-git clone <this repo> my-agent && cd my-agent
+git clone <this repo> crimanalyze && cd crimanalyze
+python -m pip install -e . pytest
 cp .env.example .env        # set exactly ONE provider key (Anthropic / Gemini / OpenRouter)
-```
-
-Then open a Hermes session anchored to the repo and **just say what you want, in plain
-English** — no slash command, no setup:
-
-```
-Build me an agent that monitors my Shopify store for low-inventory products and drafts restock emails.
-```
-
-Hermes auto-loads `.hermes.md` from the repo, which **routes your request to exactly one of
-three skills** and follows it:
-
-- **build** — "build me an agent that…", "add X to it" → creates the agent / adds a capability
-- **fix** — "it's erroring on…", "the tests fail", "X doesn't work" → diagnoses + fixes, then verifies
-- **sync** — "make the code match the spec", "reconcile the drift" → reconciles spec ↔ code (spec wins)
-
-You never pick the skill — describe the goal and the harness chooses. One deep intake (which
-also collects your API key into `.env`), then the build runs one phase at a time and stops at
-each boundary with a live URL for you to test.
-
-**Prefer the `/zero-shot-build` slash command?** Hermes only recognises slash commands that
-are registered — an unregistered one returns "Unknown command". Register this clone once
-(one line, tracks the repo, nothing to re-run after `git pull`), then restart Hermes:
-
-```yaml
-# ~/.hermes/config.yaml
-skills:
-  external_dirs:
-    - /absolute/path/to/this/clone/harness/skills
-```
-
-*Optional:* `uv sync && uv run python agent.py` runs a doctor over the baseline (deps, `.env`,
-app, unit tests). Not required to build.
-
-## What Happens
-
-```
-Your idea
-    ↓
-INTAKE — multi-round product questions (idea-specific, multi-select), then one technical
-         round; fill .env; the key is VALIDATED with a real call before building
-    ↓
-[spec-writer]  → full spec: capabilities + architecture + agent graph + phased plan
-    ↓
-[root session] → feature branch + PR (base = the branch you were on, never main)
-    ↓
-per phase:  code-generator per slice → qa-auditor gates each → boot + live smoke
-            → commit + push → HUMAN GATE (live URL + what-worked checklist)
-    ↓
-repeat per phase → final drift audit → SHIP
-```
-
----
-
-## Running the Baseline
-
-```bash
-# all commands run from the repo root
-uv sync
-cp .env.example .env         # set ONE of: AGENT_ANTHROPIC_API_KEY / AGENT_GEMINI_API_KEY / AGENT_OPENROUTER_API_KEY
-uv run python agent.py       # verify setup (doctor)
-uv run python agent.py --run # migrations (if any) + start the server
+python agent.py             # verify setup (doctor)
+python agent.py --run       # start server
 ```
 
 | URL | What |
 |-----|------|
-| `http://localhost:8001/app/` | **UI** — the transform form (the capability slot) |
-| `http://localhost:8001/health` | Health + active provider (never key values) |
-| `http://localhost:8001/docs` | Interactive API docs (Swagger) |
+| `http://localhost:8001/app/` | **UI** — upload CSVs, ask a question |
+| `http://localhost:8001/health` | Health + active provider |
+| `http://localhost:8001/docs` | Interactive API docs |
 
 Tests:
-
 ```bash
-uv run pytest tests/unit -q          # no key needed — green on a fresh clone
-uv run pytest tests -q               # + integration against the REAL provider (key in .env)
+python -m pytest tests/unit -q          # no key needed
+python -m pytest tests/ -q              # + integration against real provider (needs key in .env)
 ```
 
+---
+
 ## Repo Layout
 
 ```
-src/                ← baseline agent: api/ config/ db/ domain/ graph/ llm/ prompts/ observability/
-frontend/public/    ← zero-build static UI (index.html + styles.css + app.js), served at /app
+src/                ← agent: api/ config/ db/ domain/ graph/ llm/ prompts/ observability/ summarizer.py
+frontend/public/    ← UI: index.html + styles.css + app.js
 tests/              ← unit/ (no key) + integration/ (real key)
-spec/               ← your spec: roadmap, architecture, capabilities/, data, api, ui, agent
+spec/               ← roadmap, architecture, capabilities/analyze.md, data, api, ui, agent
 harness/
   rules/            ← ai-agents, git, secret-hygiene
   patterns/         ← spec-driven, phases, project-layout, tech-stack, code, test-driven,
-                      ui-ux, agentic-ai, engineering-practices
-  skills/           ← zero-shot-build / zero-shot-fix / zero-shot-sync (SKILL.md each)
-  agents/           ← spec-writer, code-generator, qa-auditor (dual-mode role files)
-AGENTS.md           ← the session entry point
-agent.py            ← doctor (default) / --run (serve)
-alembic/            ← migrations, wired (empty until the first schema change)
-.env.example
+                       ui-ux, agentic-ai, engineering-practices
+  skills/           ← zero-shot-build / zero-shot-fix / zero-shot-sync
+  agents/           ← spec-writer, code-generator, qa-auditor
+AGENTS.md           ← session entry point
+agent.py            ← doctor / --run serve
 ```
 
-**Capability slot** — the three surfaces to replace for your agent:
-- `src/graph/nodes.py` — replace `transform_text` with your logic
-- `src/prompts/transform.md` — replace with your system prompt
-- `frontend/public/` — replace the transform form with your UI
+**Capability surfaces** for this agent:
+- `src/graph/nodes.py` → `analyze_data` node
+- `src/prompts/analyze.md` → system prompt
+- `frontend/public/` → multi-file analyst UI
+- `src/summarizer.py` → CSV schema + head/tail summarization for the prompt
+
+---
+
+## Phase 1 — Multi-CSV Analyst
+
+Upload 1–12 CSV files, ask a natural-language question, get an insight grounded in the data with a chart spec.
+
+## Phase 2 — MsSQL Federation + Cache
+
+> Assumed: read-only MsSQL connector with query fingerprinting, SQLite-backed aggregate cache, configurable TTL, cache-hit telemetry in run metadata.
 
-Everything else (graph wiring, API, DB, settings, providers, tests) is already working.
+Repeated identical questions answer from cache; cache latency under 1s; live query latency under 5s. Adds `AGENT_DATABASE_URL_MSSQL` to `.env`.
 
 ---
 
 ## FAQ
 
 **What if I already have a stack in mind?**
-State it in the idea: `/zero-shot-build [idea] — use Python + FastAPI + PostgreSQL`. Stack
-choices are binding.
+State it at intake and the spec-writer records it as binding.
 
 **What if something breaks?**
-`/zero-shot-fix [what's broken]` — qa-auditor classifies SPEC vs CODE, the generator role
-fixes, qa-auditor re-gates, the root commits + pushes.
+`/zero-shot-fix [what's broken]` — qa-auditor classifies SPEC vs CODE, the generator fixes, then re-gates.
 
 **What if spec and code drift?**
 `/zero-shot-sync` — qa-auditor audits, generators fix, spec wins.
 
 **Why did my build branch not merge to main?**
-By design. `main` is boilerplate-only — ABSOLUTELY. Builds live on feature branches whose
-PRs target the branch they were cut from.
+By design. `main` is boilerplate-only — ABSOLUTELY. Builds live on feature branches whose PRs target the branch they were cut from.

frontend/public/app.js

@@ -1,5 +1,4 @@
-// Zero-build baseline frontend. Single-origin: the page is served by the
-// backend at /app, so API calls are same-origin relative paths.
+// CrimAnalyze frontend — multi-file CSV upload + officer question.
 "use strict";
 
 const $ = (id) => document.getElementById(id);
@@ -22,33 +21,48 @@ async function loadHealth() {
   }
 }
 
-async function runTransform() {
+async function runAnalyze() {
   const btn = $("run-btn");
   const status = $("status");
   const errBox = $("error");
   const wrap = $("result-wrap");
 
-  const text = $("text").value.trim();
+  const files = ($("files").files || []);
   const instruction = $("instruction").value.trim();
 
   errBox.hidden = true;
   wrap.hidden = true;
 
-  if (!text) {
-    errBox.textContent = "Paste some text first — the input can't be empty.";
+  if (!files.length) {
+    errBox.textContent = "Upload at least one CSV file.";
+    errBox.hidden = false;
+    return;
+  }
+  if (!instruction) {
+    errBox.textContent = "Type an investigator question first.";
+    errBox.hidden = false;
+    return;
+  }
+  if (files.length > 12) {
+    errBox.textContent = "You can upload at most 12 files in one run.";
     errBox.hidden = false;
     return;
   }
 
   btn.disabled = true;
-  status.textContent = "Running… (one real LLM call)";
+  status.textContent = `Analyzing ${files.length} file(s)... (one real LLM call)`;
   status.hidden = false;
 
   try {
+    const form = new FormData();
+    form.append("instruction", instruction);
+    for (const file of files) {
+      form.append("files", file, file.name || "data.csv");
+    }
+
     const res = await fetch("/runs", {
       method: "POST",
-      headers: { "content-type": "application/json" },
-      body: JSON.stringify({ text, instruction }),
+      body: form,
     });
     const body = await res.json();
 
@@ -60,9 +74,9 @@ async function runTransform() {
     if (run.status === "failed") {
       throw new Error(run.error_message || "The agent run failed.");
     }
-    $("result").textContent = run.output_text;
+    $("result").textContent = run.output_text || "";
     $("result-meta").textContent =
-      `run ${run.run_id} · ${run.provider} · ${run.model}`;
+      `run ${run.run_id} · ${run.provider} · ${run.model} · ${run.file_count} file(s)`;
     wrap.hidden = false;
   } catch (err) {
     errBox.textContent = err.message;
@@ -73,5 +87,5 @@ async function runTransform() {
   }
 }
 
-$("run-btn").addEventListener("click", runTransform);
+$("run-btn").addEventListener("click", runAnalyze);
 loadHealth();

frontend/public/index.html

@@ -3,31 +3,36 @@
 <head>
   <meta charset="utf-8" />
   <meta name="viewport" content="width=device-width, initial-scale=1.0" />
-  <title>Zero-Shot Agent — Transform</title>
+  <title>CrimAnalyze — UP Police Data Analyst</title>
   <link rel="stylesheet" href="styles.css" />
 </head>
 <body>
   <main class="shell">
     <header class="topbar">
-      <h1>Zero-Shot Agent</h1>
+      <h1>CrimAnalyze</h1>
       <span id="provider-badge" class="badge" title="Active LLM provider">…</span>
     </header>
 
-    <!-- CAPABILITY SLOT — replace this form with your agent's UI -->
-    <section class="card" id="transform-card">
-      <h2>Transform text</h2>
-      <p class="hint">The baseline capability: give it text and an instruction; it applies the
-        instruction with one real LLM call.</p>
+    <section class="card">
+      <h2>Investigative data analyst</h2>
+      <p class="hint">Upload 1–12 CSV files, ask a natural-language question, and get an insight grounded in the data.</p>
 
-      <label for="instruction">Instruction</label>
-      <input id="instruction" type="text"
-             value="Summarize the text in one short paragraph." />
+      <div class="form">
+        <div class="field">
+          <label for="files">Dataset files (CSV)</label>
+          <input id="files" type="file" accept=".csv,text/csv" multiple />
+        </div>
 
-      <label for="text">Text</label>
-      <textarea id="text" rows="8"
-                placeholder="Paste the text to transform…"></textarea>
+        <div class="field">
+          <label for="instruction">Investigator question</label>
+          <input id="instruction" type="text"
+                 value="Summarize the combined data and answer the question." />
+        </div>
+      </div>
 
-      <button id="run-btn">Run</button>
+      <div class="actions">
+        <button id="run-btn">Analyze</button>
+      </div>
 
       <div id="status" class="status" hidden></div>
       <div id="error" class="error" hidden></div>