Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
30 changes: 20 additions & 10 deletions .env.example
Original file line number Diff line number Diff line change
@@ -1,15 +1,25 @@
AGENT_DATABASE_URL=sqlite:///./data/agent.db

# LLM provider — auto-detected from whichever key is set below.
# Override explicitly: anthropic | gemini
# AGENT_LLM_PROVIDER=anthropic
# LLM provider — one of: gemini, anthropic, openrouter
AGENT_LLM_PROVIDER=gemini
AGENT_LLM_MODEL=gemini-2.5-flash

# Override model (uses provider default when blank):
# AGENT_LLM_MODEL=claude-sonnet-4-6 (Anthropic default)
# AGENT_LLM_MODEL=gemini-3.1-pro (Gemini default)

# Set exactly ONE provider key:
AGENT_ANTHROPIC_API_KEY=
# Set exactly ONE provider key (Pydantic SecretStr; never echoed):
AGENT_GEMINI_API_KEY=
AGENT_ANTHROPIC_API_KEY=

# Mirror — leave blank to use the in-process mock mirror (default in dev).
# Set to an SQL Server URL to flip into live mode (Phase 3 connector).
AGENT_CCTNS_MIRROR_URL=

# Bounded executor
AGENT_ROW_CAP=1000
AGENT_STATEMENT_TIMEOUT_MS=10000

# Server
AGENT_PORT=8001
AGENT_LOG_LEVEL=INFO

PORT=8001
# LangSmith (Phase 3 only — turn on if/when wired)
# LANGCHAIN_TRACING_V2=true
# LANGCHAIN_API_KEY=
6 changes: 6 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,9 @@ node_modules/
.next/
dist/
build/
frontend/out/
playwright-report/
test-results/

# DB
*.db
Expand All @@ -27,3 +30,6 @@ data/
# Logs / OS
*.log
.DS_Store

# Hermes tmp sentinel dirs (do not track)
.hermes-tmp.*/
246 changes: 90 additions & 156 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,176 +1,110 @@
# Zero Shot SDD Harness for Building Agents

Give it a one-line idea. Walk away with a working, tested, phased agent.

A lean, Claude-Code-native harness for building agentic software **spec-first**. One person with an idea and one API key can drive a real, production-shaped agent into existence — and a senior engineer opening the result finds a conventional, reviewable stack, not generated mush.

---

## The Spirit

Six convictions the whole repo is built around:

1. **Spec is the source of truth.** The spec is written before the code, always. When spec and code disagree, the spec wins and the code is fixed (`/zero-shot-sync`). Every AI session reads the same requirements instead of re-deriving them.
2. **Built for two audiences at once.** A non-coder drives it with a single sentence; a senior engineer inherits a clean FastAPI + LangGraph stack they can read, review, and own. Neither audience is an afterthought.
3. **Lean harness, not a framework.** `harness/` is engineering *mindfulness* — rules and patterns that keep every session consistent — deliberately Claude-Code-only and kept small. The product runtime stays provider-agnostic; the harness does not.
4. **Smallest first-time-right win, phase by phase.** Each phase ships the smallest increment a human can actually test, and it must work the *first* time they test it — real on the tested path, with clearly-labelled stubs for everything still to come. No rough edges on the path you're handed.
5. **A human gates every phase.** The build is autonomous *within* a phase and stops at each boundary for you to test the increment. You stay in control of what "done" means.
6. **Real LLM/API or it doesn't count.** Gates, tests, and evals run against the real model with keys from `.env`. A stubbed pass is not a pass.

---

## What This Is

A starting point for building AI agents spec-first. The repo ships with:

- A working **baseline agent** in `src/` (FastAPI + LangGraph + SQLite, provider-agnostic LLM — Anthropic or Gemini, `transform_text` as the capability slot) — tests pass out of the box
- A **spec template** in `spec/` covering roadmap, architecture, capabilities, data model, API, UI, and agent graph
- Three **zero-shot skills** (`/zero-shot-build`, `/zero-shot-fix`, `/zero-shot-sync`)
- A four-agent **team** — agent-builder orchestrates (plans, fans out, owns git/PR); spec-writer is the single design authority; code-generator implements one slice per instance (parallelised); qa-auditor reviews and gates
- Engineering rules and patterns in `harness/` so every Claude Code session is consistent
- **Human testing gate between phases** — autonomous within a phase, you test each increment before the next starts

---

## How to Use This

### Step 1 — Clone

```bash
git clone https://github.com/smallTechOrg/zero-shot-sdd-harness.git my-agent
cd my-agent
```

### Step 2 — Open in Claude Code

```bash
claude
```

### Step 3 — Build

```
/zero-shot-build An agent that monitors my Shopify store for low-inventory products and drafts restock emails to suppliers
# CCTNS Analyst

Natural-language → bounded SQL → short-answer analyst over the **CCTNS
mirror**, tuned for low latency and low load on the source database. Built
for UP Police analysts.

> **All commands below run from the project root** (`E:\smalltech\hermes\zero-shot-hermes-harness`).
> The package slug is `cctns_analyst` and lives at `src/cctns_analyst/`; the
> Next.js UI lives at `frontend/` and is mounted by FastAPI at `/app/`. The
> `harness/` directory is engineering rules for AI agents — leave it alone.

## What it does

1. The user types an analyst question into the UI.
2. The backend (FastAPI) routes through a **LangGraph** state machine:
- `nl_to_sql` — LLM (Gemini `gemini-2.5-flash`) drafts a single SELECT
against `cctns_mirror.*` (schema only; data-locality safe).
- `execute_sql` — runs the SELECT on the configured mirror under strict
bounds (row_cap=1000, statement_timeout=10s).
- `validate_result` — one self-correction retry on validation failure.
- `summarize_answer` — second LLM call turns the bounded result into a
short prose summary.
- `finalize` — persists an `AnswerRun` row.
3. The UI shows the prose answer, results table, the SQL, and a latency badge.

## Stack (binding)

- Python 3.11 + FastAPI + LangGraph + SQLAlchemy 2.0
- Gemini (`gemini-2.5-flash`) — provider key in `.env`
- ICC Mode: **mock** mirror (default in dev, in-process, ≥ 500 synthetic FIR rows)
or **live** mirror via `CCTNS_MIRROR_URL` (Phase 3 connector)
- Next.js 15 + React 19 + Tailwind v4 static export at `/app/`
- SQLite for our own state (`AnswerRun`, `CctnsTable`)

## `.env`

```env
AGENT_DATABASE_URL=sqlite:///./data/agent.db
AGENT_GEMINI_API_KEY=<your-gemini-key>
AGENT_LLM_MODEL=gemini-2.5-flash
AGENT_LLM_PROVIDER=gemini
AGENT_ROW_CAP=1000
AGENT_STATEMENT_TIMEOUT_MS=10000
AGENT_LOG_LEVEL=INFO
AGENT_PORT=8001

# Flip to live mode:
# AGENT_CCTNS_MIRROR_URL=mssql+pyodbc://user:pass@host/cctns?driver=ODBC+Driver+18+for+SQL+Server
```

One intake round (scope, stack, API keys → fill `.env`), then the agent builds phase by phase and stops at each boundary for you to test.
These are also documented in `.env.example`.

---
## Run

## What Happens (Intake → Phase by Phase)

```
Your idea
INTAKE — scope, stack, LLM provider, constraints; fill .env with the required API key
[spec-writer] → Full spec: architecture + agent-graph + phased plan (self-reviewed)
[agent-builder] → Feature branch + PR, scaffold
per phase — all slices concurrently:
[code-generator: slice-a] ──→ [qa-auditor: slice-a] ─┐
[code-generator: slice-b] ──→ [qa-auditor: slice-b] ─┤→ commit + push
[code-generator: slice-c] ──→ [qa-auditor: slice-c] ─┘
HUMAN TESTING GATE — exact run commands + expected result; you confirm before next phase
(issue → qa-auditor classifies SPEC-vs-CODE → code-generator fixes → re-gate)
repeat per phase → SHIP
```
From the **repo root**:

Phase 1 is the smallest first-time-right win — real on the tested path, with labelled stubs for everything coming later. Each later phase wires one more stub into real functionality.
```bash
# 1. install
uv sync --group dev

---
# 2. database
uv run alembic upgrade head
uv run alembic current # verify the migration applied (not blank)

## Repo Layout
# 3. build the frontend (once; rerun after frontend changes)
cd frontend && npm install --silent && npm run build && cd ..

# 4. start the server (serves API + static UI on :8001)
uv run python -m src
```
src/ ← baseline agent (FastAPI + LangGraph + SQLite, Anthropic/Gemini)
api/ ← FastAPI routers (create_app, health, runs)
config/ ← Pydantic BaseSettings
db/ ← SQLAlchemy models + session
domain/ ← Pydantic request/response models
graph/ ← LangGraph nodes, edges, state, runner ← CAPABILITY SLOT
llm/ ← LLM client + providers/ (anthropic, gemini)
prompts/ ← prompt templates (.md)
observability/
frontend/ ← Next.js static export (served by FastAPI at /app)
tests/
unit/ ← passes with no API key
integration/ ← requires real key in .env
spec/ ← your spec: roadmap, architecture, capabilities/, data, api, ui, agent
harness/
rules/ ← ai-agents, git, secret-hygiene
patterns/ ← spec-driven, phases, project-layout, tech-stack, code, test-driven, ui-ux, agentic-ai, engineering-practices
.claude/
skills/ ← /zero-shot-build, /zero-shot-fix, /zero-shot-sync
agents/ ← agent-builder, spec-writer, code-generator, qa-auditor
CLAUDE.md
pyproject.toml
alembic.ini ← Alembic migrations (alembic/)
agent.py ← verify setup (default); --run to start the server
.env.example
```

**Capability slot** — the three files to replace for your agent:
- `src/graph/nodes.py` — replace `transform_text` with your logic
- `src/prompts/transform.md` — replace with your system prompt
- `frontend/src/app/page.tsx` — replace the transform form with your UI

Everything else (graph wiring, API, DB, settings, tests) is already working.

---
Then open **http://localhost:8001/app/** in a browser.

## Running the Baseline
## Test

```bash
cp .env.example .env
# edit .env: set exactly ONE provider key —
# AGENT_ANTHROPIC_API_KEY=<your key> or AGENT_GEMINI_API_KEY=<your key>
# the provider is auto-detected from whichever key is set
uv sync
python agent.py # verify tools, .env, deps, tests (default)
python agent.py --run # migrations + frontend build + start server
uv run pytest -q # unit + integration
cd tests/e2e && npm install --silent && npx playwright install --with-deps chromium && cd ../..
npx playwright test tests/e2e/ --reporter=line
```

Once running:
The integration suite uses a **recording stub LLM provider** so it runs
without burning a real Gemini call. Data-locality, bounded-query, and
full-data correctness gates are part of `uv run pytest -q`.

| URL | What |
|-----|------|
| `http://localhost:8001/app/` | **UI** — transform form (the capability slot) |
| `http://localhost:8001/health` | API health check |
| `http://localhost:8001/docs` | Interactive API docs (Swagger) |
## Layout

Tests:

```bash
uv run pytest tests/unit/ -v # no key needed
uv run pytest tests/ -v # requires real key in .env
```
src/cctns_analyst/
├── api/ FastAPI routers, app factory
├── config/ Pydantic Settings (env_prefix=APP_)
├── db/ SQLAlchemy 2.0 models + session
├── domain/ Pydantic body/response shapes
├── graph/ LangGraph state machine
├── llm/ LLMClient + Gemini provider
├── observability/ structlog JSON
├── prompts/ .md templates
└── tools/ mock_mirror + cctns_mirror (live in Phase 3)
alembic/ migrations
frontend/ Next.js 15 + Tailwind v4 (static export)
tests/ unit, integration, e2e/
spec/ source of truth (this build's product spec)
```

---

## Rules AI Agents Follow

Full rules in `harness/rules/ai-agents.md`. Summary:

- Read the full spec before writing any code
- Never skip a phase; commit every logical unit
- Tests run against the real LLM/API using keys from `.env` — stubbed runs do not count as passing
- Each phase is tested by the human before the next phase starts
- The build record is git history + the PR + the per-phase test-handoffs

---

## FAQ

**What if I already have a stack in mind?**
State it in the idea: `/zero-shot-build [idea] — use Python + FastAPI + PostgreSQL`. Stack choices are binding.

**What if something breaks?**
Run `/zero-shot-fix [what's broken]` — qa-auditor classifies the problem (SPEC vs CODE), the right generator fixes it, qa-auditor re-gates.
## What's a stub

**What if spec and code drift?**
Run `/zero-shot-sync` — qa-auditor classifies each divergence, generators fix, spec wins.
The UI panels for follow-up questions, conversation history, role-based
filtering, and switching to live CCTNS are **labelled stubs** saying
"Coming in Phase 2/3". They are deliberately not wired and never look
broken.
38 changes: 38 additions & 0 deletions alembic.ini
Original file line number Diff line number Diff line change
@@ -0,0 +1,38 @@
[alembic]
script_location = alembic
sqlalchemy.url = sqlite:///./data/agent.db
prepend_sys_path = src

[loggers]
keys = root,sqlalchemy,alembic

[handlers]
keys = console

[formatters]
keys = generic

[logger_root]
level = WARN
handlers = console
qualname =

[logger_sqlalchemy]
level = WARN
handlers =
qualname = sqlalchemy.engine

[logger_alembic]
level = INFO
handlers =
qualname = alembic

[handler_console]
class = StreamHandler
args = (sys.stderr,)
level = NOTSET
formatter = generic

[formatter_generic]
format = %(levelname)-5.5s [%(name)s] %(message)s
datefmt = %H:%M:%S
Loading