Cl3mi · Cl3mi · Jul 24, 2026 · Jul 24, 2026 · Jul 24, 2026 · Jul 24, 2026
diff --git a/bun.lock b/bun.lock
diff --git a/docs/superpowers/plans/2026-07-23-partner-eval-comparison.md b/docs/superpowers/plans/2026-07-23-partner-eval-comparison.md
diff --git a/eval-dist/.env.example b/eval-dist/.env.example
@@ -1,7 +1,5 @@
-# Bearer token the client uses to authenticate against /api/external/*.
-# Pick a long random string; share it out-of-band with the eval operator
-# so both sides know they are talking to the right deployment.
-EVAL_API_KEY=replace-with-long-random-string
+# Shared secret. Must match the Authorization: Bearer token your app sends.
+EVAL_API_KEY=change-me-to-a-long-random-string
 
-# Judge model. Default fits in ~6GB VRAM (GTX 1660 Super-class).
-OLLAMA_JUDGE_MODEL=qwen2.5:7b
+# Label written into the report so the two systems are distinguishable.
+SYSTEM_LABEL=partner-org
diff --git a/eval-dist/Dockerfile b/eval-dist/Dockerfile
@@ -67,12 +67,16 @@ ENV EVAL_SERVER_MODE=1
 COPY --from=deps  /app/node_modules ./node_modules
 COPY --from=build /app/build        ./build
 COPY --from=build /app/package.json ./package.json
+COPY --from=build /app/eval-dist/seed/questions.json ./seed/questions.json
+RUN mkdir -p /app/data
 
 # Drop any leftover dev artifacts and verify no .ts / .map remain
 RUN find . -name '*.ts' -not -path './node_modules/*' -delete \
  && find . -name '*.map' -not -path './node_modules/*' -delete \
  && find . -name 'tsconfig*.json' -not -path './node_modules/*' -delete \
  && find . -name 'svelte.config*' -not -path './node_modules/*' -delete
 
+RUN node -e "require('better-sqlite3'); console.log('better-sqlite3 OK')"
+
 EXPOSE 3000
 CMD ["node", "build/index.js"]
diff --git a/eval-dist/README.md b/eval-dist/README.md
@@ -1,217 +1,98 @@
-# web-rag eval-server
+# web-rag eval collector
 
-A self-contained Docker bundle that scores **your** RAG system on the same
-test queries and with the same LLM-as-judge methodology used by the
-operator's reference system. You run it locally, point your retrieval +
-generation code at it, and send back the resulting JSON report. The
-operator runs the equivalent evaluation on their side and compares the two
-reports.
-
-The container only exposes four HTTP endpoints under `/api/external/*`.
-There is no UI, no shell, and no access to the operator's source code,
-prompts, or pipelines.
-
----
+A single, self-contained container that scores **your** RAG system on a shared
+set of website questions using the same retrieval metrics as the operator's
+reference system. It needs **no GPU, no model download, and no database of its
+own**. You run it locally, POST each answered question to it, then export one
+`report.json` and send it back. The operator runs the LLM-as-judge over both
+reports and produces the side-by-side comparison.
 
 ## 1. Prerequisites
-
 - Docker 24+ and `docker compose`
-- ~12 GB free disk (image + judge model weights)
-- NVIDIA GPU recommended (CPU works but is ~10× slower for the judge step)
 - A `read:packages` PAT for `ghcr.io` — see [`LICENSE`](./LICENSE)
 
----
-
-## 2. Pull and start
-
+## 2. Start
 ```bash
-# Authenticate against GHCR using the credentials in LICENSE
 echo "$GHCR_PAT" | docker login ghcr.io -u "$GHCR_USERNAME" --password-stdin
-
-# Configure
-cp .env.example .env
-# → edit .env and set EVAL_API_KEY to a long random string
-
-# Boot (CPU)
+cp .env.example .env          # set EVAL_API_KEY and SYSTEM_LABEL
 docker compose up -d
-# …or with NVIDIA GPU
-docker compose --profile gpu up -d
-
-# Wait for the judge model to download (first run only, ~5 min)
-docker compose logs -f ollama
 ```
-
-The eval-server listens on `127.0.0.1:3000`. Verify it is up:
-
+The collector listens on `127.0.0.1:3000`. Verify:
 ```bash
 curl -H "Authorization: Bearer $EVAL_API_KEY" \
      http://127.0.0.1:3000/api/external/queries | jq '.queries | length'
 ```
 
----
-
-## 3. The four endpoints
-
-All endpoints require `Authorization: Bearer $EVAL_API_KEY`.
+## 3. Endpoints
+All require `Authorization: Bearer $EVAL_API_KEY`.
 
 ### `GET /api/external/queries`
-Returns the shared test query set. Every query carries the ground-truth
-URLs the operator's documents resolve to. **You must ingest the same URLs
-into your own system before evaluating** so retrieval can be compared.
-
+Returns the shared question set + ground-truth URLs. Ingest the same source
+URLs into your system so retrieval can be compared.
 ```json
-{ "queries": [
-    { "id": "…", "query": "…", "category": "…", "difficulty": "easy",
-      "expectedUrls": ["https://…", "https://…"] }
-]}
+{ "questionSetVersion": "v1",
+  "queries": [{ "id": "q1", "query": "…", "category": "…",
+                "difficulty": "easy", "expectedUrls": ["https://…"] }] }
 ```
 
-### `POST /api/external/evaluate`
-Computes retrieval metrics for one query from the document URLs your
-retriever returned.
-
+### `POST /api/external/run`
+Call once per answered question, at answer-generation time.
 ```json
 {
-  "sessionId":     "client-run-2026-06-29",
-  "queryId":       "<id from /queries>",
-  "query":         "<text from /queries>",
-  "expectedUrls":  ["https://…"],
-  "retrievedUrls": ["https://…", "https://…", "…"],
-  "k":             10
+  "sessionId":     "run-2026-07-23",
+  "systemLabel":   "partner-org",
+  "queryId":       "q1",
+  "answer":        "<your model's answer>",
+  "context":       "<context your retriever passed to the LLM>",
+  "retrievedUrls": ["https://…", "…"],
+  "latency":       { "retrievalMs": 120, "generationMs": 800 }
 }
 ```
-Returns Recall@5/10, MRR, nDCG, Hit@1/5/10. URLs are normalised
-(lowercased, trailing slash stripped) before matching, so trivial
-differences are fine.
-
-### `POST /api/external/judge`
-Runs the LLM-as-judge (3× with majority vote) over your generated answer.
-
-```json
-{
-  "sessionId":  "client-run-2026-06-29",
-  "queryId":    "<id from /queries>",
-  "query":      "<question>",
-  "context":    "<context your retriever passed to the LLM>",
-  "answer":     "<your model's answer>",
-  "judgeRuns":  3,
-  "recallAtK":  0.8
-}
-```
-Returns groundedness, completeness, correctness, hallucination,
-answerable-from-context, judgeMean, and an overall failureType
-(`normal | generation_failure | retrieval_failure | robust_generation |
-hallucination_failure`).
+The container looks up the ground-truth URLs by `queryId`, computes
+Recall@5/10, MRR, nDCG, Hit@1/5/10, records latency, and returns the metrics.
 
 ### `GET /api/external/report?sessionId=…`
-Downloads the full JSON report (per-query rows + aggregate summary).
-Send this file back to the operator.
-
----
+Downloads the full `report.json` (per-query rows + aggregate summary incl.
+latency p95). Send this file back to the operator.
 
 ## 4. Minimal Python integration
-
-Add this single module to your project. It is the only client code you
-have to write.
-
 ```python
-# eval_client.py
-import os, requests, uuid
-
-BASE = os.environ["EVAL_BASE", "http://127.0.0.1:3000"]
+import os, requests
+BASE = os.environ.get("EVAL_BASE", "http://127.0.0.1:3000")
 KEY  = os.environ["EVAL_API_KEY"]
 HDR  = {"Authorization": f"Bearer {KEY}"}
-SESSION = os.environ.get("EVAL_SESSION", f"client-{uuid.uuid4().hex[:8]}")
+SESSION = os.environ.get("EVAL_SESSION", "run-1")
+LABEL   = os.environ.get("SYSTEM_LABEL", "partner-org")
 
 def queries():
     r = requests.get(f"{BASE}/api/external/queries", headers=HDR, timeout=30)
-    r.raise_for_status()
-    return r.json()["queries"]
-
-def submit_retrieval(q, retrieved_urls, k=10):
-    r = requests.post(f"{BASE}/api/external/evaluate", headers=HDR, json={
-        "sessionId": SESSION,
-        "queryId": q["id"],
-        "query": q["query"],
-        "expectedUrls": q["expectedUrls"],
-        "retrievedUrls": retrieved_urls,
-        "k": k,
-    }, timeout=30)
-    r.raise_for_status()
-    return r.json()["metrics"]
-
-def submit_judge(q, context, answer, recall_at_k):
-    r = requests.post(f"{BASE}/api/external/judge", headers=HDR, json={
-        "sessionId": SESSION,
-        "queryId": q["id"],
-        "query": q["query"],
-        "context": context,
-        "answer": answer,
-        "recallAtK": recall_at_k,
-    }, timeout=600)
-    r.raise_for_status()
-    return r.json()["scores"]
-
-def fetch_report():
-    r = requests.get(f"{BASE}/api/external/report",
-                     headers=HDR, params={"sessionId": SESSION}, timeout=60)
-    r.raise_for_status()
-    return r.json()
-```
+    r.raise_for_status(); return r.json()["queries"]
 
-Wire it into your existing run loop — typically a 5-line patch:
-
-```python
-from eval_client import queries, submit_retrieval, submit_judge, fetch_report
+def submit(q, answer, context, retrieved_urls, retrieval_ms, generation_ms):
+    r = requests.post(f"{BASE}/api/external/run", headers=HDR, json={
+        "sessionId": SESSION, "systemLabel": LABEL, "queryId": q["id"],
+        "answer": answer, "context": context, "retrievedUrls": retrieved_urls,
+        "latency": {"retrievalMs": retrieval_ms, "generationMs": generation_ms},
+    }, timeout=60)
+    r.raise_for_status(); return r.json()
 
 for q in queries():
-    hits   = my_retriever(q["query"], top_k=10)            # <— your code
-    ctx    = "\n\n".join(h["text"] for h in hits)
-    answer = my_generator(q["query"], ctx)                 # <— your code
-
-    metrics = submit_retrieval(q, [h["url"] for h in hits])
-    scores  = submit_judge(q, ctx, answer, recall_at_k=metrics["recallAt10"])
-
-report = fetch_report()
-open("report.json", "w").write(__import__("json").dumps(report, indent=2))
+    import time
+    t0 = time.time(); hits = my_retriever(q["query"], top_k=10); t1 = time.time()
+    ctx = "\n\n".join(h["text"] for h in hits)
+    answer = my_generator(q["query"], ctx); t2 = time.time()
+    submit(q, answer, ctx, [h["url"] for h in hits],
+           int((t1-t0)*1000), int((t2-t1)*1000))
 ```
 
-That is the full client-side surface — no other code in your project needs
-to change.
-
----
-
-## 5. Sending the report back
-
-When the loop finishes:
-
+Fetch and send the report:
 ```bash
 curl -H "Authorization: Bearer $EVAL_API_KEY" \
      "http://127.0.0.1:3000/api/external/report?sessionId=$EVAL_SESSION" \
      -o report.json
 ```
 
-Send `report.json` to the operator (email, SFTP, signed S3 URL — whatever
-was agreed). The operator runs the identical evaluation against the same
-queries on their reference system and produces a side-by-side comparison.
-
----
-
-## 6. Troubleshooting
-
-| Symptom | Cause | Fix |
-|---|---|---|
-| `401 Unauthorized` | Wrong / missing bearer token | Re-export `EVAL_API_KEY` in your shell; it must match `.env` |
-| `502` from `/api/external/judge` | Judge model not yet downloaded | `docker compose logs ollama` and wait for `Ollama ready` |
-| All Recall@K = 0 | URL mismatch | Confirm `expectedUrls` and your `retrievedUrls` use the same canonical form (your system must ingest the same source URLs) |
-| Slow judge calls | Running CPU profile | Use `docker compose --profile gpu up` if you have NVIDIA + nvidia-container-toolkit |
-
----
-
-## 7. Tear-down
-
+## 5. Tear-down
 ```bash
-docker compose down -v   # stops containers AND deletes the eval Postgres
+docker compose down -v   # stops the container AND deletes the SQLite scratch data
 ```
-
-This wipes all locally-stored results. Do this after sending the report.