feat: RAG Q&A, per-compartment templates, eval harness, doctor upgrades

AGENTS.md

@@ -31,12 +31,14 @@ Dendrite receives raw captures (Telegram, webhook, CLI), classifies them with an
 ```bash
 npm run build && npm test          # 31 checks — run before/after changes
 
-dendrite doctor [--stats]
+dendrite doctor [--stats] [--json]   # + embedding coverage, queue health, dangling links
 dendrite ingest "text" [--dry-run]
+dendrite ask "question"              # RAG answer over the vault, with [[wikilink]] citations
 dendrite sort [--dry-run]            # inbox + unfiled imports → brain/
 dendrite repair [--dry-run]          # split junk-drawer notes
 dendrite migrate [--dry-run]         # upgrade frontmatter schema
 dendrite embed [--force]             # build semantic vectors (hybrid search)
+dendrite eval                        # classifier accuracy on golden dataset (dry-run)
 dendrite remove --last
 dendrite reindex
 dendrite mcp
@@ -45,7 +47,7 @@ dendrite serve
 
 ### Telegram (`dendrite serve`)
 
-`/start` `/help` `/inbox` `/recent` `/compartments` `/sort` `/undo`
+`/start` `/help` `/inbox` `/recent` `/compartments` `/ask` `/sort` `/undo`
 
 - **`/sort`** — dry-run preview with ✅ Confirm / ❌ Cancel buttons (10 min expiry).
 
@@ -55,6 +57,7 @@ dendrite serve
 |------|----------|
 | `describe_schema` | **Call first** — compartments + frontmatter contract |
 | `search_vault` | Keyword + hybrid semantic search (if embeddings enabled) |
+| `answer_question` | RAG answer from the vault with `[[wikilink]]` citations |
 | `read_note` | Read note by vault-relative path |
 | `vault_catalog` | Full index snapshot |
 | `list_compartments` | Compartment list + counts |
@@ -101,6 +104,7 @@ Archives: `brain/_dendrite/imported/` (sort), `brain/_dendrite/repaired/` (repai
 3. **`create_new` honored** on splits — no FTS junk-drawer appends.
 4. **Near-dup guard:** title keywords must appear in new text to append.
 5. **Hybrid search:** when `index.embeddings.enabled` + vectors exist, crosslink and MCP search blend FTS + cosine similarity (`hybrid_weight`).
+6. **Per-compartment templates:** optional `templates/<compartment>.md` files customize frontmatter + body of newly created notes (dynamic core frontmatter still wins).
 
 ## Config knobs
 
@@ -136,7 +140,7 @@ Covers: classification, laundry-list, multi-split, idempotency, sort/migrate/rep
 
 ## Not yet built (v0.3+)
 
-Per-compartment templates, email input, MCP `capture_note`, merge-back correction. See [ROADMAP.md](ROADMAP.md).
+Email input, MCP `capture_note`, merge-back correction. See [ROADMAP.md](ROADMAP.md) and [SPEC.md](SPEC.md).
 
 ## Cursor Cloud specific instructions
 

CHANGELOG.md

@@ -1,5 +1,14 @@
 # Changelog
 
+## 0.3.0 — Unreleased
+
+### Features
+
+- RAG question-answering: `dendrite ask "..."` answers from vault notes only via hybrid FTS + embeddings retrieval, cites sources as `[[wikilinks]]`, is read-only, and refuses (without calling the LLM) when nothing relevant is found. Also `/ask` (Telegram) and `answer_question` (MCP). Flags: `--compartment`, `-k`, `--json`.
+- Per-compartment templates: optional `templates/<compartment>.md` customize frontmatter + body of newly created notes with `{{variable}}` placeholders; existing notes are never rewritten. Config: `templates.enabled`, `templates.dir`.
+- `dendrite doctor` upgrades: reports embedding coverage, ingest-queue health (pending/processing/dead), and dangling `[[wikilink]]` count; new `--json` flag exits non-zero on critical issues.
+- Classification eval harness: `dendrite eval` runs a golden `eval/dataset.jsonl` through the classifier in dry-run and reports routing accuracy + per-compartment breakdown. Flags: `--limit`, `--min`, `--json`, `--dataset`.
+
 ## 0.1.0 — 2026-07-07
 
 First public beta.

DOCS.md

@@ -100,6 +100,39 @@ dendrite reindex
 
 ---
 
+## Asking your vault (RAG)
+
+`dendrite ask` answers a natural-language question using **only** the notes already
+in your vault. It retrieves relevant notes with the existing hybrid FTS + embeddings
+search, feeds them to the LLM as context, and cites the notes it used inline as
+`[[wikilinks]]`.
+
+```bash
+dendrite ask "what did I learn about agent orchestration?"
+```
+
+It is **read-only** — it never writes to the vault. When nothing relevant is
+retrieved, it **refuses without calling the LLM** and replies that there is no
+matching note.
+
+### Flags
+
+| Flag | Description |
+|------|-------------|
+| `-c, --config <path>` | Config file path |
+| `--compartment <name>` | Restrict retrieval to one compartment |
+| `-k <n>` | Number of notes to retrieve (default from config) |
+| `--json` | Machine-readable output: `{ question, answer, sources[], usedNotes, refused }` |
+
+Like classification, `ask` reuses the LLM provider config and needs a reachable LLM.
+
+### Telegram and MCP
+
+- **Telegram:** `/ask <question>`
+- **MCP:** `answer_question({ question, compartment?, k? })`
+
+---
+
 ## Configuration
 
 Main file: `dendrite.config.yaml` (copy from `dendrite.config.example.yaml`).
@@ -184,6 +217,28 @@ Then build vectors:
 dendrite embed
 ```
 
+### Retrieval (RAG)
+
+Controls how `dendrite ask` retrieves context from the vault:
+
+```yaml
+retrieval:
+  k: 8                 # notes retrieved for `dendrite ask`
+  max_context_chars: 6000
+  min_score: 0
+```
+
+### Templates
+
+```yaml
+templates:
+  enabled: true
+  dir: templates
+```
+
+No templates ship by default, so behavior is unchanged until you add a
+`templates/<compartment>.md` file. See [Per-compartment templates](#per-compartment-templates).
+
 ---
 
 ## Brain compartments
@@ -311,10 +366,12 @@ curl http://localhost:8787/health
 | Command | Description |
 |---------|-------------|
 | `dendrite init` | Interactive setup wizard |
-| `dendrite doctor [--stats]` | Health check + metrics |
+| `dendrite doctor [--stats] [--json]` | Health check + metrics (embedding coverage, queue health, dangling links) |
 | `dendrite ingest "text"` | Classify and write one capture |
 | `dendrite ingest --dry-run "text"` | Preview without writing |
 | `dendrite ingest --file audio.ogg` | Transcribe + ingest audio |
+| `dendrite ask "question"` | RAG answer from your vault, with citations |
+| `dendrite eval` | Classification accuracy on a golden dataset |
 | `dendrite serve` | Run daemon |
 | `dendrite mcp` | MCP read-server (stdio) |
 | `dendrite reindex` | Rebuild search index from vault |
@@ -383,6 +440,64 @@ TIL agent orchestration uses a DAG not a chain. Related: [[related-note]].
 
 ---
 
+## Per-compartment templates
+
+Drop a `templates/<compartment>.md` file to customize the layout and frontmatter of
+**newly created** notes in that compartment (e.g. `templates/reads.md`,
+`templates/tasks.md`). Existing notes are never rewritten — templates only affect the
+first write that creates a note.
+
+A template may contain optional YAML frontmatter (extra static fields) plus a Markdown
+body with `{{variable}}` placeholders.
+
+### Available variables
+
+| Variable | Value |
+|----------|-------|
+| `{{title}}` | Note title |
+| `{{summary}}` | One-line summary |
+| `{{date}}` | Capture date |
+| `{{source}}` | Capture source (e.g. `telegram-text`) |
+| `{{compartment}}` | Target compartment |
+| `{{entities}}` | Extracted entities (comma-joined) |
+| `{{tags}}` | Tags (comma-joined) |
+| `{{links}}` | Cross-links (comma-joined) |
+| `{{capture}}` | The timestamped capture section |
+
+If a template omits `{{capture}}`, the capture section is appended after the template
+body. Dynamic core frontmatter (`compartment`, `title`, `created`, `updated`, `source`,
+`confidence`, `entities`, `tags`, `links`, `dendrite_version`, `summary`) always wins
+over template static fields.
+
+### Example — `templates/reads.md`
+
+```markdown
+---
+rating:
+status: to-read
+---
+
+# {{title}}
+
+> {{summary}}
+
+**Source:** {{source}}
+
+{{capture}}
+```
+
+Enable/locate templates in config:
+
+```yaml
+templates:
+  enabled: true
+  dir: templates
+```
+
+No templates ship by default, so behavior is unchanged until you add a template file.
+
+---
+
 ## Vault maintenance
 
 ### Sort unfiled notes
@@ -454,6 +569,7 @@ Register in Cursor / Claude Code / Hermes:
 |------|---------|
 | `describe_schema` | Compartments + frontmatter contract |
 | `search_vault` | Keyword + hybrid semantic search |
+| `answer_question` | RAG answer from the vault with `[[wikilink]]` citations |
 | `read_note` | Read note by path |
 | `vault_catalog` | Full index grouped by compartment |
 | `list_compartments` | Compartment list + counts |

README.md

@@ -176,6 +176,7 @@ flowchart LR
 | **Correction loop** | Telegram inline keyboard corrections feed few-shot examples into future classifications. |
 | **Idempotent ingest** | Same `dump.id` twice → no-op. Safe for webhook retries. |
 | **Soft undo** | `dendrite remove --last` or Telegram `/undo` — section remove or move to inbox. |
+| **Per-compartment templates** | Optional `templates/<compartment>.md` customize frontmatter + body of newly created notes. |
 
 ### Inputs
 
@@ -195,13 +196,16 @@ flowchart LR
 | `dendrite migrate` | Upgrade note frontmatter to current `dendrite_version`. |
 | `dendrite embed` | Build embedding vectors for hybrid semantic search. |
 | `dendrite backfill` | Classify vault-root / scratch notes into brain folders. |
+| `dendrite ask` | RAG question-answering over the vault, with `[[wikilink]]` citations. |
+| `dendrite eval` | Run a golden labeled dataset through the classifier to measure routing accuracy. |
 
 ### Agent interface (MCP)
 
 | Tool | Description |
 |------|-------------|
 | `describe_schema` | Compartments + frontmatter contract — call this first. |
 | `search_vault` | Keyword + hybrid semantic search over the index. |
+| `answer_question` | RAG answer from your vault with `[[wikilink]]` citations. |
 | `read_note` | Read any note by vault-relative path. |
 | `vault_catalog` | Full index snapshot grouped by compartment. |
 | `get_capture_siblings` | Reconstruct a multi-segment capture by `split_group`. |
@@ -216,6 +220,8 @@ dendrite init              # interactive setup wizard
 dendrite doctor [--stats]  # health check + local metrics
 dendrite ingest "text"     # classify + write
 dendrite ingest --dry-run  # preview without writing
+dendrite ask "question"    # RAG answer from your vault, with citations
+dendrite eval              # classification accuracy on a golden dataset
 dendrite serve             # daemon: telegram + webhook + crons
 dendrite mcp               # MCP read-server (stdio)
 dendrite reindex           # rebuild SQLite index from vault
@@ -229,7 +235,7 @@ dendrite backfill          # classify vault-root imports only
 dendrite pattern-scan      # weekly digest now
 ```
 
-Telegram: `/help` `/inbox` `/recent` `/compartments` `/sort` `/undo`
+Telegram: `/help` `/inbox` `/recent` `/compartments` `/ask` `/sort` `/undo`
 
 ## Configuration
 

eval/dataset.jsonl

@@ -0,0 +1,22 @@
+# Dendrite classification golden set. Lines starting with '#' and blank lines are ignored.
+# Routing case: {"text": "...", "expected": "<compartment>"}  -> pass if any segment matches.
+# Split case:   {"text": "...", "expected_min_segments": <n>} -> pass if results.length >= n.
+{"text": "My sister Anna works as a nurse at a hospital in Munich.", "expected": "memories", "note": "durable personal/biographical fact about a relative"}
+{"text": "I'm allergic to penicillin and my doctor's office is on Oak Street.", "expected": "memories", "note": "durable personal health fact"}
+{"text": "TIL that in Rust the borrow checker prevents data races at compile time.", "expected": "learnings", "note": "TIL / new concept learned"}
+{"text": "Learned a neat technique today: you can debounce input handlers with a trailing setTimeout to avoid extra renders.", "expected": "learnings", "note": "technique learned"}
+{"text": "Need to book a dentist appointment before Friday.", "expected": "tasks", "note": "clear to-do with deadline"}
+{"text": "Remember to renew my passport, it expires next month.", "expected": "tasks", "note": "follow-up action item"}
+{"text": "Had coffee, walked the dog, and felt kind of tired this afternoon.", "expected": "journal", "note": "ephemeral mundane daily log"}
+{"text": "What if we built an AI code reviewer that leaves inline comments on diffs automatically?", "expected": "ideas", "note": "unformed product idea"}
+{"text": "Read a great article about the MCP protocol; key takeaway is that tools are exposed as JSON schemas.", "expected": "reads", "note": "article consumed with takeaway"}
+{"text": "Finished the book Thinking Fast and Slow this weekend, loved the section on cognitive biases.", "expected": "reads", "note": "book consumed"}
+{"text": "I realised I communicate way better in writing than in live meetings, and I should lean into async updates.", "expected": "reflections", "note": "personal growth insight"}
+{"text": "Hit a nasty bug in the auth refactor today, the refresh token rotation breaks when the clock skews.", "expected": "projects", "note": "per-project progress/blocker"}
+{"text": "Honestly the new deploy process is infuriating, every single release feels like pulling teeth and nobody documents anything.", "expected": "rants", "note": "raw emotional thought stream"}
+{"text": "qqqxxzz asdf lorem zzz ttttt gibber", "expected": "inbox", "note": "gibberish, low confidence -> inbox"}
+{"text": "asdkfj 9282 ;;; ---- wut", "expected": "inbox", "note": "unparseable noise -> inbox"}
+{"text": "My parents live in Germany. Need to book a dentist before Friday. TIL Rust prevents data races.", "expected_min_segments": 3, "note": "memory + task + learning -> 3 segments"}
+{"text": "I read a fascinating paper on transformers today. Also, remind me to email the landlord about the leak.", "expected_min_segments": 2, "note": "reads + task -> 2 segments"}
+{"text": "My grandmother turns 90 next year. I want to prototype a birthday reminder app. Also I felt anxious all morning.", "expected_min_segments": 3, "note": "memory + idea + journal -> 3 segments"}
+{"text": "Learned that Postgres uses MVCC for isolation. Need to add an index on the orders table. Feeling great about the sprint.", "expected_min_segments": 2, "note": "learning + task + journal -> at least 2 segments"}

scripts/ci-smoke.mjs

@@ -4,7 +4,7 @@
  * Full LLM integration: npm test (local or with GitHub secrets).
  */
 import { spawn } from "node:child_process";
-import { existsSync } from "node:fs";
+import { existsSync, readFileSync } from "node:fs";
 import { join } from "node:path";
 import { fileURLToPath } from "node:url";
 
@@ -106,6 +106,94 @@ async function main() {
     else fail(label, `exit ${code}`);
   }
 
+  console.log("\n=== CONFIG DEFAULTS ===");
+  try {
+    const { loadConfig } = await import(join(ROOT, "dist", "config.js"));
+    const { config } = loadConfig();
+    const k = config.retrieval?.k;
+    const templatesEnabled = config.templates?.enabled;
+    const maxCtx = config.retrieval?.max_context_chars;
+    if (k === 8 && templatesEnabled === true && maxCtx === 6000)
+      pass("config: retrieval+templates defaults");
+    else
+      fail(
+        "config: retrieval+templates defaults",
+        `retrieval.k=${k} templates.enabled=${templatesEnabled} retrieval.max_context_chars=${maxCtx}`,
+      );
+  } catch (e) {
+    fail("config defaults", e.message);
+  }
+
+  console.log("\n=== TEMPLATE RENDER (unit) ===");
+  try {
+    const { renderVars, renderTemplateBody } = await import(
+      join(ROOT, "dist", "pipeline", "template.js")
+    );
+    const vars = {
+      title: "T",
+      summary: "S",
+      source: "cli",
+      date: "2026-01-01 00:00",
+      compartment: "reads",
+      entities: "a, b",
+      tags: "x",
+      links: "",
+      capture: "## SECTION\nbody",
+    };
+
+    const a = renderVars("# {{title}} — {{summary}}", vars);
+    if (a === "# T — S") pass("template render A");
+    else fail("template render A", `got ${JSON.stringify(a)}`);
+
+    const b = renderTemplateBody({ frontmatter: {}, body: "# {{title}}", hasCapture: false }, vars);
+    if (b.includes("# T") && b.includes("## SECTION")) pass("template render B");
+    else fail("template render B", `got ${JSON.stringify(b)}`);
+
+    const c = renderTemplateBody({ frontmatter: {}, body: "{{capture}}", hasCapture: true }, vars);
+    if (c === vars.capture) pass("template render C");
+    else fail("template render C", `got ${JSON.stringify(c)}`);
+  } catch (e) {
+    fail("template render", e.message);
+  }
+
+  console.log("\n=== EVAL DATASET ===");
+  try {
+    const datasetPath = join(ROOT, "eval", "dataset.jsonl");
+    if (!existsSync(datasetPath)) {
+      fail("eval dataset present", "eval/dataset.jsonl missing");
+    } else {
+      const raw = readFileSync(datasetPath, "utf8");
+      const lines = raw
+        .split("\n")
+        .map((l) => l.trim())
+        .filter((l) => l.length > 0 && !l.startsWith("#"));
+      let badLine = null;
+      let n = 0;
+      for (const line of lines) {
+        let obj;
+        try {
+          obj = JSON.parse(line);
+        } catch (err) {
+          badLine = `invalid JSON: ${line.slice(0, 80)}`;
+          break;
+        }
+        const hasExpected =
+          typeof obj.expected === "string" || typeof obj.expected_min_segments === "number";
+        const hasText = typeof obj.text === "string" && obj.text.length > 0;
+        if (!hasExpected || !hasText) {
+          badLine = `missing fields: ${line.slice(0, 80)}`;
+          break;
+        }
+        n++;
+      }
+      if (badLine) fail("eval dataset valid", badLine);
+      else if (n >= 10) pass("eval dataset valid", `${n} cases`);
+      else fail("eval dataset valid", `only ${n} cases`);
+    }
+  } catch (e) {
+    fail("eval dataset", e.message);
+  }
+
   return summary();
 }