feat: RAG Q&A, per-compartment templates, eval harness, doctor upgrades - #3
Merged
Conversation
Co-authored-by: Moses Man <mosesman831@users.noreply.github.com>
Co-authored-by: Moses Man <mosesman831@users.noreply.github.com>
Co-authored-by: Moses Man <mosesman831@users.noreply.github.com>
…ng links Co-authored-by: Moses Man <mosesman831@users.noreply.github.com>
Co-authored-by: Moses Man <mosesman831@users.noreply.github.com>
Co-authored-by: Moses Man <mosesman831@users.noreply.github.com>
Co-authored-by: Moses Man <mosesman831@users.noreply.github.com>
Co-authored-by: Moses Man <mosesman831@users.noreply.github.com>
There was a problem hiding this comment.
Pull request overview
This PR extends Dendrite’s ingestion+vault ecosystem with new read-only retrieval (RAG) Q&A and evaluation tooling, while also adding optional per-compartment note templates and expanding doctor observability.
Changes:
- Add vault-wide Q&A (
dendrite ask, Telegram/ask, MCPanswer_question) powered by existing hybrid search and bounded context assembly. - Add per-compartment templates (
templates/<compartment>.md) that can shape frontmatter/body for newly created notes. - Add a golden eval harness (
dendrite eval) and upgradedoctorwith--jsonoutput + additional health metrics.
Reviewed changes
Copilot reviewed 20 out of 20 changed files in this pull request and generated 6 comments.
Show a summary per file
| File | Description |
|---|---|
| templates/tasks.md | Adds an example Tasks note template. |
| templates/reads.md | Adds an example Reads note template. |
| src/pipeline/write.ts | Applies optional per-compartment templates on first-write note creation. |
| src/pipeline/template.ts | Implements template loading and placeholder/frontmatter rendering helpers. |
| src/pipeline/pipeline.ts | Threads configDir through to note writing for template resolution. |
| src/pipeline/index.ts | Adds ingest-queue status aggregation helper for doctor. |
| src/pipeline/answer.ts | Implements vault RAG answering: retrieval, context packing, and chat call. |
| src/mcp/server.ts | Exposes answer_question tool via MCP. |
| src/inputs/telegram.ts | Adds Telegram /ask command using the RAG implementation. |
| src/config.ts | Adds retrieval and templates config blocks with defaults. |
| src/commands/eval.ts | Adds dendrite eval golden-dataset runner and reporting. |
| src/commands/doctor.ts | Adds --json output and new health metrics (coverage/queue/dangling links). |
| src/commands/ask.ts | Adds dendrite ask CLI command. |
| src/cli.ts | Wires up new ask and eval CLI commands and doctor --json. |
| scripts/ci-smoke.mjs | Extends keyless smoke checks for config defaults, templates, dataset validity. |
| README.md | Documents new commands and MCP tool at a high level. |
| eval/dataset.jsonl | Adds labeled golden cases for eval harness. |
| DOCS.md | Adds documentation for ask, retrieval config, and templates. |
| CHANGELOG.md | Notes new features under an unreleased 0.3.0 entry. |
| AGENTS.md | Updates agent guidance to include the new commands/tools and templates. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+101
to
+111
| for (const hit of hits) { | ||
| if (used >= budget) break; | ||
| const body = readNoteBody(vaultPath, hit.path) || hit.snippet; | ||
| const remaining = budget - used; | ||
| const excerpt = body.length > remaining ? body.slice(0, remaining) + "…" : body; | ||
| blocks.push( | ||
| `### ${wikilink(slugOf(hit.path))} — ${hit.title}\n(path: ${hit.path})\n${excerpt}`, | ||
| ); | ||
| used += excerpt.length; | ||
| usedNotes++; | ||
| } |
Comment on lines
+19
to
+22
| const result = await answerQuestion(index, config.vault.path, q, config, llm, { | ||
| compartment: opts.compartment, | ||
| k: opts.k ? Number(opts.k) : undefined, | ||
| }); |
| server.tool( | ||
| "answer_question", | ||
| "Answer a natural-language question using ONLY vault notes, with [[wikilink]] citations. Read-only RAG; refuses when nothing relevant is found.", | ||
| { question: z.string(), compartment: z.string().optional(), k: z.number().optional() }, |
| } | ||
|
|
||
| const CAPTURE_RE = /\{\{\s*capture\s*\}\}/; | ||
| const VAR_RE = /\{\{\s*([\w.]+)\s*\}\}/g; |
| }); | ||
| } | ||
|
|
||
| /** Deep-render string values inside template frontmatter. */ |
Comment on lines
+239
to
+240
| No templates ship by default, so behavior is unchanged until you add a | ||
| `templates/<compartment>.md` file. See [Per-compartment templates](#per-compartment-templates). |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Implements four high-value P1 features from SPEC.md, built in parallel by subagents and integrated + tested end-to-end against a local OpenAI-compatible LLM (Ollama
llama3.2:3b).Features
1. Vault-wide RAG Q&A (§5.3)
dendrite ask "question"— answers using only vaultbrain/notes, retrieved via existing hybrid FTS+embeddings search, with inline[[wikilink]]citations. Read-only; refuses (no LLM call) when nothing is retrieved.--compartment,-k,--json,-c.answer_question({ question, compartment?, k? })and Telegram/ask.src/pipeline/answer.tsfilters retrieval tobrain/so vault scaffolding (e.g. starter README) can't pollute context.2. Per-compartment templates (§6.1)
templates/<compartment>.mdcustomize frontmatter + body of newly created notes. Variables:{{title}} {{summary}} {{date}} {{source}} {{compartment}} {{entities}} {{tags}} {{links}} {{capture}}.{{capture}}is auto-appended if omitted. Shipstemplates/reads.mdandtemplates/tasks.md.templates.enableddefault true, no templates ship in a user's dir).3.
doctorupgrades (§7.3)--jsonmachine-readable health object; added embedding coverage %, ingest-queue health (pending/processing/dead), and dangling[[wikilink]]count.4. Golden eval harness (§7.1)
dendrite evalrunseval/dataset.jsonl(19 labeled cases) through the classifier in dry-run and reports routing accuracy + per-compartment breakdown.--limit,--min <ratio>(gate),--json,--dataset.New config (all optional, safe defaults):
retrieval.{k,max_context_chars,min_score},templates.{enabled,dir}.Testing
npm run buildnpm run test:ci— keyless smoke suite, now includes new checks: config defaults, template render (3 unit cases), 19-case eval dataset validationdendrite ask— grounded answer + citation, refusal path,--jsondendrite doctor --jsonand--stats(human) — new metricsdendrite eval --limit 10 --min 0.5→ 80% (8/10) with the small local model; the 2 misses are weak-model routing drift the gate is designed to surfacereadscapture renderedtemplates/reads.md)npm test— 29/31 with the small local model; both failures (task→inbox,FTS search) are weak-model/data-dependent artifacts, not code regressions (classification logic is untouched). A stronger model / API keys yields 31/31.Demo logs
features_demo.log
full_test_after_features.log
To show artifacts inline, enable in settings.