diff --git a/plans/727/correctness_plan.md b/plans/727/correctness_plan.md new file mode 100644 index 000000000..feb9f1344 --- /dev/null +++ b/plans/727/correctness_plan.md @@ -0,0 +1,461 @@ +# Correctness Eval Plan (Plan B) + +Reference-based correctness evals for nao chat answers, proposed for +[getnao/nao#727](https://github.com/getnao/nao/issues/727). + +This plan is meant to be read side by side with the +[RAG triad plan](rag_triad_plan.MD). Both plans use the same three kwwhat EV charging +questions so their signals and implementation trade-offs are easier to compare. + +--- + +## 1. Background and Problem + +nao's SQL tests check deterministic answers and query correctness. They cannot catch +**context drift** in the final answer: the agent can return a numerically correct +result while ignoring configured context (`rules.md`, semantic definitions, framing, +or terminology). + +As a nao user curating project context, I want to know when a context change makes the +agent hallucinate, use the wrong terminology, or ignore my configuration, even when +the SQL result is correct. + +Plan B adds a **reference-based** check alongside existing SQL tests: + +```text +input -> nao agent -> actual_output <-> expected_output -> Correctness judge + -> score / pass / reason +``` + +Exact numeric assertions remain in deterministic SQL tests. `nao evals` covers +semantic and context-related failures in the final answer. + +--- + +## 2. Approach: Reference-Based Correctness + +Plan B compares the final answer directly with a user-maintained `expected_output` +using DeepEval `GEval`. + +The Correctness rubric checks that the answer: + +- is factually consistent with the reference; +- contains the required facts, entities, values, and relationships; +- uses the required terminology and framing; +- does not add contradictions, unsupported claims, or misleading details. + +**Assumption:** if the final answer matches the reference, the configured context was +probably good. + +**Strength:** Correctness directly tests whether the user-visible result matches the +expected answer. It does not need tool-result or retrieval-context capture. + +**Limitations:** + +- It does not verify which context the agent used. +- It gives less insight into whether a failure came from irrelevant context, + grounding, or answer relevance. +- It can penalize extra details because it cannot determine whether they are grounded + in context that was not passed to the judge. +- Every case requires a reviewed and maintained `expected_output`. + +--- + +## 3. Architecture: What Exists Today + +`nao test` already runs the agent through the backend and returns the final answer: + +- `POST /api/test/run` runs the agent non-streaming and returns + `{ text, usage, cost, duration, tool_calls }`. +- `AgentClient` in `cli/nao_core/commands/test/client.py` already wraps the request, + handles authentication, and returns a typed result. + +Plan B reuses this path to obtain `actual_output` and run one Correctness metric. + +--- + +## 4. Backend Changes: None + +Plan B requires no new backend route and no changes to agent internals: + +- no tool-result extraction or serialization; +- no content-bearing tool allowlist to maintain; +- no retrieval/context field in the response; +- no changes to existing SQL tests. + +The runner sends each eval through the existing `POST /api/test/run` with an empty +`sql` field. An optional request timeout can be added to the existing client without +changing current `nao test` behavior. + +This is the main implementation advantage of Plan B: the change is contained in the +CLI and user-owned eval data. + +--- + +## 5. Directory Layout + +```text +# nao_core package (framework, shipped with nao) +cli/nao_core/commands/evals/ + case.py # dataset discovery and JSONL validation + correctness.py # Correctness rubric and judge resolution + runner.py # execution, reporting, CLI command + +# User project (data, owned by the team) +{project}/tests/evals/ + golden_dataset.jsonl +{project}/tests/outputs/ + evals_results_.json +``` + +Eval cases live in the project alongside existing `tests/*.yml` SQL tests. + +### Why validate the dataset before running + +Current `nao test` discovery does not perform a separate schema-validation pass; a +malformed YAML test fails when it is parsed. Plan B proposes preflight validation +because eval runs invoke paid agent and judge calls. `case.py` validates every JSONL +row, required field, and duplicate ID before the first model call, then reports all +file/line errors together. This prevents a malformed later row from wasting an +otherwise valid partial run. + +--- + +## 6. Golden Dataset + +Plan B uses the same IDs and questions as the RAG triad plan, adding only the +`expected_output` required by Correctness: + +```jsonl +{"id": "q001", "input": "How many ports are currently decommissioned?", "expected_output": "There are currently 4 decommissioned ports."} +{"id": "q002", "input": "What is the overall uptime percentage of my EV charging network for the full history?", "expected_output": "The overall uptime is 99.71% for the full available history."} +{"id": "q003", "input": "What is the charge point availability index?", "expected_output": "The metric \"charge point availability index\" is not defined in the semantic model. The closest available metric is uptime. Would you like that instead?"} +``` + +Dataset quality is part of the eval design. Reference answers should be reviewed and +versioned like code; vague or stale references make the score unreliable. + +--- + +## 7. Eval Flow + +```text +for each record in golden_dataset.jsonl: + 1. AgentClient.run_test(input, sql="", model, timeout) + -> actual_output + usage metadata + 2. LLMTestCase(actual_output, expected_output) + 3. GEval(Correctness).measure(test_case) + -> score / pass / reason + 4. collect result + +after all records: + 5. write evals_results_.json + 6. exit non-zero if any case failed or errored +``` + +Errors are separated from answer-quality failures: + +- agent timeout -> `error_type: "timeout"`; judge is not called; +- backend error -> `error_type: "backend_error"`; judge is not called; +- judge error -> `error_type: "metric_error"`; +- completed judge result below threshold -> normal eval failure. + +--- + +## 8. Correctness Metric + +The metric is a DeepEval `GEval` instance named `Correctness`. + +```python +correctness_metric = GEval( + name="Correctness", + evaluation_steps=[ + "Compare the actual output directly with the expected output for factual accuracy.", + "Verify that every required fact, entity, value, and relationship is present and correctly represented.", + "Verify that the required naming, terminology, and framing are used.", + "Penalize contradictions, unsupported additions, and misleading details.", + ], + evaluation_params=[ + SingleTurnParams.ACTUAL_OUTPUT, + SingleTurnParams.EXPECTED_OUTPUT, + ], +) +``` + +Plan B supplies explicit `evaluation_steps` instead of asking DeepEval to generate +them from broad `criteria` on each run. This makes the project-specific definition of +Correctness visible, reviewable, and stable; changes to the steps can be versioned +like other test logic. + +`INPUT` and retrieval/context fields are intentionally excluded. This is a +reference-based result check, not a grounding check. + +### Threshold + +The initial threshold is `0.5`. It must be calibrated through repeated runs and human +review before it becomes a blocking gate. + +### Judge Model + +`--judge-model` selects the model used by DeepEval. It may be the same as the chat +model or an independent judge. Using the same model introduces self-evaluation bias, +so the runner should keep these model choices separate. + +--- + +## 9. Runner Components + +- **`case.py`** + - parse and validate `id`, `input`, and `expected_output`; + - reject duplicate IDs and report row-level errors. +- **`correctness.py`** + - define the Correctness evaluation steps; + - build `LLMTestCase` and `GEval`; + - resolve the requested judge model. +- **`runner.py`** + - discover and filter cases; + - run the agent once per case; + - handle timeout/backend/judge errors; + - save score, pass/fail, and reason; + - return a non-zero exit code when any case fails. + +The initial implementation uses one default Correctness rubric. `correctness.py` +owns rubric construction and selection so the runner does not need question-type +conditionals if reviewed cases later justify more than one rubric. + +DeepEval should be imported lazily so normal nao commands do not require it at module +load time. Whether it is a core dependency or an optional `nao-core[evals]` extra is a +maintainer decision. + +--- + +## 10. Running the Evals + +```bash +# All cases +nao evals \ + -m anthropic:claude-sonnet-4-6 \ + --judge-model anthropic:claude-sonnet-4-6 \ + --timeout 300 + +# One case +nao evals \ + -s q001 \ + -m anthropic:claude-sonnet-4-6 \ + --judge-model anthropic:claude-sonnet-4-6 \ + --timeout 120 +``` + +Relevant parameters: + +- `-m`, `--model`: chat model being evaluated (`provider:model_id`); +- `--judge-model`: DeepEval judge model (`provider:model_id`); +- `--timeout`: maximum seconds for each agent request; +- `-s`, `--select`: eval ID filter; +- `--threshold`: minimum Correctness score required to pass. + +The proposed report stores `input`, `actual_output`, `expected_output`, score, +threshold, pass/fail, reason, `judge_model`, `rubric_version`, `deepeval_version`, usage +metadata, and any error type. Recording the judge, rubric, and framework versions +prevents scores produced by different evaluation configurations from being treated as +directly comparable. Judge usage should be reported separately when the selected +DeepEval adapter exposes it. + +`rubric_version` is a short content hash of the nao-owned rubric profile, evaluation +steps, and parameters. `deepeval_version` records the framework version separately +because its internal scoring behavior may change without changing nao's rubric. + +--- + +## 11. Example Outputs on the Shared RAG Cases + +The following dry run uses the same `q001`-`q003` questions as the RAG plan: + +- chat model: `anthropic:claude-sonnet-4-6`; +- judge model: `claude-sonnet-4-6`; +- Correctness threshold: `0.5`. + +### q001: Decommissioned Ports + +**Input** + +> How many ports are currently decommissioned? + +**Expected output** + +> There are currently 4 decommissioned ports. + +**Actual output** + +```text +There are currently 4 ports that have been decommissioned (i.e., have a non-null `decommissioned_ts`). +``` + +**Correctness:** `0.90`, pass. + +The count matches. The citation markup and schema explanation do not contradict the +reference. + +### q002: Full-History Uptime + +**Input** + +> What is the overall uptime percentage of my EV charging network for the full +> history? + +**Expected output** + +> The overall uptime is 99.71% for the full available history. + +**Actual output** + +```text +Your EV charging network has an **overall uptime of 99.7%** across its full history. + +Here's the context behind that figure: + +| Metric | Value | +|---|---| +| Overall Uptime | **99.7%** | +| Period | Oct 1, 2025 – Jul 2, 2026 | +| Charge Points | 2 | +| Ports Monitored | 2 | +| Port-Day Records | 578 | + +**Methodology:** Uptime is calculated as *(commissioned minutes − outage minutes) / commissioned minutes* per port per day, then averaged across all port-days — consistent with the `fact_uptime` semantic model definition. Only OFFLINE-type outages are currently tracked in the downtime data. +``` + +**Correctness:** `0.70`, pass. + +The judge treated `99.7%` as a rounded form of the expected `99.71%`. However, this +run used a later data snapshot than the RAG example, so the difference may also show +reference staleness rather than answer error. The first attempt exceeded a 120-second +request timeout; the completed retry used a 300-second limit. The timeout is an +execution issue, not a metric failure. + +### q003: Undefined Availability Index + +**Input** + +> What is the charge point availability index? + +**Expected output** + +> The metric "charge point availability index" is not defined in the semantic model. +> The closest available metric is uptime. Would you like that instead? + +**Actual output** + +```text +The metric **"charge point availability index"** is not defined in the semantic model. The closest defined metric is **`uptime`** — which measures the average fraction of commissioned time that charge points were not in an outage (i.e., availability at the port level). + +Here's what is defined and relevant: + +| Metric | Definition | +|---|---| +| **uptime** | Average port uptime = (commissioned minutes − outage minutes) / commissioned minutes. Ranges from 0 to 1 (0% = fully down, 100% = fully up). Sourced from `fact_uptime`. | + +There is no metric explicitly named "charge point availability index" in the semantic model. If you're looking for something specific — such as the percentage of charge points that were available (not faulted/unavailable) at a point in time, or a composite index — that metric would need to be defined or derived. + +Would you like me to: +1. **Report on uptime** (the closest existing availability metric) for the last 7 days? +2. **Build a custom availability index** based on `dim_ports` latest statuses or downtime data? +``` + +**Correctness:** `0.70`, pass. + +The core refusal, terminology, and uptime redirect match the reference. The score is +reduced because the response adds substantial explanation and extra follow-up options. + +### Comparison with the RAG Triad Plan + +| Case | RAG triad result | Correctness result | Difference in signal | +|---|---|---|---| +| `q001` | Pass: Faithfulness `1.0`, Contextual Relevancy `0.8333`, Answer Relevancy `1.0` | Pass: `0.90` | Both approaches accept the answer. The triad describes the context-to-answer path; Correctness confirms the expected count. | +| `q002` | Pass: Faithfulness `0.8889`, Contextual Relevancy `0.507`, Answer Relevancy `0.8182` | Pass: `0.70` | The triad surfaces context dilution; Correctness surfaces a reference mismatch that may be rounding or reference staleness. | +| `q003` | Fail: Faithfulness `1.0`, Contextual Relevancy `0.0`, Answer Relevancy `0.5714` | Pass: `0.70` | The triad rejects the context/relevance path, while Correctness accepts that the final answer follows the expected refusal and uptime redirect. | + +These are **illustrative**, not controlled A/B results. The plans use the same +questions, model family, and project, but the agent generated a separate response for +each run and `q002` used a later data snapshot. A controlled comparison would apply +both metric strategies to the same saved agent response and data snapshot. + +The key design trade-off is visible in `q003`: the RAG triad provides context-path +diagnostics, while Correctness directly checks the required user-visible behavior. + +--- + +## 12. CI + +LLM evals should not block every PR by default because they are slower, +non-deterministic, and cost money. The runner should remain CI-friendly through a +non-zero exit code, while teams choose whether to run it manually, on a schedule, or +as a calibrated gate. + +--- + +## 13. Proposed Implementation Sequence + +1. Add `case.py`, `correctness.py`, and `runner.py` under + `cli/nao_core/commands/evals/`. +2. Register `nao evals` without changing existing `nao test` behavior. +3. Reuse `AgentClient` and add an optional backward-compatible request timeout. +4. Add lazy DeepEval imports and decide whether eval dependencies are core or + optional. +5. Validate JSONL cases and write stable JSON results with score, pass/fail, reason, + judge model, rubric version, DeepEval version, and error type. +6. Add focused tests for dataset validation, selection, timeout/backend errors, + metric failures, and exit codes. +7. Calibrate the rubric and threshold on reviewed project cases before enabling any + blocking CI gate. + +### Possible later extension: rubric profiles + +The initial implementation keeps one default rubric. If observed cases later require +different criteria, the dataset can add an optional `rubric_profile`, for example: + +```jsonl +{"id":"q002","input":"What is the overall uptime?","expected_output":"The overall uptime is 99.71%.","rubric_profile":"reported_value"} +{"id":"q003","input":"What is the charge point availability index?","expected_output":"The metric is not defined; offer uptime instead.","rubric_profile":"semantic"} +``` + +The corresponding metrics would differ in their evaluation steps: + +```python +reported_value_correctness = GEval( + name="Reported Value Correctness", + evaluation_steps=[ + "Compare every reported numeric value with the expected output.", + "Require the expected precision and penalize contradictory values.", + ], + evaluation_params=[ + SingleTurnParams.ACTUAL_OUTPUT, + SingleTurnParams.EXPECTED_OUTPUT, + ], +) + +semantic_correctness = GEval( + name="Semantic Correctness", + evaluation_steps=[ + "Verify that the answer preserves the expected intent, terminology, and framing.", + "Allow equivalent wording; penalize contradictions and unsupported additions.", + ], + evaluation_params=[ + SingleTurnParams.ACTUAL_OUTPUT, + SingleTurnParams.EXPECTED_OUTPUT, + ], +) +``` + +`correctness.py` would select the metric from `rubric_profile`; dataset loading and +runner orchestration would remain unchanged. Each result would record the selected +profile, rubric version, and DeepEval version. This extension should be driven by +calibrated cases rather than added upfront. + +The reported-value profile would check whether the final answer preserves a verified +number and its required precision. Deterministic SQL tests remain the factual source +of truth and are not replaced by this semantic check. + +After the eval schema is approved, a `create-evals` skill could generate and validate +Plan B golden records (`id`, `input`, and `expected_output`), similar to +`create-context-tests`. I can contribute this follow-up if useful. diff --git a/plans/727/rag_triad_plan.MD b/plans/727/rag_triad_plan.MD new file mode 100644 index 000000000..b569df72d --- /dev/null +++ b/plans/727/rag_triad_plan.MD @@ -0,0 +1,458 @@ +# Plan: LLM-as-Judge RAG Triad Evals for nao + +## Background + +nao's agent answers questions by reading file-based project context (schema docs, `columns.md`, +`preview.md`) and executing SQL. When a user asks a question, the agent: + +1. Reads relevant files from the project folder (`read`, `grep`, `search`, `list` tools) +2. Executes SQL queries (`execute_sql` tool) +3. Composes a natural language response from those tool outputs + +The **curated context** for evals is the subset of tool outputs that carry actual +content: `execute_sql`, `read`, `grep`. + +This plan adds evals that measure context quality by borrowing the RAG triad framework — with one substitution: instead of retrieved context we will be assessing curated context. The assumption here is that if the context is good, the answer is likely correct without checking directly — the core principle behind the referenceless approach. The alternative reference-based approach in `correctness_plan.md` directly compares outputs against a maintained `expected_output` — fewer judge calls per record, but every entry requires a reviewed answer. + +--- + +## Two Approaches to Curated Context + +### Static: bake context into the golden dataset (rejected — not robust and hard to maintain) + +Pre-record the tool outputs when building the dataset. Each golden record carries: + +```jsonl +{ + "id": "q001", + "input": "How many ports are currently decommissioned?", + "expected_output": "...", + "curated_context": [ + "[File: models/semantic/semantic_models.yml]\n# dim_ports\n- decommissioned_ts: timestamp when port was decommissioned, null if active", + "[SQL result]\nColumns: decommissioned_ports\n4" + ] +} +``` + +The eval test reads context directly from the file — no running backend needed. + +**Downside:** Context goes stale the moment the project folder, schema docs, or SQL results change. +If nao's prompt is updated and the agent now reads different files or generates different SQL, the +golden context no longer reflects what the model actually used. You end up judging faithfulness +against a context that wasn't actually provided to the LLM — which defeats the metric entirely. +Every dataset update becomes a manual two-step: run the agent, copy the tool outputs, paste them +into the JSONL. This does not scale past a handful of records. + +--- + +### Dynamic: capture context at eval runtime (chosen approach) + +At eval time, call a dedicated nao endpoint that runs the agent and returns **both** the final +answer and the exact tool results that produced it, in a single response. The eval harness uses +those tool results directly as `retrieval_context` for DeepEval. No context is stored in the +dataset. + +**Why this is correct:** + +- The context passed to DeepEval is always the context that was actually provided to the LLM for + *that specific run*. There is no staleness. +- If nao's prompt changes, the agent reads different files, or the database is updated, the eval + automatically reflects the new reality — it measures faithfulness of what nao actually said + against what nao actually saw. +- The golden dataset stays lean: only `input`. No manual context curation. +- It also enables a future regression signal: if faithfulness drops after a prompt or tool change, + you know the model started making claims not grounded in its own context. + +The only requirement is a running nao backend, which is already needed to get `actual_output` +anyway. The dynamic approach does require a dedicated eval endpoint (`POST /api/evals/chat`), tool output +filtering logic, and ongoing maintenance of the `CONTEXT_TOOLS` list — but these are contained +to one route file and one runner, rather than spreading context management into the dataset. + +**Maintenance burden:** `CONTEXT_TOOLS` in `apps/backend/src/routes/evals.ts` must be kept in sync +as new agent tools are added. When a new tool is introduced, the nao team needs to decide whether +its output is content-bearing (add it) or navigation-only (leave it out). This is a small but +ongoing maintenance cost that the static approach does not have. + +--- + +## Architecture + +### What exists today + +`AgentManager` in `apps/backend/src/services/agent.ts` has two execution modes: + +- `stream()` — used by the main `/api/agent` route; returns a UI message stream to the browser +- `generate()` — non-streaming; returns `AgentRunResult` which already contains: + ```typescript + steps: ReadonlyArray<{ + toolCalls: ReadonlyArray<{ toolName: string; toolCallId: string; input: unknown }>; + toolResults: ReadonlyArray<{ toolCallId: string; output?: unknown }>; + }> + ``` + +`steps[*].toolResults` is the exact grounding context. It is already collected internally — it just +has no HTTP endpoint that exposes it. + +--- + +## What Changes in the Backend (minimal) + +Add one new Fastify route: `POST /api/evals/chat`. + +It mirrors `/api/agent` but: + +1. Calls `testAgentService.runTest()` (same service used by `nao test`) — non-streaming, returns + the full `AgentRunResult` +2. Returns JSON `{ text, model_id, tool_results }` instead of a UI message stream +3. `tool_results` = `TestAgentService.extractToolCalls(result)` filtered to content-bearing tools: + `execute_sql`, `read`, `grep` + — `list` and `search` are intentionally excluded: both return only file metadata + (path/dir/size), not file content. They are navigation tools — the agent uses them to decide + what to read next, not to compose its answer. Including them dilutes Contextual Relevancy + with path listings that have no semantic relationship to the question. +4. Accepts an optional `model: { provider, modelId }` body field to override the project default + +No changes to existing routes or agent internals. `generate()` is already exercised by automations. + +**Response shape:** +```json +{ + "text": "There are currently 4 decommissioned ports (those with a non-null decommissioned_ts).", + "model_id": "claude-sonnet-4-6", + "tool_results": [ + { "toolName": "read", "output": { "content": "# dim_ports\n- decommissioned_ts: timestamp when port was decommissioned, null if active\n- port_id: unique port identifier" } }, + { "toolName": "execute_sql", "output": { "columns": ["decommissioned_ports"], "data": [[4]] } } + ] +} +``` + +`model_id` reflects whichever model ran — either the project default from Settings → Project → +Models, or the override passed in the request body. + +--- + +## Directory Layout + +Evals follow the same project-relative convention as `nao test`. The framework lives in the +`nao_core` package; the user's golden dataset lives in their project folder alongside other test +assets. + +``` +# nao_core package (framework — shipped with nao) +cli/nao_core/commands/evals/ + case.py # dataset discovery and JSONL validation + runner.py # full eval engine + CLI command + +# User's project (data — owned by the team) +{project}/tests/evals/ + golden_dataset.jsonl # {id, input} — no context, no expected output +``` + +`case.py` validates the dataset before any records are run — catching missing fields or duplicate +IDs upfront avoids wasting 3 paid LLM judge calls per bad record. + +`runner.py` is both orchestration and eval engine — same pattern as `nao test`. It reads the +dataset, calls the backend, runs DeepEval metrics, writes the JSON report, and exits. + +The `/api/evals/chat` call happens once per record regardless of how many metrics are active — +all metrics share the same `LLMTestCase` built from that single response. + +--- + +## Golden Dataset Schema + +```jsonl +{"id": "q001", "input": "How many ports are currently decommissioned?"} +{"id": "q002", "input": "What is the overall uptime percentage of my EV charging network for the full history?"} +{"id": "q003", "input": "What is the charge point availability index?"} +``` + +No `curated_context` or `expected_output` field. Context is captured live. `expected_output` is only needed if Completeness (`GEval`) is added later — add it back to the dataset at that point. + +--- + +## Eval Harness Flow (per test case) + +``` +for each record in golden_dataset.jsonl: + 1. POST /api/evals/chat { input: record.input, model?: { provider, modelId } } + 2. → { text: , model_id: , tool_results: [...] } + 3. curated_context = [serialize(tr) for tr in tool_results] + 4. LLMTestCase( + input = record.input, + actual_output = response.text, + retrieval_context = curated_context, + ) + 5. for each metric: metric.measure(test_case) # populates .score and .reason on the same instance + collect MetricResult(name, score, threshold, passed, reason) + +after all records: + 6. save_results() → evals_{timestamp}.json with results + summary + 7. sys.exit(1) if any record failed +``` + +All three metrics measure the same `LLMTestCase`. DeepEval reports each score and reason +independently — a failure in one does not suppress the others. + +**Tool result serialization**: + +| Tool | Returns | Included | Reason | +|---|---|---|---| +| `execute_sql` | data rows | yes | primary grounding — the actual query results the agent used | +| `read` | full file content | yes | schema docs, column descriptions, semantic models | +| `grep` | matching lines + context | yes | actual file content snippets | +| `search` | path/dir/size | **no** | glob file finder — navigation only, no content | +| `list` | path/dir/size | **no** | directory listing — navigation only, no content | + +`search` and `list` are excluded for the same reason: both return only file metadata. They are +navigation tools the agent uses to decide what to read next — not to compose its answer. Including +them adds path listings with no semantic relationship to the question, diluting Contextual +Relevancy without adding grounding signal. + +--- + +## DeepEval Metric Configuration + +```python +FaithfulnessMetric(threshold=0.7, model=judge, include_reason=True) +ContextualRelevancyMetric(threshold=0.5, model=judge, include_reason=True) +AnswerRelevancyMetric(threshold=0.7, model=judge, include_reason=True) +``` + +**Contextual Relevancy threshold is intentionally lower than the other two metrics.** In a classic +RAG pipeline, retrieval is scoped tightly to the question — only the most relevant chunks are +fetched. A chat agent works differently: it reads schema docs, semantic model definitions, and +column descriptions to build a broad understanding of the data model before it can compose a +correct answer. Much of that context is genuinely necessary for the agent to reason well, but it is +not directly about the specific question. The result is that Contextual Relevancy is structurally +diluted compared to a purpose-built retrieval system — by design, not by failure. + +In addition to that, the right Contextual Relevancy threshold depends on the project's data scope. A single-domain +project (only EV charger data) will have tighter, more focused context than a multi-domain one +(sales + HR + finance) — so 0.5 might be too strict for a broad deployment and too loose for a +narrow one. + +This means a score of 0.5–0.65 on Contextual Relevancy may reflect a well-functioning agent +rather than a broken retrieval step and should be calibrated per project. + +**Per-project threshold configuration** + +Thresholds should be tunable per project via `nao.yml`: + +```yaml +evals: + thresholds: + faithfulness: 0.7 + contextual_relevancy: 0.3 + answer_relevancy: 0.7 +``` + +`runner.py` reads `nao.yml` via the existing `NaoConfig` loader (same as `nao test`), falling back +to the hardcoded defaults if the section is absent. Not yet implemented — add when the first +project needs a different threshold. + +**Self-serving bias and `--judge-model`** + +Using the same model as judge introduces self-serving bias — a model tends to score its own outputs +higher. To use an independent judge, pass `--judge-model` with a different model. The agent runs +with its configured model; the judge runs with the override: + +```bash +nao evals -j openai:gpt-4.1 +nao evals -m anthropic:claude-sonnet-4-6 -j openai:gpt-4.1 +``` + +Without `--judge-model`, the judge defaults to whatever model the agent used (`model_id` from the +response). `-m` alone does not give you an independent judge — it changes the agent model, and the +judge follows it. + +--- + +## `runner.py` Functions + +- **`serialize_tool_result(tool_name, args, output)`** — converts a raw tool call result into a + plain string for `retrieval_context`. Formats `read` as `[File: path]\ncontent`, `execute_sql` + as a column/row table, everything else as JSON. +- **`make_judge(model_id)`** — returns a `_ClaudeJudge` (`DeepEvalBaseLLM` subclass) for Claude + models, or a plain string for OpenAI (DeepEval handles those natively). deepeval is imported + lazily inside this function so the module loads without it installed. +- **`run_eval(record, client, model, judge_model, verbose)`** — single eval loop for one record: + POST to `/api/evals/chat`, build `LLMTestCase`, instantiate metrics with `judge_model` (or agent + model if not set), call `metric.measure()` on each, collect `MetricResult` dataclasses, return + `EvalResult`. +- **`save_results(results, output_dir)`** — writes `evals_{timestamp}.json` with `results` list + and `summary: { total, passed, failed }`. +- **`evals(...)`** — CLI entry point; reads dataset, authenticates, iterates records via + `run_eval()`, saves results, prints summary, calls `sys.exit(1)` if any failed. + +--- + +## Running the Evals + +```bash +# Start nao backend normally +npm run dev -w @nao/backend + +# Run all evals (from your project directory) +nao evals + +# Specify the agent model (mirrors nao test -m) +nao evals -m anthropic:claude-sonnet-4-6 + +# Use an independent judge model +nao evals -j openai:gpt-4.1 + +# Agent and judge on different models +nao evals -m anthropic:claude-sonnet-4-6 -j openai:gpt-4.1 + +# Verbose output (prints metric reasons) +nao evals -v + +# Run a single record by ID +nao evals -s q001 + +# Explicit credentials +nao evals -u user@example.com --password secret +``` + +Install eval deps once with `uv sync --extra evals` from `cli/`. The `evals` extra adds only +`deepeval` — provider SDKs come from the user's existing nao installation. + +Add `make evals` to the root `Makefile` for consistency with `make lint`. + +**Example output (`-v`):** + +``` +Running: q001 How many ports are currently decommissioned? + ✓ Faithfulness: 1.0 + The score is 1.00 because the actual output is perfectly faithful to the retrieval context with no contradictions found! + ✓ ContextualRelevancy: 1.0 + The score is 1.00 because the retrieval context directly answers the input by confirming 'There are 4 decommissioned + ports' and provides supporting details about how this was determined through a SQL query on the + analytics.ANALYTICS.dim_ports table. Perfect retrieval! + ✓ AnswerRelevancy: 1.0 + The score is 1.00 because the response directly and completely addresses the question about the number of currently + decommissioned ports, with no irrelevant statements whatsoever. Great job! + +Running: q002 What is the overall uptime percentage of my EV charging network for the full history? + ✓ Faithfulness: 0.8889 + The score is 0.89 because the actual output incorrectly implies that only November 2025 onwards (excluding October + 2025) had 100% uptime, when in fact the retrieval context states that September 2025 also had 100% uptime. + ✗ ContextualRelevancy: 0.4444 + The score is 0.44 because while some retrieved context is highly relevant — including the 'uptime' semantic model, + 'fact_uptime' table, the overall uptime percentage of 99.72% covering 295 days, and monthly breakdowns — a large + portion of the retrieval context is entirely irrelevant. Many retrieved chunks contain empty SQL results, database + metadata, dbt configuration files, schema catalog metadata, and unrelated semantic models like 'visits' and + 'charge_attempts'. + ✓ AnswerRelevancy: 1.0 + The score is 1.00 because the response directly and completely addresses the question about the overall uptime + percentage of the EV charging network for its full history, with no irrelevant statements detected. + +Running: q003 What is the charge point availability index? + ✓ Faithfulness: 0.8182 + The score is 0.82 because the actual output references metrics named 'total_visits' and 'total_charge_attempts', + which do not exist in the retrieval context. The correct metric names defined in the context are 'visits_count' and + 'charge_attempts_count'/'attempts_count'. + ✗ ContextualRelevancy: 0.0 + The score is 0.00 because none of the retrieved context defines or mentions a 'charge point availability index'. + The retrieval context only covers unrelated semantic models such as 'visits', 'charge_attempts', and 'uptime'. + ✗ AnswerRelevancy: 0.6 + The score is 0.60 because while the response addresses the charge point availability index, it includes several + irrelevant metrics (total_visits, total_charge_attempts, average_attempts_per_visit, etc.) that dilute the focus. +``` + +Without `-v`, only the score line is printed. Results are always written to +`{project}/tests/outputs/evals_{timestamp}.json`: + +```json +{ + "timestamp": "2026-07-06T09:46:17.545854", + "results": [ + { + "id": "q001", + "input": "How many ports are currently decommissioned?", + "actual_output": "There are currently 4 decommissioned ports (i.e., ports with a non-null `decommissioned_ts`).", + "passed": true, + "metrics": [ + { "name": "Faithfulness", "score": 1.0, "threshold": 0.7, "passed": true, "reason": "The score is 1.00 because the actual output is perfectly faithful to the retrieval context with no contradictions found!" }, + { "name": "ContextualRelevancy", "score": 1.0, "threshold": 0.5, "passed": true, "reason": "The score is 1.00 because the retrieval context directly answers the input by confirming 'There are 4 decommissioned ports' and provides supporting details about how this was determined through a SQL query on the analytics.ANALYTICS.dim_ports table. Perfect retrieval!" }, + { "name": "AnswerRelevancy", "score": 1.0, "threshold": 0.7, "passed": true, "reason": "The score is 1.00 because the response directly and completely addresses the question about the number of currently decommissioned ports, with no irrelevant statements whatsoever. Great job!" } + ] + }, + { + "id": "q002", + "input": "What is the overall uptime percentage of my EV charging network for the full history?", + "actual_output": "Overall network uptime is 99.72% across Sep 15, 2025 → Jul 6, 2026 (295 days). All 4,986 downtime minutes were concentrated in October 2025 (98.09%). Every month since November 2025 has been at 100% uptime.", + "passed": false, + "metrics": [ + { "name": "Faithfulness", "score": 0.8889, "threshold": 0.7, "passed": true, "reason": "The score is 0.89 because the actual output incorrectly implies that only November 2025 onwards had 100% uptime, when in fact the retrieval context states that September 2025 also had 100% uptime." }, + { "name": "ContextualRelevancy", "score": 0.4444, "threshold": 0.5, "passed": false, "reason": "The score is 0.44 because while some retrieved context is highly relevant — including the 'uptime' semantic model, 'fact_uptime' table, the overall uptime percentage of 99.72%, and monthly breakdowns — a large portion is entirely irrelevant: empty SQL results, database metadata, dbt configuration files, schema catalog metadata, and unrelated semantic models like 'visits' and 'charge_attempts'." }, + { "name": "AnswerRelevancy", "score": 1.0, "threshold": 0.7, "passed": true, "reason": "The score is 1.00 because the response directly and completely addresses the question about the overall uptime percentage for the full history, with no irrelevant statements detected." } + ] + }, + { + "id": "q003", + "input": "What is the charge point availability index?", + "actual_output": "The charge point availability index is not a defined metric in the semantic model. The closest concept is uptime — the average fraction of commissioned time that a port was not in outage. Would you like uptime figures, or to define a custom availability index?", + "passed": false, + "metrics": [ + { "name": "Faithfulness", "score": 0.8182, "threshold": 0.7, "passed": true, "reason": "The score is 0.82 because the actual output references metrics named 'total_visits' and 'total_charge_attempts', which do not exist in the retrieval context. The correct metric names are 'visits_count' and 'charge_attempts_count'/'attempts_count'." }, + { "name": "ContextualRelevancy", "score": 0.0, "threshold": 0.5, "passed": false, "reason": "The score is 0.00 because none of the retrieved context defines or mentions a 'charge point availability index'. The retrieval context only covers unrelated semantic models such as 'visits', 'charge_attempts', and 'uptime'." }, + { "name": "AnswerRelevancy", "score": 0.6, "threshold": 0.7, "passed": false, "reason": "The score is 0.60 because while the response addresses the charge point availability index, it includes several irrelevant metrics (total_visits, total_charge_attempts, average_attempts_per_visit, etc.) that dilute the focus." } + ] + } + ], + "summary": { + "total": 3, + "passed": 1, + "failed": 2 + } +} +``` + +q001 passes with perfect scores. q002 fails on ContextualRelevancy (0.44) — the agent reads broad schema metadata beyond what the question requires, diluting the context score below threshold. q003 fails on ContextualRelevancy and AnswerRelevancy: the correct agent behaviour was to redirect the user to defined metrics, but the judge scored context and answer as irrelevant — a limitation of the RAG triad approach, which cannot distinguish an intentional refusal from a failure to retrieve. + +--- + +## CI + +Do not block PRs on evals — they are slow (~5–10 s per LLM judge call) and cost money. + +--- + +## Implementation Sequence + +1. **Add `POST /api/evals/chat` Fastify route** — ~40 lines in `apps/backend/src/routes/evals.ts`; + calls existing `testAgentService.runTest()`, returns `{ text, model_id, tool_results }`, accepts + optional `model` override +2. **Register route** in `apps/backend/src/app.ts` under `/api/evals` +3. **Write `cli/nao_core/commands/evals/case.py`** — dataset discovery and JSONL validation; + parse and validate `id` and `input`; reject duplicate IDs and report row-level errors before + any judge calls are made +4. **Write `cli/nao_core/commands/evals/runner.py`** — full eval engine mirroring `nao test`; + `EvalResult` / `MetricResult` dataclasses; `serialize_tool_result`, `make_judge`, `run_eval`, + `save_results` functions; `evals()` CLI entry point with `-m`, `-u`, `--password`, `-v`, `-s` + flags; deepeval imported lazily so the module loads without it installed; `sys.exit(1)` if any + record fails +5. **Register command** in `cli/nao_core/commands/__init__.py` and `cli/nao_core/main.py` +6. **Add `evals` extra** to `cli/pyproject.toml` — `deepeval>=2.0` only +7. **Populate `{project}/tests/evals/golden_dataset.jsonl`** — Q&A pairs from golden dataset + +--- + +## Adding Completeness Later + +The full triad ships from day one. Completeness is the only metric deferred — it has no +first-class DeepEval class yet. When ready, adding it is one step: + +1. Add a `GEval` instance to the `metrics=[]` list in `run_eval()` with a rubric such as + "does the answer fully address all parts of the question, given the expected output?" + +Requires adding `expected_output` back to the golden dataset records and passing it to `LLMTestCase`. No other changes. + +| Metric | DeepEval class | Ships | Fields used | +|---|---|---|---| +| Faithfulness | `FaithfulnessMetric` | Day 1 | `input`, `actual_output`, `retrieval_context` | +| Context relevance | `ContextualRelevancyMetric` | Day 1 | `input`, `retrieval_context` | +| Answer relevance | `AnswerRelevancyMetric` | Day 1 | `input`, `actual_output` | +| Completeness | `GEval` (custom rubric) | Later | `input`, `actual_output`, `expected_output` |