Plans: propose chat context eval designs

[VoxelPrincess]

Hi — before jumping into evals implementation of #727 , I’d like to align on two design decisions: the measurement strategy and how much of the judge behavior nao should own.
Proposed designs: RAG triad vs Correctness, with Both as a design-only combined option.

Problem: As a nao user curating project context, I want to know when a change to my rules.md or semantic model definitions makes the chat agent hallucinate, use wrong terminology, or ignore my configuration — even when the SQL query returns the correct number.

Goal: add context-quality evals alongside SQL tests. Exact numeric correctness stays in deterministic SQL tests; nao evals would cover semantic/context failures in the final answer. Known context failures become repeatable eval cases, and thresholds would be calibrated on real runs before they are used for pass/fail decisions.

1. Measurement strategy: referenceless vs reference-based

There are three options:

• Plan A — RAG triad only — referenceless. It evaluates Contextual Relevancy, Faithfulness, and Answer Relevancy. It assumes that if relevant context was used, the answer was grounded in it, and it addressed the question, then a correct answer is likely. It does not verify correctness directly. It also requires nao to capture content-bearing tool outputs and maintain a list of tools such as execute_sql, read, and grep.

• Plan B — Correctness only — reference-based. It compares the final answer with an expected answer. It assumes that if the final answer matches the expected output, then the context was likely good. It does not verify which context the agent used or explain whether a failure came from missing context, hallucination, or another cause. Every test case needs a maintained expected_output.

• Plan C — Both — avoids relying on either assumption alone: the RAG triad checks the context-to-answer path, while Correctness verifies the final answer against the expected output. Judge ownership is still chosen in §2.

Summary of the three measurement options (§1):

	Plan A (RAG triad)	Plan B (Correctness)	Plan C (Both)
Measurement	Referenceless	Reference-based	Both
Data per case	id, input	id, input, expected_output	id, input, expected_output
Context capture	Required	Not required	Required
Backend changes	New /api/evals/chat	None	Same as A
Judge metrics per case	3	1	4
Agent runs per case	1	1	1 shared run
Diagnostic signal	Context-to-answer path	Reference mismatch	Both
Main maintenance	Content-tool list	Expected outputs	Both

Decision needed: RAG triad, Correctness, or both?

2. Judge implementation: ownership

This decision is independent of the measurement strategy: it determines who owns the judge prompts and scoring behavior.

• DeepEval built-in metrics and prompts — used by the Plan A prototype. DeepEval owns the triad prompts and scoring; nao owns wiring, thresholds, and reporting.
• DeepEval with nao-specific evaluation steps — what the Plan B prototype uses. DeepEval runs GEval; nao defines the Correctness rubric.
• No eval framework / fully custom judge — a possible alternative if nao wants to avoid a framework dependency. It is not prototyped in these plans and would require a separate design for prompts, scoring, parsing, and calibration.

Decision needed: how much of the judge behavior should nao own?

Shared requirements

• User-defined eval cases live in the project alongside existing SQL tests.
• Existing SQL tests remain unchanged.
• Evals are single-turn.
• Results include pass/fail, score, and reason per test case.
• No UI changes or mandatory PR-blocking CI gate by default.
  Both plans include prototype evidence. A short reply on the preferred measurement strategy and judge ownership, including a phased approach if appropriate, is enough to unblock implementation.

[github-actions]

This PR was auto-closed. Only contributors approved with lgtm can open PRs. Open an issue first.

Maintainers review auto-closed issues daily. Issues that do not meet the quality bar in CONTRIBUTING.md will not be reopened or receive a reply.

If a maintainer replies lgtmi, your future issues will stay open. If a maintainer replies lgtm, your future issues and PRs will stay open.

See CONTRIBUTING.md.