Purpose
The project had no way to verify that the ConversationAgent behaves correctly at the LLM level. Unit tests cover Python logic, but they can't catch regressions like "the model stopped calling the right tool" or "the model started inventing URLs it shouldn't". Without executable behavioural specifications, any change to the agent's instructions, tools, or model configuration could silently break documented behaviour.
Solution
Behavioral eval framework (chat/evals/)
Built on top of pydantic-evals, a lightweight eval layer sits alongside the agent code. Each dataset is a YAML file of cases with typed inputs and expected outcomes. Each eval config wires a dataset to its evaluators and optionally to a custom agent subclass. A central registry maps dataset names to configs so the runner can discover them automatically.
3 evaluator strategies are used:
- Deterministic (
UrlRegexEvaluator): regex-based, zero cost, instant feedback — flags any invented http(s):// URL not present in the tool
output.
- Semantic (
LLMJudge): used when the correct behaviour requires understanding intent, not just pattern matching.
- Span-based (
HasMatchingSpan): checks that specific tool calls were (or were not) emitted by inspecting pydantic_ai traces — used for the
self_documentation tool.
Two initial datasets
| Dataset |
What it guards |
url_hallucination |
The agent never invents https:// URLs; only uses URLs from tool output |
self_documentation |
The self_documentation tool is called when and only when the user asks about the assistant itself |
Management command (run_evals)
A Django management command (make eval) runs any subset of datasets inside Docker, with flags for filtering by case, repeating runs for consistency checks, verbose output, and skipping the LLM judge when the model endpoint doesn't support structured output. It guards against accidentally running against a mock model.
Purpose
The project had no way to verify that the
ConversationAgentbehaves correctly at the LLM level. Unit tests cover Python logic, but they can't catch regressions like "the model stopped calling the right tool" or "the model started inventing URLs it shouldn't". Without executable behavioural specifications, any change to the agent's instructions, tools, or model configuration could silently break documented behaviour.Solution
Behavioral eval framework (
chat/evals/)Built on top of
pydantic-evals, a lightweight eval layer sits alongside the agent code. Each dataset is a YAML file of cases with typed inputs and expected outcomes. Each eval config wires a dataset to its evaluators and optionally to a custom agent subclass. A central registry maps dataset names to configs so the runner can discover them automatically.3 evaluator strategies are used:
UrlRegexEvaluator): regex-based, zero cost, instant feedback — flags any inventedhttp(s)://URL not present in the tooloutput.
LLMJudge): used when the correct behaviour requires understanding intent, not just pattern matching.HasMatchingSpan): checks that specific tool calls were (or were not) emitted by inspectingpydantic_aitraces — used for theself_documentationtool.Two initial datasets
url_hallucinationhttps://URLs; only uses URLs from tool outputself_documentationself_documentationtool is called when and only when the user asks about the assistant itselfManagement command (
run_evals)A Django management command (
make eval) runs any subset of datasets inside Docker, with flags for filtering by case, repeating runs for consistency checks, verbose output, and skipping the LLM judge when the model endpoint doesn't support structured output. It guards against accidentally running against a mock model.