feat: end-to-end QA scoring stage with pluggable LLM runners

[groksrc]

Summary

Adds the missing stage between retrieval metrics and publishable benchmark numbers: per-query answer generation over retrieved context, followed by LLM-judged grading against the expected answer. Retrieval metrics (recall/MRR/content-hit) measure the search layer; QA accuracy is what memory-system comparisons are actually made on.

What's new

llm/runners.py — LLMRunner abstraction with two transports:

• claude:<model> — Claude Code CLI print mode (claude -p --output-format json --max-turns 1). Bills the operator's Claude subscription plan; no API key required.
• openai-compat:<model>@<base_url> — any OpenAI-compatible chat-completions endpoint (Ollama, LM Studio, vLLM, OpenAI).

Both retry transient failures and report token usage + latency per call.

scoring/qa.py — fixed, auditable prompts (they ship in the repo so any number can be traced to its exact rubric):

• Answer prompt: answer from retrieved memories only; abstain with exactly "I don't know" when the context lacks the answer.
• Judge prompt: strict binary verdict as JSON. Abstention is correct only when the gold answer marks the question unanswerable (LoCoMo adversarial); abstaining on an answerable question is incorrect.
• Malformed judge output or runner failure is recorded as an explicit per-case error (scored incorrect), never silently passed.

run qa CLI command — reads per-query-retrieval.jsonl, writes per-query-qa.jsonl + qa-summary.json with per-category accuracy, abstain/error counts, mean answer latency, and token totals. The same answerer and judge are used for every provider in a run, holding the model constant across systems.

Fairness properties

• Identical answerer + judge models for all providers in a run.
• Prompts are fixed in source, not configurable per provider.
• Judge model recorded in every artifact row for auditability.

Testing

• tests/llm/test_runners.py — spec parsing, claude transport parsing/retry/error paths (subprocess mocked).
• tests/test_qa_scoring.py — prompt construction, verdict parsing (incl. code-fenced JSON), correct/incorrect/abstain scoring, category breakdown, error recording, token/latency accounting, artifact writing.
• Live smoke test against the real claude CLI (haiku answerer, sonnet judge): answerable case answered and judged correct; adversarial case abstained and judged correct.

🤖 Generated with Claude Code