P0: Strengthen retrieval eval into an evidence-level acceptance gate

[catoncat]

Review umbrella: #69

Problem

Sherlog has usable message/session recall, but the public eval path is not yet strong enough to protect future retrieval and ranking changes. The next work should build an evidence-level acceptance gate before tuning ranking, adding bundle output, or hardening more adapters.

This maps directly to the paper's evaluation questions for overall effectiveness and retrieval fidelity. It also supports dynamic update robustness by making stale/missing/coverage failures observable rather than anecdotal.

Local Evidence

• eval/manual-eval-core.ts:evaluateManualQuery() only supports title_or_summary, cwd, and snippet predicates.
• eval/dogfood-eval-core.ts:evaluateDogfoodItem() already models stronger private checks: acceptable session UUIDs, cwd, match_source, selected rank/topK, and context mustContain.
• eval/dogfood-eval-core.ts:desiredContextMode() chooses read-range for message hits and read-page for session-only hits.
• src/query-session-fields.test.ts verifies session-level title/compact/reasoning recall and explicitly checks matchSource = "session" with matchSeq = null.
• src/query-flow.test.ts covers the happy path sync -> find -> read-range -> read-page, but that is not yet an evidence-level acceptance suite.
• AGENTS.md says not to claim a strong gold set, rubric, or error taxonomy exists yet.

Scope

Build a repo-shareable retrieval gate with sanitized/synthetic fixtures. It should verify evidence identity and readability, not just whether a snippet contains a word.

Assertions should cover:

• source id
• session ref or session UUID
• matchSource
• nullable matchSeq
• cwd
• top-k rank
• context mustContain after read-range or read-page
• coverage/freshness status where relevant

Fixture cases should include:

• message hits
• session-only hits from title/summary/compact/reasoning fields
• Codex source behavior
• Claude Code and Pi public source behavior where feasible
• CJK fallback
• cwd selectors
• --sort ended
• --exclude-session
• stale/missing coverage nextAction

Acceptance Criteria

• A single command produces pass/fail evidence-level retrieval results.
• Failures are categorized at least as: no recall, wrong source, wrong session, wrong match source, unreadable context, coverage/staleness issue, or ranking outside top-k.
• Session-only hits with matchSeq = null are explicitly covered.
• The gate is strong enough to block Make query profile scoring real, or remove the stale abstraction #71 and Assemble evidence for session-only recall hits without adding a top-level command #72 regressions.
• Private dogfood data remains private.

Non-Goals

• Do not tune ranking as part of this issue except when required to make the fixture harness testable.
• Do not add heavy retrieval systems before the gate exists.

Trace

• deep review request: cxs_deep_review_correction_20260626T040901Z_api_append
• issue update request: cxs_update_issues_after_review_20260626T044426Z_api_append

[catoncat]

Implementation progress for request_id: cxs_continue_roadmap_to_completion_20260626T054330Z_api_append:

Opened PR #75: #75

This PR implements the evidence-level acceptance gate slice for this issue.

Local evidence / scope:

• Adds an acceptance eval gate around the dogfood retrieval fixtures, so hard retrieval cases assert not just session discovery but also evidence-level expectations.
• Keeps the gate runnable through the project eval path and targeted tests, rather than converting this issue into a broad roadmap item.

Verification:

• npm run eval:acceptance -> passed.
• npx vitest run eval/acceptance-gate.test.ts eval/dogfood-eval-core.test.ts -> passed.
• npm run check -> passed, 31 files / 205 tests.
• Remote PR checks for Add evidence-level acceptance eval gate #75: test passed, cubic · AI code reviewer passed.

Branching:

• PR Add evidence-level acceptance eval gate #75 targets main and is the foundation for the stacked query-profile/source-aware eval work.
• Suggested merge order for the stack: Add evidence-level acceptance eval gate #75 -> Make query profile scoring affect ranking #79 -> Add source-aware acceptance eval fixtures #80.