Make query profile scoring real, or remove the stale abstraction

[catoncat]

Review umbrella: #69

Problem

Sherlog has query profile classification, but current scoring does not actually branch on broad vs exact query kind. That makes the abstraction easy to misunderstand and risky to tune by intuition.

The paper's retrieval/query-routing module argues for evaluating routing precision explicitly. For Sherlog, that does not mean adding a natural-language planner; it means either using the existing profile in a small eval-backed scoring branch or deleting the stale abstraction.

Local Evidence

• src/ranking.ts:classifyQueryProfile() defines broad/exact/path-like query metadata.
• src/ranking.ts comments state that v3 scoring no longer branches on kind and primarily uses local row/session signals.
• src/query-profile.test.ts verifies classification for broad terms, multi-term exact queries, digits, and path-like command strings.
• src/query-session-ranking.test.ts already covers several ranking behaviors: title hit vs incidental mention, sustained broad evidence vs title-only hit, distinct sessions with collapsed titles, and path-like command query ranking.
• docs/ROADMAP.md and AGENTS.md both identify broad/exact scoring as not fully landed.

Scope

After #70 exists, choose one of two paths:

1. Make query profile affect scoring in a narrow, measured way.
2. Remove or de-emphasize classifyQueryProfile() if the current heuristic ranking is intentionally profile-agnostic.

If path 1 is chosen, candidate changes should stay conservative:

• Exact/path-like queries can reward phrase adjacency, command/path boundedness, and full term coverage.
• Broad single-term queries can reward sustained session evidence and avoid over-weighting incidental title/cwd hits.
• Time sorting should remain explicit through --sort ended; do not infer time scopes from free text in this issue.

Acceptance Criteria

• P0: Strengthen retrieval eval into an evidence-level acceptance gate #70 contains eval cases that fail before and pass after any scoring change.
• Scoring behavior either uses query profile kind meaningfully or the stale abstraction is removed.
• Ranking docs and tests explain the chosen behavior.
• Existing source-aware/session-level recall behavior remains stable.

Non-Goals

• No broad natural-language planner.
• No vector or graph retrieval.
• No ranking changes without evidence-level eval coverage.

Trace

• deep review request: cxs_deep_review_correction_20260626T040901Z_api_append
• issue update request: cxs_update_issues_after_review_20260626T044426Z_api_append

[catoncat]

Progress update for request_id cxs_continue_roadmap_to_completion_20260626T054330Z_api_append:

Opened stacked PR #79 for the measured "make query profile affect scoring" path.

Scope covered:

• src/ranking.ts now carries QueryProfile.kind into scoring.
• broad queries keep stronger sustained-evidence/session-level weighting.
• exact/path-like queries reward adjacent phrase evidence, full term coverage, and readable message-row phrase hits.
• exact queries slightly reduce split title/cwd term weighting so metadata fragments do not outrank adjacent message evidence.
• eval/acceptance-gate.ts now includes exact-query-profile-phrase, a hard P0: Strengthen retrieval eval into an evidence-level acceptance gate #70 gate case for this behavior.
• src/ranking.test.ts adds the focused unit regression.
• docs/RANKING_WEIGHTS.md documents the chosen branch and weights.

Verification:

• npx vitest run src/ranking.test.ts src/query-session-ranking.test.ts eval/acceptance-gate.test.ts passed: 3 files / 14 tests.
• npm run eval:acceptance passed: 6 synthetic sessions, 4/4 hard cases, 0 hard failures.
• npm run check passed: TypeScript plus 31 test files / 206 tests.
• git diff --check -- docs/RANKING_WEIGHTS.md eval/acceptance-gate.test.ts eval/acceptance-gate.ts src/ranking.test.ts src/ranking.ts passed.

Branching:

• PR Make query profile scoring affect ranking #79 base: agent/sherlog-memory-review-stack
• Depends on: Add evidence-level acceptance eval gate #75 / P0: Strengthen retrieval eval into an evidence-level acceptance gate #70
• Suggested merge order: Add evidence-level acceptance eval gate #75 first, then Make query profile scoring affect ranking #79 after stack/base alignment. Independent PRs Add deterministic evidence reads to find JSON #76, Extend perf bench operation cost telemetry #77, and Document source adapter projection contracts #78 do not block it.

Remaining #71 status:

• Implementation slice is ready for review in Make query profile scoring affect ranking #79.
• Keep this issue open until the stacked dependency is resolved and the PR is retargeted/merged into the final integration path.