PRD: Agent-first recall quality packets

[catoncat]

Problem Statement

Sherlog and ctx are highly overlapping local agent-history retrieval tools. The comparison above shows ctx is not merely broader provider support: it has a richer history archive model, event-level retrieval, citations, file/source metadata, SQL access, and compact display modes. In natural dogfood tasks, ctx often matches Sherlog and sometimes finds broader cross-stage evidence.

Sherlog should not respond by becoming a second generic history archive. Sherlog's core principle is that modern coding agents are already strong enough to reason, synthesize, and judge if they are given compact, faithful, source-grounded evidence. The product problem is therefore: Sherlog must serve agents better than a generic archive search tool by returning fast, minimal, high-signal, verifiable recall packets that help an agent answer or act without drowning in transcript noise.

Today Sherlog has real advantages: explicit coverage/sync semantics, source-aware sessionRef reads, deterministic session#seq evidence, relatively low-latency searches, and a developing dogfood gate. But these strengths are not yet packaged as a coherent agent-first recall experience. Large low-signal messages can still dominate context, recall quality is not yet protected by a strong acceptance gate, performance/DB growth are not tracked as first-class product constraints, and Sherlog lacks some of ctx's useful evidence metadata and event/window affordances.

Solution

Build an Agent-First Recall Quality initiative for Sherlog. The goal is not to add more providers or copy ctx's full archive model. The goal is to make the existing Sherlog command surface produce better agent-consumable evidence:

• find should identify likely historical sessions and explain why they are worth reading.
• read-range and read-page should provide compact, deterministic, query-centered evidence without hiding provenance.
• The JSON contract should expose enough metadata for agents to make good next-step decisions without parsing raw transcripts.
• The evaluation suite should move from weak recall smoke tests toward an acceptance gate that measures whether an agent can answer realistic history questions from Sherlog evidence.
• Performance, DB size, and returned-context noise should be tracked as product metrics, not incidental implementation details.

The highest-level product seam is the existing CLI behavior: find followed by read-range / read-page, plus the existing dogfood evaluation runner. Avoid adding a new top-level command. Prefer additive JSON fields, deterministic projection helpers, and eval coverage over a larger command surface.

User Stories

1. As a coding agent, I want find results to include a compact evidence plan, so that I know which read-* command to run next without inventing a strategy.
2. As a coding agent, I want find to explain whether a match came from message text, title, summary, compact text, or reasoning summary, so that I can judge how much to trust the hit.
3. As a coding agent, I want find to return source-aware session references, so that I can read Codex, Pi, and Claude-code sessions without guessing source-specific IDs.
4. As a coding agent, I want read-range to preserve exact session sequence anchors, so that I can cite evidence precisely.
5. As a coding agent, I want read-range to avoid dumping huge low-signal blocks by default, so that my context budget is spent on evidence rather than noise.
6. As a coding agent, I want long pasted messages to be deterministically elided around the relevant query span, so that I still see what matched without consuming hundreds of thousands of characters.
7. As a coding agent, I want elided content to include original size and elision markers, so that I know evidence was shortened and can explicitly request more if needed.
8. As a coding agent, I want read-page to make session-level matches useful even when there is no exact message sequence match, so that summary-level recall can still be verified.
9. As a coding agent, I want Sherlog to distinguish “no result” from “coverage stale/missing,” so that I do not prematurely conclude history does not exist.
10. As a coding agent, I want coverage guidance to stay explicit and conservative, so that read-only history search does not unexpectedly mutate local indexes.
11. As a coding agent, I want query suggestions or broadening hints when an exact long query returns zero results, so that I can recover from over-specific natural-language searches.
12. As a coding agent, I want Chinese and mixed English/Chinese queries to degrade gracefully, so that natural user wording works without manual token surgery.
13. As a coding agent, I want session-diverse results by default, so that one noisy session does not fill all top hits.
14. As a coding agent, I want dense within-session reading once I pick a session, so that I can inspect all relevant local hits in context.
15. As a coding agent, I want source and cwd metadata in results, so that I can decide whether a hit is from the right project or a nearby but irrelevant context.
16. As a coding agent, I want freshness and coverage metadata in the same JSON response as search results, so that I can decide whether to trust, refine, or sync.
17. As a coding agent, I want stable JSON fields for evidence reads, so that skills can build reliable workflows without scraping text output.
18. As a coding agent, I want text output to remain compact and readable, so that humans can still use Sherlog directly.
19. As a coding agent, I want JSON output to be rich enough for automation but not so verbose that it consumes my context window.
20. As a coding agent, I want find to prefer high-signal user/assistant decision turns over repeated boilerplate, logs, or scaffolding noise, so that recall answers are grounded in decisions.
21. As a coding agent, I want huge command output or transcript dumps to be discoverable but not dominant, so that evidence retrieval stays useful.
22. As a coding agent, I want dogfood cases to reflect real user questions rather than tool-specific keyword prompts, so that eval results predict actual agent usefulness.
23. As a coding agent, I want dogfood cases to record expected answer facets, not just expected session IDs, so that recall quality measures whether the agent can answer.
24. As a coding agent, I want dogfood failures to be classified by layer, so that coverage issues, skill guidance issues, ranking issues, and stale golden expectations are fixed differently.
25. As a maintainer, I want candidate dogfood cases to be non-blocking until promoted, so that noisy early cases do not freeze development.
26. As a maintainer, I want hard dogfood cases to block when they represent confirmed recall regressions, so that core recall quality does not slip.
27. As a maintainer, I want performance benchmarks to run on representative query sets, so that Sherlog stays fast as indexes grow.
28. As a maintainer, I want DB size and table-size reports in performance output, so that storage growth is visible before it becomes a user problem.
29. As a maintainer, I want returned-context size metrics, so that improvements can reduce agent context waste without harming recall.
30. As a maintainer, I want top-result diversity metrics, so that ranking changes do not overfit one session family.
31. As a maintainer, I want false-positive and false-negative examples captured from real dogfood, so that future ranking work is evidence-driven.
32. As a maintainer, I want the skill instructions to teach agents the preferred search/read loop, so that agents use Sherlog in the way it is designed.
33. As a maintainer, I want the skill to recommend dogfood capture only when a reproducible recall/ranking/context issue appears, so that private goldens stay curated.
34. As a maintainer, I want the CLI to remain source-aware but not provider-chasing, so that Sherlog differentiates through agent recall quality rather than support breadth.
35. As a maintainer, I want improvements to preserve the existing command surface, so that installed skills and existing users do not need to relearn the tool.
36. As a maintainer, I want additive JSON contract changes, so that existing consumers keep working.
37. As a maintainer, I want current source-of-truth docs to describe what is actually implemented, so that agents do not rely on aspirational architecture.
38. As a user, I want Sherlog to answer “what did we decide before?” quickly, so that I can resume work without re-reading old transcripts.
39. As a user, I want Sherlog to show the evidence behind an answer, so that I can trust or challenge the agent's memory.
40. As a user, I want Sherlog to stay quiet and local, so that it does not mutate indexes or expose private history unless asked.
41. As a user, I want Sherlog to be better than grep for historical intent, so that I get decisions and context rather than arbitrary text matches.
42. As a user, I want Sherlog to be more focused than a generic archive, so that agents get the smallest useful history slice.
43. As a user, I want local history search to avoid self-hits from the current session when possible, so that current prompts do not pollute recall.
44. As a user, I want find and read-* to work well with modern agents, so that the agent can do synthesis rather than the CLI pretending to be the intelligence layer.
45. As a user, I want Sherlog to acknowledge when ctx-like richer archive features are useful, so that the roadmap is honest and competitive.

Implementation Decisions

• Preserve the existing top-level command surface. Do not add a new public command for this initiative.
• Treat the find -> read-range/read-page loop as the main agent workflow and the main product seam.
• Additive JSON fields are preferred over breaking contract changes. Text output should remain concise.
• Introduce an internal “recall packet” concept as a product/API shape, not necessarily a new command. A recall packet should include candidate sessions, match provenance, suggested evidence reads, coverage/freshness status, result diversity metadata, and context-size hints.
• Keep full local evidence available, but make default read output more context-budget aware through deterministic projection and elision.
• Do not use LLM-generated summaries as the default index-time compression layer. Modern agents can synthesize from evidence at query time; Sherlog should provide faithful spans and metadata.
• Add deterministic low-signal handling for very large messages. Large content should remain traceable, but default output should show matched spans, head/tail context, omitted character counts, and explicit expansion instructions.
• Improve long-query fallback. When a natural-language query is over-constrained and returns zero results, return structured guidance for broadening instead of silently looking empty.
• Preserve explicit coverage semantics. sync remains the mutation boundary; read-only commands should not refresh implicitly.
• Prefer session diversity at the search stage and dense context at the read stage.
• Track performance, DB size, and returned-context size as first-class metrics in the eval/perf tooling.
• Expand dogfood from session-ID recall toward answer-facet acceptance. A good case should ask a natural history question and verify that retrieved evidence can support the expected answer.
• Keep dogfood private and curated. Ordinary implementation agents may run existing dogfood gates but must not create or promote goldens without explicit user workflow.
• Use comparison with ctx as an external pressure test, not a runtime dependency and not a product identity. Sherlog should learn from ctx's event/citation ergonomics without becoming a generic archive clone.
• Update the distributable skill so agents naturally use the evidence loop and self-report recall-quality issues in a way that can become dogfood candidates.

Testing Decisions

• Test at the CLI boundary first. The highest-value seam is observable behavior from find, read-range, read-page, and the dogfood/perf runners.
• Unit tests should cover deterministic projection behavior for large messages, elision markers, matched-span preservation, and source/session reference stability.
• Integration tests should cover the full find -> read-* loop using fixture sessions with message-level matches, session-level summary matches, huge pasted content, CJK/mixed-language queries, stale coverage metadata, and zero-result broadening hints.
• Dogfood tests should use natural questions and expected answer facets. They should not reward prompts that are overfit to a specific tool or command.
• Candidate dogfood failures should report but not block. Confirmed hard failures should block local acceptance.
• Performance tests should record median and tail latency for representative queries, result counts, output sizes, DB size, and table-size distribution.
• Regression tests should verify that existing JSON consumers continue to work when new fields are added.
• Skill tests or smoke checks should verify that the installed Sherlog skill still guides agents to use find and read-* before answering historical questions.
• Prior art exists in the current dogfood eval runner, acceptance gate, manual eval runner, compare runner, and performance bench. Extend these rather than introducing a parallel eval framework.

Out of Scope

• Adding broad new provider support solely to match ctx.
• Building a GUI or daemon/watch mode.
• Replacing Sherlog's explicit sync model with implicit search-time refresh.
• Deleting or pruning user history as part of normal read/search.
• Building a full ctx-compatible event archive model in this PRD.
• Adding an LLM reranker or index-time LLM summarizer as the first response.
• Publishing private dogfood goldens or turning dogfood capture into a public user feature.
• Changing the public top-level command list.

Further Notes

The comparison with ctx changed the strategic framing. Sherlog should not claim that it is simply better at recall. The evidence says ctx is competitive and sometimes broader. Sherlog's defensible path is narrower and sharper: serve coding agents with reliable, compact, provenance-rich recall packets under explicit local control.

This PRD assumes the product principle that modern agents are strong. Sherlog should therefore avoid over-modeling intelligence inside the CLI. It should give agents better evidence, better anchors, better context budgeting, better coverage truth, and better acceptance gates.