Deep review: evidence-first agent memory gaps for Sherlog

[catoncat]

Deep review: evidence-first agent memory gaps for Sherlog

This issue is the single umbrella for the June 26, 2026 deep review of Sherlog against the agent-memory framework in arXiv:2606.24775. It replaces the earlier generic roadmap wording with a code-aware review summary.

This is not a roadmap release and not a request to replace Sherlog's current architecture. The goal is to keep the issue tracker aligned with the current codebase and with executable next work.

Paper Lens

The review uses the paper's four-module framework:

• Memory representation and storage
• Memory extraction
• Memory retrieval and query routing
• Memory maintenance

It also uses the paper's five evaluation questions:

• Overall effectiveness
• Retrieval fidelity
• Dynamic update robustness
• Long-horizon stability
• Operation cost

Current Sherlog Baseline

Sherlog is best understood as a local evidence locator for agent session logs, not a full autonomous memory bank. The implemented retrieval chain is:

status -> sync -> message/session recall -> heuristic rerank -> read-range/read-page

Local evidence:

• AGENTS.md fixes the command surface to status, sync, find, read-range, read-page, list, and stats; it also says sync is the only writer and there is no watcher/daemon/realtime sync.
• docs/ARCHITECTURE.md describes the current chain as message/session recall plus heuristic rerank, not a full resource-level reranker.
• docs/INDEX_COVERAGE_DESIGN.md defines the key invariant: content answers come from the index, while raw sessions are used for inventory/sync planning.
• src/db/schema.ts defines the session/message/FTS/coverage representation.
• src/indexer.ts implements strict sync, snapshot checks, transactions, coverage writing, and optional prune.
• src/query/search.ts, src/query/find.ts, and src/ranking.ts implement message/session FTS, CJK fallback, and heuristic session ranking.
• src/query/read.ts exposes the raw transcript context through range/page reads.

What Is Already Solid

• Representation and storage are strong for Sherlog's scope: sessions, messages, messages_fts, sessions_fts, and coverage preserve source-aware transcript evidence.
• Extraction is intentionally conservative: Codex user/assistant event_msg records become messages; compact/reasoning material becomes session-level recall text.
• Dynamic freshness is stronger than a plain append-only store: coverage, source fingerprints, file-set fingerprints, stale reasons, and recommended actions are implemented.
• The design preserves raw transcript context instead of replacing it with generated memory facts.

Main Gaps

• Public evidence-level eval is still too weak. eval/manual-eval-core.ts only checks title/summary, cwd, and snippet; eval/dogfood-eval-core.ts has stronger private predicates but they are not a shareable gate.
• Session-level recall is implemented but evidence assembly is incomplete. src/query-session-fields.test.ts verifies matchSource = "session" and matchSeq = null; src/query/read.ts currently anchors precise ranges through message hits.
• Query profile classification exists but is not actually used as a scoring/routing branch. src/query-profile.test.ts covers classification; src/ranking.ts documents that v3 scoring does not branch on broad/exact kind.
• Source adapter hardening is partly implemented, especially for Claude Code, but parity/contract coverage still needs work.
• Operation cost tracking exists in fragments, but not yet as a memory-systems RQ5 view across sync/find/read/source/freshness dimensions.

Linked Executable Work

• P0: Strengthen retrieval eval into an evidence-level acceptance gate #70: P0 evidence-level eval gate
• Make query profile scoring real, or remove the stale abstraction #71: query profile scoring or removal of stale abstraction
• Assemble evidence for session-only recall hits without adding a top-level command #72: session-only hit evidence assembly
• Harden experimental source adapters with parity privacy and contract fixtures #73: source adapter contract/privacy fixture hardening
• Track local memory operation costs across sync, find, read, and freshness #74: local operation cost telemetry

Non-Goals For Now

• Do not add vector/graph/heavy reranking before P0: Strengthen retrieval eval into an evidence-level acceptance gate #70 can catch retrieval regressions.
• Do not add a new top-level command for evidence assembly; use the existing find surface if this work proceeds.
• Do not replace raw transcript evidence with extracted facts or summaries.
• Do not add watcher/daemon/realtime/implicit sync.

Trace

• deep review request: cxs_deep_review_correction_20260626T040901Z_api_append
• issue update request: cxs_update_issues_after_review_20260626T044426Z_api_append

[catoncat]

Roadmap implementation checkpoint for request_id: cxs_continue_roadmap_to_completion_20260626T054330Z_api_append:

Current implementation coverage from the deep-review roadmap:

• P0: Strengthen retrieval eval into an evidence-level acceptance gate #70 -> PR Add evidence-level acceptance eval gate #75: evidence-level acceptance eval gate. Remote checks passed. Base main.
• Make query profile scoring real, or remove the stale abstraction #71 -> PR Make query profile scoring affect ranking #79: query profile scoring affects ranking. Remote checks passed. Base agent/sherlog-memory-review-stack, depends on Add evidence-level acceptance eval gate #75.
• Assemble evidence for session-only recall hits without adding a top-level command #72 -> PR Add deterministic evidence reads to find JSON #76: deterministic evidence reads in find JSON output. Remote checks passed. Base main.
• Harden experimental source adapters with parity privacy and contract fixtures #73 -> PR Document source adapter projection contracts #78: source adapter projection/privacy contracts. Remote checks passed. Base main.
• Harden experimental source adapters with parity privacy and contract fixtures #73 -> PR Add source-aware acceptance eval fixtures #80: source-aware acceptance eval fixtures for Claude Code and Pi. Remote checks passed. Base agent/issue-71-query-profile, depends on Add evidence-level acceptance eval gate #75 and Make query profile scoring affect ranking #79.
• Harden experimental source adapters with parity privacy and contract fixtures #73 -> PR Add source adapter format-drift smoke tests #81: malformed JSONL / format-drift source adapter smoke tests. GitHub Actions test passed; AI review check was still in progress when checked. Base main.
• Track local memory operation costs across sync, find, read, and freshness #74 -> PR Extend perf bench operation cost telemetry #77: local operation-cost telemetry in perf bench. Remote checks passed. Base main.

Branching / merge map:

• Independent PRs that can merge into main: Add deterministic evidence reads to find JSON #76, Extend perf bench operation cost telemetry #77, Document source adapter projection contracts #78, Add source adapter format-drift smoke tests #81.
• Stacked line: Add evidence-level acceptance eval gate #75 -> Make query profile scoring affect ranking #79 -> Add source-aware acceptance eval fixtures #80.
• Add evidence-level acceptance eval gate #75 should merge before Make query profile scoring affect ranking #79, and Make query profile scoring affect ranking #79 before Add source-aware acceptance eval fixtures #80. This avoids pretending dependent eval/ranking/source-aware work can merge independently.

Current blocker state:

No additional local implementation slice is obvious from the current roadmap without review/merge feedback. The remaining work is PR review/merge sequencing and then closing the corresponding issues once the linked PRs land.

[catoncat]

Integration branch checkpoint for request_id: cxs_status_and_merge_integration_20260626T063048Z_api_append:

Unified review branch updated:

• Branch: agent/sherlog-memory-review-stack
• Head commit: 9657abc44d19103abe47b5925f1e50e0a75bd334
• GitHub branch: https://github.com/catoncat/sherlog/tree/agent/sherlog-memory-review-stack

Included roadmap work:

• P0: Strengthen retrieval eval into an evidence-level acceptance gate #70 / PR Add evidence-level acceptance eval gate #75: evidence-level acceptance eval gate (f40fa32 already present on integration branch).
• Make query profile scoring real, or remove the stale abstraction #71 / PR Make query profile scoring affect ranking #79: query profile scoring affects ranking (80a970e, now merged into the integration branch because Make query profile scoring affect ranking #79 targeted that branch).
• Harden experimental source adapters with parity privacy and contract fixtures #73 / PR Add source-aware acceptance eval fixtures #80: source-aware acceptance eval fixtures (1c97c73).
• Assemble evidence for session-only recall hits without adding a top-level command #72 / PR Add deterministic evidence reads to find JSON #76: deterministic evidence reads for find JSON (b765fc5, merged into integration as 322dc6b).
• Track local memory operation costs across sync, find, read, and freshness #74 / PR Extend perf bench operation cost telemetry #77: operation-cost telemetry (592c700, merged into integration as 1873fa7).
• Harden experimental source adapters with parity privacy and contract fixtures #73 / PR Document source adapter projection contracts #78: source adapter projection contracts (dc7fbc1, merged into integration as 1c3abf7).
• Harden experimental source adapters with parity privacy and contract fixtures #73 / PR Add source adapter format-drift smoke tests #81: malformed JSONL / format-drift source adapter smoke tests (bae64c6, merged into integration as 9657abc).

Verification after integration:

• npm run check -> passed, 32 test files / 212 tests.
• npm run eval:acceptance -> passed, 6/6 cases, 0 failures.
• git diff --check origin/agent/sherlog-memory-review-stack...HEAD -> passed.

Important status note:

• main was not updated.
• PR Make query profile scoring affect ranking #79 is marked merged because its base was the integration branch.
• PRs Add evidence-level acceptance eval gate #75, Add deterministic evidence reads to find JSON #76, Extend perf bench operation cost telemetry #77, Document source adapter projection contracts #78, Add source-aware acceptance eval fixtures #80, and Add source adapter format-drift smoke tests #81 remain open unless/until separately merged through their configured targets.