Harden experimental source adapters with parity privacy and contract fixtures

[catoncat]

Review umbrella: #69

Problem

Claude Code and Pi source support are public but experimental. The deep review found that Claude Code privacy filtering is not empty work; it is already partially covered. The remaining issue is to turn source adapter behavior into a clear, parity-tested contract across supported public sources.

This maps to the paper's memory extraction and maintenance modules: source adapters decide what becomes memory, what is ignored, and how robust the system is under upstream format drift.

Local Evidence

• src/sources/types.ts:SessionSourceAdapter defines the adapter boundary: default root, root resolution, file collection, inventory, snapshot, and parseFile.
• src/sources/registry.ts registers codex, claude-code, and pi.
• src/sources/codex-parser.ts parses Codex event_msg user/assistant messages and stores compact/reasoning as session-level fields.
• src/sources/claude-code-parser.ts accepts only policy-approved text records and leaves compact/reasoning empty.
• src/sources/pi-parser.ts accepts message and compaction records and builds compact text.
• src/sources/claude-code.test.ts already verifies that meta, sidechain, tool_result, thinking, and attachment-like records do not leak into searchable projection.
• docs/ARCHITECTURE.md and docs/USAGE.md describe Claude/Pi support as public but experimental.

Scope

Make source adapter contracts explicit and parity-tested.

Work items:

• Document accepted vs rejected record classes for each public source.
• Add or extend negative fixtures for private/diagnostic/tool/attachment/thinking/sidechain-like records.
• Add malformed transcript fixtures and format-drift smoke tests.
• Add source-aware recall/eval cases that cover find, read-range, and read-page across public sources where fixtures exist.
• Keep Claude Code and Pi experimental until the contract and fixtures justify stronger claims.

Acceptance Criteria

• Adapter tests distinguish already-supported behavior from new hardening.
• Unsupported/private record types are filtered by default and covered by tests.
• Source-aware session refs remain deterministic and isolated.
• Docs accurately state experimental status and contract limits.
• P0: Strengthen retrieval eval into an evidence-level acceptance gate #70 has at least basic source-aware eval coverage.

Non-Goals

• Do not broaden accepted record types just to improve recall.
• Do not index tool outputs, attachments, diagnostics, or private metadata without an explicit privacy design.
• Do not promise stable upstream format support beyond tested contracts.

Trace

• deep review request: cxs_deep_review_correction_20260626T040901Z_api_append
• issue update request: cxs_update_issues_after_review_20260626T044426Z_api_append

[catoncat]

Progress update for request_id cxs_continue_roadmap_to_completion_20260626T054330Z_api_append:

Opened PR #78 for the independent source-adapter contract slice.

Scope covered:

• Added docs/SOURCE_CONTRACTS.md documenting accepted/rejected searchable projection classes for Codex, Claude Code, and Pi.
• Linked the contract from docs/ARCHITECTURE.md while keeping Claude Code and Pi marked experimental.
• Added src/sources/codex.test.ts coverage proving malformed Codex JSONL, unsupported records, tool-result-like payloads, and internal transcript markers do not enter the searchable projection.

Verification:

• npx vitest run src/sources/codex.test.ts passed: 1 file / 3 tests.
• npm run check passed: TypeScript plus 30 test files / 204 tests.
• git diff --check -- docs/ARCHITECTURE.md docs/SOURCE_CONTRACTS.md src/sources/codex.test.ts passed.

Branching:

• PR Document source adapter projection contracts #78 base: main
• Depends on: none
• Merge order: independent of Add evidence-level acceptance eval gate #75, Add deterministic evidence reads to find JSON #76, and Extend perf bench operation cost telemetry #77.

Remaining #73 work:

• Source-aware recall/eval coverage should stack on agent/sherlog-memory-review-stack after P0: Strengthen retrieval eval into an evidence-level acceptance gate #70/Add evidence-level acceptance eval gate #75 is available there.
• Additional malformed/format-drift parity fixtures for non-Codex sources can follow as smaller PRs if needed.

[catoncat]

Progress update for the source-aware eval slice: opened #80.

What #80 covers:

• Adds synthetic acceptance fixtures for claude-code and pi in eval/acceptance-gate.ts.
• Keeps the existing Codex acceptance fixture path as the compatibility baseline.
• Verifies Claude Code source-qualified find -> read-range for message evidence.
• Verifies Pi source-qualified session-level compaction find -> read-page for context.
• Leaves production parser/source adapter projection behavior unchanged.

Local evidence:

• eval/acceptance-gate.ts: new per-source fixture roots and per-source sync summaries.
• eval/acceptance-gate.test.ts: acceptance gate now asserts Codex + Claude Code + Pi sync and 6/6 hard-case pass coverage.
• Existing source-aware read path remains through src/query/read.ts:getMessageRange and src/query/read.ts:getMessagePage with source-qualified session refs.

Verification on branch agent/issue-73-source-aware-eval / commit 1c97c73:

• npx vitest run eval/acceptance-gate.test.ts -> passed.
• npm run eval:acceptance -> 6 hard cases, 6 pass, 0 fail; Codex added 6, Claude Code added 1, Pi added 1.
• npm run check -> 31 files passed, 206 tests passed.
• git diff --check -- eval/acceptance-gate.ts eval/acceptance-gate.test.ts -> passed.

Dependency note:

#80 is stacked on #79, which is stacked on #75. Suggested merge order is #75 -> #79 -> #80. This issue should remain open until the broader #73 contract/parity work is either fully merged or explicitly trimmed.

request_id: cxs_continue_roadmap_to_completion_20260626T054330Z_api_append

[catoncat]

Progress update for request_id: cxs_continue_roadmap_to_completion_20260626T054330Z_api_append:

Opened PR #81: #81

This is an independent, test-only slice for the source-adapter hardening work. It adds malformed JSONL / format-drift smoke coverage for the experimental Claude Code and Pi adapters.

Local evidence:

• src/sources/claude-code.test.ts: adds a fixture with malformed JSON, unsupported record types, accepted user/assistant messages, and attachment-like content; verifies unsupported text does not leak into findSessions or getMessagePage.
• src/sources/pi.test.ts: adds a fixture with malformed JSON, unsupported future events, accepted messages, unsupported tool-call content, and a compaction summary; verifies unsupported text does not leak while accepted session/message evidence remains searchable.

Verification:

• npx vitest run src/sources/claude-code.test.ts src/sources/pi.test.ts -> passed, 2 files / 20 tests.
• npm run check -> passed, TypeScript plus 30 files / 205 tests.
• git diff --check -- src/sources/claude-code.test.ts src/sources/pi.test.ts -> passed.

Branching:

• PR Add source adapter format-drift smoke tests #81 targets main and has no dependency on the stacked Add evidence-level acceptance eval gate #75 -> Make query profile scoring affect ranking #79 -> Add source-aware acceptance eval fixtures #80 line.
• It complements Document source adapter projection contracts #78 and Add source-aware acceptance eval fixtures #80 and can merge independently.