Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 3 additions & 1 deletion docs/ARCHITECTURE.md
Original file line number Diff line number Diff line change
Expand Up @@ -34,6 +34,8 @@
- 核心层负责 selector、coverage、DB、query/read/list/stats。
- `find` 默认跨 public source fanout 并合并结果;切换 `--source claude-code` 或 `--source pi` 时走同一组命令但只查目标 source。当前非 Codex 语义仍限于 allowlisted transcript text。

Accepted/rejected projection boundaries live in [SOURCE_CONTRACTS.md](SOURCE_CONTRACTS.md). Treat that file as the code-facing contract for parser privacy fixtures; it is not an upstream raw-format stability promise.

Codex adapter 会把原有 `sessionUuid` 映射为 source-aware identity:

- `sourceId = "codex"`
Expand Down Expand Up @@ -144,7 +146,7 @@ CLI 默认 `find` 会对 public sources 执行单源 `findSessions()` fanout,

下面这些不要误写成现状:

- 真正按 broad/exact query profile 分权的 scoring
- 真正按 broad/exact query profile 分权的 scoring(真实 A/B 未证明有效,当前仍不落地)
- richer projection / event replay / range cache
- duplicate collapse / diversity control
- 强约束 gold set / rubric / error taxonomy
Expand Down
3 changes: 2 additions & 1 deletion docs/RANKING_WEIGHTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -46,12 +46,13 @@
```
normalizedBm25
+ (contentPhrase ? 8 : 0)
+ commandSequenceBonus
+ termCoverage * 2
+ (matchSource === "message" ? 4 : 0)
+ (matchRole === "user" ? 2 : 0)
```

四项的设计意图是:**bm25 + 三个 token-free 的硬证据**。bm25 处理“词义相关性”,这三项处理 bm25 看不到的元信息
这些项的设计意图是:**bm25 + token-free 的硬证据**。bm25 处理“词义相关性”,其余项处理 bm25 看不到的短语邻接、命令 token 邻近、消息可回读性与作者角色

### `contentPhrase ? 8 : 0`

Expand Down
68 changes: 68 additions & 0 deletions docs/SOURCE_CONTRACTS.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,68 @@
# Source Adapter Contracts

This document describes the searchable projection contract for Sherlog public source adapters. It is not a promise that upstream raw transcript formats are stable. `claude-code` and `pi` remain experimental transcript-reader support until upstream formats and fixtures are stronger.

## Shared Invariants

- Source adapters implement the boundary in `src/sources/types.ts`: root resolution, file collection, inventory, snapshot, and `parseFile`.
- Indexing stores only accepted transcript projections: session metadata, raw user/assistant text where allowed, and a small set of session-level handoff fields.
- Unsupported/private records are filtered by default. Tool outputs, attachments, diagnostics, sidechain/meta records, and thinking blocks must not enter `messages_fts` or `sessions_fts` unless a future issue adds an explicit privacy design.
- Read isolation is source-aware. `sessionRef` values returned by `find` can be passed back to `read-range` or `read-page` without guessing the source.
- Malformed JSONL records are skipped instead of failing the whole file.

## Codex

Accepted projection:

- `session_meta` id and cwd metadata.
- `turn_context` model and cwd metadata.
- `event_msg` records whose payload type is `user_message` or `agent_message` and whose `message` is non-empty text.
- `compacted` payload text as session-level `compactText`.
- `response_item` reasoning summaries as session-level `reasoningSummaryText`.

Rejected projection:

- malformed JSONL lines.
- unrelated record types.
- `event_msg` records with other payload types, including tool-result-like records.
- empty user/assistant messages.
- internal approval/evaluation transcript markers filtered by `looksInternal()`.

## Claude Code

Accepted projection:

- policy-approved user and assistant text records from `claude-code-policy`.
- accepted session id, cwd, and timestamp metadata only from accepted records.

Rejected projection:

- meta records.
- sidechain records.
- tool results.
- thinking blocks.
- attachment-like non-text content.
- skipped records when deriving inventory grouping and snapshots.

## Pi

Accepted projection:

- `session` metadata with id, cwd, and timestamp.
- latest `model_change` model id.
- user/assistant text content from `message` records.
- `compaction` summary text as session-level `compactText`.

Rejected projection:

- tool-result roles.
- tool-call content.
- thinking content.
- unsupported content parts.
- malformed or incomplete records.

## Current Contract Limits

- The contract covers what Sherlog indexes and reads from synthetic fixtures, not every upstream raw variant.
- Do not broaden accepted record classes to improve recall without adding privacy-specific tests first.
- Keep source-specific parser changes paired with negative fixtures that prove private or diagnostic data is still filtered.
26 changes: 26 additions & 0 deletions eval/acceptance-gate.test.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,26 @@
import { describe, expect, test } from "vitest";
import { runAcceptanceGate } from "./acceptance-gate";

describe("acceptance gate", () => {
test("passes synthetic evidence-level retrieval fixtures", async () => {
const result = await runAcceptanceGate();

expect(result.sync.added).toBe(4);
expect(result.sourceSyncs["claude-code"].added).toBe(1);
expect(result.sourceSyncs.pi.added).toBe(1);
expect(result.scoreboard).toMatchObject({
total: 5,
pass: 5,
fail: 0,
hardFail: 0,
});
expect(result.rows.map((row) => row.id)).toEqual([
"message-hit-context",
"session-only-compact-context",
"cjk-message-hit",
"claude-code-message-range-context",
"pi-session-page-context",
]);
expect(result.rows.every((row) => row.predicates.length > 0)).toBe(true);
});
});
Loading