diff --git a/.plans/29-unified-transcript-log.md b/.plans/29-unified-transcript-log.md new file mode 100644 index 0000000..0a7ea59 --- /dev/null +++ b/.plans/29-unified-transcript-log.md @@ -0,0 +1,301 @@ +# Unified neutral transcript (one canonical log) — plan v3 + +Introduce a single provider-agnostic, JSON-serializable conversation transcript as the harness's canonical record of a run, and make `resume_state` (and the approval-pause `provider_state`) a projection of it instead of a per-provider native blob. + +Revised after two multi-review rounds (`.reviews/plans/unified-transcript-log/*-v{1,2}.md`). All reviewers endorsed the core design across both rounds; v2 fixed the v1 correctness bugs (OpenAI deferred replay, system re-injection, Anthropic coalescing, notice preservation, version bump), and v3 resolves the v2 implementation-spec precision items (chiefly: the backing store is explicitly **dual-store / additive**, not a replacement). Findings-resolution tables for both rounds are at the end. + +## Goal + +A run's durable state should have **one representation**, not three. Today the conversation exists as: + +1. Provider-native session state (`AnthropicMessagesSession.messages` in Anthropic block format, `OpenRouterSession.messages` in chat format, `OpenAIResponsesSession.previous_response_id` as a server-side pointer). +2. `HarnessResult.resume_state` — whatever (1) serializes to, tagged by provider `kind`. +3. The live `StreamEvent` queue and OTel spans — separate ephemeral projections. + +This plan collapses (1) and (2) into a neutral `Transcript` that every provider renders to/from. After this change: + +- `resume_state` is provider-agnostic: a run captured on `anthropic:claude-...` can be resumed on `openai:gpt-...` (with documented graceful degradation), and vice versa. +- `resume_state` is self-contained and no longer depends on OpenAI server-side response retention (the ~30-day expiry footgun in `.plans/08-resume.md`). +- The approval-pause envelope's `provider_state` is the same neutral transcript, so both resume paths unify. +- The neutral transcript is positioned to become the single source the event stream and tracing derive from (deferred — see "Out of scope"). + +This is the architecture pydantic-ai, strands, and agno converged on. The cost is bounded because thinharness has only three built-in providers and is already half-neutral on the harness↔provider boundary. + +## What this is and is not + +**Is:** a neutral *transcript / log* — one durable representation of what happened — added *alongside* the existing provider-native in-run request builders, which are unchanged. + +**Is not:** a neutral *capability surface* (LiteLLM-style lowest-common-denominator), and **not** a replacement of the in-run request path. Provider-specific request settings (`ModelSettings.extra_body`, Anthropic `max_tokens`, structured-output modes) keep flowing through unchanged. The transcript records the conversation, not the request knobs, and is consulted only to produce/restore durable state. + +## Current state (file:line) + +The harness↔provider boundary is **already neutral in both directions**: + +- **Out:** every provider returns `ModelTurn` (`providers.py:31` — `text`, `tool_calls: list[ModelToolCall]`, `raw`). The run loop never reads provider-native responses; it appends `turn.raw` to `run_ctx.responses` (`core.py:526`) and reads `turn.text` / `turn.tool_calls`. +- **In:** the loop drives sessions with neutral inputs — `start(prompt=...)`, `continue_with_tools(outputs: list[ToolOutput])`, `continue_with_user_message`, `continue_with_user_prompt`, plus `ModelNotice` / `StructuredOutputRequest` (`providers.py:115-173`). + +What is **not** neutral is only the accumulated transcript each session keeps, appended after each completion: + +- `AnthropicMessagesSession` — Anthropic block format; appends at `providers.py:614` (start user), `:634` (tool_result batch), `:653` (corrective user), `:669` (resumed user prompt), `:697` (assistant turn). Notices are baked into user content via `append_notices_to_text` at `:614/:653/:669` and as a trailing text block in the tool batch at `:632-633`. +- `OpenRouterSession` — chat format; appends at `:762/:780/:797/:811/:845`. Notices baked in at `:764/:797/:811` and a trailing user message at `:781-783`. +- `OpenAIResponsesSession` — no client transcript; only `previous_response_id`, set from each response id (`:541`). Notices appended to input text at `:440/:495/:517` and as a user message item in the tool path at `:467-471`. + +`TurnDriver` can attach `ModelNotice`s to `start`, resumed prompts, corrective messages, and tool outputs (`runtime.py:67/:81/:122`), so notices are model-visible on every entry point, not just tool continuations. + +`dump_state()` (`providers.py:171`) serializes whichever native blob the session kept, tagged with a provider `kind`. That tag is the entire reason resume is provider-bound. Both durable-state consumers funnel through `dump_state()`: + +- `HarnessResult.resume_state` via `run_ctx.finalize(..., require_dump_state=model_supports_resume)` (`core.py:532-538`). +- The approval-pause envelope's `provider_state` (built in `run_ctx.pause_for_approval`, `core.py:577`; restored via `self._resume_approval_session(approval_pause.provider_state)` → `resume_session(...)`, `core.py:465`, `:720`). + +So neutralizing `dump_state`/`resume_session` fixes **both** durable paths. + +The validation/lifecycle rules from `.plans/08-resume.md` (clean-`end_turn`-only, no resume after `final_result`, no transcript repair) stay in force. + +## Design + +### 1. The neutral `Transcript` type (`providers.py`, new leaf types) + +An ordered list of neutral entries. Pure data, JSON-serializable via an explicit to-dict/from-dict mapping with a `role` discriminator (dataclasses are not directly `json.dumps`-able, and the durable path does `json.loads(json.dumps(state))` — `runtime.py`/`approvals.py:87`). + +```python +@dataclass +class AssistantEntry: + text: str + tool_calls: list[ModelToolCall] # reuse existing leaf type + +@dataclass +class UserEntry: + content: str # fully rendered, provider-neutral user text + notice: bool = False # True only for batch-accompanying notice text (grouping hint) + +@dataclass +class ToolResultEntry: + call_id: str + output: str + +TranscriptEntry = AssistantEntry | UserEntry | ToolResultEntry +``` + +**Serialization (load-bearing — specify exactly).** `dump_state` emits plain dicts with a `role` discriminator; the validator/renderers reconstruct from those dicts: + +```python +{"role": "user", "content": "...", "notice": false} +{"role": "assistant", "text": "...", "tool_calls": [{"id": "...", "name": "...", "arguments": "..."}]} +{"role": "tool", "call_id": "...", "output": "..."} +``` + +`ModelToolCall.arguments` stays a JSON string in the transcript. Renderers needing a parsed object (Anthropic `tool_use.input`) call `json.loads(arguments)` (mirroring `_extract_anthropic_tool_calls`'s `json.dumps(input)` at `providers.py:994`). + +**Notice text is captured in `UserEntry.content`, faithfully (no notice is dropped).** Because notices are model-visible on *all* entry points (current state above), v3 records them wherever they appear: + +- `start` / `continue_with_user_prompt` / `continue_with_user_message`: the `UserEntry.content` stores the **notice-appended** text — exactly the value `append_notices_to_text(text, notices)` produces and that current in-run tests pin (`test_providers.py:148/:154-155/:172`) — not the raw prompt. `notice=False` (it is a normal user turn that happens to carry appended guidance). +- `continue_with_tools`: tool results become `ToolResultEntry`s, and any rendered notice text becomes a trailing `UserEntry(notice=True)` so the Anthropic renderer can fold it back into the tool-result user message (role alternation; §3). + +Dropping stale notices from the durable transcript is a separate, intentional behavior change deferred to a follow-up; v3 preserves current behavior. + +Notes: + +- **No `system` in the stream.** The system prompt and tool schemas are the run's "versioned constant" (supplied by `HarnessConfig` each turn via `instructions`/`tools`), not part of the transcript. Resume re-derives them from live config — see §3. +- **v1 stores reasoning as text only.** No opaque provider blobs (Anthropic thinking `signature`, OpenAI `encrypted_content`/reasoning item ids). Same-provider reasoning fidelity is deferred (Out of scope). +- **No per-entry `provider_name` and no `_parse_extras` in v1.** Envelope-level `origin_provider` covers diagnostics; per-entry provenance is only needed by deferred fidelity work. +- **Assistant content ordering is intentionally flattened.** `AssistantEntry` stores concatenated `text` + a separate `tool_calls` list, matching what the harness already sees (`ModelTurn.text` is the concatenation of all text blocks via `_extract_anthropic_text`, `providers.py:998`). A `text → tool_use → text` native turn round-trips as `text + tool_use`. This is consistent with every existing downstream consumer; an accepted, tested degradation, not a regression. (See §2 — it affects only `dump_state`, never the in-run request.) +- `ModelToolCall.id` is the portable correlation key between an `AssistantEntry.tool_calls[i]` and the matching `ToolResultEntry.call_id`. + +### 2. Provider contract: dual store (native in-run builder + parallel neutral transcript) + +**This is Option A and it is explicit: the native in-run request builders are retained; the neutral transcript is purely additive.** Concretely: + +- `AnthropicMessagesSession.messages`/`.system`, `OpenRouterSession.messages`, and `OpenAIResponsesSession.previous_response_id` **stay** and remain the source of every in-run request, byte-for-byte as today (OpenRouter still stores the raw assistant message at `providers.py:845`, preserving any `reasoning`/`reasoning_details` during the live run; OpenAI still chains `previous_response_id` and sends only new items). +- Each session **also** maintains a neutral `Transcript`, appending entries as it already receives/produces them: `start`/`continue_with_user_*` append a `UserEntry` (notice-appended content per §1); `continue_with_tools` appends `ToolResultEntry`s + optional `UserEntry(notice=True)`; `_complete` appends an `AssistantEntry(text, tool_calls)` from the existing `_extract_*` helpers (`providers.py:968-1009`). +- `_render_input(entries, *, instructions)` is invoked **only on the first post-resume turn**, never during a normal in-run turn. It is gated on a per-session resume flag (`_resume_entries` for stateless providers, `_pending_replay` for OpenAI) that is set by `resume_session` and consumed exactly once. + +Consequence (and the implementation guard): **all in-run payload tests must remain byte-identical** — `test_harness.py:226-242`, `test_providers.py:60-74`, and `test_providers.py:92-175` (minus the resume sub-case at `:106-120`). If any in-run payload test changes, the dual-store boundary leaked (Option B crept in) and must be corrected. The assistant-text flattening and any per-turn re-rendering apply to resumed turns only. + +The four `continue_with_*` bodies stay distinct — each renders to its native wire format and gains one neutral-append line. They do not "collapse." + +### 3. Provider-agnostic `dump_state` / `resume_session` lifecycle + +```python +# neutral envelope — same shape for every provider; allowed keys are exactly this set +{"kind": "transcript", "version": 2, + "origin_provider": "anthropic", # normalized model-ref prefix (see below); required + "origin_model": "claude-...", # required + "entries": [ {"role": "user", ...}, {"role": "assistant", ...}, {"role": "tool", ...}, ... ]} +``` + +- `dump_state()` serializes the neutral transcript. `origin_provider` is the **normalized lowercase prefix** via `provider_prefix(self.model.provider.name)` (`providers.py:888-895`) — `self.model.provider.name` is capitalized (`"Anthropic"`/`"OpenAI"`/`"OpenRouter"`, `:279/:303/:329`), so it must be normalized to match model-ref prefixes. `origin_provider`/`origin_model` are **required** envelope fields; the allowed-key set is exactly `{kind, version, origin_provider, origin_model, entries}`. +- `resume_session(state)` (any provider) validates the neutral envelope and prepares the session for full replay. It **must not make a model request** (it runs before hooks/`_running`), so the rendered replay is deferred to the first post-resume turn, not produced inside `resume_session`. + +**Replay injection is entry-point-agnostic.** A resumed run's first provider call is `continue_with_user_prompt` (normal resume) or `continue_with_tools` (approval resume — `core.py:492` sets `skip_user_prompt`; `_resume_approval_batch` enters through `send_tool_outputs` → `continue_with_tools`, `core.py:778`). The first such turn renders the transcript prefix and injects it, then proceeds normally; the resume flag is consumed so subsequent turns use the native in-run path. + +- **Anthropic / OpenRouter (stateless).** `resume_session` stores `self._resume_entries = entries` (it cannot render yet — no `instructions`). The first post-resume turn renders entries into `self.messages`, sets/prepends the system prompt from the live `instructions`, appends the new turn's content, completes, and clears `_resume_entries`. +- **OpenAI (server-side).** `resume_session` stores `self._pending_replay = entries`, leaves `previous_response_id = None`. The first post-resume turn renders entries into Responses `input` items and produces the final `input` as `[*replay_items, *entry_point_items]`, sends with **no** `previous_response_id`, then clears `_pending_replay`. **`continue_with_user_prompt` builds a *string* `input`** (`providers.py:516`); when `_pending_replay` is set, that string must first be wrapped as a `{type:message, role:user, content:[{type:input_text, text}]}` item before prepending replay items (you cannot prepend list items to a string). `continue_with_tools` already builds a list, so it prepends directly. Once the response yields an `id`, chaining resumes. + +**System prompt re-injection** (reverses plan-08's "instructions ignored on resume" — `.plans/08-resume.md:226/:236`): + +- **Anthropic** — the first resumed turn sets `self.system = instructions` (today it ignores `instructions`, relying on rehydrated `self.system`, `providers.py:656-670`). Gated on the resume flag so non-resumed in-run turns are untouched. +- **OpenRouter** — the resume render prepends `{"role": "system", "content": instructions}` as `messages[0]` (today system is the rehydrated `messages[0]`, `providers.py:800-812`). +- **OpenAI** — already passes `instructions` live every turn (`providers.py:407`); no change. + +**Per-provider render (`_render_input`):** + +- **Anthropic** — entries → `messages`. `AssistantEntry` → one assistant message with `{type:tool_use, id, name, input: json.loads(arguments)}` per call, **plus a leading `{type:text}` block only when `text` is non-empty** (an empty text block would shift `content[0]` off the `tool_use` block that `test_resume.py:61` pins). **A maximal run of consecutive `ToolResultEntry`s + an immediately-following `UserEntry(notice=True)` coalesces into one `user` message** with N `tool_result` blocks (+ trailing `text` block) — separate user messages would violate role alternation. A `UserEntry(notice=True)` with *zero* preceding tool results (defensive: shouldn't occur) renders as a plain user message with a text block. Plain `UserEntry` → its own user message. +- **OpenRouter** — entries → chat messages. `AssistantEntry` → `{role:assistant, content, tool_calls:[{id, type:function, function:{name, arguments}}]}`; each `ToolResultEntry` → `{role:tool, tool_call_id, content}` (consecutive tool messages allowed); `UserEntry` → `{role:user, content}`. +- **OpenAI Responses** — entries → `input` items: `UserEntry` → `{type:message, role:user, content:[{type:input_text, text}]}`; `AssistantEntry` → optional `{type:message, role:assistant, content:[{type:output_text, text}]}` (omit when text empty) + one `{type:function_call, call_id, name, arguments}` per call; `ToolResultEntry` → `{type:function_call_output, call_id, output}`. + +**Tool-call id portability (claim softened).** Replaying both sides of a call/result pair from the stored `ModelToolCall.id` preserves *internal* correlation, but does **not** guarantee the receiving provider accepts a foreign-format id (e.g. OpenAI may reject a `toolu_*` `call_id`). Acceptance must be verified against the real API (residual risks). Likewise, on a cross-provider replay into Anthropic, `json.loads(arguments)` could raise if a non-Anthropic origin stored non-JSON `arguments` (the OpenAI/OpenRouter extractors pass the raw provider string, `providers.py:973/:1008`); treat a parse failure as a documented cross-provider failure mode adjacent to the foreign-`call_id` risk. + +### 4. Cross-provider / cross-model resume + version + validator coupling + +- `_validate_resume_state`'s model/provider mismatch rejection (`providers.py:197-202`) is **removed**. `origin_provider`/`origin_model` are retained for diagnostics only. +- **`version` bumps to `2`.** The envelope shape changed, so per plan-08's contract the version must bump. The validator rejects old state with a clear, **`"resume_from"`-prefixed** `HarnessError` instructing the caller to regenerate — both old `version: 1` *and* old `kind` values (`"openai"`/`"anthropic"`/`"openrouter"`). thinharness is greenfield (no deployed persisted state); no migration shim. +- **The validator must keep the `"resume_from"` message prefix.** `_resume_approval_session` relabels only errors whose message `startswith("resume_from")` into `"approval state provider_state…"` (`core.py:724-728`); `test_approvals.py:733` depends on this. If the new validator changes the prefix, either preserve it or update `core.py:726` to match — state the coupling. +- Validation still rejects: non-dict, wrong/missing `version`, missing/non-list/malformed `entries`, missing/wrong-typed `origin_*`, unknown top-level keys, non-JSON-serializable payloads, malformed entry dicts (bad `role`/missing fields). `HarnessError`, not `ProviderError`. +- v1 degradation is simple because reasoning is text: switching providers loses nothing structural — message/tool history replays cleanly. + +### 5. `resume_kind`, capability gates, and the custom-model boundary + +`resume_kind` is no longer used for envelope validation (`kind` is fixed to `"transcript"`), but it is **kept as the capability marker** for the two runtime `hasattr` gates that opt a model into resume: `core.py:460` (`model_supports_resume`, also drives finalize's `require_dump_state`) and `core.py:1135` (`_model_supports_approval_resume`, also referenced by approval-tool config validation in `.plans/24-human-in-the-loop.md`). Both gates and the fakes/tests that set `resume_kind` stay. The field is intentionally vestigial for the envelope and load-bearing only for capability detection — documented as such. + +**Custom-model boundary.** The neutral-transcript contract binds the **three built-in providers** (and the real-provider-backed test fakes). Custom `ResumableModel` implementations keep their own opaque `dump_state`/`resume_session` protocol — the harness delegates to `model.resume_session(state)` and does not impose the neutral schema on them. The scripted/sequence test fakes are exactly such custom models (`kind == "scripted"`) and are **left unchanged** (see §Tests). Document this boundary so adapter authors don't assume they must emit `{"kind": "transcript", ...}`. + +### 6. Approval-envelope unification (`approvals.py`, `core.py`) + +`build_approval_envelope`'s `provider_state` already carries `dump_state()` output, restored via `resume_session` (`core.py:465`, `:720-727`). With §3 it becomes the neutral transcript automatically. Required work: + +- Update the field's documented shape; note that the envelope now embeds the full transcript inside `provider_state` (net-new size growth for OpenAI, previously a single id). +- `APPROVAL_ENVELOPE_VERSION` **stays `1`** — intentionally. The outer envelope shape is unchanged; only the *inner* `provider_state` reshapes. The two versions are independent. An approval envelope captured under the old code carries a native `provider_state` that now fails at `resume_session` time (not at `validate_approval_pause_state`, which only checks `provider_state` is a dict, `approvals.py:110`) with the relabeled `"approval state provider_state…"` error — consistent with the greenfield/no-migration stance. State this rather than leaving it silent. +- Fix `test_approval_resume_labels_inner_provider_state_errors` (`test_approvals.py:731-734`): it mutates `provider_state["kind"]="wrong"`; since `kind` is fixed to `"transcript"`, switch the malformed trigger to a bad `version`/`entries` so it still exercises the relabeling path (and the error must keep the `"resume_from"` prefix per §4). The scripted-backed approval envelopes (`test_approvals.py:860/:884`) use the custom protocol and are unaffected. + +### 7. `final_result` / lifecycle parity + +Unchanged from `.plans/08-resume.md`: `resume_state` only on clean `end_turn`, never after the synthetic `final_result` tool, never on non-clean exits. `finalized_via_output_tool` gating in `run_ctx.finalize` (`core.py:536`) stays. + +### 8. Low-level escape hatch — a resume *semantic regression* to document + +`OpenAIResponsesSession.start(previous_response_id=...)` (the documented escape hatch, `providers.py:438`, exercised by `test_providers.py:76-90`) still works in-run. But if a caller seeds a session from an *external* OpenAI response id and the run later dumps state, the neutral transcript captures only the new prompt onward — the externally-seeded prior turns are **absent from a full replay**. Today those turns survive via server-side chaining; after this change they do not. Because this changes resume *semantics* for an existing supported escape hatch (not merely an absent feature), it is surfaced in `behavior.md` (RESUME-6), not just noted here. + +## Behavior changes (update `docs/behavior.md` after review, before implementation) + +Add a `RESUME` section per the template (`docs/behavior.md:6-22`): + +- **RESUME-1** — `resume_state` is a provider-agnostic transcript; resume across providers and across models is supported. +- **RESUME-2** — `resume_state` is self-contained and does not depend on any provider continuation token (e.g. OpenAI server-side response retention); an OpenAI run that never received a response id is still resumable. +- **RESUME-3** — v1 does not preserve provider-specific reasoning chains across resume (reasoning replays as text). +- **RESUME-4** — `resume_state` `version` is `2`; v1-shaped state (old `version` or old provider `kind`) is rejected with a regenerate error. +- **RESUME-5** — on resume, the *live* system prompt (from the resuming harness's config) is re-injected; the captured system prompt is not stored or restored. +- **RESUME-6** — a session seeded via the `start(previous_response_id=…)` escape hatch loses its externally-seeded prior turns on resume (the transcript captures only from the new prompt onward). + +## Implementation steps + +1. **`providers.py`** — add transcript leaf types + dict (de)serialization; add the neutral-envelope validator (drop kind/model rejection, require `version == 2`, keep `"resume_from"` prefix, reject old `kind` values, exact allowed-key set); per session add the parallel transcript (dual-store) + `_resume_entries`/`_pending_replay` flags, neutral-append in `start`/`continue_*`/`_complete`, `_render_input` (Anthropic coalescing + omit-empty-text + system set; OpenAI pending-replay with string→item wrap; OpenRouter system prepend), rewrite `dump_state`/`resume_session` to the neutral envelope with normalized `origin_provider`. Keep native in-run stores and `resume_kind`. +2. **`core.py`** — no structural loop change; confirm `_resume_approval_session` (`:720`) passes the neutral envelope through and the relabel prefix still matches; both capability gates (`:460`, `:1135`) unchanged. +3. **`approvals.py`** — update `provider_state` docs/shape; record version-independence + size-growth notes. +4. **Tests — targeted audit (§"Tests").** Update only real-provider-backed fakes; **leave the scripted/sequence fakes unchanged**. +5. **`docs/behavior.md`** — add `RESUME-1..6` (after review). +6. **Docs surfaces** — update `docs/docs.md` stale contract at `:248` (approval wraps provider resume state), `:252`, `:558`, `:562`, `:564` (same provider/model + provider-owned details), `:570` (size growth), `:572`; **add** a resume section to `README.md` (none exists — `:295` is a one-line bullet); regenerate site artifacts (`scripts/build_site.py`) if part of the docs workflow. +7. **Run** `uv run pyright`, ruff, and the full pytest suite (per project `CLAUDE.md`). + +## Tests + +**Do NOT migrate the scripted/sequence fakes.** `ScriptedModel.resume_session`/`ScriptedSession.dump_state` (`fakes.py:177-183/:210/:240-242`) and `SequenceSession` defaults (`test_mcp.py:102`, `test_streaming.py:62`) model custom resumable models with their own `kind == "scripted"` protocol; they never touch `_render_input` or the neutral validator. `test_stream_resume_from_emits_resume_kind` (`test_streaming.py:352`) **passes unchanged** — drop it from the change list. Migrating these would cascade-break `test_resume.py:179/:201/:223/:264/:305/:346/:405`, `test_tracing.py:358`, `test_mcp.py:102`, `test_streaming.py:62`, `test_approvals.py:860/:884`. Only `FakeClient`/`FakeAnthropicProvider`/`FakeOpenRouterProvider`-backed tests change. + +Existing assertions to **invert or restructure** (not "migrate"): + +- `test_resume.py:30` — resume-turn `previous_response_id` assertions invert; **also** `:43` (`input == "follow-up"` string→list) and `:46` (`"first" not in json.dumps` — now `"first"` *is* present). In-run `payloads[1]["previous_response_id"]` stays. +- `test_resume.py:88` — remove `kind`/`model` rejection branches (cross-provider/cross-model now succeed); keep version + unknown-key branches, retargeted to `version: 2`. +- `test_resume.py:116` (`test_resume_rejects_malformed_shapes_before_hooks_fire`) — **restructure** against the neutral schema. Every sub-case uses dead v1 field names (`:132/:134/:139/:141`), and the JSON-serializable case (`:143-147`) now trips the unknown-key check (which runs before `json.dumps`, `providers.py:208-214`), so it would raise "unknown keys" not "JSON-serializable". Rebuild the cases (and assert the JSON-serializable case still raises the JSON error — adjust validator order if needed). +- `test_resume.py:248` (`test_no_openai_response_id_omits_resume_state`) — **inverts**: a no-id OpenAI run is now resumable (RESUME-2). +- `test_resume.py:361-374` (detachment) — index changes from `["messages"]` to `["entries"]`; the mutated element must be a `UserEntry` (`entries[0]["content"]`), since `entries[1]` is the `AssistantEntry` (no `content` key). +- `test_resume.py:399` (`assert session.messages == []`) — **stays valid** under dual-store (native attribute retained). +- `test_approvals.py:600/:609/:629` — provider-native `provider_state` + OpenAI `previous_response_id`-replay assertions become `kind:"transcript"` + full-replay. +- `test_approvals.py:731-734` — change malformed trigger per §6; keep the `"resume_from"` prefix so relabeling fires. +- `test_providers.py:106-120` — neutral envelope + full-replay shape. + +New cases: + +1. **Cross-provider resume after a real tool round-trip** — capture on an Anthropic-shaped fake *through an actual tool turn*, resume on OpenAI- and OpenRouter-shaped fakes; assert the assistant tool call and its result replay with matching ids. +2. **Cross-model same-provider resume** — previously rejected; now succeeds. +3. **OpenAI approval-resume full replay** — pause on an approval-required tool (OpenAI-shaped fake), resume; assert the first `continue_with_tools` request replays the full prior input (user + assistant `function_call`) + the `function_call_output`, with **no** `previous_response_id`. +4. **OpenAI normal-resume shape** — assert `payloads[2]["input"]` is the list `[{user "first"}, {assistant function_call}, {function_call_output}, {user "follow-up"}]` with no `previous_response_id`, and `payloads[1]["previous_response_id"]` (in-run chaining) preserved. +5. **Multi-tool batch resume** — assistant turn with ≥2 tool calls: Anthropic render = one `user` message with N `tool_result` blocks; OpenRouter = N `role:tool` messages. (Requires fakes emitting multiple tool calls — current `echo_tool` fakes emit one, `fakes.py:133/:154`.) +6. **System-prompt re-derivation on resume** — resume with a harness whose system prompt differs from the capturing one; assert the replayed input carries the *new* system text (Anthropic `system` / OpenRouter `messages[0]` / OpenAI `instructions`), never `""`/absent (RESUME-5). +7. **Notice preservation, tool and non-tool** — (a) resume after a `continue_with_tools` carrying a background-completion/cancellation notice and a limit warning; assert the notice text survives and (Anthropic) is folded into the tool-result user message; (b) attach a limit notice to `start`/`continue_with_user_prompt` (e.g. low `max_model_requests`), resume, assert the rendered `` text survives in the replayed `UserEntry.content`. +8. **Round-trip serialization** — `json.loads(json.dumps(dump_state()))` equals `dump_state()` for a transcript with all three entry kinds and a multi-tool assistant turn; the non-JSON-serializable-dump test (`test_resume.py:402`) still raises. +9. **OpenAI no-id resume** — `resume_state` non-`None` for a no-id run; resume full-replays (RESUME-2). +10. **Version + kind rejection** — v1-shaped state (`version:1` *and* old `kind` values) raises `HarnessError` (RESUME-4). +11. **In-run byte-identical guard** — confirm `test_harness.py:226-242`, `test_providers.py:60-74`, `test_providers.py:92-175` (minus `:106-120`) are unchanged; any change means Option B leaked. +12. **Wire-shape pins still pass** — `test_resume.py:49/:61/:68` (string-content plain user messages; exact `tool_use`/`tool_result` and `type:"function"` shapes). +13. Carry over (retargeted): `final_result` ⇒ no state, non-clean exits ⇒ no state, malformed envelopes ⇒ `HarnessError`, fresh-harness persistence. + +A test fake that **rejects an unpaired `function_call_output`** is recommended so the OpenAI renderer's pairing is proven against something stricter than permissive `FakeClient`. Live notice tests (`test_providers.py:300/:324`, skipped without keys) seed `session.system`/`session.messages` directly on a fresh session — confirm dual-store keeps that working. + +## Out of scope (deferred) + +- **Same-provider reasoning fidelity** — a `provider_extras` field stamped with `origin_provider`, re-emitted only on same-provider resume (Anthropic thinking `signature`, OpenAI `encrypted_content`), with per-entry `provider_name` added then. v1 stores reasoning as text. +- **Dropping stale notices** from the durable transcript — intentional behavior change; v1 preserves current behavior. +- **Tracing/event-stream as projections of the transcript** — pydantic-ai has each canonical part own its OTel serialization; thinharness's `tracing.py`/`events.py` could pull payload-building from the transcript. Separate change. +- **Extended-thinking in-run block-ordering constraints** — current code does not enable extended thinking; revisit if it does. +- **OpenAI same-provider `previous_response_id` fast-path on resume** — v1 favors uniform full replay (intentional; full replay is larger payloads for the common same-provider case, accepted for uniformity + retention-independence). + +## Residual risks + +- **Real-provider divergence from fakes (highest).** The OpenAI render path (full `input`-item replay, foreign `call_id` acceptance, `output_text`/`input_text` content types, `function_call`/`function_call_output` pairing without `previous_response_id`, reasoning-item-free replay against gpt-5-class models) is only exercised against fakes. Gate "RESUME-1 cross-provider supported" (vs experimental) on at least one real-API smoke test per provider pair: capture on Anthropic, resume on OpenAI Responses and OpenRouter; confirm acceptance or capture the exact rejection. +- **Cross-provider malformed `arguments`** — `json.loads(arguments)` in the Anthropic render can raise if a non-Anthropic origin stored non-JSON `arguments`; documented failure mode adjacent to foreign-`call_id`. +- **OpenAI `input` wire shape** changes from string to list on the resumed first turn (handled per §3); tests asserting string `input` update. +- **Size growth** — OpenAI `resume_state`/approval-envelope grows O(1)→O(conversation); also in-run *stored* state for OpenAI grows O(1)→O(n) (the parallel transcript), bounded by `max_model_requests`. Document for OpenAI callers. + +## LOC estimate + +~400–600 net lines added against the 1009-line `providers.py`: per-provider `_render_input`, Anthropic coalescing, OpenAI pending-replay (with string→item wrap), system re-injection, dual-store bookkeeping, and explicit entry (de)serialization. A few hundred net lines, not a doubling. + +## Findings resolution — v1 review round + +| Finding (reviewer) | Resolution | +|---|---| +| OpenAI replay can't run in `resume_session`; approval-resume re-enters via `continue_with_tools` (claude H1, glm H1) | §3 deferred entry-point-agnostic pending-replay; tests 3-4. | +| System prompt empty on Anthropic/OpenRouter resume (claude #3, glm H2) | §3 system re-injection from live `instructions`; RESUME-5; test 6. | +| Anthropic must coalesce consecutive tool results into one user message (claude #2, glm M2) | §3 coalescing rule; test 5. | +| Notices dropped from durable transcript (codex #1) | §1 notice capture; test 7. | +| Entry dataclasses not JSON-serializable / no discriminator (claude #4, glm L2) | §1 explicit dict shape + `json.loads(arguments)`. | +| Test blast radius understated; assertions invert (codex, claude #5, glm H3) | §Tests enumerates inversions. | +| No-id OpenAI run becomes resumable (claude #6) | RESUME-2; test 9. | +| `provider_name`/`_parse_extras` speculative (claude #7, glm L3) | Dropped from v1 (§1). | +| `version` not bumped (glm M1) | Bumped to `2`; v1 rejected (§4); RESUME-4. | +| `resume_kind` / two gates (glm M3) | §5 vestigial capability marker; both gates unchanged. | +| Tool-call id cross-provider acceptance over-claimed (glm M4) | §3 softened; real-API residual risk. | +| Assistant content ordering loss (codex #2) | §1 accepted+documented flattening; test 5. | +| Stale `docs/docs.md`; README has no section (codex #3, glm L1) | Step 6 updates docs.md + adds README section. | +| "Collapse `continue_with_*`" inaccurate (glm L4) | §2 corrected. | +| Wrong append-site citations (glm L5) | "Current state" anchored on real sites. | +| Escape-hatch interaction (claude #9, glm residual) | §8 + RESUME-6. | +| Envelope size growth (claude open-q1, glm residual) | §6 + residual risks. | +| behavior.md should be `RESUME-*` (glm/claude open-q) | RESUME-1..6. | +| OpenAI replay only tested vs fake (claude #8) | Residual risk + strict fake. | + +## Findings resolution — v2 review round + +| Finding (reviewer) | Resolution | +|---|---| +| Backing-store wording self-contradictory; pick Option A (claude #1 High, glm #2) | §2 rewritten: explicit dual-store; in-run byte-identical guard (test 11). | +| OpenAI `continue_with_user_prompt` builds a string `input` (glm #1) | §3 string→message-item wrap before prepending replay items. | +| Notice asymmetry for non-tool paths (codex #1, claude #4) | §1 `UserEntry.content` stores notice-appended text on those paths; test 7b. | +| Don't migrate scripted/sequence fakes (claude #3) | §5 custom-model boundary; §Tests leaves them unchanged; `test_streaming.py:352` dropped from change list. | +| `test_resume.py:116` malformed-shapes needs restructure (validator order) (glm #3) | §Tests restructure + assert JSON-serializable case still raises that error. | +| `test_resume.py:361-374` detachment retarget (`messages`→`entries`, mutate `UserEntry`) (claude #2, glm #3) | §Tests explicit retarget. | +| `test_resume.py:43/:46` invert (glm #3) | §Tests enumerated. | +| `_resume_approval_session` relabel depends on `"resume_from"` prefix (claude #6, glm #4) | §4 validator keeps the prefix (or update `core.py:726`); stated coupling. | +| behavior.md missing system-on-resume change (glm #5) | RESUME-5. | +| Escape-hatch is a semantic regression, belongs in behavior.md (claude #9, glm #5) | RESUME-6 + §8. | +| `origin_provider`/`origin_model` required + normalized (glm #6) | §3 required, exact allowed-key set, `provider_prefix` normalization. | +| Anthropic omit empty assistant text block (glm #7) | §3 leading text block only when non-empty. | +| docs.md enumeration incomplete (claude #7) | Step 6 adds `:248/:562/:564/:570`. | +| Approval envelope version independence (claude #8, glm #10) | §6 stays `1`, independent, old envelopes rejected at resume time. | +| Coalescing zero-tool-results edge (glm #8) | §3 defensive: plain user message. | +| Cross-provider malformed `arguments` (glm #9) | §3 + residual risks documented failure mode. | +| OpenAI in-run memory O(1)→O(n) (glm #11) | Residual risks note. | +| Custom `ResumableModel` schema boundary (codex open-q) | §5 custom-model boundary. | + +## Open questions + +None blocking. Deferred items are listed under "Out of scope." diff --git a/.plans/30-tracing-consolidation.md b/.plans/30-tracing-consolidation.md new file mode 100644 index 0000000..1997a1f --- /dev/null +++ b/.plans/30-tracing-consolidation.md @@ -0,0 +1,142 @@ +# Tracing and event projection from transcript deltas + +## Goal + +Reduce duplicated conversation-shape construction in tracing and runtime now that built-in providers maintain a neutral transcript. The durable transcript is the canonical model-visible conversation state; tracing and streaming should be projections from the same per-request model-visible delta where that makes sense. + +This is intentionally a behavior cleanup, not a backwards-compatibility exercise. Trace input should reflect the exact model-visible content, including rendered `` text. Keeping a separate "logical but not quite what the model saw" trace view is unnecessary complexity. + +## Current state + +The new transcript introduced by `.plans/29-unified-transcript-log.md` is the durable model-visible log: + +- `UserEntry(content, notice=False)` +- `AssistantEntry(text, tool_calls)` +- `ToolResultEntry(call_id, output)` +- `UserEntry(content, notice=True)` for notice text accompanying tool-result batches + +Tracing currently has a parallel request-log model: + +- `ModelTraceSnapshot` in `tracing.py` carries `kind`, `prompt`, `tool_outputs`, `notices`, and `structured_output`. +- `TurnDriver` builds that snapshot independently in `runtime.py` for `start`, `resume`, `send_tool_outputs`, and `send_user_message`. +- `tracing.py` then rebuilds model input payloads again via `_model_request_input()` and `_otel_input_messages()`. +- `tracing.py` rebuilds assistant output again via `_otel_output_messages(turn)`. +- `runtime.py` separately emits `ModelMessageEvent` by mapping `ModelTurn.tool_calls` to `StreamToolCall`. + +The event stream is not just the transcript. It includes operational lifecycle notifications that are not durable model-visible conversation entries: + +- run start/completion/failure +- model request start +- tool call start/completion +- background task start/completion +- retry and limit notifications +- approval resume + +So the right target is not "make events.py equal transcript." The right target is "use one model-visible request/response delta as the source for durable transcript, model tracing payloads, and model-message stream projection." Operational events remain separate. + +## Design sketch + +Introduce an internal `ModelRequestDelta` (name flexible) near the provider/runtime boundary: + +```python +@dataclass(frozen=True) +class ModelRequestDelta: + kind: Literal["start", "resume", "tool_outputs", "approval_resume", "correction", "output_retry_tool", "background_completion"] + entries: list[TranscriptEntry] + notices: list[ModelNotice] = field(default_factory=list) + structured_output: str | None = None +``` + +The delta represents the exact new model-visible input for one provider request, after hooks/background/limit notices have been rendered into user text where applicable. `notices` remains as structured metadata for observability, but the primary trace input should include the rendered notice text because that is what the model saw. + +Examples: + +- start/resume/correction: one `UserEntry(content=append_notices_to_text(prompt, notices))` +- tool output request: N `ToolResultEntry`s plus optional `UserEntry(notice=True)` if notices are sent as user text +- background completion as user message: one `UserEntry(content=...)` +- approval resume: same as tool output request, with `kind="approval_resume"` + +Then project it to: + +- tracing input attributes from `entries` plus structured `notices` metadata (`gen_ai.input.messages`, `gen_ai.prompt`, `langfuse.observation.input`, `thinharness.model.notices`) +- request-kind stream metadata (`ModelRequestStartedEvent.request_kind`) +- provider transcript append logic, if provider sessions can accept already-rendered entries without leaking into their native in-run request builders + +For assistant output, add a helper from `AssistantEntry` / `ModelTurn` to: + +- tracing output attributes (`gen_ai.output.messages`, `langfuse.observation.output`) +- `ModelMessageEvent` +- durable transcript append, if provider sessions can share that helper + +## Boundaries + +Keep these separate: + +- **Conversation log:** model-visible entries only. This is what durable resume uses. +- **Operational stream:** lifecycle notifications. It includes some conversation projections, but also events that happen before or outside model-visible transcript entries. +- **Trace spans:** operational timing plus optional model-visible payload capture. + +Do not add operational metadata to durable transcript entries. Avoid storing timing, stream sequence, tracing capture policy, or output-mode details in `resume_state`. + +Public stream event dataclasses should stay mostly stable, with one intended simplification: remove `StreamOptions.include_model_text`. The stream is an observability/log surface; suppressing assistant text makes it less useful and adds branching that other agent frameworks do not appear to expose as a core option. If callers need redaction, that should be a separate filtering layer outside the core event schema. + +## Implementation steps + +1. Add projection helpers in a small internal module, e.g. `thinharness/projections.py`, so `providers.py` does not learn about tracing or stream-event concerns: + - `trace_input_messages_from_entries(entries)` + - `trace_output_messages_from_assistant(entry_or_turn)` + - `model_request_input_from_delta(delta)` + - `stream_tool_calls_from_assistant(entry_or_turn)` + +2. Replace `ModelTraceSnapshot.prompt/tool_outputs/notices` with `ModelRequestDelta.entries/notices`, or adapt `ModelTraceSnapshot` into this shape as an intermediate step. Do not keep separate logical/model-visible entry lists. + +3. Update `TurnDriver` to build one delta per model request after notices are known: + - It currently builds `ModelTraceSnapshot` before `advance_model`, but final limit/background notices are only known inside `RunContext.advance_model`. + - The likely place to finalize the delta is inside `advance_model`, after `all_notices` is computed and before `annotate_model_request`. + +4. Update `annotate_model_request()` to consume the delta and remove `_otel_input_messages()` branches that duplicate request-kind logic. Trace input should include rendered notices because the delta is model-visible. + +5. Update `annotate_model_span()` to use the same assistant-entry projection helper used by stream `ModelMessageEvent`. + +6. Keep stream lifecycle event emission in `runtime.py` and tool execution code. Use the shared assistant projection for `ModelMessageEvent` and always include assistant text. + +7. Remove `StreamOptions.include_model_text` and the associated runtime branch. Keep `StreamOptions.include_subagents` unless a separate review shows it is also unnecessary. + +8. Leave provider-native in-run request builders untouched. This must preserve the dual-store boundary from plan 29: normal in-run provider payloads should stay byte-identical. + +## Tests + +This is mostly a refactor, with two intentional behavior/API changes: + +- trace input messages now represent exact model-visible input, including rendered notices +- stream events no longer support suppressing assistant text via `StreamOptions.include_model_text` + +Required checks: + +- Existing tracing tests in `tests/test_tracing.py` should be updated where they currently expect notices to be absent from trace input messages. +- Existing streaming tests in `tests/test_streaming.py` and approval streaming tests should be updated if they cover `StreamOptions.include_model_text`. +- Existing provider payload tests should remain byte-identical; any changed in-run payload means transcript projection leaked into provider request construction. +- Full `uv run pyright`, relevant `ruff`, and full `pytest`. + +New or adjusted tests worth adding: + +- One tracing test that asserts a tool-output request with a notice includes the rendered notice in trace input and also preserves structured notice metadata. +- One tracing test that asserts a prompt-path notice includes the rendered notice in trace input. +- One tracing test that asserts assistant text + tool calls uses the same projection as `ModelMessageEvent`. +- One regression test for resume first-turn tracing: replayed transcript itself should not be double-counted as the new request delta unless that is intentionally exposed. +- One streaming test update proving model text is always included in `ModelMessageEvent`. + +## Risks + +- Tracing has sink-specific conventions (`gen_ai.*`, Langfuse) that are not identical to durable transcript shape. The shared helper should project transcript entries into those conventions rather than forcing tracing attributes to become transcript dictionaries. +- Including rendered notices in trace input is a trace-shape behavior change. This is accepted for simplicity and fidelity to what the model saw, but docs/tests should make it explicit. +- Limit notices are computed inside `advance_model`, while request kind/prompt/tool outputs are prepared in `TurnDriver`. Moving delta finalization too early will miss notices; moving too late can obscure the original request kind. +- Stream events include operational timing and partial lifecycle states. Treating the stream as "just the transcript" would lose useful host notifications like `tool_call_started`. + +## Out of scope + +- Broad changes to public stream event schemas beyond removing `StreamOptions.include_model_text`. +- Adding token streaming. +- Making tracing required or changing capture defaults. +- Moving tool execution records into durable transcript. +- Reducing provider-native in-run state or changing the transcript resume envelope. diff --git a/.plans/31-reasoning-fidelity.md b/.plans/31-reasoning-fidelity.md new file mode 100644 index 0000000..9ff36a8 --- /dev/null +++ b/.plans/31-reasoning-fidelity.md @@ -0,0 +1,147 @@ +# Same-provider reasoning fidelity in the neutral transcript — plan v1 + +Capture native model reasoning into the neutral `Transcript` and re-emit it natively when a run is resumed on the **same provider**, so reasoning models do not lose their chain-of-thought across a resume boundary. Cross-provider resume degrades reasoning to text (unchanged intent of RESUME-3). + +This executes the item plan-29 (`.plans/29-unified-transcript-log.md`) explicitly deferred under "Out of scope": *"Same-provider reasoning fidelity — a `provider_extras` field stamped with `origin_provider`, re-emitted only on same-provider resume (Anthropic thinking `signature`, OpenAI `encrypted_content`)… v1 stores reasoning as text."* + +## Goal + +Plan-29 made `resume_state` a neutral transcript of `UserEntry`/`AssistantEntry`/`ToolResultEntry`. `AssistantEntry` keeps only `text` + `tool_calls`; `_append_assistant_turn` discards `turn.raw` (`providers.py:306-307`), which is the only place native reasoning lives. Consequences today: + +- **OpenAI Responses (sharpest regression).** Before plan-29, OpenAI `dump_state` serialized just `previous_response_id`; on resume OpenAI rebuilt the full prior conversation server-side **including reasoning items**. Plan-29 switched to self-contained client-side full replay with **no** `previous_response_id` and a transcript that carries no reasoning items → reasoning is dropped on every resume of a reasoning model. The retention-independence win was real; the casualty was the server-held reasoning. +- **OpenRouter.** `reasoning_details` arrive on the raw assistant message and are kept in-run via `self.messages`, but never reach the transcript → lost on resume. +- **Anthropic.** Only relevant once extended thinking is enabled (the harness does not enable it by default; see §6). When enabled, thinking blocks + signatures are kept in-run via `self.messages` but lost on resume. + +After this change a run captured and resumed on the same provider preserves native reasoning; a run resumed on a different provider keeps the reasoning **text** (a leading ``-tagged block) and drops the opaque blob. + +## What this is and is not + +**Is:** an additive `reasoning` field on `AssistantEntry`, populated from each turn's raw response and rendered back natively only when the resuming provider matches the originating provider. + +**Is not:** a change to the in-run request/response normalization (`ModelTurn.text`/`tool_calls` are unchanged), and **not** a cross-provider reasoning translator. The neutral part is an opaque carrier gated by provenance, exactly as pydantic-ai's `ThinkingPart` does. + +## Live API verification (done — gates the residual risk) + +The plan-29 residual risk was "real-provider divergence from fakes." The happy path (one tool round-trip, a multiply tool, low reasoning effort) was verified against all three live APIs on 2026-06-22: + +- **OpenAI** (`gpt-5-mini`): with `store` defaulted true (harness in-run mode) **plus** `include=["reasoning.encrypted_content"]`, the response's `reasoning` item carries `encrypted_content`. In-run `previous_response_id` chaining still works alongside `include`. A reasoning blob captured under `store=true` **replays statelessly** — full `input` `[reasoning, function_call, function_call_output]`, `store=false`, **no** `previous_response_id` → **200**, terse cached-reasoning answer. Replaying with `store` defaulted true (no `previous_response_id`) also → 200. Dropping the reasoning item also → 200 (model re-reasons). +- **Anthropic** (`claude-sonnet-4-6`, `thinking={type:enabled,budget_tokens:1024}`): T1 returns `[thinking{thinking,signature}, text, tool_use]`. Reconstructing the assistant message as `[thinking(signature), text, tool_use]` + a user `tool_result` → **200**. (Missing thinking block also accepted here, but native preservation is the goal.) +- **OpenRouter** (`reasoning={effort:"low"}`): `openai/gpt-5-mini` returns `reasoning_details=[{type:"reasoning.encrypted", data:"…"}]`; `anthropic/claude-sonnet-4.5` returns `[{type:"reasoning.text", text, format:"anthropic-claude-v1", index, signature}]`. Echoing `reasoning_details` verbatim on the assistant message + a `role:tool` result → **200** for both. + +Smoke scripts are reproducible from `.env` keys; see §Tests for the cases to keep as a guarded live suite. **Not yet verified** (residual risks below): redacted_thinking, multiple reasoning items per turn, interleaved-thinking constraints, long conversations, and signature acceptance across a model-version change. + +## Design + +### 1. Neutral `ReasoningPart` + `AssistantEntry.reasoning` (`providers.py`, leaf types) + +```python +@dataclass +class ReasoningPart: + text: str = "" # plain reasoning text — always kept; cross-provider fallback + signature: str | None = None # opaque blob: Anthropic signature / redacted data, + # OpenAI encrypted_content, OpenRouter signature|data + id: str | None = None # provider reasoning-item id (OpenAI rs_…; "redacted_thinking" marker) + provider_name: str | None = None # origin provider prefix; native re-emit only when this matches + provider_details: Json | None = None # spillover: OpenAI summary raw_content; OpenRouter raw reasoning_details entry + +@dataclass +class AssistantEntry: + text: str + tool_calls: list[ModelToolCall] + reasoning: list[ReasoningPart] = field(default_factory=list) # additive, defaulted +``` + +A list because a turn can carry several reasoning parts (Anthropic thinking + redacted_thinking; OpenAI multiple summaries sharing one id; OpenRouter multiple `reasoning_details` entries). + +`ModelTurn` gains `reasoning: list[ReasoningPart] = field(default_factory=list)`, populated in each `_complete` from `turn.raw`; `_append_assistant_turn` copies `turn.reasoning` (deep-copied) into the `AssistantEntry`. + +**Provenance is per-part, not per-envelope.** A transcript can accumulate entries from multiple providers after cross-provider resumes, so the gate keys on `ReasoningPart.provider_name`, not the envelope's `origin_provider`. This mirrors pydantic-ai's `provider_name == self.system` gate (`models/anthropic.py:1348`, `models/openai.py:3044`). + +### 2. Serialization (load-bearing) + +- Add `ReasoningPart` to/from dict; `signature`/`id`/`provider_name`/`provider_details` are `None`-omitted-or-present per the existing explicit style. +- Extend the assistant branch of `_transcript_entry_to_dict`/`_transcript_entry_from_dict` (`providers.py:258-289`) to include `"reasoning": [ … ]`. +- **`_TRANSCRIPT_ENTRY_KEYS["assistant"]` (`providers.py:213`) must gain `"reasoning"`.** The validator does strict `set(value) != _TRANSCRIPT_ENTRY_KEYS[role]` (`:274`), so an assistant entry without the key would fail — hence the version bump (§4) and: `dump_state` always emits `"reasoning"` (empty list when none), keeping the key-set exact. +- Round-trips through `json.loads(json.dumps(...))` like every other entry, so it is also persisted incrementally by the transcript-delta tracing path (`tracing.py:467`). + +### 3. Per-provider capture (IN) and native re-emit (OUT) + +The gate, applied in every `_render_*_transcript` when rendering an `AssistantEntry.reasoning[i]`: + +``` +native_ok = part.provider_name == and and +if native_ok: emit native reasoning block +elif part.text: emit a leading "\n{part.text}\n" block (text fallback) +else: drop +``` + +The text fallback is emitted as a leading content block (Anthropic `text` / OpenRouter `content` / OpenAI `output_text` message), **before** the assistant text and tool calls, matching pydantic-ai's `thinking_tags` degradation. + +**OpenAI Responses.** +- *Capture:* add `include=["reasoning.encrypted_content"]` to `build_payload` (`providers.py:493-513`) so every response's `reasoning` items carry `encrypted_content` even under in-run `store=true` chaining (verified). In `_complete`, extract items where `type == "reasoning"`: `ReasoningPart(text=joined summary text or "", signature=encrypted_content, id=rs_id, provider_name="openai", provider_details={"raw_content": [...]} if present)`. +- *Re-emit:* in `_render_openai_transcript` (`providers.py:1133`), for a matching part emit `{"type":"reasoning","id":part.id,"encrypted_content":part.signature,"summary":[]}` **before** the assistant `output_text` message and the `function_call` items. The resume path already sends no `previous_response_id`; default `store` is accepted for the replay (verified), so **no `store` toggle is required** (optionally set `store=false` for ZDR — call-out, not a requirement). + +**Anthropic Messages.** +- *Capture:* in `_complete` (`providers.py:793`), extract from `response["content"]`: `thinking` → `ReasoningPart(text=thinking, signature=signature, provider_name="anthropic")`; `redacted_thinking` → `ReasoningPart(text="", signature=data, id="redacted_thinking", provider_name="anthropic")`. +- *Re-emit:* in `_render_anthropic_transcript` (`providers.py:1060`), **prepend** reasoning blocks to the assistant `content` (before the optional text block and the `tool_use` blocks — Anthropic requires thinking first): `{"type":"thinking","thinking":part.text,"signature":part.signature}`, or `{"type":"redacted_thinking","data":part.signature}` when `id == "redacted_thinking"`. +- *Constraint:* Anthropic only accepts thinking blocks when the **resuming** request enables thinking. Gate the native emit additionally on "thinking enabled for this run" (derived from `self.model.settings.extra_body`); otherwise use the text fallback. Because the harness does not enable thinking by default (no `thinking` key is sent anywhere today), Anthropic native preservation is effectively inert until a caller turns thinking on — consistent with plan-29's note that extended thinking is out of scope. The in-run block-ordering constraints for live extended thinking remain out of scope (deferred in plan-29). + +**OpenRouter.** +- *Capture:* in `_complete` (`providers.py:944`), read `message.get("reasoning_details")`. Store each entry faithfully: `ReasoningPart(text=entry.get("text",""), signature=entry.get("signature") or entry.get("data"), id=entry.get("id"), provider_name="openrouter", provider_details=entry)` — keeping the full raw entry in `provider_details` so the self-describing OpenRouter shape (`type`/`format`/`index`) round-trips exactly. +- *Re-emit:* in `_render_openrouter_transcript` (`providers.py:1101`), reattach `message["reasoning_details"] = [part.provider_details for matching parts]` on the assistant message (verbatim from `provider_details`). OpenRouter accepted both encrypted and text forms echoed verbatim (verified). + +### 4. Version + validator coupling + +- **`version` bumps `2` → `3`.** The assistant entry shape changed (new required `reasoning` key in the exact-key-set check). Per plan-08/plan-29's contract a shape change bumps the version. Greenfield (no deployed persisted state); reject old `version: 2` and `version: 1` plus old `kind` values with the existing **`"resume_from"`-prefixed** `HarnessError` instructing regeneration (`providers.py:227-230`). +- The `_resume_approval_session` relabel still keys on the `"resume_from"` prefix (`core.py:724-728`, `test_approvals.py:733`); keep it. +- `_validate_anthropic_tool_arguments` (`providers.py:310`) is unaffected (reasoning carries no tool-arg JSON). + +### 5. Capability/boundary parity (unchanged) + +`resume_kind`, both capability gates (`core.py:460`, `:1135`), and the custom-`ResumableModel` boundary (custom models keep their opaque protocol) are untouched. The neutral-reasoning contract binds only the three built-in providers and their real-provider-backed fakes. + +## Behavior changes (update `docs/behavior.md` after review, before implementation) + +- **Update RESUME-3** — was "v1 does not preserve provider-specific reasoning chains across resume (reasoning replays as text)." Now: *same-provider* resume preserves native reasoning (Anthropic thinking signatures, OpenAI `encrypted_content`, OpenRouter `reasoning_details`); *cross-provider* resume degrades reasoning to text. +- **Add RESUME-7** — for OpenAI Responses, the harness requests `include=["reasoning.encrypted_content"]` so reasoning survives resume; this is captured into `resume_state` (which therefore contains encrypted reasoning blobs — treat as sensitive, consistent with the existing local-trace sensitivity note). +- **RESUME-6 is unchanged and still applies** — the `start(previous_response_id=…)` escape hatch still loses externally-seeded prior turns on resume. This plan does **not** fix RESUME-6; state that explicitly so "reasoning regression fixed" is not misread as "all resume regressions fixed." +- `version` is now `3` (extends RESUME-4's regenerate-on-old-state rule). + +## Implementation steps + +1. **`providers.py`** — add `ReasoningPart`; add `reasoning` to `AssistantEntry` and `ModelTurn`; `ReasoningPart` (de)serialization; extend assistant entry (de)serialization + `_TRANSCRIPT_ENTRY_KEYS["assistant"]`; bump validator to `version == 3` (reject 1/2 + old `kind`, keep `"resume_from"` prefix); per-provider capture in the three `_complete`s; `_append_assistant_turn` copies `turn.reasoning`; OpenAI `include` in `build_payload`; native re-emit + text fallback in the three `_render_*_transcript`s with the per-part provenance gate (Anthropic also gated on thinking-enabled). +2. **`docs/behavior.md`** — update RESUME-3, add RESUME-7, restate RESUME-6 scope, note `version: 3` (after review). +3. **`README.md` / `docs/docs.md`** — update the Resume section: line 303 ("Provider-specific reasoning chains are not preserved") becomes "preserved on same-provider resume, degraded to text cross-provider"; note the OpenAI `include`/sensitivity point. +4. **Tests** — unit (fakes) + the guarded live suite (§Tests). +5. **Run** `uv run pyright`, ruff, full pytest (project `CLAUDE.md`). + +## Tests + +Real-provider-backed fakes (`FakeClient`/`FakeAnthropicProvider`/`FakeOpenRouterProvider`) only; leave scripted/sequence fakes unchanged (per plan-29 §Tests). Fakes must be extended to emit reasoning in their raw responses (OpenAI `reasoning` item w/ `encrypted_content`; Anthropic `thinking` block w/ `signature`; OpenRouter `reasoning_details`). + +Unit cases: +1. **Capture** — each provider's `_complete` populates `AssistantEntry.reasoning` with the right `provider_name`, `signature`/`id`, and (OpenAI) `include` is present in the payload. +2. **Same-provider native re-emit** — resume on the same provider renders the native block (OpenAI `reasoning` item with `encrypted_content` ahead of the `function_call`, no `previous_response_id`; Anthropic `thinking{signature}` first in assistant content; OpenRouter `reasoning_details` reattached). +3. **Cross-provider text fallback** — capture on Anthropic-shaped fake, resume on OpenAI/OpenRouter-shaped fakes: assert a leading ``-tagged text block, no opaque blob, no foreign reasoning item. +4. **Anthropic thinking-disabled fallback** — same-provider resume but thinking not enabled in the resuming config → text fallback, not a `thinking` block. +5. **redacted_thinking** — Anthropic `redacted_thinking` round-trips to `{"type":"redacted_thinking","data":…}`. +6. **Multi-part turn** — ≥2 reasoning parts and ≥1 tool call render in the right order. +7. **Round-trip serialization** — `json.loads(json.dumps(dump_state()))` equals `dump_state()` for an assistant turn carrying reasoning; `version == 3`. +8. **Version rejection** — `version: 2` and `version: 1` state raise the `"resume_from"`-prefixed `HarnessError`. +9. **In-run guard** — Anthropic/OpenRouter in-run payloads stay byte-identical; OpenAI in-run payload changes **only** by the added `include` key (assert exactly that delta — the deliberate exception to plan-29's byte-identical guard). + +Live suite (guarded behind keys, mirrors the verified smoke tests; one tool round-trip each): OpenAI stateless reasoning replay; Anthropic reconstructed thinking(signature) acceptance; OpenRouter `reasoning_details` echo for one encrypted and one text model. Gate "same-provider reasoning preserved" on these passing. + +## Residual risks + +- **Verified only on the happy path.** redacted_thinking, multiple reasoning items per turn, interleaved-thinking ordering, long multi-turn conversations, and signature acceptance across a model-version change between capture and resume are **not** yet verified — add live cases or document as experimental. +- **`resume_state` now contains encrypted reasoning blobs** for OpenAI/OpenRouter (and signed thinking for Anthropic). Bigger payloads and sensitive content; documented in RESUME-7. +- **Anthropic native re-emit depends on thinking being enabled** in the resuming run; mismatched config silently uses the text fallback (intended, but worth a doc line). +- **OpenAI `include` changes every in-run payload** (the one accepted exception to plan-29's byte-identical guard). + +## Out of scope (deferred) + +- Cross-provider reasoning translation (kept as text by design). +- In-run extended-thinking block-ordering constraints (deferred in plan-29; revisit if the harness enables live thinking). +- RESUME-6 (`previous_response_id` escape-hatch seeding) — unrelated; not addressed here. +- OpenAI same-provider `previous_response_id` fast-path on resume (plan-29 chose uniform full replay). diff --git a/.plans/image-inputs.md b/.plans/image-inputs.md new file mode 100644 index 0000000..81dd4dd --- /dev/null +++ b/.plans/image-inputs.md @@ -0,0 +1,26 @@ +# Image Inputs + +ThinHarness should support images by making model input richer while keeping plain text prompts unchanged. The public API should continue to accept `Harness.run("prompt")`, and additionally accept an ordered sequence of input parts such as text plus image content. + +The core type should be provider-neutral: + +```python +UserInput = str | Sequence[UserInputPart] +UserInputPart = TextPart | ImageUrlPart | BinaryImagePart +``` + +`TextPart` can stay optional at first because bare strings inside the sequence are enough for most callers. `ImageUrlPart` should hold a URL, optional MIME type, optional provider metadata, and an optional `force_download` flag. `BinaryImagePart` should hold bytes, MIME type, and optional provider metadata, with helpers such as `from_path(...)` and `from_data_uri(...)`. + +The harness should normalize prompt handling once near the run boundary. Hooks, tracing, stream events, and model session APIs should receive the neutral input shape instead of assuming a string. Text-only callers should see the same behavior and payloads they see today. + +Provider adapters should own wire-format mapping: + +- OpenAI Responses: map text to `input_text` and images to `input_image`; binary images should use data URIs, URL images should use URLs unless `force_download` is set. +- Anthropic Messages: map text to text blocks and images to image blocks; binary images should use base64 source blocks, URL images should use URL source blocks when supported. +- OpenRouter chat completions: map text and images to chat content parts using `text` and `image_url`; binary images should use data URIs. + +Unsupported image cases should fail loudly with a provider error. The harness should not silently stringify images or drop them. + +Keep the first implementation limited to images. Do not include audio, video, PDFs, uploaded provider files, or prompt-cache markers yet. The type shape should leave room for those later, but the code should not implement speculative modalities. + +Tests should cover backward-compatible text prompts, mixed text/image ordering, URL image mapping, binary image mapping, provider-specific unsupported cases, notice appending with rich input, and tracing redaction or placeholder behavior so raw image bytes are not accidentally written into local traces. diff --git a/README.md b/README.md index 52b216a..9dc0945 100644 --- a/README.md +++ b/README.md @@ -26,7 +26,7 @@ ThinHarness exists for the gap between building the agent loop yourself and adop It owns a focused set of agent-loop primitives that generalize well and are tedious to rebuild, leaving the rest of the application stack for you to own. -I started building ThinHarness after running into this gap in practice. Filesystem-enabled agents are simple but powerful, yet most frameworks don't include them out of the box, and the ones that do are among the largest and heaviest options (Claude Code -> Claude Agent SDK, LangChain -> deepagents, Agno). I usually needed only a small slice of the functionality, but that slice came with coupled assumptions that didn't match my application. Making it fit meant writing enough wrappers, adapters, and fixes that I ended up owning framework-shaped code anyway. +I started building ThinHarness after running into this gap in practice. Filesystem-enabled agents are simple yet powerful, but you mostly get them by adopting a large framework with layers of abstraction. I usually needed only a small slice of the functionality, but that slice came with coupled assumptions that didn't match my application. Making it fit meant writing enough wrappers, adapters, and fixes that I ended up owning framework-shaped code anyway.
@@ -49,39 +49,36 @@ I started building ThinHarness after running into this gap in practice. Filesyst LOC1 Tool
retries2 Subagents - Structured
output Skills FS
tools OTel
tracing - + ThinHarness - 8,241 - ✅ + 8,658 ✅ ✅ ✅ ✅ ✅ - + -  Claude Agent SDK3 +  Claude Agent SDK - 8,263 + 8,2633 ❌ ✅ - ❌ ✅ ✅ ⚠️ - + @@ -90,126 +87,118 @@ I started building ThinHarness after running into this gap in practice. Filesyst 9,840 ❌ ✅ - ✅ ❌ ❌ - ✅ + ⚠️ - + -  deepagents4 +  deepagents - 17,039 + 17,6644 ❌ ✅ - ❌ ✅ ✅ ❌ - +  AWS Strands - 28,157 + 32,526 ⚠️ ✅ ✅ ❌ - ❌ ✅ - +  Microsoft
Agent Framework - 40,514 + 41,331 ❌ ✅ ✅ ✅ - ❌ ✅ - +  Pydantic AI - 59,034 + 59,087 ✅ ❌ - ✅ ❌ ❌ ✅ - +  Google ADK - 64,890 + 65,799 ⚠️ ✅ ✅ ✅ - ❌ ✅ - +  OpenAI Agents SDK - 73,139 - ✅ - ✅ - ✅ - ❌ + 73,796 ❌ ✅ + ✅ + ⚠️ + ⚠️ - +  Agno - 111,539 + 113,477 ⚠️ ✅ ✅ ✅ - ✅ - ✅ + ⚠️ -

* Table focuses on harness-level features that differentiate the libraries. All listed also support MCP, lifecycle hooks, multi-turn conversations, and human-in-the-loop. It intentionally does not compare framework/platform features like vector DB integrations, hosted deployment, memory/session stores, or broad SaaS connectors.

+

* Table focuses on harness-level features that differentiate the libraries. All listed also support MCP, lifecycle hooks, multi-turn conversations, structured output, and human-in-the-loop. It intentionally does not compare framework/platform features like vector DB integrations, hosted deployment, memory/session stores, or broad SaaS connectors.

1. LOC excludes anything that is not the core agent harness framework. See raw README source comments for exact commands.
2. Tool retries: a documented primitive (e.g. Pydantic AI's ModelRetry) that lets tools signal "model passed bad args — retry with this feedback," distinct from generic exception propagation.
3. Claude Agent SDK shells out to the Claude Code CLI binary, which is 200k+ LOC.
- 4. deepagents is a thin wrapper over LangChain/LangGraph; effective import surface is ≈111k LOC. + 4. deepagents is a thin wrapper over LangChain/LangGraph; effective import surface is ≈112k LOC.

@@ -226,13 +215,9 @@ ThinHarness has opinions. They are the reason it stays small. **No bash by default.** Purpose-built business agents usually don't need a shell. Bash is a broad security and reliability surface: it gives the model open-ended authority instead of typed, bounded actions. ThinHarness keeps bash out of the default and built-in tool sets, but exposes an opt-in `BashTool` for exploratory runs before the workflow is hardened with typed tools. -**Skills are tools, not auto-discovery.** Skills live in directories you point at explicitly. The agent calls `skill_read` and `skill_run` like any other tool. No interactive scan of the workspace, no global skill marketplace, no magic. SDK use is deliberate; the auto-discovery design is for interactive coding agents and doesn't belong here. - **Search is a top priority.** The `search` tool exposes ripgrep as compact grouped path/line results, tuned for document and business-workflow agents rather than code navigation. There's also a `jsonl_search` variant, because JSONL is the right shape when you're replacing RAG with agent-driven search over structured data: ripgrep row prefiltering, jq-style field projection, `where` filters, range filters, and snippets from large multiline fields. -**Parallel LLM calls, built in.** Fan out from inside the harness when a workflow needs reliability beyond a single agent loop — majority vote, ensembled extraction. Set `builtin_parallel_llm_model` to enable the default `parallel_llm` tool for plain-text batches; for validated structured output per call, instantiate `ParallelLlmTool` yourself with `output_type` (a Pydantic model). Each call is stateless, and large batches can write JSON to `output_file`. - -**Background tools are simple.** Some long-running tools can start in the background so the agent can keep working. There is no detached job queue, polling API, or job-control surface; the current run still owns the task, and the completion is sent back to the model when it finishes. +**Parallel LLM calls, built in.** Fan out from inside the harness when a workflow needs efficient parallel processing or majority vote for reliability. Set `builtin_parallel_llm_model` to enable the default `parallel_llm` tool for plain-text batches; for validated structured output per call, instantiate `ParallelLlmTool` yourself with `output_type` (a Pydantic model). Each call is stateless, and large batches can write JSON to `output_file`. **No token streaming.** Streaming is for workflow progress, not live chatbot text. ThinHarness emits run, model-turn, tool, retry, limit, background, and subagent events, but it does not stream provider token deltas. Token streaming would add provider-specific plumbing, event merging, cancellation edge cases, and more surface area to keep stable. For workflow-style agents, step-level updates are usually the useful signal. @@ -292,7 +277,7 @@ Streaming emits coarse run, model, tool, background, retry, limit, and subagent - **Subagents:** opt-in delegation through a built-in `subagent` tool and explicit `SubAgentConfig`. - **Parallel LLM:** opt-in `parallel_llm` fan-out for batches of independent one-shot prompts, plus `ParallelLlmTool(...).spec()` for renameable tools with explicit model, path, prompt, and retry settings. - **Skills:** explicit `skill_read` and `skill_run` tools for selected skill directories, with Python, shell, JavaScript, and Go script runners. -- **Resume:** clean new-turn continuation through opaque provider session state. +- **Resume:** clean new-turn continuation through self-contained transcript state that can replay across built-in providers and models, preserving native reasoning on same-provider resume and degrading it to text across providers. - **MCP:** optional MCP client support with lazy tool discovery and collision checks. - **Parallel tool calls:** same-turn tool batches run concurrently when every called tool is parallel-safe. - **Background tools:** opt-in long-running tool calls return a start notice immediately, keep the agent loop moving, and deliver completion back to the model when ready. @@ -302,15 +287,32 @@ Streaming emits coarse run, model, tool, background, retry, limit, and subagent - **Limits and notices:** configured request, tool-call, output-retry, and tool-retry budgets bound each run; near-limit guidance can warn the model before request or tool-call budgets are exhausted. - **Tracing:** local plaintext JSONL traces plus OpenTelemetry-compatible spans for runs, provider calls, tools, and subagents. -## Tracing +## Examples + +Three agents built on ThinHarness, from a self-contained demo to a benchmark run to one I use live. + +### 1. Web Research Report + +A market-landscape research agent that plans, runs batched Exa search, triages and fetches sources, extracts structured source notes with `parallel_llm`, drafts a report, and runs a `citation_critic` subagent before finalizing. + +It isn't meant to be a state-of-the-art research agent; it's a worked example showing the harness drive a non-trivial agentic loop correctly — real multi-tool use across 15 model turns, ending in a reasonable cited report. The full run is browsable on the [docs site](https://ryanbbrown.com/thinharness/examples). + +### 2. LongMemEval-V2 Reproduction + +I ran ThinHarness on a retrieval-heavy 127 question subset of a benchmark for long-term agent memory, and did a local reproduction of the benchmark's optimized harness on the same subset. + +- **Performance:** Matched-or-better accuracy (74.0% vs 72.4% on the 127 dynamic questions) with ~46% less token usage (62M vs. 116M). See [my fork](https://github.com/ryanbbrown/LongMemEval-V2) for more details. +- **Simpler Setup:** ThinHarness only had its built-in filesystem tools (with `jsonl_search` doing the heavy lifting), while the benchmark harness was a full Codex instance with shell and a custom Python tool designed for the task. + +### 3. Personal Opinions Agent -Local tracing is on by default. It writes full plaintext JSONL traces under `~/.thinharness/traces//`, including prompts, model outputs, tool arguments, and tool results, so treat that directory as sensitive local data. +An agent I run live for myself, inspired by [this Substack post](https://blog.kunchenguid.com/p/everyone-should-have-an-opinionsmd). On a schedule, it reads my Readwise highlights + surrounding context, draws on what it's learned in past runs, and proposes conceptual changes to a durable `OPINIONS.md`. -Set `local_tracing=False` or `THINHARNESS_DISABLE_LOCAL_TRACING=1` to disable local trace files. External tracing is generic OpenTelemetry: pass any tracer with `start_as_current_span(...)` or `start_span(...)` in `TracingOptions`, and each sink keeps its own capture policy. +Approval happens over Telegram, and it can be simple accept/reject or involve multi-turn revision + discussion. Under the hood it exercises filesystem tools over a JSONL corpus, native structured output, long-term memory, a custom validation tool, and resumable conversations that pick up across each round. See [here](https://github.com/ryanbbrown/opinions-agent) for the code. ## Status -Pre-1.0. APIs may shift, but I don't expect dramatic changes. Forking is a real option, not just a theoretical one: the codebase is small enough that pulling upstream changes into your fork by hand stays cheap. Each major feature (MCP, subagents, jsonl_search, parallel_llm, background tools, skills) lives in its own file with no hidden dependencies. If you don't use one, that's even less code to worry about. If you want to delete it entirely, that's a one-shot 10-word prompt to a coding agent. +Pre-1.0. APIs may shift, but I don't expect dramatic changes. Forking is a real option, not just a theoretical one: the codebase is small enough that pulling upstream changes into your fork by hand stays cheap. Each major feature (MCP, subagents, jsonl_search, parallel_llm, skills) lives in its own file with no hidden dependencies. If you don't use one, that's even less code to worry about. If you want to delete it entirely, that's a one-shot 10-word prompt to a coding agent. ThinHarness was built with coding agents, but isn't vibe-coded. I have used it, iterated on it, and reviewed its design + behavior. The [docs site](https://ryanbbrown.com/thinharness/) includes a [codebase explainer](https://ryanbbrown.com/thinharness/explainer.html) that I iterated on to understand the library, and the [web research example](https://ryanbbrown.com/thinharness/examples.html) has the transcript from a non-trivial agent run to show that it works effectively. diff --git a/docs/behavior.md b/docs/behavior.md index b42f2b1..3fefadd 100644 --- a/docs/behavior.md +++ b/docs/behavior.md @@ -62,3 +62,34 @@ Use this section only when ordering, lifecycle, concurrency, retries, streaming, - JSONL-TYPED-EQUALITY-3: Date equality filters compare ISO-like date and datetime strings, compare date-only values by calendar date, and treat aware/naive datetime mismatches as non-comparable. - JSONL-TYPED-EQUALITY-4: Non-comparable row values do not match either `eq` or `ne` and increment `compare_warnings` once per candidate row where a typed equality comparison was attempted. - JSONL-TYPED-EQUALITY-5: Invalid typed equality filter definitions fail before scanning rows with `invalid where filter`. + +## Resume State + +### Purpose + +Built-in provider resume state is a self-contained, provider-agnostic transcript that can be replayed by any built-in provider or model while preserving the run lifecycle rules for when resume state is available. + +### Requirements + +- RESUME-1: `resume_state` is a provider-agnostic transcript; resume across built-in providers and across built-in models is supported. +- RESUME-2: `resume_state` is self-contained and does not depend on provider continuation tokens such as OpenAI `previous_response_id`; an OpenAI run that never received a response id is still resumable. +- RESUME-3: Resuming on the originating provider preserves native reasoning (Anthropic thinking signatures, OpenAI `encrypted_content`, OpenRouter `reasoning_details`); resuming on a different provider degrades each reasoning part to a leading ``-tagged text block and drops the opaque blob. Native re-emit additionally requires the resuming run to be able to accept the block: OpenAI re-emits the native reasoning item only when the resuming model is reasoning-capable, and Anthropic only when extended thinking is enabled; otherwise both use the text fallback. So a reasoning-model capture resumed on a non-reasoning model of the same provider degrades to text. +- RESUME-4: Built-in provider resume state uses `version` 3; version 1 and version 2 state and old provider-native `kind` values are rejected with a regenerate error. +- RESUME-5: On resume, the live system prompt from the resuming harness config is re-injected; captured system prompts are not stored or restored. +- RESUME-6: A session seeded via `OpenAIResponsesSession.start(previous_response_id=...)` captures only new transcript entries, so externally seeded prior turns are not present when later resumed from `resume_state`. This is unrelated to reasoning fidelity and is not changed by RESUME-3/RESUME-7. +- RESUME-7: For reasoning-capable OpenAI Responses models the harness requests `include=["reasoning.encrypted_content"]` so reasoning survives resume; non-reasoning models are unaffected. Captured `resume_state` therefore contains encrypted reasoning blobs (OpenAI/OpenRouter) and signed thinking (Anthropic) and should be treated as sensitive, consistent with the local-trace sensitivity note. + +## Model Observability Projections + +### Purpose + +Tracing and streaming expose projections of the same neutral per-request model-visible input and assistant output without changing provider-native in-run request construction. + +### Requirements + +- MODEL-OBSERVABILITY-1: Model trace input is built from the new model-visible entries for that provider request, including rendered `` text whenever notices are sent to the model. +- MODEL-OBSERVABILITY-2: Structured model notice metadata remains available on model spans separately from rendered input messages. +- MODEL-OBSERVABILITY-3: Replayed resume transcript entries are not counted as new model request input for the first resumed provider request; only the new resume prompt or continuation delta is traced as request input. +- MODEL-OBSERVABILITY-4: `ModelMessageEvent.text` always includes assistant text from the completed provider turn; stream text suppression is not part of `StreamOptions`. +- MODEL-OBSERVABILITY-5: Stream lifecycle events remain operational events and are not stored in durable provider transcript entries. +- MODEL-OBSERVABILITY-6: Core tracing emits OTel/GenAI-oriented attributes and does not include sink-specific display namespaces. diff --git a/docs/docs.md b/docs/docs.md index c6ba132..6dfe3b4 100644 --- a/docs/docs.md +++ b/docs/docs.md @@ -63,7 +63,7 @@ Stream events are high-level workflow events intended for app consumption: - `ToolCallStartedEvent.arguments` includes the model-requested tool arguments. - `ToolCallCompletedEvent.output` and `BackgroundTaskCompletedEvent.output` include model-visible tool output. - Raw provider response JSON is not part of stream events; use `HarnessResult.responses` for raw provider responses after completion. -- `ModelMessageEvent.text` is included by default; set `include_model_text=False` to hide it. +- `ModelMessageEvent.text` includes assistant text from the completed provider turn. - Child subagent events are flattened by default; set `include_subagents=False` to keep only the parent `subagent` tool lifecycle. Workflow UIs should group nested work with `kind`, `run_id`, `parent_run_id`, and `parent_tool_call_id`. The final successful `RunCompletedEvent` carries the same full `HarnessResult` returned by `run()`, including `responses`, `tool_call_records`, and `resume_state`; treat that terminal result as completion data rather than a progress update. @@ -245,11 +245,11 @@ Set `requires_approval=True` on a custom `ToolSpec` when the host application mu The paused result includes: - `pending_approvals`: call id, tool name, and raw JSON arguments for each approval-required call. -- `resume_state`: one JSON-serializable approval envelope that wraps provider resume state, the full paused tool batch, run history, usage, metadata, and accounting needed to continue the same logical run. +- `resume_state`: one JSON-serializable approval envelope that wraps the provider resume payload, the full paused tool batch, run history, usage, metadata, and accounting needed to continue the same logical run. For built-in providers, the nested `provider_state` is the full neutral transcript. Resume with `resume_approvals(...)`, `stream_approvals(...)`, or `resume_approvals_sync(...)` and one `ApprovalDecision` per pending approval. Approved calls execute through the normal tool machinery, including hooks, tracing, retry accounting, and stream events. Rejected calls do not execute or fire tool hooks; the model receives a failed tool result with `error_type="ApprovalRejected"` and can explain, recover, or request another tool. -Approval-required tools need a resumable model because the harness must continue the exact provider session after the paused assistant tool-call turn. They cannot use background execution, and they are not supported inside child subagent harnesses. Built-in tools remain non-approval tools in this version; wrap built-in behavior in a custom `ToolSpec` when host review is required. +Approval-required tools need a resumable model because the harness must continue after the paused assistant tool-call turn. They cannot use background execution, and they are not supported inside child subagent harnesses. Built-in tools remain non-approval tools in this version; wrap built-in behavior in a custom `ToolSpec` when host review is required. ### Bash Prototype Tool @@ -559,21 +559,24 @@ The contract: - Save `result.resume_state` exactly as JSON. - Pass it back as `resume_from` with the next user message. -- Use the same provider, model, system prompt, and tools as the run that produced it. +- Built-in provider state is a self-contained transcript and can be resumed by any built-in provider or model. +- The resuming harness supplies the live system prompt and tool schemas; captured system prompts are not stored or restored. - Expect no state after failed, cancelled, partial, or exhausted runs. -- Treat the contents as provider-owned details; do not read or construct them. +- Treat the contents as harness-owned details; persist them exactly, but do not construct them by hand. `resume_from` is a new-turn API. The prior run completed, and the next call appends a new user message. It is not a retry mechanism, interrupted-tool-call recovery, or a way to continue the assistant's previous response. Approval pauses use a separate resume path. When `stop_reason == "approval_required"`, persist the returned `resume_state` envelope and call `resume_approvals(...)` with approval decisions instead of passing that envelope to `run(..., resume_from=...)`. The post-resume result carries the full logical run history: pre-pause responses, tool records, usage counters, and metadata are restored before the approved or rejected batch is processed. -Budgets span the pause. The paused batch counts against `usage.tool_calls` exactly once at pause time, and a resumed run can immediately hit `limit_reached` if the logical run was already at its configured model-request or tool-call limit. The approval envelope also includes raw provider responses, so its stored size grows with run length. +Budgets span the pause. The paused batch counts against `usage.tool_calls` exactly once at pause time, and a resumed run can immediately hit `limit_reached` if the logical run was already at its configured model-request or tool-call limit. The approval envelope also includes the full nested provider transcript and raw provider responses, so its stored size grows with run length. `APPROVAL_ENVELOPE_VERSION` remains independent from the nested provider-state version; old nested provider states fail when the provider resume step validates them. -Provider behavior differs internally: +Built-in provider resume details: -- OpenAI Responses stores conversation state server-side. `resume_state` contains the previous response id. -- Anthropic Messages stores the full message transcript in `resume_state`. -- OpenRouter chat completions stores the full chat transcript in `resume_state`. +- `resume_state["kind"] == "transcript"` and `version == 3`. +- The transcript is provider-agnostic and no longer depends on OpenAI server-side response retention. +- Provider-specific reasoning chains are preserved on same-provider resume (Anthropic thinking signatures, OpenAI `encrypted_content`, OpenRouter `reasoning_details`) and degraded to a leading ``-tagged text block on cross-provider resume. Anthropic native re-emit also requires extended thinking to be enabled in the resuming run. For reasoning-capable OpenAI models the harness adds `include=["reasoning.encrypted_content"]`, so `resume_state` can contain encrypted reasoning blobs — treat it as sensitive. +- Cross-provider resume is supported by the built-in renderers, but real providers may reject foreign-format tool-call ids or malformed tool-call argument JSON. +- `OpenAIResponsesSession.start(previous_response_id=...)` remains available as a low-level escape hatch, but later resume state captures only the new prompt onward, not the externally seeded prior turns. The same `resume_state` can be reused for sequential branching: diff --git a/docs/site/about/index.html b/docs/site/about/index.html index 0e495f5..369d390 100644 --- a/docs/site/about/index.html +++ b/docs/site/about/index.html @@ -41,7 +41,6 @@ install use features - tracing status license @@ -49,7 +48,7 @@
// why this exists

Why this exists

-

Production agents rarely stop at framework configuration. Things like orchestration, permissions, user/session storage, and deployment become specific to the application and its users.

ThinHarness exists for the gap between building the agent loop yourself and adopting a large agent runtime where the loop comes bundled with assumptions you don’t need and can’t easily change.

It owns a focused set of agent-loop primitives that generalize well and are tedious to rebuild, leaving the rest of the application stack for you to own.

I started building ThinHarness after running into this gap in practice. Filesystem-enabled agents are simple but powerful, yet most frameworks don't include them out of the box, and the ones that do are among the largest and heaviest options (Claude Code -> Claude Agent SDK, LangChain -> deepagents, Agno). I usually needed only a small slice of the functionality, but that slice came with coupled assumptions that didn't match my application. Making it fit meant writing enough wrappers, adapters, and fixes that I ended up owning framework-shaped code anyway.

+

Production agents rarely stop at framework configuration. Things like orchestration, permissions, user/session storage, and deployment become specific to the application and its users.

ThinHarness exists for the gap between building the agent loop yourself and adopting a large agent runtime where the loop comes bundled with assumptions you don’t need and can’t easily change.

It owns a focused set of agent-loop primitives that generalize well and are tedious to rebuild, leaving the rest of the application stack for you to own.

I started building ThinHarness after running into this gap in practice. Filesystem-enabled agents are simple yet powerful, but you mostly get them by adopting a large framework with layers of abstraction. I usually needed only a small slice of the functionality, but that slice came with coupled assumptions that didn't match my application. Making it fit meant writing enough wrappers, adapters, and fixes that I ended up owning framework-shaped code anyway.

@@ -69,7 +68,6 @@

How small, exactly

LOC1 Tool
retries2 Sub-
agents - Structured
output Skills FS
tools OTel
tracing @@ -78,63 +76,63 @@

How small, exactly

ThinHarness
- 8,241 - ●●●●●● + 8,658 + ●●●●● -
Claude Agent SDK3
- 8,263 - ○●○●●◑ +
Claude Agent SDK
+ 8,2633 + ○●●●◑
smolagents
9,840 - ○●●○○● + ○●○○◑ -
deepagents4
- 17,039 - ○●○●●○ +
deepagents
+ 17,6644 + ○●●●○
AWS Strands
- 28,157 - ◑●●○○● + 32,526 + ◑●●○●
Microsoft Agent Framework
- 40,514 - ○●●●○● + 41,331 + ○●●●●
Pydantic AI
- 59,034 - ●○●○○● + 59,087 + ●○○○●
Google ADK
- 64,890 - ◑●●●○● + 65,799 + ◑●●●●
OpenAI Agents SDK
- 73,139 - ●●●○○● + 73,796 + ○●●◑◑
Agno
- 111,539 - ◑●●●●● + 113,477 + ◑●●●◑ -

Table focuses on harness-level features that differentiate the libraries. All listed also support MCP, lifecycle hooks, multi-turn conversations, and human-in-the-loop. It intentionally does not compare framework/platform features like vector DB integrations, hosted deployment, memory/session stores, or broad SaaS connectors.

+

Table focuses on harness-level features that differentiate the libraries. All listed also support MCP, lifecycle hooks, multi-turn conversations, structured output, and human-in-the-loop. It intentionally does not compare framework/platform features like vector DB integrations, hosted deployment, memory/session stores, or broad SaaS connectors.

1. LOC excludes anything that is not the core agent harness framework. See raw README source comments for exact commands.

2. Tool retries: a documented primitive (e.g. Pydantic AI's ModelRetry) that lets tools signal "model passed bad args — retry with this feedback," distinct from generic exception propagation.

3. Claude Agent SDK shells out to the Claude Code CLI binary, which is 200k+ LOC.

-

4. deepagents is a thin wrapper over LangChain/LangGraph; effective import surface is ≈111k LOC.

+

4. deepagents is a thin wrapper over LangChain/LangGraph; effective import surface is ≈112k LOC.

See docs/table.md for per-cell rationale and how the LOC numbers are measured.

@@ -146,10 +144,8 @@

Opinions

purpose_built

Purpose-built agents, not universal agents

ThinHarness is for bounded agent loops inside software you control, not open-ended interactive assistants. For business use cases, focused agent loops orchestrated by deterministic code are usually a better fit than sprawling multi-agent systems with broad authority.

no_bash

No bash by default

Purpose-built business agents usually don't need a shell. Bash is a broad security and reliability surface: it gives the model open-ended authority instead of typed, bounded actions. ThinHarness keeps bash out of the default and built-in tool sets, but exposes an opt-in BashTool for exploratory runs before the workflow is hardened with typed tools.

-
skills

Skills are tools, not auto-discovery

Skills live in directories you point at explicitly. The agent calls skill_read and skill_run like any other tool. No interactive scan of the workspace, no global skill marketplace, no magic. SDK use is deliberate; the auto-discovery design is for interactive coding agents and doesn't belong here.

search

Search is a top priority

The search tool exposes ripgrep as compact grouped path/line results, tuned for document and business-workflow agents rather than code navigation. There's also a jsonl_search variant, because JSONL is the right shape when you're replacing RAG with agent-driven search over structured data: ripgrep row prefiltering, jq-style field projection, where filters, range filters, and snippets from large multiline fields.

-
parallel_llm

Parallel LLM calls, built in

Fan out from inside the harness when a workflow needs reliability beyond a single agent loop — majority vote, ensembled extraction. Set builtin_parallel_llm_model to enable the default parallel_llm tool for plain-text batches; for validated structured output per call, instantiate ParallelLlmTool yourself with output_type (a Pydantic model). Each call is stateless, and large batches can write JSON to output_file.

-
background_tools

Background tools are simple

Some long-running tools can start in the background so the agent can keep working. There is no detached job queue, polling API, or job-control surface; the current run still owns the task, and the completion is sent back to the model when it finishes.

+
parallel_llm

Parallel LLM calls, built in

Fan out from inside the harness when a workflow needs efficient parallel processing or majority vote for reliability. Set builtin_parallel_llm_model to enable the default parallel_llm tool for plain-text batches; for validated structured output per call, instantiate ParallelLlmTool yourself with output_type (a Pydantic model). Each call is stateless, and large batches can write JSON to output_file.

no_token_streaming

No token streaming

Streaming is for workflow progress, not live chatbot text. ThinHarness emits run, model-turn, tool, retry, limit, background, and subagent events, but it does not stream provider token deltas. Token streaming would add provider-specific plumbing, event merging, cancellation edge cases, and more surface area to keep stable. For workflow-style agents, step-level updates are usually the useful signal.

providers

Three providers, no matrix

ThinHarness ships small provider classes for OpenAI, Anthropic, and OpenRouter. If your gateway speaks one of those protocols, you swap a base URL and move on. If not, the provider classes are small enough to fork or replace, and ignoring the bundled ones costs you nothing.

no_compaction

No compaction

Compaction is a workaround for context windows filling up across long, accumulating runs — useful for interactive coding sessions that sprawl over hours. For SDK-based business agents, the right answer to "context is getting big" is almost always better task decomposition: shorter runs, separate harness instances, narrower subagents.

@@ -193,7 +189,7 @@

Features

Subagents

Opt-in delegation through a built-in subagent tool and explicit SubAgentConfig.

Parallel LLM

Opt-in parallel_llm fan-out for batches of independent one-shot prompts, plus ParallelLlmTool(...).spec() for renameable tools with explicit model, path, prompt, and retry settings.

Skills

Explicit skill_read and skill_run tools for selected skill directories, with Python, shell, JavaScript, and Go script runners.

-
Resume

Clean new-turn continuation through opaque provider session state.

+
Resume

Clean new-turn continuation through self-contained transcript state that can replay across built-in providers and models, preserving native reasoning on same-provider resume and degrading it to text across providers.

MCP

Optional MCP client support with lazy tool discovery and collision checks.

Parallel tool calls

Same-turn tool batches run concurrently when every called tool is parallel-safe.

Background tools

Opt-in long-running tool calls return a start notice immediately, keep the agent loop moving, and deliver completion back to the model when ready.

@@ -205,13 +201,6 @@

Features

-
-
// tracing
-

Tracing

-

Local tracing is on by default. It writes full plaintext JSONL traces under ~/.thinharness/traces/<encoded-project-root>/, including prompts, model outputs, tool arguments, and tool results, so treat that directory as sensitive local data.

-

Set local_tracing=False or THINHARNESS_DISABLE_LOCAL_TRACING=1 to disable local trace files. External tracing is generic OpenTelemetry: pass any tracer with start_as_current_span(...) or start_span(...) in TracingOptions, and each sink keeps its own capture policy.

-
-
// status

Status

diff --git a/docs/site/assets/site.css b/docs/site/assets/site.css index 7d7f3c8..c62ed3d 100644 --- a/docs/site/assets/site.css +++ b/docs/site/assets/site.css @@ -1812,3 +1812,27 @@ body.page-transcripts { background: #fff; } } + +/* ---- examples page: LongMemEval / transcript toggle ---- */ +.ex-toggle{display:flex;justify-content:center;gap:8px;width:min(1180px,calc(100vw - 48px));margin:18px auto 6px;} +.ex-toggle button{font-family:var(--mono);font-size:13px;padding:8px 16px;border:1px solid var(--line);border-radius:8px;background:var(--panel);color:var(--ink-soft);cursor:pointer;transition:color .15s,background .15s,border-color .15s;} +.ex-toggle button:hover{color:var(--green-deep);border-color:var(--green-line);} +.ex-toggle button.is-active{color:var(--green-deep);background:var(--green-wash);border-color:var(--green-line);font-weight:600;} +.ex-panel[hidden]{display:none !important;} +.md-render{width:min(860px,calc(100vw - 48px));margin:0 auto;padding:14px 0 64px;color:var(--ink);font-family:var(--sans);line-height:1.62;} +.md-render .md-eyebrow{font-family:var(--mono);font-size:13px;color:var(--green);letter-spacing:.04em;font-weight:500;margin:6px 0 6px;} +.md-render h1{font-size:30px;margin:0 0 8px;letter-spacing:-.02em;font-weight:700;} +.md-render h2{font-size:20px;margin:30px 0 10px;color:var(--green-deep);border-bottom:1px solid var(--line);padding-bottom:6px;} +.md-render h3{font-size:16px;margin:20px 0 8px;} +.md-render p{margin:10px 0;color:var(--ink-soft);} +.md-render ul{margin:10px 0;padding-left:22px;color:var(--ink-soft);} +.md-render li{margin:7px 0;} +.md-render a{color:var(--green);text-decoration:underline;text-underline-offset:2px;} +.md-render code{font-family:var(--mono);font-size:.86em;background:var(--green-wash);padding:1px 5px;border-radius:4px;color:var(--green-deep);} +.md-render strong{color:var(--ink);} +.md-table-wrap{overflow-x:auto;margin:14px 0;border:1px solid var(--line);border-radius:8px;max-width:620px;} +.md-render table{width:100%;border-collapse:collapse;font-size:13px;} +.md-render th,.md-render td{padding:8px 12px;border-bottom:1px solid var(--line);text-align:left;vertical-align:top;} +.md-render td:first-child,.md-render th:first-child{font-family:var(--mono);font-size:12px;color:var(--ink);white-space:nowrap;} +.md-render thead th{background:var(--green-wash);color:var(--green-deep);font-weight:600;} +.md-render tbody tr:last-child td{border-bottom:none;} diff --git a/docs/site/examples/index.html b/docs/site/examples/index.html index 4f9ca92..3a207a4 100644 --- a/docs/site/examples/index.html +++ b/docs/site/examples/index.html @@ -3,7 +3,7 @@ - Web Research Example — ThinHarness + Examples — ThinHarness @@ -22,11 +22,30 @@
-
// example run
-

Web Research Example

-
A complete DeepSeek + Exa run: plan, search, source notes, citation critique, final report.
+
// examples
+

Examples

+
Real ThinHarness runs. Toggle between the LongMemEval-V2 benchmark write-up and the Web Research transcript.
-
+
+ + +
+
+
// benchmark reproduction

LongMemEval-V2 - ThinHarness Benchmark

+

Context

+

I used a fork of the LongMemEval-V2 benchmark to test whether ThinHarness is a strong general-purpose agent harness on a nontrivial non-coding task: information retrieval over long trajectory haystacks.

+

I scoped the comparison to the 127 dynamic questions in the small tier: dynamic-environment and dynamic-environment-abs across the web and enterprise domains. After reviewing the benchmark categories, this subset looked like the best test for my purposes: given a long history of interaction traces, can the memory layer efficiently and accurately find the state change, UI behavior, or environment fact needed to answer the question?

+

Run Details

+

To get a local baseline for more detailed metrics than the leaderboard provides, I ran AgentRunbook-C with the resumable wrapper in evaluation/scripts/run_agentrunbook_c_dynamic.sh. That script plans the 127-question dynamic set, runs one question per output directory, and can be re-run to fill only missing questions. Running one question at a time is slower but was helpful when encountering intermittent issues running on my local machine. The matching ThinHarness wrapper is evaluation/scripts/run_thinharness_dynamic.sh.

+

This is a best-faith reproduction rather than an exact reproduction of the paper. The paper setup expects a local Qwen/Qwen3.5-9B reader deployment; I used qwen/qwen3.5-9b through OpenRouter. The paper uses Codex v0.117.0 for Codex and AgentRunbook-C; my rerun used the local Codex CLI available, codex-cli 0.141.0. Both the AgentRunbook-C rerun and ThinHarness run used gpt-5.4-mini with xhigh reasoning for the query-time memory agent and gpt-5.2 for the evaluator.

+

For the ThinHarness run, I used only its generic built-in filesystem tools (read, search, jsonl_search, list, and glob). I did restructure the memory files into a JSONL-friendly corpus so its generic jsonl_search tool could work well; that seems acceptable here because AgentRunbook-C also creates a custom trajectory structure rather than using the vanilla Codex raw layout. I tried to keep the query-time system prompt as close as practical to AgentRunbook-C, changing the tool instructions only where ThinHarness needed to know how to use its built-in tools.

+

Results

+

Across all 127 dynamic questions in the small tier:

+
MetricAgentRunbook-C rerunThinHarness
Dynamic score72.4% (92/127)74.0% (94/127)
Non-abstention86.0% (74/86)84.9% (73/86)
Abstention43.9% (18/41)51.2% (21/41)
Memory query time151.9s avg, 129.1s median99.7s avg, 87.9s median
Memory-agent tokens / usage114.77M input, 1.32M output, 116.09M total60.14M input, 2.10M output, 62.24M total
Dynamic-subset LAFS3.769.73 (+5.96)
+

The 72.4% accuracy for AgentRunbook-C matches the paper, but I would not treat this single consolidated run as a statistically signficant claim that ThinHarness has higher accuracy than AgentRunbook-C--I saw meaningful variance on a portion of the questions when doing targeted reruns. The result does make me reasonably confident that ThinHarness at least matches AgentRunbook-C's performance on this slice, and the published leaderboard reference for vanilla Codex is materially lower than both.

+

The memory query time is the harness-measured time around memory.query(...): it includes the query-time memory retrieval agent, but not the downstream reader, scorer, or prior runtime input generation. The timing comparison isn't perfect (local Codex CLI vs. OpenAI API), but the 46.4% lower token usage indicates that ~34% time savings is probably in the right ballpark. Note that the paper only provides a single aggregate query time figure across all questions, 108.3s, which is far lower than the 151.9s above but includes all questions in the small tier (some of which may have been faster).

+
+ diff --git a/docs/site/index.html b/docs/site/index.html index d23e268..42af279 100644 --- a/docs/site/index.html +++ b/docs/site/index.html @@ -36,7 +36,7 @@

A minimal, opinionated agent harness.
install.sh
-
copy$ uv add thinharness
# or: pip install thinharness
# requires python 3.11+

✓ resolved · 22 files · 8,241 LOC
+
copy$ uv add thinharness
# or: pip install thinharness
# requires python 3.11+

✓ resolved · 23 files · 8,658 LOC

@@ -46,10 +46,10 @@

A minimal, opinionated agent harness.
purpose_built

Purpose-built agents

ThinHarness is for bounded agent loops inside software you control, not open-ended interactive assistants.

no_bash

No bash by default

Bash stays out of the default tools, with an opt-in BashTool only for prototyping before typed tools.

-
skills

Skills are tools, not auto-discovery

Skills live in directories you point at explicitly. The agent calls them like any other tool. No magic scan, no marketplace.

-
search

Search is a top priority

Ripgrep exposed as compact grouped results, tuned for documents and business workflows — plus JSONL search with field projection, range filters, and multiline snippets.

-
parallel_llm

Parallel LLM calls, built in

Fan out from inside the harness when a workflow needs reliability beyond a single loop — majority vote, ensembled extraction.

-
background_tools

Background tools are simple

Long-running tools can start in the background, but the current run still owns the task and receives completion.

+
search

Search is a top priority

Ripgrep exposed as compact grouped results, tuned for documents and business workflows — plus a custom JSONL search tool for structured corpuses.

+
parallel_llm

Parallel LLM calls, built in

Fan out from inside the harness when a workflow needs efficient parallel processing or majority vote for reliability.

+
no_compaction

No compaction

Compaction makes sense for sprawling coding sessions. For business agents the fix is smarter task decomposition and context management

+
no_deployment

No deployment layer

Serving, auth, durable jobs, and session storage stay yours. ThinHarness owns the agent loop, not the production stack around it.

diff --git a/docs/table.md b/docs/table.md index 4d4d36e..aaaf714 100644 --- a/docs/table.md +++ b/docs/table.md @@ -12,33 +12,35 @@ The README is the canonical rendered table. The raw README source comments keep the command beside each row. To reproduce locally, clone each upstream repo at the pinned commit and run the command shown below. -Measured 2026-06-15 for upstream libraries. ThinHarness was remeasured from the -current working tree on 2026-06-20. +Measured 2026-06-22 for upstream libraries (pinned commits below). ThinHarness +was measured from the current working tree on 2026-06-23. ## LOC Commands - **ThinHarness** — `tokei thinharness/ -t Python` -- **Claude Agent SDK** — `tokei src/claude_agent_sdk/ -t Python --exclude testing` at `anthropics/claude-agent-sdk-python @ 634c2f6` -- **smolagents** — `tokei src/smolagents/ -t Python --exclude cli.py --exclude gradio_ui.py --exclude vision_web_browser.py` at `huggingface/smolagents @ e8b988d` -- **deepagents** — `tokei libs/deepagents/deepagents/ -t Python` at `langchain-ai/deepagents @ 5975503` -- **AWS Strands** — `tokei strands-py/src/strands/ -t Python --exclude experimental --exclude vended_plugins --exclude multiagent/a2a` at `strands-agents/sdk-python @ a92502f` -- **Microsoft Agent Framework** — `tokei python/packages/core/agent_framework/ -t Python --exclude _evaluation.py --exclude a2a --exclude ag_ui --exclude chatkit --exclude declarative --exclude devui --exclude hyperlight --exclude lab --exclude orchestrations --exclude mem0 --exclude redis --exclude microsoft` at `microsoft/agent-framework @ ed4ff18` -- **Pydantic AI** — `tokei pydantic_ai_slim/pydantic_ai/ -t Python --exclude _a2a.py --exclude ag_ui.py --exclude ui --exclude durable_exec --exclude embeddings --exclude ext` at `pydantic/pydantic-ai @ fabeacc` -- **Google ADK** — `tokei src/google/adk/ -t Python --exclude a2a --exclude apps --exclude cli --exclude cloud --exclude code_executors --exclude environment --exclude evaluation --exclude examples --exclude integrations --exclude optimization --exclude platform` at `google/adk-python @ 22adbe1` -- **OpenAI Agents SDK** — `tokei src/agents/ -t Python --exclude realtime --exclude voice --exclude extensions/experimental --exclude extensions/visualization.py` at `openai/openai-agents-python @ c359c20` -- **Agno** — `tokei libs/agno/agno/{agent,agents,approval,compression,factory,guardrails,hooks,memory,models,reasoning,registry,run,session,skills,team,tools,tracing,utils} -t Python` at `agno-agi/agno @ 5cf1ed7` +- **Claude Agent SDK** — `tokei src/claude_agent_sdk/ -t Python --exclude testing` at `anthropics/claude-agent-sdk-python @ 315df97` +- **smolagents** — `tokei src/smolagents/ -t Python --exclude cli.py --exclude gradio_ui.py --exclude vision_web_browser.py` at `huggingface/smolagents @ 526069c` +- **deepagents** — `tokei libs/deepagents/deepagents/ -t Python` at `langchain-ai/deepagents @ eb9de75` +- **AWS Strands** — `tokei strands-py/src/strands/ -t Python --exclude experimental --exclude vended_plugins --exclude multiagent/a2a` at `strands-agents/sdk-python @ a5a2cf9` +- **Microsoft Agent Framework** — `tokei python/packages/core/agent_framework/ -t Python --exclude _evaluation.py --exclude a2a --exclude ag_ui --exclude chatkit --exclude declarative --exclude devui --exclude hyperlight --exclude lab --exclude orchestrations --exclude mem0 --exclude redis --exclude microsoft` at `microsoft/agent-framework @ 2999f74` +- **Pydantic AI** — `tokei pydantic_ai_slim/pydantic_ai/ -t Python --exclude _a2a.py --exclude ag_ui.py --exclude ui --exclude durable_exec --exclude embeddings --exclude ext` at `pydantic/pydantic-ai @ 53e0641` +- **Google ADK** — `tokei src/google/adk/ -t Python --exclude a2a --exclude apps --exclude cli --exclude cloud --exclude code_executors --exclude environment --exclude evaluation --exclude examples --exclude integrations --exclude optimization --exclude platform` at `google/adk-python @ 8c9fff8` +- **OpenAI Agents SDK** — `tokei src/agents/ -t Python --exclude realtime --exclude voice --exclude extensions/experimental --exclude extensions/visualization.py` at `openai/openai-agents-python @ a9b7b7e` +- **Agno** — `tokei libs/agno/agno/{agent,agents,approval,compression,factory,guardrails,hooks,memory,models,reasoning,registry,run,session,skills,team,tools,tracing,utils} -t Python` at `agno-agi/agno @ 16f33c1` Claude Agent SDK also shells out to the Claude Code CLI binary, which is 200k+ LOC. The table counts the Python SDK package and footnotes that relationship. ## Notes on the Marks -- **Tool retries** — only Pydantic AI (`ModelRetry`) and OpenAI Agents (`ModelRetryAdvice` / `ModelRetrySettings`) ship a documented, named primitive that lets a tool function signal "model passed bad args — please retry with this feedback," distinct from generic exception propagation. AWS Strands has hook-based retry via `AfterToolCallEvent.retry=True`, Google ADK has a `ReflectAndRetryToolPlugin`, and Agno has a `RetryAgentRun` exception that retries the whole agent run rather than a single tool — these are marked `⚠️`. Claude Agent SDK, smolagents, deepagents, and Microsoft Agent Framework have no named primitive and are marked `❌`. -- **Subagents** — Pydantic AI documents an "agent delegation" pattern, where one agent is called inside another's tool function, but ships no class, decorator, or middleware for it. Its own multi-agent docs point users to deepagents for that case, so it is marked `❌`. -- **Structured output** — Claude Agent SDK and deepagents return free-form messages with no built-in validation step, so they are marked `❌`. -- **Skills** — Pydantic AI, OpenAI Agents SDK, smolagents, and AWS Strands have no Markdown/frontmatter skills primitive, so they are marked `❌`. -- **Built-in FS tools** — A `✅` means the project ships a model-facing filesystem toolkit with read/write/edit or search-style primitives. OpenAI Agents SDK is marked `⚠️` because it has hosted `apply_patch` and shell tools, but not a full read/write/search toolkit. Google ADK is marked `⚠️` because its experimental environment tools include `ReadFile`, `WriteFile`, and `EditFile`, but those are outside the strict core LOC scope above. Pydantic AI, smolagents, AWS Strands, and Microsoft Agent Framework are marked `❌` because they do not ship a comparable toolkit in the core package. Generic shell or code-exec tools do not count as full filesystem tools. -- **OTel tracing** — deepagents leans on LangSmith rather than emitting OTel from its own code, so it is marked `❌`. Claude Agent SDK is marked `⚠️` because the Python SDK itself ships no instrumentation beyond W3C traceparent propagation into the CLI subprocess, while the Claude Code CLI it shells out to has beta OTel support. +Marks reflect shipped, documented, first-class capability judged on the public API — independent of whether a feature is gated experimental or lives in a directory excluded from the strict LOC count. + +- **Tool retries** — only Pydantic AI (`ModelRetry`) ships a documented, named primitive that lets a tool function signal "model passed bad args — please retry with this feedback," distinct from generic exception propagation. OpenAI Agents' `ModelRetryAdvice` / `ModelRetrySettings` are runner-managed retries for the *model HTTP call* (network/timeout/HTTP-status backoff), not a tool-feedback primitive. AWS Strands has hook-based retry via `AfterToolCallEvent.retry=True`, Google ADK has a `ReflectAndRetryToolPlugin`, and Agno has a `RetryAgentRun` exception that retries the whole agent run rather than a single tool — these are marked `⚠️`. Claude Agent SDK, smolagents, deepagents, Microsoft Agent Framework, and OpenAI Agents SDK have no named primitive and are marked `❌`. +- **Subagents** — Pydantic AI documents an "agent delegation" pattern, where one agent is called inside another's tool function, but ships no class, decorator, or middleware for it in core (real subagent primitives live in external community packages), so it is marked `❌`. +- **Structured output** — every library now ships a built-in output-validation step, so the column is uniformly `✅`. Claude Agent SDK (`output_format` / `structured_output`, validated and re-prompted in the CLI it shells out to) and deepagents (`response_format` on `create_deep_agent`) added theirs in 2026. +- **Skills** — Pydantic AI and smolagents have no Markdown/frontmatter skills primitive in core, so they are marked `❌`. OpenAI Agents SDK (sandbox `Skills`) and AWS Strands (top-level `Skill` / `AgentSkills`) added one in 2026. +- **Built-in FS tools** — A `✅` means the project ships a model-facing filesystem toolkit with read/write/edit or search-style primitives. Google ADK (`ReadFile` / `WriteFile` / `EditFile`) and Microsoft Agent Framework (`FileAccessProvider` — read/write/delete/list/search) both ship one; both are gated experimental, but the mark reflects shipped capability regardless of maturity or LOC scope. OpenAI Agents SDK is marked `⚠️` because it ships only hosted `apply_patch` (create/update/delete) plus shell — not a full read/write/search toolkit (read and search come through bash, which doesn't count). Pydantic AI, smolagents, and AWS Strands ship no comparable toolkit in core, so they are marked `❌`. Generic shell or code-exec tools do not count as full filesystem tools. +- **OTel tracing** — AWS Strands, Microsoft Agent Framework, Pydantic AI, and Google ADK emit OpenTelemetry spans from their own code (`✅`). smolagents, OpenAI Agents SDK, and Agno are marked `⚠️` because they emit no spans from their own code: tracing comes from a separate external instrumentor (OpenInference, for smolagents and Agno) or a proprietary exporter with OTel reachable only via third-party processors (OpenAI). Claude Agent SDK is marked `⚠️` because the Python SDK itself ships no instrumentation beyond W3C traceparent propagation into the CLI subprocess, while the Claude Code CLI it shells out to has beta OTel support. deepagents leans on LangSmith rather than emitting OTel from its own code, so it is marked `❌`. ## What "Strict Framework-Only" Excludes @@ -59,4 +61,4 @@ Per-library exclusions: - **Pydantic AI** — `_a2a.py`, `ag_ui.py`, `ui/`, `durable_exec/`, `embeddings/`, `ext/` (A2A, UI, durable execution runtime, embedding models, ext). - **Google ADK** — `a2a/`, `apps/`, `cli/`, `cloud/`, `code_executors/`, `environment/`, `evaluation/`, `examples/`, `integrations/`, `optimization/`, `platform/`. - **OpenAI Agents SDK** — `realtime/`, `voice/`, `extensions/experimental`, `extensions/visualization.py`. -- **Agno** — `api/`, `client/`, `cloud/`, `db/`, `integrations/`, `knowledge/`, `learn/`, `os/`, `remote/`, `scheduler/`, `vectordb/`, `context/`, `culture/`, plus boundary cases `workflow/` and `eval/`. As shipped, Agno is 262,687 LOC. +- **Agno** — `api/`, `client/`, `cloud/`, `db/`, `integrations/`, `knowledge/`, `learn/`, `os/`, `remote/`, `scheduler/`, `vectordb/`, `context/`, `culture/`, plus boundary cases `workflow/` and `eval/`. As shipped, Agno is 265,195 LOC. diff --git a/examples/longmemeval.md b/examples/longmemeval.md new file mode 100644 index 0000000..15f607c --- /dev/null +++ b/examples/longmemeval.md @@ -0,0 +1,33 @@ +# LongMemEval-V2 - ThinHarness Benchmark + +## Context + +I used a [fork](https://github.com/ryanbbrown/LongMemEval-V2) of the LongMemEval-V2 benchmark to test whether ThinHarness is a strong general-purpose agent harness on a nontrivial non-coding task: information retrieval over long trajectory haystacks. + +I scoped the comparison to the 127 dynamic questions in the small tier: `dynamic-environment` and `dynamic-environment-abs` across the web and enterprise domains. After reviewing the benchmark categories, this subset looked like the best test for my purposes: given a long history of interaction traces, can the memory layer efficiently and accurately find the state change, UI behavior, or environment fact needed to answer the question? + +## Run Details + +To get a local baseline for more detailed metrics than the leaderboard provides, I ran AgentRunbook-C with the resumable wrapper in `evaluation/scripts/run_agentrunbook_c_dynamic.sh`. That script plans the 127-question dynamic set, runs one question per output directory, and can be re-run to fill only missing questions. Running one question at a time is slower but was helpful when encountering intermittent issues running on my local machine. The matching ThinHarness wrapper is `evaluation/scripts/run_thinharness_dynamic.sh`. + +This is a best-faith reproduction rather than an exact reproduction of the paper. The paper setup expects a local `Qwen/Qwen3.5-9B` reader deployment; I used `qwen/qwen3.5-9b` through OpenRouter. The paper uses Codex v0.117.0 for Codex and AgentRunbook-C; my rerun used the local Codex CLI available, `codex-cli 0.141.0`. Both the AgentRunbook-C rerun and ThinHarness run used `gpt-5.4-mini` with `xhigh` reasoning for the query-time memory agent and `gpt-5.2` for the evaluator. + +For the ThinHarness run, I used only its generic built-in filesystem tools (`read`, `search`, `jsonl_search`, `list`, and `glob`). I did restructure the memory files into a JSONL-friendly corpus so its generic `jsonl_search` tool could work well; that seems acceptable here because AgentRunbook-C also creates a custom trajectory structure rather than using the vanilla Codex raw layout. I tried to keep the query-time system prompt as close as practical to AgentRunbook-C, changing the tool instructions only where ThinHarness needed to know how to use its built-in tools. + +## Results + +Across all 127 dynamic questions in the small tier: + +| Metric | AgentRunbook-C rerun | ThinHarness | +| --- | --- | --- | +| Dynamic score | 72.4% (92/127) | 74.0% (94/127) | +| Non-abstention | 86.0% (74/86) | 84.9% (73/86) | +| Abstention | 43.9% (18/41) | 51.2% (21/41) | +| Memory query time | 151.9s avg, 129.1s median | 99.7s avg, 87.9s median | +| Memory-agent tokens / usage | 114.77M input, 1.32M output, 116.09M total | 60.14M input, 2.10M output, 62.24M total | +| Dynamic-subset LAFS | 3.76 | 9.73 (+5.96) | + +The 72.4% accuracy for AgentRunbook-C matches the paper, but I would not treat this single consolidated run as a statistically signficant claim that ThinHarness has higher accuracy than AgentRunbook-C--I saw meaningful variance on a portion of the questions when doing targeted reruns. The result does make me reasonably confident that ThinHarness at least matches AgentRunbook-C's performance on this slice, and the published leaderboard reference for vanilla Codex is materially lower than both. + +The memory query time is the harness-measured time around `memory.query(...)`: it includes the query-time memory retrieval agent, but not the downstream reader, scorer, or prior runtime input generation. The timing comparison isn't perfect (local Codex CLI vs. OpenAI API), but the 46.4% lower token usage indicates that ~34% time savings is probably in the right ballpark. Note that the paper only provides a single aggregate query time figure across all questions, 108.3s, which is far lower than the 151.9s above but includes all questions in the small tier (some of which may have been faster). + diff --git a/scripts/build_site.py b/scripts/build_site.py index 21e70c2..bbf6f4c 100644 --- a/scripts/build_site.py +++ b/scripts/build_site.py @@ -46,10 +46,8 @@ def slug_for_opinion(title: str) -> str: known_tags = { "Purpose-built agents, not universal agents": "purpose_built", "No bash by default": "no_bash", - "Skills are tools, not auto-discovery": "skills", "Search is a top priority": "search", "Parallel LLM calls, built in": "parallel_llm", - "Background tools are simple": "background_tools", "Three providers, no matrix": "providers", "No compaction": "no_compaction", "No deployment layer": "no_deployment", @@ -98,22 +96,31 @@ def __init__(self) -> None: self._row: list[dict[str, str]] | None = None self._cell: dict[str, str] | None = None self._text: list[str] = [] + self._sup: list[str] = [] + self._in_sup = False def handle_starttag(self, tag: str, attrs: list[tuple[str, str | None]]) -> None: if tag == "tr": self._row = [] elif tag in {"td", "th"} and self._row is not None: - self._cell = {"text": "", "img": ""} + self._cell = {"text": "", "img": "", "sup": ""} self._text = [] + self._sup = [] + self._in_sup = False elif tag == "img" and self._cell is not None: attrs_dict = dict(attrs) self._cell["img"] = attrs_dict.get("src") or "" + elif tag == "sup" and self._cell is not None: + self._in_sup = True elif tag == "br" and self._cell is not None: self._text.append(" ") def handle_endtag(self, tag: str) -> None: - if tag in {"td", "th"} and self._cell is not None and self._row is not None: + if tag == "sup" and self._cell is not None: + self._in_sup = False + elif tag in {"td", "th"} and self._cell is not None and self._row is not None: self._cell["text"] = normalize_table_text("".join(self._text)) + self._cell["sup"] = normalize_table_text("".join(self._sup)) self._row.append(self._cell) self._cell = None elif tag == "tr" and self._row is not None: @@ -122,7 +129,11 @@ def handle_endtag(self, tag: str) -> None: self._row = None def handle_data(self, data: str) -> None: - if self._cell is not None: + if self._cell is None: + return + if self._in_sup: + self._sup.append(data) + else: self._text.append(data) @@ -149,12 +160,8 @@ def mark(value: str) -> str: def library_cell(cell: dict[str, str], asset_prefix: str = "assets/") -> str: name = cell["text"] - superscript = "" - match = re.search(r"(\d+)$", name) - if match and name not in {"3"}: - name = name[: -len(match.group(1))].strip() - superscript = f"{match.group(1)}" - display_name = html.escape(name).replace("Claude Agent SDK", "Claude Agent SDK").replace("OpenAI Agents SDK", "OpenAI Agents SDK") + superscript = f"{html.escape(cell['sup'])}" if cell.get("sup") else "" + display_name = html.escape(name) if name == "ThinHarness": return f'
ThinHarness
' if name == "Agno": @@ -169,12 +176,14 @@ def comparison_table(markdown: str, asset_prefix: str = "assets/") -> str: body_rows = [] for row in rows[1:]: library = row[0] + loc = row[1] + loc_sup = f'{html.escape(loc["sup"])}' if loc.get("sup") else "" css = ' class="me"' if library["text"] == "ThinHarness" else "" marks = "".join(f"{mark(cell['text'])}" for cell in row[2:]) body_rows.append( f""" {library_cell(library, asset_prefix)} - {html.escape(row[1]["text"])} + {html.escape(loc["text"])}{loc_sup} {marks} """ ) @@ -204,7 +213,6 @@ def render_about(markdown: str) -> str: use = section(markdown, "Use") use_code = code_highlight(fenced_code(use, "python")) use_paragraph = paragraphs(use.split("```", 2)[2].strip())[0] - tracing = paragraphs(section(markdown, "Tracing")) opinion_items = "\n".join( f'
{tag}

{html.escape(title)}

{body}

' @@ -219,7 +227,7 @@ def render_about(markdown: str) -> str: ) table_summary = ( "Table focuses on harness-level features that differentiate the libraries. All listed also support MCP, " - "lifecycle hooks, multi-turn conversations, and human-in-the-loop. It intentionally does not compare " + "lifecycle hooks, multi-turn conversations, structured output, and human-in-the-loop. It intentionally does not compare " "framework/platform features like vector DB integrations, hosted deployment, memory/session stores, or broad SaaS connectors." ) retry_footnote = ( @@ -274,7 +282,6 @@ def render_about(markdown: str) -> str: install use features - tracing status license @@ -302,7 +309,6 @@ def render_about(markdown: str) -> str: LOC1 Tool
retries2 Sub-
agents - Structured
output Skills FS
tools OTel
tracing @@ -318,7 +324,7 @@ def render_about(markdown: str) -> str:

1. LOC excludes anything that is not the core agent harness framework. See raw README source comments for exact commands.

2. {retry_footnote}

3. Claude Agent SDK shells out to the Claude Code CLI binary, which is 200k+ LOC.

-

4. deepagents is a thin wrapper over LangChain/LangGraph; effective import surface is ≈111k LOC.

+

4. deepagents is a thin wrapper over LangChain/LangGraph; effective import surface is ≈112k LOC.

See docs/table.md for per-cell rationale and how the LOC numbers are measured.

@@ -354,13 +360,6 @@ def render_about(markdown: str) -> str: -
-
// tracing
-

Tracing

-

{inline_markdown(tracing[0])}

-

{inline_markdown(tracing[1])}

-
-
// status

Status

diff --git a/scripts/build_transcripts.py b/scripts/build_transcripts.py index 3fea81c..547dbdf 100644 --- a/scripts/build_transcripts.py +++ b/scripts/build_transcripts.py @@ -1,6 +1,7 @@ from __future__ import annotations import argparse +import html import json import re from pathlib import Path @@ -8,6 +9,7 @@ REPO_ROOT = Path(__file__).resolve().parents[1] EXAMPLES_ROOT = REPO_ROOT / "examples" +LONGMEMEVAL_MD = EXAMPLES_ROOT / "longmemeval.md" DEFAULT_OUTPUT = REPO_ROOT / "docs" / "site" / "examples.html" LONG_PREVIEW_CHARS = 1200 WEB_RESEARCH_REPORT_META = { @@ -33,6 +35,72 @@ AGENT_META = {"web_research_report": WEB_RESEARCH_REPORT_META} +def md_inline(text: str) -> str: + """Render inline markdown (code, bold, links) to HTML, preserving links.""" + placeholders: list[str] = [] + + def hold(value: str) -> str: + placeholders.append(value) + return f"\0{len(placeholders) - 1}\0" + + text = re.sub(r"`([^`]+)`", lambda m: hold(f"{html.escape(m.group(1))}"), text) + text = re.sub( + r"\[([^\]]+)\]\(([^)]+)\)", + lambda m: hold(f'{html.escape(m.group(1))}'), + text, + ) + escaped = html.escape(text, quote=False).replace("\n", " ") + escaped = re.sub(r"\*\*([^*]+)\*\*", r"\1", escaped) + for index, value in enumerate(placeholders): + escaped = escaped.replace(f"\0{index}\0", value) + return escaped + + +def md_table(block: str) -> str: + """Render a GFM table block to an HTML table wrapped for horizontal scroll.""" + def cells(row: str) -> list[str]: + return [cell.strip() for cell in row.strip().strip("|").split("|")] + + def is_separator(row: str) -> bool: + return set(row.replace("|", "").replace("-", "").replace(":", "").strip()) <= {" ", ""} + + rows = [cells(line) for line in block.splitlines() if line.strip().startswith("|") and not is_separator(line)] + head, *body = rows + thead = "".join(f"{md_inline(cell)}" for cell in head) + tbody = "".join("" + "".join(f"{md_inline(cell)}" for cell in row) + "" for row in body) + return f'
{thead}{tbody}
' + + +def render_markdown(md: str) -> str: + """Render the LongMemEval excerpt (headings, paragraphs, one table) to HTML. + + Block-level only, matching the regex-based markdown approach used in build_site.py; the + leading h1 gets a site-style eyebrow so it reads like the rest of the docs pages. + """ + out: list[str] = [] + eyebrow_done = False + for block in re.split(r"\n[ \t]*\n", md.strip()): + block = block.strip() + if not block: + continue + if block.lstrip().startswith("|"): + out.append(md_table(block)) + elif block.startswith("### "): + out.append(f"

{md_inline(block[4:].strip())}

") + elif block.startswith("## "): + out.append(f"

{md_inline(block[3:].strip())}

") + elif block.startswith("# "): + title = md_inline(block[2:].strip()) + if eyebrow_done: + out.append(f"

{title}

") + else: + out.append(f'
// benchmark reproduction

{title}

') + eyebrow_done = True + else: + out.append(f"

{md_inline(block)}

") + return "\n".join(out) + + def parse_jsonish(value: Any) -> Any: if not isinstance(value, str) or not value.strip(): return value @@ -445,12 +513,18 @@ def load_agents() -> list[dict[str, Any]]: def render_html(agents: list[dict[str, Any]], *, template_path: Path | None = None) -> str: data = json.dumps({"agents": agents}, ensure_ascii=False) script_data = data.replace(")(.*?)()', re.S) - if pattern.search(template): - return pattern.sub(lambda match: f"{match.group(1)}{script_data}{match.group(3)}", template, count=1) - raise ValueError("examples template must contain )', re.S) + if not trace_pattern.search(template): + raise ValueError("examples template must contain