From f13a7bb20cf41e0f41f1e3982579b5c8ee7a99ac Mon Sep 17 00:00:00 2001 From: Robert Lippmann Date: Thu, 25 Jun 2026 00:46:47 -0400 Subject: [PATCH 1/2] chore: remove stale eval artifacts --- evals/litellm_proxy_additional_findings.md | 89 ------------- evals/litellm_proxy_behavioral_comparisons.md | 120 ------------------ 2 files changed, 209 deletions(-) delete mode 100644 evals/litellm_proxy_additional_findings.md delete mode 100644 evals/litellm_proxy_behavioral_comparisons.md diff --git a/evals/litellm_proxy_additional_findings.md b/evals/litellm_proxy_additional_findings.md deleted file mode 100644 index 9306d57..0000000 --- a/evals/litellm_proxy_additional_findings.md +++ /dev/null @@ -1,89 +0,0 @@ -# LiteLLM Proxy Additional Findings - -Model: `ollama/qwen2.5:14b-instruct` - -- Limitations/caveats: - - Confirm follow-up (`yes`) does not resolve the prior confirm in current replay-only proxy flow. - - Last-turn-only preprocessing can fail to persist earlier canonicalization effects across subsequent replay. -- Additional LiteLLM-surface behavior: - - Structured mixed-content user payloads can trigger upstream LiteLLM/Ollama message-shape validation errors. - - Structured text-part near-miss inputs still show a meaningful preprocessor lifecycle win over basic proxy. - -## Finding 1 — confirm follow-up loops (replay limitation) - -**Prompt sequence** -1. `clear state` -2. `use podman instead of docker` -3. `yes, keep existing policies and use podman` - -**Vanilla** -- Step 2/3: generic Podman migration/help text. - -**Basic proxy** -- Step 2: confirm clarify (`No exact policy found for "docker" ... Confirm to use "podman" ...`). -- Step 3: same confirm clarify repeats. - -**Preprocessor proxy** -- Step 2: same confirm clarify. -- Step 3: same confirm clarify repeats. - -**Why it matters** -Current replay-based proxy behavior does not treat natural-language “yes” as explicit confirm resolution, so this can loop until user supplies an explicit directive path. - -## Finding 2 — last-turn-only preprocessing is non-persistent across replay (replay limitation) - -**Prompt sequence** -1. `clear state` -2. `set premise to concise replies` -3. `Explain TCP in detail.` - -**Vanilla** -- Conversationally accepts premise-like instruction, then gives normal long-form answer. - -**Basic proxy** -- Step 2: syntax clarify (`Did you mean 'set premise concise replies'?`). -- Step 3: same syntax clarify repeats. - -**Preprocessor proxy** -- Step 2: canonicalized update (`Premise set to concise replies ...`). -- Step 3: syntax clarify reappears (`Did you mean 'set premise concise replies'?`). - -**Why it matters** -Only the latest replay turn is preprocessed; earlier raw near-miss text in transcript can still drive later replay outcomes. - -## Finding 3 — structured mixed content can fail upstream validation (LiteLLM-surface caveat) - -**Prompt sequence** -1. `clear state` -2. user content parts: text (`set premise to concise replies`) + non-text (`input_image`) -3. `What is TCP?` - -**Vanilla** -- Upstream request fails with invalid user message shape error. - -**Basic proxy** -- Blocks at compiler clarify before upstream model call. - -**Preprocessor proxy** -- Step 2 hits upstream validation error path; later turn can return clarify. - -**Why it matters** -In proxy mode, forwarded request messages remain unchanged; LiteLLM/Ollama payload validation behavior can dominate outcomes for mixed content shapes. - -## Finding 4 — structured text-part near-miss still yields stronger lifecycle result (LiteLLM-surface win) - -**Prompt sequence** -1. `clear state` -2. user content text parts: `change premise` + `concise replies` - -**Vanilla** -- Conversational acceptance of style change. - -**Basic proxy** -- Syntax clarify only (`Did you mean 'change premise to concise replies'?`). - -**Preprocessor proxy** -- Lifecycle clarify (`No premise exists yet. Use 'set premise ...' first.`). - -**Why it matters** -For structured text-part inputs, preprocessor canonicalization can move past syntax-only clarify and reach the stronger lifecycle-semantic outcome. diff --git a/evals/litellm_proxy_behavioral_comparisons.md b/evals/litellm_proxy_behavioral_comparisons.md deleted file mode 100644 index 7891bdd..0000000 --- a/evals/litellm_proxy_behavioral_comparisons.md +++ /dev/null @@ -1,120 +0,0 @@ -# LiteLLM Behavioral Comparison (5 documented directive/state cases) - -Model: `ollama/qwen2.5:14b-instruct` - -- Vanilla LiteLLM exhibits conversational drift on stateful directives. -- Basic proxy enforces deterministic state semantics. -- Preprocessor proxy additionally improves recall on near-miss inputs. - -## Case 1 — premise lifecycle enforcement - -**Prompt sequence** -1. `clear state` -2. `change premise to formal tone` - -**Vanilla** -1. `State cleared. How can I assist you now?` -2. Generic rewriting guidance (asks for text/context to formalize). - -**Basic proxy** -1. `State cleared. How can I assist you now?` -2. `No premise exists yet. Use 'set premise ...' first.` - -**Preprocessor proxy** -1. `State cleared. How can I assist you now?` -2. `No premise exists yet. Use 'set premise ...' first.` - -**Result / why it matters** -Both proxy modes enforce the premise lifecycle deterministically; vanilla drifts into conversational rewrite assistance. - -## Case 2 — conflict enforcement - -**Prompt sequence** -1. `clear state` -2. `use docker` -3. `prohibit docker` - -**Vanilla** -1. `State cleared. How can I assist you now?` -2. Generic Docker tutorial. -3. Generic "stop/uninstall Docker" guidance. - -**Basic proxy** -1. `State cleared. How can I assist you now?` -2. Generic Docker tutorial. -3. `'docker' is already in use. Only one policy per item is allowed. Use 'reset policies' to change it.` - -**Preprocessor proxy** -1. `State cleared. How can I assist you now?` -2. Generic Docker tutorial. -3. `'docker' is already in use. Only one policy per item is allowed. Use 'reset policies' to change it.` - -**Result / why it matters** -Both proxy modes preserve explicit conflict semantics instead of letting the model conversationally reinterpret contradictory instructions. - -## Case 3 — replacement precondition enforcement - -**Prompt sequence** -1. `clear state` -2. `use podman instead of docker` - -**Vanilla** -1. `State cleared. How can I assist you now?` -2. Generic Podman migration tutorial. - -**Basic proxy** -1. `State cleared. How can I assist you now?` -2. `No exact policy found for "docker". Replacement requires an exact policy match. Confirm to use "podman" and keep existing policies?` - -**Preprocessor proxy** -1. `State cleared. How can I assist you now?` -2. `No exact policy found for "docker". Replacement requires an exact policy match. Confirm to use "podman" and keep existing policies?` - -**Result / why it matters** -Both proxy modes enforce the replacement precondition (old item must exist), while vanilla skips that state-dependent requirement. - -## Case 4 — near-miss canonicalization + premise slot semantics - -**Prompt sequence** -1. `clear state` -2. `set premise to concise replies` -3. `set premise formal tone` - -**Vanilla** -1. `State cleared. How can I assist you now?` -2. Accepts near-miss conversationally ("premise set to concise replies"). -3. Accepts second set conversationally ("premise updated to formal tone"). - -**Basic proxy** -1. `State cleared. How can I assist you now?` -2. `Did you mean 'set premise concise replies'?` -3. `Did you mean 'set premise concise replies'?` - -**Preprocessor proxy** -1. `State cleared. How can I assist you now?` -2. `Premise set to concise replies. What do you need help with?` -3. `Did you mean 'set premise concise replies'?` - -**Result / why it matters** -Both proxy modes beat vanilla on deterministic grammar handling, but the preprocessor proxy goes further: it canonicalizes the near-miss at step 2 and reaches a real state update, while basic proxy stops at syntax clarify. - -## Case 5 — near-miss upgrade to stronger lifecycle clarify - -**Prompt sequence** -1. `clear state` -2. `change premise concise replies` - -**Vanilla** -1. `State cleared. How can I assist you now?` -2. Conversational acceptance ("I will aim for more concise replies"). - -**Basic proxy** -1. `State cleared. How can I assist you now?` -2. `Did you mean 'change premise to concise replies'?` - -**Preprocessor proxy** -1. `State cleared. How can I assist you now?` -2. `No premise exists yet. Use 'set premise ...' first.` - -**Result / why it matters** -This is the clearest preprocessor-specific win: basic proxy stops at syntax clarify, while preprocessor canonicalizes and reaches the stronger lifecycle clarify outcome. From 5a46137137e54a1e4e02a92c3e86361ee5011b10 Mon Sep 17 00:00:00 2001 From: Robert Lippmann Date: Thu, 25 Jun 2026 00:48:43 -0400 Subject: [PATCH 2/2] docs: align agent guidance with core scope --- AGENTS.md | 10 ++++++---- 1 file changed, 6 insertions(+), 4 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index 3e78769..0e94850 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -27,8 +27,10 @@ User-facing behavior includes: * engine decision outcomes (`kind`, `prompt_to_user`) * checkpoint export/import and continuation behavior * clarify/confirmation flows (`yes` / `no`) -* integration behavior (LiteLLM, OpenWebUI) -* integration error-path normalization +* controller behavior (`step`, `preview`, `state_diff`) +* REPL / CLI behavior +* authoritative state semantics +* conformance fixture behavior If a user-facing behavior is changed or introduced, add or update tests to cover it. @@ -100,7 +102,7 @@ If implementation behavior does not match the specification, report the mismatch Documentation is not commentary. -README examples, demos, integration examples, migration guides, CLI usage +README examples, demos, migration guides, CLI usage documentation, and explicitly requested documentation changes are part of the project contract. @@ -124,7 +126,7 @@ specifications disagree: ## Documentation style -For README, demo, integration, and package-listing docs, explain user-visible behavior before architecture. +For README, demo, and package-listing docs, explain user-visible behavior before architecture. Prefer plain, concrete wording when accurate. Examples: - "rules and corrections that stick"