Conversation
e2e/llm/harness.py drives a real model (OpenRouter) against the pddlpy MCP server served from the PUBLISHED package (uvx --from pddlpy[mcp]): the model gets the courier problem in NL plus write_file + the MCP tools, and the run is judged sound iff validate was used, solve returned the 7-toll optimum, and the final answer narrates it. Each run captures a markdown transcript (gitignored; the winning one gets committed with the docs/llm-interaction.md replacement). --dry-run replays the doc's scripted turns through the full pipeline offline — verified green. claude_driver.sh covers mid-tier models via claude -p with an MCP config. make e2e-llm MODEL=... as the manual entry point; deliberately not in CI (non-deterministic). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…nscript The previous commit appended the transcripts ignore pattern to a .gitignore lacking a trailing newline, fusing it with the '.claude/' line — which un-ignored both, letting 'git add -A' stage the agent worktree gitlinks and the dry-run transcript. Restore the two rules and drop the three paths from the index. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Round 1 exposed two weak-model failure modes: double-escaped newlines in JSON tool arguments (PDDL written as one line of literal \n -> ANTLR token errors) and invented cost syntax (:cost fields, fantasy function blocks) that burned all turns in the validate loop. write_file now decodes literal \n/\t when the content has no real newlines, the system prompt gains a 4-line action-costs syntax reminder (the moral equivalent of the agent skill), and max turns go 12 -> 20. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This was referenced Jul 13, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #106 (the harness half — the doc replacement lands after we run real rounds).
What
e2e/llm/— a manual, non-CI harness that reproducesdocs/llm-interaction.mdwith a real model, per the issue's steps 1–7:uvx --from "pddlpy[mcp]" pddlpy-mcp— the published package in a throwaway environment, nothing from the checkout.harness.py(OpenRouter, for small/open models): hands the model the courier problem in NL plus tools — a localwrite_fileand the pddlpy MCP tools bridged to OpenRouter function-calling. Captures a full markdown transcript per run (gitignored; the winning one gets committed with the doc update).validatewas actually used,solvereturned the 7-toll optimum, and the final answer narrates cost + route. Exit 0/1/2.claude_driver.sh(mid-tier models): drives the same MCP server throughclaude -p --mcp-config— zero custom harness, as agreed.codex execworks analogously.make e2e-llm MODEL=...as the entry point;--dry-runreplays the doc's scripted turns through the entire pipeline (MCP server, dispatch, transcript, soundness) with no API key — verified green locally.Notes
OPENROUTER_API_KEY, never in CI..gitignorebug (pattern appended to a file with no trailing newline fused with the.claude/rule) and untracks accidentally staged paths.🤖 Generated with Claude Code