Skip to content

docs: real LLM transcript replaces the simulation (#106 step 8)#116

Merged
hfoffani merged 1 commit into
mainfrom
doc-real-106
Jul 13, 2026
Merged

docs: real LLM transcript replaces the simulation (#106 step 8)#116
hfoffani merged 1 commit into
mainfrom
doc-real-106

Conversation

@hfoffani

Copy link
Copy Markdown
Owner

Final step of #106 (the harness merged in #113): docs/llm-interaction.md now tells the story with a real captured conversation instead of a scripted one.

The transcript (qwen/qwen-2.5-72b-instruct, 2026-07-13)

Better than the simulation we wrote by hand, because the failure→fix loop actually happened:

  1. Model translates the courier problem into clean PDDL (two write_file calls)
  2. validate passes — but solve rejects the domain: (:types ...) used without :typing, caught by requirements enforcement (Follow the level of expressiveness specified in :requirements #9)
  3. The model reads the structured error, fixes the one line, re-validates, re-solves
  4. ucs returns the provable optimum (cost 7.0, three-hop back road) and the model narrates it correctly

Honesty rules kept

  • Tool outputs verbatim; the model's PDDL re-indented for readability, nothing else edited
  • The capture box states model, date, harness, and that an earlier round with the same model was unsound (invented cost syntax) — the thesis in miniature
  • No fabricated dialogue remains anywhere in the doc; the "LLMs get this wrong" claim now leans on the references + the recorded unsound round instead of a scripted quote
  • The winning transcript is committed (force-added; the directory is otherwise gitignored) as provenance, linked from the doc

Unchanged

examples/pddl/courier-*.pddl stay as the curated reference (still pinned by tests/test_llm_doc.py, passing), and the References section (LLM+P, PlanBench) stays.

🤖 Generated with Claude Code

Step 8 of #106: docs/llm-interaction.md now walks through the captured
qwen/qwen-2.5-72b-instruct conversation (2026-07-13, e2e harness) —
the model writes PDDL, validate passes, solve REJECTS it (missing
:typing caught by requirements enforcement #9), the model fixes one
line, re-runs, and narrates the ucs optimum (cost 7). Tool outputs
verbatim; PDDL re-indented only. The 'what is simulated' box becomes
'how this was captured', noting the earlier unsound round as evidence
of the thesis. Winning transcript force-added (dir is gitignored) as
provenance.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@hfoffani
hfoffani merged commit 22225d5 into main Jul 13, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant