diff --git a/e2e/llm/README.md b/e2e/llm/README.md index be7b92c..1a4d583 100644 --- a/e2e/llm/README.md +++ b/e2e/llm/README.md @@ -36,6 +36,17 @@ same MCP server through the agent's own harness: The driver prints where it saved the transcript; judge soundness with the same criteria (validate used, cost 7.0, route narrated). +## Observed results + +| Model | Rounds | Outcome | +|---|---|---| +| `qwen/qwen-2.5-72b-instruct` | 2 | R1 **unsound** (double-escaped newlines, invented `:cost` syntax → harness hardened). R2 **sound**: wrote clean PDDL, recovered from a real `solve` rejection (missing `:typing`), narrated the optimum — now the transcript in `docs/llm-interaction.md`. | +| `meta-llama/llama-3.1-8b-instruct` | 3 | All **unsound**, each differently: mangled the map + drifted into pseudo-code instead of tool calls; emitted malformed JSON tool arguments (now answered with an error result instead of crashing); fired 7 tool calls in one batch without reading any result. | + +The gap is the point: the 72B model *uses* the fix loop, the 8B model can't +drive it — and the soundness gate correctly refused every unsound round. +Below ~8B, the missing skill isn't PDDL, it's sustained tool use. + ## Iterating (issue steps 5–8) Repeat runs (vary model / round) until a transcript is sound *and reads diff --git a/e2e/llm/harness.py b/e2e/llm/harness.py index 5b0de4e..8e58d97 100644 --- a/e2e/llm/harness.py +++ b/e2e/llm/harness.py @@ -248,9 +248,17 @@ async def main() -> int: break for tc in calls: fn = tc["function"] - fargs = json.loads(fn["arguments"]) - result = await run_tool(session, workdir, fn["name"], fargs) - convo.tool_results.append((fn["name"], fargs, result)) + # Small models emit malformed JSON arguments; answer + # with an error result instead of crashing the run. + try: + fargs = json.loads(fn["arguments"]) + result = await run_tool(session, workdir, + fn["name"], fargs) + convo.tool_results.append((fn["name"], fargs, result)) + except (json.JSONDecodeError, KeyError, TypeError) as exc: + result = ("Error: tool call arguments were not valid " + "JSON (%s). Repeat the call with valid, " + "complete JSON arguments." % exc) convo.messages.append({"role": "tool", "tool_call_id": tc["id"], "content": result})