Benchmark a typed interpreter-call protocol against the current code-block loop

[shanev]

Problem

The current loop combines several protocol decisions:

• extract generated Python from model text;
• execute it and feed output back into refinement;
• detect completion through the answer["content"] / answer["ready"] convention;
• repair malformed answers and use a bounded extract fallback after exhaustion.

Each mechanism is individually useful, but together they create formatting and recovery surface area. A narrower protocol in which the root model emits a typed interpreter call and later returns a typed final result may remove glue without changing Droste's core method: the model still writes programs over data and delegates bounded semantic work.

This issue is an evidence-gathering experiment, not authorization for a rewrite.

Experiment

Add a benchmark arm that holds model, task, data, prompt content, and compute budget constant while varying only the execution protocol:

1. Current protocol: generated code block + refinement loop + answer dictionary + repair/extract fallback.
2. Typed interpreter call: a structured python.eval request with typed execution results, followed by either another interpreter request or a typed final answer.
3. Optional hybrid: current code generation with a typed completion/result envelope, to isolate code extraction from answer signaling.

The interpreter remains behind the Capability/Broker ABI (#9). This experiment must not introduce a second sandbox or bypass policy, budget, and trace enforcement.

Measurements

Record per task and in aggregate:

• answer accuracy and task completion;
• malformed model-output rate;
• code-extraction and tool-argument parse failures;
• repair attempts, extract fallbacks, and unrecovered failures;
• iterations and capability/subcall counts;
• root and subcall tokens separately;
• latency and cost;
• terminal status correctness, including successful output that begins with ERROR: (Trajectory records need structured execution status #58).

Include factual, semantic, aggregation, and adversarial protocol fixtures. Preserve failed and timed-out attempts as typed benchmark artifacts rather than dropping them.

Decision rule

• If the typed arm matches or improves quality while materially reducing repair/fallback frequency or protocol code, open a separate design issue for the smallest migration.
• If it does not, retain the current protocol and fold any isolated improvements into prompt packs or result typing.
• Do not change the public run/trace contract until the benchmark report identifies which behavior is load-bearing.

Deliverables

• Reproducible benchmark manifest and immutable per-task artifacts.
• A report comparing the protocol arms with confidence intervals across repeated runs.
• A short inventory of engine code that each arm makes necessary or removable.
• A recommendation that explicitly separates measured results from implementation preference.

Related

• Prompt packs + RLM skills: move all strategy out of the engine into versioned data #3 — prompt packs and versioned prompt data
• Brokered Capability ABI + ledger: sole sandbox egress for data, llm_query*, and host tools #9 — Capability/Broker ABI
• Unified Trace ABI: one versioned run record for trajectory, events, usage, policy, capability calls #13 — unified run/trace record
• Trajectory records need structured execution status #58 — structured execution status

[shanev]

Separate protocol choice from interpreter-language choice

The current two-arm design could incorrectly attribute a gain from the typed eval contract to JavaScript, or a language effect to the contract. Use a factorial comparison if budget permits:

1. current fenced Python protocol;
2. typed python.eval with the same Python runtime;
3. typed javascript.eval on QuickJS with the same capability descriptors and terminal-result schema.

This makes the first decision “typed eval or current protocol?” and the second, independent decision “which interpreter runtime?”

Add runtime-specific fixtures and measurements:

• startup time and peak/resident memory;
• per-eval memory, stack, and wall-time enforcement;
• cancellation during bridged calls;
• sequential and concurrent capability fan-out;
• persistence across evals and snapshot/restore cost;
• exact integers above 2^53, decimals/money, Unicode, regex, dates, and large nested results;
• model syntax/repair failure rate for Python versus JavaScript.

Do not open a QuickJS implementation track from ecosystem preference alone. The benchmark should identify whether a second substrate earns its maintenance and compatibility cost.