Skip to content

harness: orchestrator engagement is unreliable — model skips the firmware Invariant (run iq fsm state first) in ~29% of runs #264

Description

@ccisnedev

Evidence

7 identical conducted trials (OpenCode, agent=inquiry, qwen3-coder:30b-16k). The firmware was loaded in all 7 (confirmed by two independent means: agent=inquiry recorded + identical first_turn_input=7254 tokens + no fallback warning; and an iq shim logging every real invocation). Yet the model ran the mandatory first action (iq fsm state --json) in only 5/7 runs:

Trial real iq calls engaged
1 0 no (freestyled: read+edited the file directly)
2 18 yes
3 0 no (text only, zero tools)
4 11 yes
5 3 yes
6 11 yes
7 27 yes

So ~29% of the time the model ignores the firmware's "Invariant" (run iq fsm state before responding) despite having the firmware in context, and instead freestyles or just talks. This is model-behavior variance, not a load failure (ruled out by the evidence above).

Problem

The firmware's Invariant is not reliably obeyed by a weak local model. For a valid H-vs-F experiment (and for the harness to be trustworthy on local models), engagement must be near-deterministic, or non-engaged runs must be detectable and excluded.

Direction (to design)

  • Strengthen the firmware's opening so the first iq fsm state --json is near-unconditional (more imperative / fewer escape hatches).
  • Consider a host-side kick: the conductor/runner issues the first iq fsm state (or a wrapper enforces it) so engagement doesn't depend on the model choosing to.
  • Add a cheap detector (the iq call log / DB) to flag non-engaged runs so experiments don't count them.

Found via the OpenCode local-model engagement experiments (#260 follow-up).

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions