Evidence
7 identical conducted trials (OpenCode, agent=inquiry, qwen3-coder:30b-16k). The firmware was loaded in all 7 (confirmed by two independent means: agent=inquiry recorded + identical first_turn_input=7254 tokens + no fallback warning; and an iq shim logging every real invocation). Yet the model ran the mandatory first action (iq fsm state --json) in only 5/7 runs:
| Trial |
real iq calls |
engaged |
| 1 |
0 |
no (freestyled: read+edited the file directly) |
| 2 |
18 |
yes |
| 3 |
0 |
no (text only, zero tools) |
| 4 |
11 |
yes |
| 5 |
3 |
yes |
| 6 |
11 |
yes |
| 7 |
27 |
yes |
So ~29% of the time the model ignores the firmware's "Invariant" (run iq fsm state before responding) despite having the firmware in context, and instead freestyles or just talks. This is model-behavior variance, not a load failure (ruled out by the evidence above).
Problem
The firmware's Invariant is not reliably obeyed by a weak local model. For a valid H-vs-F experiment (and for the harness to be trustworthy on local models), engagement must be near-deterministic, or non-engaged runs must be detectable and excluded.
Direction (to design)
- Strengthen the firmware's opening so the first
iq fsm state --json is near-unconditional (more imperative / fewer escape hatches).
- Consider a host-side kick: the conductor/runner issues the first
iq fsm state (or a wrapper enforces it) so engagement doesn't depend on the model choosing to.
- Add a cheap detector (the
iq call log / DB) to flag non-engaged runs so experiments don't count them.
Found via the OpenCode local-model engagement experiments (#260 follow-up).
Evidence
7 identical conducted trials (OpenCode,
agent=inquiry,qwen3-coder:30b-16k). The firmware was loaded in all 7 (confirmed by two independent means:agent=inquiryrecorded + identicalfirst_turn_input=7254tokens + no fallback warning; and aniqshim logging every real invocation). Yet the model ran the mandatory first action (iq fsm state --json) in only 5/7 runs:So ~29% of the time the model ignores the firmware's "Invariant" (run
iq fsm statebefore responding) despite having the firmware in context, and instead freestyles or just talks. This is model-behavior variance, not a load failure (ruled out by the evidence above).Problem
The firmware's Invariant is not reliably obeyed by a weak local model. For a valid H-vs-F experiment (and for the harness to be trustworthy on local models), engagement must be near-deterministic, or non-engaged runs must be detectable and excluded.
Direction (to design)
iq fsm state --jsonis near-unconditional (more imperative / fewer escape hatches).iq fsm state(or a wrapper enforces it) so engagement doesn't depend on the model choosing to.iqcall log / DB) to flag non-engaged runs so experiments don't count them.Found via the OpenCode local-model engagement experiments (#260 follow-up).