If the toolautosubmit attribute is missing on a form when the agent finishes filling out the form, the browser brings the submit button into focus, and the agent should then tell the user to check the form contents, and submit it manually.
This is currently not supported in evals-cli, so the browser would just be stuck.
We need to decide how we want to handle this. Some thoughts:
- Should evals-cli always add
toolautosubmit? This would go against the website's explicit configuration, so probably no.
- Should we automatically hook into
toolactivated events with Puppeteer to call form.requestSubmit() when needed? Probably not, because we evaluate whether the tool gets called, which in this case it does.
- Or is this something we should at least log to the console?
- Should we make this configurable in
evals.json somehow?
Related:
If the
toolautosubmitattribute is missing on a form when the agent finishes filling out the form, the browser brings the submit button into focus, and the agent should then tell the user to check the form contents, and submit it manually.This is currently not supported in evals-cli, so the browser would just be stuck.
We need to decide how we want to handle this. Some thoughts:
toolautosubmit? This would go against the website's explicit configuration, so probably no.toolactivatedevents with Puppeteer to callform.requestSubmit()when needed? Probably not, because we evaluate whether the tool gets called, which in this case it does.evals.jsonsomehow?Related:
getCrossDocumentScriptToolResult#53