What is currently possible is we can feed conversation history to the model to simulate multi-turn evaluation. While it helps agent understand the context it does not mean that the website is in correct state to perform evaluation. While websites might have an options for deep links there might be use cases where it's not supported or not easily possible to simulate certain app state.
What is requested is to add support for multi-turn evals that would, within single eval file, alternate between user message and WebMCP expected tool call to validate full trajectory.
Simplified example:
message: I want a large pizza
tool: set_pizza_size(LARGE)
message: Ok, now make it BBQ style pizza
tool: set_pizza_style(BBQ)
...
What is currently possible is we can feed conversation history to the model to simulate multi-turn evaluation. While it helps agent understand the context it does not mean that the website is in correct state to perform evaluation. While websites might have an options for deep links there might be use cases where it's not supported or not easily possible to simulate certain app state.
What is requested is to add support for multi-turn evals that would, within single eval file, alternate between user message and WebMCP expected tool call to validate full trajectory.
Simplified example:
message: I want a large pizza
tool: set_pizza_size(LARGE)
message: Ok, now make it BBQ style pizza
tool: set_pizza_style(BBQ)
...