Add evals report analyzer

[maudnals]

As discussed with team (@Kulikowski , @swissspidy): add a feature to help developers make sense of the evals reports, by summarizing and analyze the results (single report, or compare multiple across subsequent evals runs):

• As a parameter for a given evals run
• As a standalone command to analyze 1-n evals report(s) by specifying the file path

[maudnals]

Hey @Kulikowski @swissspidy, let's exchange ideas on how to design of this feature! I'd suggest we first align on the standalone command for a single report and the kinds of features we expect there, once we're good on this the rest should derive from it. Your feedback is most welcome.

1. Command could be webmcp-evals analyze <report-path> [options] with option -m, --model similar to what we have now. (Maybe more options later, see my point later below on Context as maybe this is something we'll want to configure too). Alternatively webmcp-evals summarize <report-path> [options].
2. To help the analyzer not just summarize but provide an analysis, we'll need to give it the evals report, but also details on the target web app the evals were run against, and knowledge about the WebMCP spec, for example what the API is, how to register tools (declarative / imperative), expectation on schema etc. How much exactly the analyzer will need to know in order to do a good job, we'll find out when trying it out :) I fiddled with my coding agent who provided good analyses, but it has access to the full webmcp-tools repo which includes lots of info.
3. Re: Context: eval reports are small but web application data can be very large, and WebMCP reference info will take some tokens too, so we'll need to think a bit how we optimize the context window so the analyzer stays fast. We could constraint the analyzer to not take in HTML report but only JSON (lighter) and also apply best practices to do context caching, not sure we'll need to do implicit or explicit cachine. Not sure yet also how that plays out with different models. Overall maybe this is something we want to take care of later as an optimization, but I think we do we want to be mindful of our dev users not overspending their tokens. (This all is making me think, maybe an alternative path to this feature is provide an analyzer as skill and people use their coding agent as an analyzer? Hmm)
4. When analyzing failures the analyzer could look at: model failure (tool hallucination, bad params), app/tool API issue (confusing description), what else?
5. I think for now we want to frame the analyses as really just hypotheses, to not put developers on a wrong debugging path.
6. Here's what an example analysis could look like. WDYT?

---

WebMCP Evals Analysis — Jul 20, 2026, 10:24 AM

1. Summary

75% pass overall.
All 3 failures in this run occurred in Test Case #2 (Sport Shop: Find Kid Baseball Gift Under $50), caused by trajectory/step-count mismatches due to self-correction paths and rules checks. The overall pass rate is 75.0% (3/4 runs passed), indicating a primary issue category of Test/Assertion Over-Rigidity.

2. Failed Trajectories & Deep Dives

Test Case: Sport Shop: Find Kid Baseball Gift Under $50 (Run #1)

• User Intent: "I'm looking for a gift for my kid. He likes baseball. Apply filter for 50$ if available"
• Failure Step: Step Bump vite from 7.1.2 to 7.3.1 in /demos/react-flightsearch #5/6 (Expected end of trajectory, but model called refine_search again)
• Behavior Analysis: The model initially filtered by 50-99.99 (getting a $52.99 glove). Upon inspecting the result, the model realized the item was over $50, self-corrected by applying 0-49.99, and fetched the results again.

3. Root Cause Hypotheses

• Hypothesis 1 (Assertion Rigidity - High Confidence): The test case expects a strict 3-step sequence. The model took 6 steps due to a self-correction loop. The model successfully found the correct items, but was penalized because the test case does not allow self-correction paths.
• Hypothesis 2 (Prompt Ambiguity - Medium Confidence): The user instruction "Apply filter for 50$" was interpreted as "around $50", leading the model to first try the $50-99.99 filter range before realizing the budget constraint.

[Kulikowski]

Re: Context: eval reports are small but web application data can be very large, and WebMCP reference info will take some tokens too, so we'll need to think a bit how we optimize the context window so the analyzer stays fast. We could constraint the analyzer to not take in HTML report but only JSON (lighter) and also apply best practices to do context caching, not sure we'll need to do implicit or explicit cachine. Not sure yet also how that plays out with different models. Overall maybe this is something we want to take care of later as an optimization, but I think we do we want to be mindful of our dev users not overspending their tokens. (This all is making me think, maybe an alternative path to this feature is provide an analyzer as skill and people use their coding agent as an analyzer? Hmm)

I would add a note that if you are analyzing reports with this commands you might want to switch to json reports.

TL;DR I would prefer to implement basic version (with guidance on how to best use it) and then iterate and optimize as we go.

[Kulikowski]

SKILL for analyzing reports that can be used by user preferred agent would be very cool!

We could then have interesting case to compare - popular harnesses with SKILL vs analyze command

[maudnals]

• Re: context management: Basic version and optimize later SGTM. +1 on your point on JSON reports.
• Re: skills vs command: ultimately we may want both. How about we first implement it as a command (or param) and then later as a skill? What we learn building the simple command informs how we design the skill.

[Kulikowski]

SGTM!