Background
Users evaluating MCP-dependent Skills may need the same MCP server to return different mocked data for different cases. For example, multiple cases can exercise the same project-mgmt MCP tools, but each case needs a different project state, search result, or error response.
Today, mocked MCP responses are configured at the eval level:
mcp:
servers:
- name: project-mgmt
mode: mocked
config_ref: evals/fixtures/mcp/project-mgmt.yaml
The referenced file contains global tool_responses, so every case in the eval sees the same mocked outputs for that server. From a quick code read, CaseConfig has no MCP override field, and defaultEvaluator.provisionMCPConfig() resolves evalCfg.MCP once and reuses it for all cases.
User Need
Allow different cases in the same eval to use the same MCP server name and tool names, but with case-specific mocked responses.
This is reasonable because:
- MCP-backed Skills often branch on tool results, so fixture variation is essential for meaningful coverage.
- Duplicating eval files just to vary mock data is noisy and makes benchmark/report comparison harder.
- Keeping the server name stable across cases better matches how the Skill is expected to behave in production.
Possible Design
Add a case-level MCP override or mock fixture override, for example:
# evals/cases/project-open.yaml
id: project-open
input:
prompt: Check project status
mcp:
servers:
- name: project-mgmt
mode: mocked
config_ref: evals/fixtures/mcp/project-open.yaml
Or a narrower mock-only shape:
mcp_overrides:
project-mgmt:
config_ref: evals/fixtures/mcp/project-open.yaml
Expected merge behavior should be explicit:
- Case-level config can override an eval-level server by
name.
- If a case does not specify an override, it inherits the eval-level MCP config unchanged.
- For mocked servers, the case-specific
config_ref should be resolved relative to the Skill directory, consistent with existing MCP config refs.
- The same behavior should work with parallel case execution and should not leak mock responses across cases.
Acceptance Criteria
CaseConfig supports a documented way to override mocked MCP responses per case.
- The evaluator provisions/installs the effective MCP config for each case instead of forcing one globally cached mocked response set.
- Existing eval-level MCP behavior remains backward compatible.
- Unit tests cover merge/override semantics and config_ref resolution.
- E2E or integration coverage demonstrates two cases using the same mocked MCP server/tool with different returned values.
docs/guide/writing-evals.md, docs/zh/guide/writing-evals.md, and skill template/reference docs are updated with an example.
Background
Users evaluating MCP-dependent Skills may need the same MCP server to return different mocked data for different cases. For example, multiple cases can exercise the same
project-mgmtMCP tools, but each case needs a different project state, search result, or error response.Today, mocked MCP responses are configured at the eval level:
The referenced file contains global
tool_responses, so every case in the eval sees the same mocked outputs for that server. From a quick code read,CaseConfighas no MCP override field, anddefaultEvaluator.provisionMCPConfig()resolvesevalCfg.MCPonce and reuses it for all cases.User Need
Allow different cases in the same eval to use the same MCP server name and tool names, but with case-specific mocked responses.
This is reasonable because:
Possible Design
Add a case-level MCP override or mock fixture override, for example:
Or a narrower mock-only shape:
Expected merge behavior should be explicit:
name.config_refshould be resolved relative to the Skill directory, consistent with existing MCP config refs.Acceptance Criteria
CaseConfigsupports a documented way to override mocked MCP responses per case.docs/guide/writing-evals.md,docs/zh/guide/writing-evals.md, and skill template/reference docs are updated with an example.