Skip to content

feat(mcp): support per-case mocked MCP responses #133

Description

@roark47

Background

Users evaluating MCP-dependent Skills may need the same MCP server to return different mocked data for different cases. For example, multiple cases can exercise the same project-mgmt MCP tools, but each case needs a different project state, search result, or error response.

Today, mocked MCP responses are configured at the eval level:

mcp:
  servers:
    - name: project-mgmt
      mode: mocked
      config_ref: evals/fixtures/mcp/project-mgmt.yaml

The referenced file contains global tool_responses, so every case in the eval sees the same mocked outputs for that server. From a quick code read, CaseConfig has no MCP override field, and defaultEvaluator.provisionMCPConfig() resolves evalCfg.MCP once and reuses it for all cases.

User Need

Allow different cases in the same eval to use the same MCP server name and tool names, but with case-specific mocked responses.

This is reasonable because:

  • MCP-backed Skills often branch on tool results, so fixture variation is essential for meaningful coverage.
  • Duplicating eval files just to vary mock data is noisy and makes benchmark/report comparison harder.
  • Keeping the server name stable across cases better matches how the Skill is expected to behave in production.

Possible Design

Add a case-level MCP override or mock fixture override, for example:

# evals/cases/project-open.yaml
id: project-open
input:
  prompt: Check project status
mcp:
  servers:
    - name: project-mgmt
      mode: mocked
      config_ref: evals/fixtures/mcp/project-open.yaml

Or a narrower mock-only shape:

mcp_overrides:
  project-mgmt:
    config_ref: evals/fixtures/mcp/project-open.yaml

Expected merge behavior should be explicit:

  • Case-level config can override an eval-level server by name.
  • If a case does not specify an override, it inherits the eval-level MCP config unchanged.
  • For mocked servers, the case-specific config_ref should be resolved relative to the Skill directory, consistent with existing MCP config refs.
  • The same behavior should work with parallel case execution and should not leak mock responses across cases.

Acceptance Criteria

  • CaseConfig supports a documented way to override mocked MCP responses per case.
  • The evaluator provisions/installs the effective MCP config for each case instead of forcing one globally cached mocked response set.
  • Existing eval-level MCP behavior remains backward compatible.
  • Unit tests cover merge/override semantics and config_ref resolution.
  • E2E or integration coverage demonstrates two cases using the same mocked MCP server/tool with different returned values.
  • docs/guide/writing-evals.md, docs/zh/guide/writing-evals.md, and skill template/reference docs are updated with an example.

Metadata

Metadata

Assignees

Labels

No labels
No labels

Fields

No fields configured for Feature.

Projects

Status
Backlog

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions