Skip to content

light-handle/mcpspec

BranchesTags

Repository files navigation

MCPSpec

MCPSpec

Ship reliable MCP servers with confidence

npm version npm downloads license node 22+

Record sessions, generate mock servers, gate your CI pipeline, and catch Tool Poisoning — all without writing a single line of test code. The testing and reliability platform for Model Context Protocol servers.

Why MCPSpec?

Deterministic. Unlike LLM-based testing, MCPSpec runs are fast and repeatable. No flaky tests, no token costs. Ideal for CI.

Secure. Catch Tool Poisoning (prompt injection in tool descriptions) and Excessive Agency (destructive tools without safeguards) before they reach production.

Collaborative. Record server interactions once, share mock servers with your team. Frontend developers and CI pipelines can test against mocks — no API keys, no live dependencies.

Quick Start

# Install
npm install -g mcpspec

# Explore a server interactively
mcpspec inspect "npx @modelcontextprotocol/server-filesystem /tmp"

# Record a session — no test code needed
mcpspec record start "npx my-server"

# Generate a mock for your team
mcpspec mock my-recording --generate ./mocks/server.js

# Add CI gating in 30 seconds
mcpspec ci-init

Try it in 30 seconds with a pre-built community collection — no setup required:

mcpspec test examples/collections/servers/filesystem.yaml
mcpspec test examples/collections/servers/time.yaml --tag smoke

See all 70 community tests for 7 popular MCP servers.

Record & Mock

Record a session once. Replay it to catch regressions. Mock it for CI. No API keys required.

# 1. Record a session against your real server
mcpspec record start "npx my-server"
mcpspec> .call get_user {"id": "1"}
mcpspec> .call list_items {}
mcpspec> .save my-api

# 2. Replay against a new version — catch regressions instantly
mcpspec record replay my-api "npx my-server-v2"

# 3. Start a mock server — drop-in replacement, zero dependencies
mcpspec mock my-api

# 4. Generate a standalone .js file — commit to your repo
mcpspec mock my-api --generate ./mocks/server.js
node ./mocks/server.js

Replay output shows exactly what changed:

Replaying 3 steps...

  1/3 get_user (id=1)...       [OK] 42ms → {"name": "Alice"}
  2/3 list_items...            [CHANGED] 38ms → {"items": [...]}
  3/3 create_item (name=test)  [OK] 51ms → {"id": "abc"}

Summary: 2 matched, 1 changed, 0 added, 0 removed

Mock options:

Option Effect
--mode match (default) Exact input match first, then next queued response per tool
--mode sequential Tape/cassette style — responses served in recorded order
--latency original Simulate original response times
--latency 100 Fixed 100ms delay
--on-missing empty Return empty instead of error for unrecorded tools
--generate <path> Output standalone .js file (only needs @modelcontextprotocol/sdk)

Manage recordings:

mcpspec record list                    # List saved recordings
mcpspec record delete my-session       # Delete a recording

CI/CD Integration

ci-init generates ready-to-use pipeline configurations. Deterministic exit codes and JUnit/JSON/TAP reporters for seamless CI integration.

mcpspec ci-init                                 # Interactive wizard
mcpspec ci-init --platform github               # GitHub Actions
mcpspec ci-init --platform gitlab               # GitLab CI
mcpspec ci-init --platform shell                # Shell script
mcpspec ci-init --checks test,audit,score       # Choose checks
mcpspec ci-init --fail-on medium                # Audit severity gate
mcpspec ci-init --min-score 70                  # MCP Score threshold
mcpspec ci-init --force                         # Overwrite/replace existing

Auto-detects platform from .github/ or .gitlab-ci.yml. GitLab --force surgically replaces only the mcpspec job block, preserving other jobs.

GitHub Actions example (generated by mcpspec ci-init --platform github):

name: MCP Server Tests
on: [push, pull_request]

jobs:
  mcpspec:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: '22'
      - run: npm install -g mcpspec
      - name: Run tests
        run: mcpspec test --ci --reporter junit --output results.xml
      - name: Security audit
        run: mcpspec audit "npx my-server" --fail-on high
      - name: Quality gate
        run: mcpspec score "npx my-server" --min-score 80
      - uses: mikepenz/action-junit-report@v4
        if: always()
        with:
          report_paths: results.xml

Exit codes:

Code Meaning
0 Success
1 Test failure
2 Runtime error
3 Configuration error
4 Connection error
5 Timeout
6 Security findings above threshold
7 Validation error
130 Interrupted (Ctrl+C)

Reporters: console (default), json, junit, html, tap.

Security Audit

Prevent Tool Poisoning. 8 security rules covering traditional vulnerabilities and LLM-specific threats. A safety filter auto-skips destructive tools, and --dry-run previews targets before scanning.

mcpspec audit "npx my-server"                        # Passive (safe)
mcpspec audit "npx my-server" --mode active           # Active probing
mcpspec audit "npx my-server" --fail-on medium        # CI gate
mcpspec audit "npx my-server" --exclude-tools delete  # Skip tools
mcpspec audit "npx my-server" --dry-run               # Preview targets

Security rules:

Rule Mode What it detects
Tool Poisoning Passive LLM prompt injection in descriptions, hidden Unicode, cross-tool manipulation
Excessive Agency Passive Destructive tools without confirmation params, arbitrary code execution
Path Traversal Passive ../../etc/passwd style directory escape attacks
Input Validation Passive Missing constraints (enum, pattern, min/max) on tool inputs
Info Disclosure Passive Leaked paths, stack traces, API keys in tool descriptions
Resource Exhaustion Active Unbounded loops, large allocations, recursion
Auth Bypass Active Missing auth checks, hardcoded credentials
Injection Active SQL and command injection in tool inputs

Scan modes:

  • Passive (default) — 5 rules, analyzes metadata only, no tool calls. Safe for production.
  • Active — All 8 rules, sends test payloads. Requires confirmation prompt.
  • Aggressive — All 8 rules with more exhaustive probing. Requires confirmation prompt.

Active/aggressive modes auto-skip tools matching destructive patterns (delete_*, drop_*, destroy_*, etc.) and require explicit confirmation unless --acknowledge-risk is passed.

Each finding includes severity (info/low/medium/high/critical), description, evidence, and remediation advice.

Test Collections

Write tests in YAML with 10 assertion types, environments, variable extraction, tags, retries, and parallel execution.

name: Filesystem Tests
server: npx @modelcontextprotocol/server-filesystem /tmp

tests:
  - name: Read a file
    call: read_file
    with:
      path: /tmp/test.txt
    expect:
      - exists: $.content
      - type: [$.content, string]

  - name: Handle missing file
    call: read_file
    with:
      path: /tmp/nonexistent.txt
    expectError: true

Advanced features — environments, tags, retries, variable extraction, expressions:

schemaVersion: "1.0"
name: Advanced Tests

server:
  command: npx
  args: ["my-mcp-server"]
  env:
    NODE_ENV: test

environments:
  dev:
    variables:
      BASE_PATH: /tmp/dev
  staging:
    variables:
      BASE_PATH: /tmp/staging

defaultEnvironment: dev

tests:
  - id: create-data
    name: Create data
    tags: [smoke, write]
    timeout: 5000
    retries: 2
    call: create_item
    with:
      name: "test-item"
    assertions:
      - type: schema
      - type: exists
        path: $.id
      - type: latency
        maxMs: 1000
    extract:
      - name: itemId
        path: $.id

  - id: verify-data
    name: Verify created data
    tags: [smoke, read]
    call: get_item
    with:
      id: "{{itemId}}"
    assertions:
      - type: equals
        path: $.name
        value: "test-item"
      - type: expression
        expr: "response.id == itemId"

Assertion types:

Type Description Example
schema Validate response structure type: schema
equals Exact match (deep comparison) path: $.id, value: 123
contains Array or string contains value path: $.tags, value: "active"
exists Path exists and is not null path: $.name
matches Regex pattern match path: $.email, pattern: ".*@.*"
type Type check path: $.count, expected: number
length Array/string length path: $.items, operator: gt, value: 0
latency Response time threshold maxMs: 1000
mimeType Content type validation expected: "image/png"
expression Safe expression eval expr: "response.total > 0"

Expressions use expr-eval — comparisons, logical operators, property access, math. No arbitrary code execution.

Shorthand format for common assertions:

expect:
  - exists: $.field
  - equals: [$.id, 123]
  - contains: [$.tags, "active"]
  - matches: [$.email, ".*@.*"]

Run options:

mcpspec test ./tests.yaml              # Specific file
mcpspec test --env staging             # Switch environment
mcpspec test --tag @smoke              # Filter by tag
mcpspec test --parallel 4              # Parallel execution
mcpspec test --reporter junit --output results.xml
mcpspec test --baseline main           # Compare against baseline
mcpspec test --watch                   # Re-run on file changes
mcpspec test --ci                      # CI mode (no colors)

MCP Score

A 0-100 quality rating across 5 weighted categories with opinionated schema linting. Use as a CI gate or generate a badge for your README.

mcpspec score "npx my-server"
mcpspec score "npx my-server" --badge badge.svg       # Generate SVG badge
mcpspec score "npx my-server" --min-score 80          # Fail if below threshold

Scoring categories:

Category (weight) What it measures
Documentation (25%) Percentage of tools and resources with descriptions
Schema Quality (25%) Property types, descriptions, required fields, constraints (enum/pattern/min/max), naming conventions
Error Handling (20%) Structured error responses (isError: true) vs. crashes on bad input
Responsiveness (15%) Median latency: <100ms = 100, <500ms = 80, <1s = 60, <5s = 40
Security (15%) Findings from passive security scan: 0 = 100, <=2 = 70, <=5 = 40

Schema quality uses 6 sub-criteria: structure (20%), property types (20%), descriptions (20%), required fields (15%), constraints (15%), naming conventions (10%).

The --badge flag generates a shields.io-style SVG badge for your README.

More Features

Interactive Inspector

Connect to any MCP server and explore its capabilities in a live REPL.

mcpspec inspect "npx @modelcontextprotocol/server-filesystem /tmp"
Command Description
.tools List all available tools with descriptions
.resources List all available resources (URIs)
.call <tool> <json> Call a tool with JSON input
.schema <tool> Display tool's JSON Schema input spec
.info Show server info (name, version, capabilities)
.help Show help
.exit Disconnect and exit

Performance Benchmarks

Measure latency and throughput with statistical analysis across hundreds of iterations.

mcpspec bench "npx my-server"                         # 100 iterations
mcpspec bench "npx my-server" --iterations 500
mcpspec bench "npx my-server" --tool read_file
mcpspec bench "npx my-server" --warmup 10

Output includes min/max/mean/median/P95/P99 latency, standard deviation, and throughput (calls/sec). Warmup iterations (default: 5) are excluded from measurements.

Doc Generator

Auto-generate Markdown or HTML documentation from server introspection. Zero manual writing.

mcpspec docs "npx my-server"                   # Markdown to stdout
mcpspec docs "npx my-server" --format html      # HTML output
mcpspec docs "npx my-server" --output ./docs    # Write to directory

Web Dashboard

A full React UI for managing servers, running tests, viewing audit results, and more. Dark mode included.

mcpspec ui                    # Opens localhost:6274
mcpspec ui --port 8080        # Custom port
mcpspec ui --no-open          # Don't auto-open browser

10 pages: Dashboard, Servers, Collections, Runs, Inspector, Recordings, Audit, Benchmark, Docs, Score.

Baselines & Comparison

Save test runs as baselines and detect regressions between versions.

mcpspec baseline save main                      # Save current run
mcpspec baseline list                           # List all baselines
mcpspec test --baseline main                    # Compare against baseline
mcpspec compare --baseline main                 # Explicit comparison
mcpspec compare <run-id-1> <run-id-2>           # Compare two runs

Transports

MCPSpec supports 3 transport types for connecting to MCP servers:

Transport Use case Connection
stdio Local processes Spawns child process, communicates via stdin/stdout
SSE Server-Sent Events Connects to HTTP SSE endpoint
HTTP Streamable HTTP POST requests to HTTP endpoint 
# stdio (default)
server:
  command: npx
  args: ["my-mcp-server"]

# SSE
server:
  transport: sse
  url: http://localhost:3000/sse

# HTTP
server:
  transport: http
  url: http://localhost:3000/mcp

Commands

Command Description
mcpspec record start <server> Record an inspector session — .call, .save, .steps
mcpspec record replay <name> <server> Replay a recording and diff against original
mcpspec mock <recording> Mock server from recording — --mode, --latency, --on-missing, --generate
mcpspec test [collection] Run test collections with --env, --tag, --parallel, --reporter, --watch, --ci
mcpspec audit <server> Security scan — --mode, --fail-on, --exclude-tools, --dry-run
mcpspec score <server> Quality score (0-100) — --badge badge.svg, --min-score
mcpspec ci-init Generate CI config — --platform github|gitlab|shell, --checks, --fail-on, --force
mcpspec inspect <server> Interactive REPL — .tools, .call, .schema, .resources, .info
mcpspec bench <server> Performance benchmark — --iterations, --tool, --args, --warmup
mcpspec docs <server> Generate docs — --format markdown|html, --output <dir>
mcpspec compare Compare test runs or --baseline <name>
mcpspec baseline save <name> Save/list baselines for regression detection
mcpspec record list List saved recordings
mcpspec record delete <name> Delete a saved recording
mcpspec init [dir] Scaffold project — --template minimal|standard|full
mcpspec ui Launch web dashboard on localhost:6274

Community Collections

Pre-built test suites for popular MCP servers in examples/collections/servers/:

Collection Server Tests
filesystem.yaml @modelcontextprotocol/server-filesystem 12
memory.yaml @modelcontextprotocol/server-memory 10
everything.yaml @modelcontextprotocol/server-everything 11
fetch.yaml @modelcontextprotocol/server-fetch 7
time.yaml @modelcontextprotocol/server-time 10
chrome-devtools.yaml chrome-devtools-mcp 11
github.yaml @modelcontextprotocol/server-github 9

70 tests covering tool discovery, read/write operations, error handling, security edge cases, and latency.

mcpspec test examples/collections/servers/filesystem.yaml
mcpspec test examples/collections/servers/time.yaml --tag smoke

Architecture

Package Description
@mcpspec/shared Types, Zod schemas, constants
@mcpspec/core MCP client, test runner, assertions, security scanner (8 rules), profiler, doc generator, scorer, recording/replay, mock server
@mcpspec/cli 13 CLI commands built with Commander.js
@mcpspec/server Hono HTTP server with REST API + WebSocket
@mcpspec/ui React SPA — TanStack Router, TanStack Query, Tailwind, shadcn/ui

Development

git clone https://github.com/light-handle/mcpspec.git
cd mcpspec
pnpm install && pnpm build
pnpm test   # 334 tests across core + server

License

MIT

About

No description, website, or topics provided.

Resources

Readme

License

MIT license
Activity

Stars

7 stars

Watchers

0 watching

Forks

2 forks
Report repository

Releases

No releases published

Packages

 
 
 

Contributors

Languages