This repository has been mirrored from the original source: https://github.com/SalesforceAIResearch/MCP-Universe on 13 Sep-2025.

About This Repository

This repository hosts the replication package for a study on improving tool descriptions for the Model Context Protocol (MCP). It provides tools to empirically evaluate, analyze, and augment tool descriptions to enhance the performance of Foundation Model (FM)-based agents.

The key contributions of this repository are:

• A scoring rubric with six components to systematically assess tool description quality.
• An FM-based scanner to automatically detect "smells" (defects) in tool descriptions.
• A semi-automated pipeline to resolve these smells and generate optimized tool descriptions.
• The ability to run the MCP-Universe benchmark to evaluate the impact of these augmented descriptions on agent performance, including success rates and execution costs.
• Support for ablation studies to analyze the importance of different description components.

What is MCP-Universe?

MCP-Universe is a comprehensive framework designed for developing, testing, and benchmarking AI agents. It offers a robust platform for building and evaluating both AI agents and LLMs across a wide range of task environments. The framework also supports seamless integration with external MCP servers and facilitates sophisticated agent orchestration workflows.

Unlike existing benchmarks that rely on overly simplistic tasks, MCP-Universe addresses critical gaps by evaluating LLMs in real-world scenarios through interaction with actual MCP servers, capturing real application challenges such as:

• 🎯 Long-horizon reasoning across multi-step tasks
• 🔧 Large, unfamiliar tool spaces with diverse MCP servers
• 🌍 Real-world data sources and live environments
• ⚡ Dynamic evaluation with time-sensitive ground truth

Prerequisites

• Python: Requires version 3.10-3.12 (Python 3.13 is not yet supported).
• Docker: Used for running Dockerized MCP servers.
• PostgreSQL (optional): Used for database storage and persistence.

Installation

1. Clone the repository

   git clone https://github.com/SalesforceAIResearch/MCP-Universe.git
   cd MCP-Universe

2. Create and activate virtual environment

   python3 -m venv venv
   source venv/bin/activate

3. Install dependencies

   pip install -r requirements.txt
   pip install -r dev-requirements.txt

4. Platform-specific requirements

   Linux:

   sudo apt-get install libpq-dev

   macOS:

   brew install postgresql

5. Configure pre-commit hooks

   pre-commit install

6. Environment configuration

   cp .env.example .env
   # Edit .env with your API keys and configuration

Environment Configuration

1. Initial Setup

Copy the environment template and configure your API credentials:

cp .env.example .env

2. API Keys and Configuration

Configure the following environment variables in your .env file. The required keys depend on which benchmark domains you plan to evaluate:

Core LLM Providers

Environment Variable	Provider	Description	Required For
OPENAI_API_KEY	OpenAI	API key for GPT models (gpt-5, etc.)	All domains
ANTHROPIC_API_KEY	Anthropic	API key for Claude models	All domains
GEMINI_API_KEY	Google	API key for Gemini models	All domains

Note: You only need to configure the API key for the LLM provider you intend to use in your evaluation.

Domain-Specific Services

Environment Variable	Service	Description	Setup Instructions
SERP_API_KEY	SerpAPI	Web search API for search benchmark evaluation	Get API key
GOOGLE_MAPS_API_KEY	Google Maps	Geolocation and mapping services	Setup Guide
GITHUB_PERSONAL_ACCESS_TOKEN	GitHub	Personal access token for repository operations	Token Setup
GITHUB_PERSONAL_ACCOUNT_NAME	GitHub	Your GitHub username	N/A
NOTION_API_KEY	Notion	Integration token for Notion workspace access	Integration Setup
NOTION_ROOT_PAGE	Notion	Root page ID for your Notion workspace	See configuration example below

Task Search Infrastructure

Place your .env file in the repository root (the same directory as this README.md) so the workflow helpers can discover it. Populate the following entries to enable the enhanced tool discovery flow:

Environment Variable	Description	Example
DB_URL	PostgreSQL connection string containing historical tool executions	postgresql://user:pass@localhost:5432/mcp

The task similarity search also reuses OPENAI_API_KEY from the core provider configuration to generate embeddings.

Tool Response Truncation

Environment Variable	Description	Example
MAX_TOKEN_LEN	Maximum number of tokens from a tool response to keep before invoking the LLM	16000

System Paths

Environment Variable	Description	Example
BLENDER_APP_PATH	Full path to Blender executable (we used v4.4.0)	/Applications/Blender.app/Contents/MacOS/Blender
MCPUniverse_DIR	Absolute path to your MCP-Universe repository	/Users/username/MCP-Universe

Configuration Examples

Notion Root Page ID: If your Notion page URL is:

https://www.notion.so/your_workspace/MCP-Evaluation-1dd6d96e12345678901234567eaf9eff

Set NOTION_ROOT_PAGE=MCP-Evaluation-1dd6d96e12345678901234567eaf9eff

Blender Installation:

1. Download Blender v4.4.0 from blender.org
2. Install our modified Blender MCP server following the installation guide
3. Set the path to the Blender executable

⚠️ Security Recommendations

🔒 IMPORTANT SECURITY NOTICE

Please read and follow these security guidelines carefully before running benchmarks:

• 🚨 GitHub Integration: CRITICAL - We strongly recommend using a dedicated test GitHub account for benchmark evaluation. The AI agent will perform real operations on GitHub repositories, which could potentially modify or damage your personal repositories.

• 🔐 API Key Management:

  • Store API keys securely and never commit them to version control
  • Use environment variables or secure key management systems
  • Regularly rotate your API keys for enhanced security

• 🛡️ Access Permissions:

  • Grant minimal necessary permissions for each service integration
  • Review and limit API key scopes to only required operations
  • Monitor API usage and set appropriate rate limits

• ⚡ Blender Operations: The 3D design benchmarks will execute Blender commands that may modify or create files on your system. Ensure you have adequate backups and run in an isolated environment if necessary.

Benchmark Configuration

Domain-Specific Configuration Files

Each benchmark domain has a dedicated YAML configuration file located in mcpuniverse/benchmark/configs/test/. To evaluate your LLM/agent, modify the appropriate configuration file:

Domain	Configuration File	Description
Web Search	web_search.yaml	Search engine and information retrieval tasks
Location Navigation	location_navigation.yaml	Geographic and mapping-related queries
Browser Automation	browser_automation.yaml	Web interaction and automation scenarios
Financial Analysis	financial_analysis.yaml	Market data analysis and financial computations
Repository Management	repository_management.yaml	Git operations and code repository tasks
3D Design	3d_design.yaml	Blender-based 3D modeling and design tasks

LLM Model Configuration

In each configuration file, update the LLM specification to match your target model:

kind: llm
spec:
  name: llm-1
  type: openai  # or anthropic, google, etc.
  config:
    model_name: gpt-4o  # Replace with your target model

Execution

Running Individual Benchmarks with augmented tool description

Pass --tool-description-type 1 to replace MCP-supplied tool descriptions with augmented entries stored in the mcp_servers database table whenever they are available.

# Location Navigation
python tests/benchmark/test_benchmark_location_navigation.py --tool-description-type 1

# Browser Automation  
python tests/benchmark/test_benchmark_browser_automation.py --tool-description-type 1

# Financial Analysis
python tests/benchmark/test_benchmark_financial_analysis.py --tool-description-type 1

# Repository Management
python tests/benchmark/test_benchmark_repository_management.py --tool-description-type 1

# Web Search
python tests/benchmark/test_benchmark_web_search.py --tool-description-type 1

# 3D Design
python tests/benchmark/test_benchmark_3d_design.py --tool-description-type 1

Analysis Scripts

MCP-Universe Representativeness Check

The script scripts/equivalence_mcp_universe_vs_rest.py evaluates whether the MCP-Universe benchmark subset is practically similar to the remaining tool corpus on the six description-quality rubrics.

It uses hardcoded inputs:

• all-mcp-universe-tools-after-optimization-consolidation.csv
• tool_description_quality_scores/tool-description-quality-updated-consolidated-pupose.csv
• tool_description_quality_scores/tool-description-quality-updated-consolidated-usage-guidelines.csv
• tool_description_quality_scores/tool-description-quality-updated-consolidated-limitations.csv
• tool_description_quality_scores/tool-description-quality-updated-consolidated-parameter_explanation.csv
• tool_description_quality_scores/tool-description-quality-updated-consolidated-examples.csv
• tool_description_quality_scores/tool-description-quality-updated-consolidated-length.csv

For each rubric, the script:

• splits tools into Only-MCP-Universe and Without-MCP-Universe using (mcp_server_name, tool_name)
• recomputes each tool's rubric score as the mean of gpt-41-mini, haiku-35, and qwen3-32b
• runs a Welch TOST equivalence test with a +/- 0.5 raw-score margin
• applies Holm-Bonferroni correction across the six rubric tests

Run it with:

python scripts/equivalence_mcp_universe_vs_rest.py

The CLI output reports the unmatched benchmark tools excluded from analysis, then one result block per rubric containing:

• the raw and tested split sizes
• the two group means and standard deviations
• the mean difference (Only-MCP-Universe - Without-MCP-Universe)
• the 90% confidence interval used for TOST
• the raw and Holm-adjusted equivalence p-values
• the final equivalence decision

Running Individual Benchmarks with different components of the augmented tool description

python tests/benchmark/test_benchmark_financial_analysis.py --tool-description-type 1 --components Purpose,Examples

All possible components are : Purpose, Examples, Limitations, UsageGuideline, Parameter_Explanation,

Benchmark definition

Define agent(s) and benchmark in a YAML file. Here’s a simple weather forecast benchmark:

kind: llm
spec:
  name: llm-1
  type: openai
  config:
    model_name: gpt-4o

---
kind: agent
spec:
  name: ReAct-agent
  type: react
  config:
    llm: llm-1
    instruction: You are an agent for weather forecasting.
    servers:
      - name: weather

---
kind: benchmark
spec:
  description: Test the agent for weather forecasting
  agent: ReAct-agent
  tasks:
    - dummy/tasks/weather.json

The benchmark definition mainly contains two parts: the agent definition and the benchmark configuration. The benchmark configuration is simple—you just need to specify the agent to use (by the defined agent name) and a list of tasks to evaluate. Each task entry is the task config file path. It can be a full file path or a partial file path. If it is a partial file path (like "dummy/tasks/weather.json"), it should be put in the folder mcpuniverse/benchmark/configs in this repo.

This framework offers a flexible way to define both simple agents (such as ReAct) and more complex, multi-step agent workflows.

1. Specify LLMs: Begin by declaring the large language models (LLMs) you want the agents to use. Each LLM component must be assigned a unique name (e.g., "llm-1"). These names serve as identifiers that the framework uses to connect the different components together.
2. Define an agent: Next, define an agent by providing its name and selecting an agent class. Agent classes are available in the mcpuniverse.agent package. Commonly used classes include "basic", "function-call", and "react". Within the agent specification ( spec.config), you must also indicate which LLM instance the agent should use by setting the "llm" field.
3. Create complex workflows: Beyond simple agents, the framework supports the definition of sophisticated, orchestrated workflows where multiple agents interact or collaborate to solve more complex tasks.

For example:

kind: llm
spec:
  name: llm-1
  type: openai
  config:
    model_name: gpt-4o

---
kind: agent
spec:
  name: basic-agent
  type: basic
  config:
    llm: llm-1
    instruction: Return the latitude and the longitude of a place.

---
kind: agent
spec:
  name: function-call-agent
  type: function-call
  config:
    llm: llm-1
    instruction: You are an agent for weather forecast. Please return the weather today at the given latitude and longitude.
    servers:
      - name: weather

---
kind: workflow
spec:
  name: orchestrator-workflow
  type: orchestrator
  config:
    llm: llm-1
    agents:
      - basic-agent
      - function-call-agent

---
kind: benchmark
spec:
  description: Test the agent for weather forecasting
  agent: orchestrator-workflow
  tasks:
    - dummy/tasks/weather.json

Utility Scripts

Evaluate MCP tool description quality

The evaluate_tool_descriptions CLI loads server definitions from an MCP configuration file (defaulting to mcpuniverse/mcp/configs/server_list.json), launches each server through the configured transport, and evaluates every exposed tool description using two dedicated LLM prompts. One prompt determines whether the tool is a consolidated workflow, while the other audits the description for missing best-practice elements. Results are saved to a CSV file compatible with our internal Node.js tooling, making it easy to compare outputs across implementations.

export OPENAI_API_KEY=sk-...  # or pass --api-key explicitly
python -m mcpuniverse.scripts.evaluate_tool_descriptions \
  --model gpt-4o-mini \
  --output /tmp/mcp_tool_audit.csv

To target a subset of configured servers, pass one or more --server flags.

python -m mcpuniverse.scripts.evaluate_tool_descriptions \
  --model gpt-4o-mini \
  --output /tmp/mcp_tool_audit.csv \
  --server github --server date

You can also add ad-hoc server scripts (not yet present in the config file) by supplying --config values; each path is converted into a temporary MCP configuration entry before evaluation.

python -m mcpuniverse.scripts.evaluate_tool_descriptions \
  --model gpt-4o-mini \
  --output /tmp/mcp_tool_audit.csv \
  --config mcpuniverse/mcp/servers/github/server.json

Source-code grounding for MCP tool evaluation

Use scripts/mcp_tool_source_parser_v3.py to extract implementation code for each MCP tool before running grounded description evaluation. The parser joins tool metadata with a repository map, clones missing GitHub repositories into a local cache, detects Python, JavaScript/TypeScript, or Go MCP implementations, and writes a CSV keyed by (mcp_server_name, tool_name).

The repository map must include:

mcp_server_name,repository_url
yfinance,https://github.com/SalesforceAIResearch/MCP-Universe
github,https://github.com/github/github-mcp-server

The grounding test map is already provided at mcpuniverse/mcp/configs/grounding_test_repo_map.csv.

Extract source code for the grounding test servers:

python scripts/mcp_tool_source_parser_v3.py \
  --server-list mcpuniverse/mcp/configs/grounding_test_server_list.json \
  --repo-map-csv mcpuniverse/mcp/configs/grounding_test_repo_map.csv \
  --repo-dir repos \
  --output mcp_tool_source_code_v3.csv \
  --metadata-source auto \
  --transport stdio

Useful variants:

# Reuse already-cloned repositories and leave code blank for missing repos.
python scripts/mcp_tool_source_parser_v3.py \
  --server-list mcpuniverse/mcp/configs/grounding_test_server_list.json \
  --repo-map-csv mcpuniverse/mcp/configs/grounding_test_repo_map.csv \
  --repo-dir repos \
  --output mcp_tool_source_code_v3.csv \
  --metadata-source auto \
  --skip-clone

# Avoid live MCP server startup and read tool names/descriptions only from CSV metadata.
python scripts/mcp_tool_source_parser_v3.py \
  --server-list mcpuniverse/mcp/configs/grounding_test_server_list.json \
  --tool-metadata-csv path/to/tool_metadata.csv \
  --metadata-source csv \
  --repo-map-csv mcpuniverse/mcp/configs/grounding_test_repo_map.csv \
  --output mcp_tool_source_code_v3.csv

The output CSV contains the columns expected by grounded evaluation, including mcp_server_name, tool_name, tool_description, and all_tool_code_with_dependencies.

Run the evaluator with the extracted source-code CSV:

export OPENAI_API_KEY=sk-...

python -m mcpuniverse.scripts.evaluate_tool_descriptions \
  --model openai:gpt-4.1-mini \
  --config mcpuniverse/mcp/configs/grounding_test_server_list.json \
  --metadata-source auto \
  --transport stdio \
  --grounding mcp_tool_source_code_v3.csv \
  --output mcpuniverse/scripts/rubric-eval-grounding-test-gpt-41-mini.csv

To compare grounded and ungrounded scoring, run the same command without --grounding and write to a separate output file:

python -m mcpuniverse.scripts.evaluate_tool_descriptions \
  --model openai:gpt-4.1-mini \
  --config mcpuniverse/mcp/configs/grounding_test_server_list.json \
  --metadata-source auto \
  --transport stdio \
  --output mcpuniverse/scripts/rubric-eval-grounding-test-gpt-41-mini-no-source.csv

Grounding is applied only when the CSV has a non-empty all_tool_code_with_dependencies value for the exact (server, tool) pair. Tools without matching source code are still evaluated normally.

Key flags:

Flag	Description
--model MODEL_NAME	Required. Target OpenAI model used for both evaluations.
--output PATH	Required. Destination CSV path for the combined scores.
--config PATH	Optional. Alternate MCP server configuration file (default: mcpuniverse/mcp/configs/server_list.json).
--transport {stdio,sse,auto}	Optional. Preferred transport; auto falls back to SSE when stdio is unavailable.
--server NAME	Optional. Limit evaluation to specific servers (repeatable).
--server-path PATH	Optional. Explicit path to a server script or directory. Paths are merged into the loaded MCP configuration.
--pattern GLOB	Optional. Filename pattern for locating scripts inside provided --server-path directories (default: server.py).
--metadata-source {dynamic,auto,csv}	Optional. Load tool metadata from live MCP servers, CSV files, or both.
--tool-metadata-csv PATH ...	Optional. CSV metadata used when --metadata-source is csv or auto.
--grounding PATH	Optional. Source-code CSV produced by mcp_tool_source_parser_v3.py; must include mcp_server_name, tool_name, and all_tool_code_with_dependencies.
--limit N	Optional. Evaluate only the first N discovered tools.
--dry-run	Skip LLM calls and emit placeholder rows (useful for connectivity tests).

The CLI expects access to OpenAI's Chat Completions API. Provide the API key via OPENAI_API_KEY, the --api-key flag, or a custom --base-url if you are using a compatible proxy. Each tool evaluation spawns the corresponding MCP server through its stdio transport, lists available tools, and records both LLM assessments in the output CSV.

Augment MCP tool descriptions

The optimize_tool_descriptions CLI connects to every MCP server defined in a JSON configuration file, retrieves their tools, and rewrites each tool's description with the help of an LLM following a built-in rubric. Augmented descriptions are versioned and stored in the mcp_servers database table so you can track how wording evolves over time.

python -m mcpuniverse.scripts.optimize_tool_descriptions \
  --model <MODEL_ALIAS_OR_ALIAS:MODEL_NAME> \
  [--config path/to/server_list.json] \
  [--transport stdio|sse|auto] \
  [--rubric-file path/to/custom_rubric.txt] \
  [--db-url postgres://user:pass@host:port/db] \
  [--grounding] \
  [--grounding-source-code-csv mcp_tool_source_code.csv]

Key notes:

• The --model (-m) flag accepts either a registered alias (for example openai) or a combination in the form alias:model_name such as openai:gpt-4.1-mini. When only a provider-specific model name is supplied, the CLI attempts to infer the correct alias (e.g. gpt- models map to the OpenAI client). Any API keys needed for that provider are read from the environment.
• Database connectivity defaults to the DB_URL or DATABASE_URL environment variables. Use --db-url to override them explicitly.
• Server definitions default to mcpuniverse/mcp/configs/server_list.json. Provide --config when you want to point at a different configuration file.
• Supply --rubric-file to replace the built-in rubric with custom guidance for the LLM.
• Use --grounding to add source-code context to the optimization prompt. When enabled, the script loads mcp_tool_source_code.csv by default and matches rows by (mcp_server_name, tool_name). If a matching row has all_tool_code_with_dependencies, that code is included as grounding context; tools without matching code are still optimized normally without grounding.
• Use --grounding-source-code-csv to point at a different source-code mapping CSV. The CSV must include mcp_server_name, tool_name, and all_tool_code_with_dependencies.

Before running the CLI, ensure the destination database contains the mcp_servers table with the schema expected by the script. The tool logs which server/tool pairs were updated and exits with a non-zero status if no descriptions could be stored.

Dissect MCP tool descriptions

Before dissecting descriptions, run the database migration that adds the tool_description_components column to the mcp_servers table:

python -m mcpuniverse.app.db.migration \
  --db-url postgresql+asyncpg://user:pass@host:5432/dbname

The CLI reads the database URL from --db-url or from DB_SOURCE, DB_URL, or DATABASE_URL. Provide an async-compatible SQLAlchemy URL (for PostgreSQL, use the postgresql+asyncpg:// scheme).

Once the column exists, use the dissect_tool_descriptions CLI to split each tool description into the required documentation components:

python -m mcpuniverse.scripts.dissect_tool_descriptions \
  --model <MODEL_ALIAS_OR_ALIAS:MODEL_NAME> \
  --db-url postgresql://user:pass@host:5432/dbname \
  [--additional-descriptions path/to/additional_tool_description.json] \
  [--all-versions] \
  [--include-existing] \
  [--dry-run]

Key notes:

• The CLI combines each row's tool_optimized_description with the server/tool-specific examples stored in mcpuniverse/mcp/additional_tool_description.json. The merged text and the component description (when present) are passed to the LLM.
• --model accepts the same alias or alias:model_name format used by the other scripts. Configure the relevant provider credentials before running the CLI.
• Supply a standard psycopg-compatible connection string through --db-url (or set DB_URL/DATABASE_URL). Only rows missing tool_description_components are processed by default; use --include-existing to reprocess populated rows.
• Enable --dry-run to preview the structured output without persisting it.
• Successful runs store a JSON object with the keys Purpose, UsageGuideline, Parameter_Explanation, Limitation, and Examples for each processed server/tool pair.

Evaluate optimized component descriptions from the database

The evaluate_db_tool_descriptions CLI evaluates optimized descriptions that have already been split into components in the mcp_servers table. It fetches rows where tool_description_components is populated, reconstructs a single description from Purpose, UsageGuideline, Limitation, Parameter_Explanation, and Examples, then applies the existing rubric evaluator and writes the scores to CSV.

By default, the query keeps the latest updated_at row per (mcp_server_name, tool_name). Use --version to restrict the query to a specific mcp_servers.version value before the latest-row selection is applied. If multiple rows exist for the same server/tool within that version, the most recently updated row in that version is evaluated.

PYTHONPATH=.:mcpuniverse/scripts python mcpuniverse/scripts/evaluate_db_tool_descriptions.py \
  --model openai:gpt-4.1-mini \
  --version 2 \
  --output mcpuniverse/scripts/version-2-component-evaluation-gpt-41-mini.csv

To restrict evaluation to a subset of servers, repeat --server:

PYTHONPATH=.:mcpuniverse/scripts python mcpuniverse/scripts/evaluate_db_tool_descriptions.py \
  --model openai:gpt-4.1-mini \
  --version 2 \
  --output mcpuniverse/scripts/version-2-component-evaluation-gpt-41-mini.csv \
  --server date \
  --server weather \
  --server github

Compare versioned component scores with Wilcoxon tests

The scripts/wilcoxon_version_component_scores.py script compares two component-evaluation CSVs, such as version 1 versus version 2. It performs an inner join on (mcp_server_name, tool_name), so when version 2 is a subset of version 1 the analysis automatically uses only the paired tools available in both files. It writes both the paired score table used for the analysis and the Wilcoxon signed-rank test results.

python scripts/wilcoxon_version_component_scores.py \
  --old mcpuniverse/scripts/version-1-component-evaluation-gpt-41-mini.csv \
  --new mcpuniverse/scripts/version-2-component-evaluation-gpt-41-mini.csv \
  --results-output analysis_output/wilcoxon_v1_v2_grounding_results.csv \
  --paired-output analysis_output/wilcoxon_v1_v2_grounding_paired_scores.csv

The script compares the overall description_quality_score and the six rubric dimension scores. It uses SciPy's Wilcoxon implementation when SciPy is available and falls back to a built-in normal approximation otherwise.

Compare DeepEval grounding scores with Wilcoxon tests

The scripts/wilcoxon_deepeval_component_grounding.py script compares DeepEval grounding detail CSVs, such as version 1 versus version 2. It joins on (mcp_server_name, tool_name, component) so each test is paired at the tool-component level. For every component, it runs separate Wilcoxon signed-rank tests for faithfulness_score and hallucination_score, including rank-biserial effect size and improved/declined/unchanged counts.

python scripts/wilcoxon_deepeval_component_grounding.py \
  --old analysis_output/deepeval_component_grounding_details_v1.csv \
  --new analysis_output/deepeval_component_grounding_details_v2.csv \
  --results-output analysis_output/wilcoxon_deepeval_component_grounding_results.csv \
  --paired-output analysis_output/wilcoxon_deepeval_component_grounding_paired_scores.csv \
  --table-output analysis_output/wilcoxon_deepeval_component_grounding_table.csv

If the version 1 file was written without a suffix, use analysis_output/deepeval_component_grounding_details.csv as --old.

Compare grounding and non-grounding success rates

The scripts/statistical_test_grounding_vs_non_grounding.py script compares paired binary success outcomes between the second and third columns of grounding_vs_non_grounding_statistical_test.csv. It uses McNemar's test, which is appropriate when each row is the same task evaluated under two conditions, such as success_grounding versus success_non_grounding.

python scripts/statistical_test_grounding_vs_non_grounding.py --exact

Use --exact to run the exact binomial McNemar test. Without --exact, the script uses the continuity-corrected chi-square McNemar test. The output includes the contingency table, both success rates, p-value, signed effect direction, and whether the difference is significant at alpha=0.05.

Generate synthetic benchmark tasks from tool descriptions

The generate_synthetic_tasks script connects to every MCP server defined in a config file, fetches each tool's description and input schema, and calls the OpenAI chat-completions API to produce realistic benchmark tasks at multiple difficulty levels. Each generated task still uses the benchmark task schema used by mcpuniverse/benchmark/configs/test/, but the script now writes a single aggregated JSON file per server for each run.

Difficulty assignment is intentionally biased toward harder cases first:

• 1 task per tool → very hard
• 2 tasks per tool → very hard, hard
• 3 tasks per tool → very hard, hard, medium
• >3 tasks per tool → continues filling in descending hardness order: very hard, then hard, then medium

Hard and very hard tasks deliberately target edge cases, boundary values, unusual inputs, brittle parameter combinations, or tool-specific failure boundaries.

The OpenAI API key is read exclusively from the OPENAI_API_KEY environment variable.

Minimal setup from a fresh checkout:

python3 -m venv venv
source venv/bin/activate
pip install -r requirements.txt
cp .env.example .env

Add the required keys to .env or export them in your shell. Synthetic task generation requires OPENAI_API_KEY; example and limitation discovery requires OPENROUTER_API_KEY because it runs the ReAct agent and synthesis step through OpenRouter. The target MCP servers may require their own service credentials from the Environment Configuration section.

End-to-end command sequence for one server:

source venv/bin/activate
export OPENAI_API_KEY=sk-...
export OPENROUTER_API_KEY=or-...

# 1. Generate a server task bundle from tool descriptions.
python -m mcpuniverse.scripts.utils.generate_synthetic_tasks \
  --model gpt-4.1-mini \
  --config mcpuniverse/mcp/configs/server_list.json \
  --output data/synthetic_tasks \
  --server yfinance \
  --tasks-per-tool 3 \
  --transport stdio

# 2. Pick the generated bundle for the server.
TASK_FILE=$(ls -t data/synthetic_tasks/mcp_server_yfinance_synthetic_tasks_*.json | head -n 1)

# 3. Execute the tasks and synthesize observed examples and limitations.
python -m mcpuniverse.scripts.utils.discover_examples_limitations \
  --model minimax/minimax-m2.5 \
  --synthesis-model openai/gpt-4.1 \
  --task-file "$TASK_FILE" \
  --config mcpuniverse/mcp/configs/server_list.json \
  --output data/discovered_additional_descriptions.json \
  --merge

To make the discovered examples and limitations available to the component dissection pipeline, pass the generated file explicitly:

python -m mcpuniverse.scripts.dissect_tool_descriptions \
  --model openai:gpt-4.1-mini \
  --db-url postgresql://user:pass@host:5432/dbname \
  --additional-descriptions data/discovered_additional_descriptions.json

export OPENAI_API_KEY=sk-...

# generate 3 tasks per tool for all servers
python -m mcpuniverse.scripts.utils.generate_synthetic_tasks \
  --model gpt-4.1-mini \
  --output data/synthetic_tasks

# restrict to specific servers
python -m mcpuniverse.scripts.utils.generate_synthetic_tasks \
  --model gpt-4.1-mini \
  --output data/synthetic_tasks \
  --server date --server weather \
  --tasks-per-tool 4

# use a custom server config
python -m mcpuniverse.scripts.utils.generate_synthetic_tasks \
  --model gpt-4o \
  --config path/to/my_server_list.json \
  --output data/synthetic_tasks

Output files are written as:

<output>/mcp_server_<server_name>_synthetic_tasks_<datetime>.json

Each file contains:

• server_name
• generated_at
• task_count
• tasks: the list of generated benchmark task objects

Each task also includes _meta.tool_name so bundled tasks remain traceable to their source tool after aggregation.

Key flags:

Flag	Default	Description
--model / -m	(required)	OpenAI model name, e.g. gpt-4.1-mini, gpt-4o
--config	mcpuniverse/mcp/configs/server_list.json	MCP server list JSON config
--output / -o	data/synthetic_tasks	Output directory
--server	(all)	Restrict to specific server names (repeatable)
--tasks-per-tool	3	Total tasks per tool (minimum 1; difficulty assigned in descending hardness order)
--transport	stdio	MCP transport (stdio, sse, auto)

Discover real-world examples and limitations from synthetic tasks

The discover_examples_limitations script closes the loop between synthetic task generation and documentation. It replays every task from a server task bundle through a live ReAct agent, captures the exact tool arguments and responses produced during execution, and feeds that evidence to an LLM to synthesise a concise additional_description block — containing concrete Examples and observed Limitations — in the same JSON format used by mcpuniverse/mcp/additional_tool_description.json.

Typical workflow

generate_synthetic_tasks  →  task bundle JSON
        ↓
discover_examples_limitations  →  discovered_additional_description_*.json

The agent model and synthesis model both use OpenRouter (key: OPENROUTER_API_KEY). Pass --synthesis-model to use a different model for the synthesis step while keeping a lighter model for execution.

export OPENROUTER_API_KEY=or-...

# run tasks for the yfinance server and write descriptions beside the task file
python -m mcpuniverse.scripts.utils.discover_examples_limitations \
  --model minimax/minimax-m2.5 \
  --task-file data/synthetic_tasks/mcp_server_yfinance_synthetic_tasks_20260426_130032.json

# use a stronger model for synthesis only
python -m mcpuniverse.scripts.utils.discover_examples_limitations \
  --model minimax/minimax-m2.5 \
  --synthesis-model openai/gpt-4.1 \
  --task-file data/synthetic_tasks/mcp_server_yfinance_synthetic_tasks_20260426_130032.json \
  --output data/discovered_additional_descriptions.json

The default output file is placed in the same directory as --task-file and named after the task file:

mcp_server_yfinance_synthetic_tasks_20260426_130032.json
  → discovered_additional_description_yfinance_synthetic_20260426_130032.json

By default the script merges new entries into an existing output file, overwriting only the (server, tool) pairs that were re-observed. Use --no-merge to overwrite the file completely.

Key flags:

Flag	Default	Description
--model / -m	(required)	OpenRouter model for the ReAct agent, e.g. minimax/minimax-m2.5
--task-file	(required)	Path to a server task bundle produced by generate_synthetic_tasks
--synthesis-model	same as --model	OpenRouter model for the synthesis step
--config	mcpuniverse/mcp/configs/server_list.json	MCP server list JSON config
--output / -o	auto-derived from task file	Destination JSON file
--merge / --no-merge	merge enabled	Merge into or overwrite the output file

Each entry in the output JSON has three fields:

{
  "mcp_server_name": "yfinance",
  "tool_name": "get_historical_stock_prices",
  "additional_description": "**Examples**\n...\n\n**Limitations**\n..."
}

This file can be passed directly to dissect_tool_descriptions via --additional-descriptions, or referenced wherever load_additional_tool_descriptions() is called.

Stratified random sampling of tool descriptions

The stratified_sample script draws a statistically justified random sample from a tool-description quality CSV (e.g. data/tool_description_quality.csv) using Cramér/Cochran's formula with a finite-population correction. The population is split into strata by MCP server name and each stratum is sampled proportionally, guaranteeing at least one row per server.

Sample-size formula

n₀ = (Z² × 0.5 × 0.5) / E²       # infinite-population estimate (p = 0.5, conservative)
n  = n₀ / (1 + (n₀ − 1) / N)     # finite-population correction
nₕ = ceil(n × Nₕ / N)             # proportional allocation per stratum h

python -m mcpuniverse.scripts.utils.stratified_sample \
  --input  data/tool_description_quality.csv \
  --output data/tool_description_quality_sample.csv \
  --confidence 0.80 \
  --error 0.05

Key flags:

Flag	Default	Description
--input / -i	data/tool_description_quality.csv	Input CSV file
--output / -o	<input-stem>_sample.csv	Output CSV file (saved beside the input by default)
--confidence / -C	0.80	Desired confidence level — accepts 0.80 or 80
--error / -E	0.05	Tolerable margin of error — accepts 0.05 or 5
--stratum-col	mcp_server_name	Column that defines the strata
--seed	42	Random seed for reproducibility
--db-table	mcp_servers	Table to look up tool_original_description (mcp_servers, mcp_servers_livemcpbench, mcp_servers_livemcpplus)

After sampling, the script queries the specified table for each (mcp_server_name, tool_name) pair and appends a tool_original_description column to the output CSV. Rows with no matching DB entry receive N/A. The DB connection URL is read from the DB_URL or DATABASE_URL environment variable; the lookup is skipped when neither is set.

The script prints a summary table showing the Z-score, required sample size, actual sample size, and per-stratum allocation before writing the output CSV.

Citation

If you use this paper in your research, please cite our paper:

@article{hasan2026model,
  title={Model Context Protocol (MCP) Tool Descriptions Are Smelly! Towards Improving AI Agent Efficiency with Augmented MCP Tool Descriptions},
  author={Hasan, Mohammed Mehedi and Li, Hao and Rajbahadur, Gopi Krishnan and Adams, Bram and Hassan, Ahmed E},
  journal={arXiv preprint arXiv:2602.14878},
  year={2026}
}

As our paper has used MCP-Universe paper, if you want to check anything in MCP-Universe paper, please feel free to cite them too.

@misc{mcpuniverse,
  title={MCP-Universe: Benchmarking Large Language Models with Real-World Model Context Protocol Servers},
  author={Ziyang Luo and Zhiqi Shen and Wenzhuo Yang and Zirui Zhao and Prathyusha Jwalapuram and Amrita Saha and Doyen Sahoo and Silvio Savarese and Caiming Xiong and Junnan Li},
  year={2025},
  eprint={2508.14704},
  archivePrefix={arXiv},
  primaryClass={cs.AI},
  url={https://arxiv.org/abs/2508.14704}, 
}