Tool Example Request
TheCrawler would be a useful example MCP tool for mcp-agent workflows that need web data but also need programmatic failure handling.
What it does:
- URL → markdown via
crawl_markdown
- Full crawl output via
crawl with metadata, links, images, JSON-LD, microdata, commerce/forms/analytics signals, hreflang, pagination, and redirect chain
- PDF/DOCX parsing
- Structured errors (
blocked-bot, timeout, http-4xx, js-required, etc.) so agents can branch instead of trusting empty output
- Optional schema-backed structured extraction through OpenAI-compatible LLM endpoints
- Current GitHub source includes validated extraction contract diagnostics for checking whether a URL set is extract-ready
Quick start from GitHub source:
git clone https://github.com/manchittlab/TheCrawler.git
cd TheCrawler/engine
npm install
npm run build
node dist/mcp.js
MCP engine path: https://github.com/manchittlab/TheCrawler/tree/main/engine
License: AGPL-3.0-or-later
Note: npm currently lags the GitHub source, so the GitHub-source path is the current install path for validated-contract diagnostics.
Tool Example Request
TheCrawler would be a useful example MCP tool for
mcp-agentworkflows that need web data but also need programmatic failure handling.What it does:
crawl_markdowncrawlwith metadata, links, images, JSON-LD, microdata, commerce/forms/analytics signals, hreflang, pagination, and redirect chainblocked-bot,timeout,http-4xx,js-required, etc.) so agents can branch instead of trusting empty outputQuick start from GitHub source:
git clone https://github.com/manchittlab/TheCrawler.git cd TheCrawler/engine npm install npm run build node dist/mcp.jsMCP engine path: https://github.com/manchittlab/TheCrawler/tree/main/engine
License: AGPL-3.0-or-later
Note: npm currently lags the GitHub source, so the GitHub-source path is the current install path for validated-contract diagnostics.