Research: Vercel bot detection integration options

[caffeinum]

Summary

Investigated whether we can integrate with Vercel's bot detection ecosystem to improve our AI agent classification.

Findings

Vercel BotID (botid npm package)

• Purpose: Block bots (captcha-like protection), not identify them
• Returns: { isBot: boolean } - binary, no agent name
• Limitation: Requires Vercel deployment (proxies to api.vercel.com)
• Verdict: ❌ Wrong tool - we want to track AI agents, not block them

Vercel Bot Management (WAF)

• Platform-integrated, no standalone library
• Detection happens at Vercel's edge via headers
• Can't call from Cloudflare Worker
• Verdict: ❌ Not usable outside Vercel

bots.fyi (Verified Bots Directory)

• 193 bots with categories: AI Crawler, AI Training, AI Assistant, etc.
• Has AI-specific bots: ClaudeBot, GPTBot, ChatGPT-User, Claude-User, Perplexity, etc.
• No programmatic export: No API, no JSON endpoint, no public repo
• Checked: /api, /bots.json, /.well-known/bots.json - all 404
• Verdict: ⚠️ Good taxonomy but can't sync programmatically

crawler-user-agents (npm package)

• Community-maintained list of generic crawlers
• Mostly traditional bots (Googlebot, Bingbot, etc.)
• Verdict: ❌ Not useful - we don't care about search crawlers

Current State

Our api/detect.ts already has:

• Custom AI agent detection via Accept: text/markdown header
• Specific agent identification: claude-code, opencode, codex
• Bot pattern matching for ~50 crawlers
• Categories: bot, browsing-agent, coding-agent, human

bots.fyi Taxonomy (for reference)

bots.fyi category	maps to our category
AI Crawler + AI Training	bot (filtered)
AI Assistant (User Initiated)	browsing-agent
coding agents	coding-agent (our custom detection)

Recommendation

Keep current approach in detect.ts. Our Accept header heuristics for coding agents are unique and better than what any external service provides.

Future consideration: If bots.fyi ever releases an API, could sync their AI bot patterns.

Related Links

• https://vercel.com/docs/botid/get-started
• https://vercel.com/docs/bot-management
• https://bots.fyi