Fix audit bugs, verify AI bot classifications against primary sources - #11
Merged
Conversation
categoryFilterSql() only matched exact bot_category values at the SQL level, but legacy rows persisted as 'ai_crawler' (before per-bot categories existed) were excluded — normalizeBotCategory() never got a chance to remap them. The daily trend and other SQL-filtered panels would appear flat/empty when selecting specific AI chips like 'AI agent'. Now ai_agent/ai_search/ai_training filters also match the relevant ai_crawler rows via bot_name, matching the JS-side normalization. Also adds claude-code and Google-NotebookLM to AI_AGENT_BOTS for correct legacy remapping.
…normalized status rows - Add PhindBot, Andibot, Grok-DeepSearch to AI_SEARCH_BOTS so legacy ai_crawler rows for these search bots are correctly remapped. - Add all 36 missing AI-categorized bots to AI_BOT_NAMES_SQL for a complete name-based safety net in AI confidence queries. - Merge normalized bot status rows in fetchStatusBatch to prevent duplicate React keys when legacy ai_crawler rows now correctly appear alongside their ai_agent/ai_search counterparts.
- fillDailyTrend used local-timezone date arithmetic while the server sends UTC-keyed data, shifting the whole trend window by a day for non-UTC users; now delegates to the already-correct, already-tested fillDatePeriods instead of a second, buggy reimplementation. - GoogleAgent pattern only matched the no-hyphen variant; Google's real token is the hyphenated Google-Agent, so every user-triggered-agent request was previously invisible (fell through to null, not even a generic bot). - AI_BOT_NAMES_SQL was a hand-maintained literal list that had drifted out of sync (missing ClaudeBot, Anthropic's own training crawler); now derived from PATTERNS at module load so it can't drift again. - bots-table.tsx row key and overview-view.tsx's AiCrawlsVsVisits used raw/ dead category values instead of normalized ones. - Chart mode switch (total vs. category) caused a layout shift from a height-constrained wrapper only applied in one mode. - Pre-merge SQL LIMITs on (bot_name, bot_category) groupings didn't account for JS-side category normalization collapsing two raw rows into one, silently truncating top-N results; doubled the raw limits to compensate (mirrors the existing LIMIT * 2 pattern in allBotDetails). - queryFiltered()'s category filter used naive equality instead of categoryFilterSql(), so it wouldn't have handled the "ai" pseudo-category or ai_crawler remapping if the events view ever exposed a category filter. - Category-trend view for week/month granularity didn't zero-fill empty periods like the total-trend view does; deselecting all categories still rendered empty chart axes instead of just the message; aggregateDailyCategory mutated map values in place instead of replacing them. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Two research passes: first added 15 missing bots the audit flagged, then a wider web search surfaced 9 more from the ai.robots.txt registry. Re-checking every addition against the operating company's own documentation (rather than third-party aggregators) found real problems, now corrected: - Removed CopilotBot, ExaBot, GoogleAgent-Mariner, Google-Gemini-CLI, Google-Firebase, Cursor, and Trae — no reliable evidence any of these UA tokens actually exist (ExaBot in particular collides with an unrelated, decades-old Exalead crawler; GoogleAgent-Mariner's underlying product, Project Mariner, was discontinued by Google before this was even added). - Recategorized KagiBot, PetalBot, and Google-CloudVertexBot out of the ai_* buckets: each was tagged ai_search/ai_training on "the operator makes AI products elsewhere" reasoning, but their own docs describe them as general search-index or owner-initiated agent-building crawlers, not bots that train models or feed AI-generated answers. - Kept the well-corroborated remainder (GoogleAgent-URLContext, TavilyBot, PanguBot, Ai2Bot-Dolma, Devin, Manus-User, NovaAct, TongyiBot, YiyanBot, ICC-Crawler, Google-GeminiNotebook, kagi-fetcher, KimiBot/Kimi-User, TikTokSpider), each confirmed by the operator's own docs or multiple independent trackers. - Corrected "Manus Bot" (speculative, no evidence) to the verified "Manus-User" token. Added a classification-rule comment at the top of bots.ts codifying the bar for ai_training/ai_search/ai_agent, so this mistake isn't repeated: the operating company's own docs must state THIS SPECIFIC bot's purpose, not just that the company is an AI business. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Builds a realistic synthetic User-Agent for each bots.ts PATTERNS entry and asserts detectBot() resolves it back to that exact name/category. Catches mis-ordered patterns (a more specific token accidentally matched by an earlier, broader pattern) and version/qualifier requirements (e.g. Devin\/\d needing a version suffix) that a name-only glance wouldn't reveal. A small override map covers entries where the display name intentionally differs from the literal UA token the regex matches (e.g. "Slack" vs. "Slackbot-LinkExpanding"). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
GoogleAgentpattern not matching Google's real hyphenatedGoogle-Agenttoken, a hand-maintainedAI_BOT_NAMES_SQLlist that had silently drifted (missingClaudeBot), plus several smaller correctness/UX issues (row-key normalization, dead code, chart layout shift, pre-merge SQL LIMIT truncation, category filter onqueryFiltered, empty-period gaps, empty-selection chart skeleton, a map-mutation bug).ExaBoteven collided with an unrelated Exalead crawler), and recategorized 3 more (KagiBot,PetalBot,Google-CloudVertexBot) that had been tagged AI-search/training on "the company does AI stuff" reasoning rather than the bot's own documented purpose.bots.tsso that mistake isn't repeated.bots-smoke.test.ts) that builds a realistic User-Agent for everyPATTERNSentry and assertsdetectBot()resolves it correctly — catches pattern-ordering and version-qualifier mistakes automatically.Test plan
npx vitest run src— 199/199 passnpx tsc --noEmit— cleannpx eslint src— cleannpm run build— production build succeedsCo-Authored-By: Claude Sonnet 5 noreply@anthropic.com