docs(seo): note IndexNow via Cloudflare - #83
Merged
Conversation
- robots.txt: comment marking the open policy as deliberate so AI crawlers don't get blocked by a future "tidy up" edit - CLAUDE.md: document that IndexNow is handled by Cloudflare Crawler Hints (now enabled) — a GEO win since Bing feeds ChatGPT/Copilot — and not to re-implement it in code Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Contributor
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Documents how IndexNow is handled and makes the open robots.txt policy deliberate. No behavioral/code changes — docs + a comment.
public/robots.txt— adds a two-line comment marking the open-crawler policy as intentional, so a future "tidy up" doesn't addDisallowfor AI bots (GPTBot/ClaudeBot/etc.) and quietly hurt GEO.CLAUDE.md(SEO & GEO → "What's already wired") — records that IndexNow is handled at the Cloudflare layer via Crawler Hints (now enabled), why it's a GEO win (Bing's index feeds ChatGPT search & Copilot), and a guardrail not to re-implement it in the repo. Notes that Google doesn't use IndexNow.Why
We're GEO-first. Crawler Hints auto-pings IndexNow on content changes so Bing/Yandex/Naver re-index faster — and faster Bing indexing ≈ faster quotability in ChatGPT/Copilot. Capturing this in
CLAUDE.mdprevents someone re-building the IndexNow key-file + ping pipeline that Cloudflare already covers.🤖 Generated with Claude Code