feat: serve raw markdown at .md URLs for AI agents#4572
Open
lennessyy wants to merge 11 commits into
Open
Conversation
Adds a Docusaurus plugin that copies raw markdown source files to the build output at their corresponding URL paths with a .md extension. This lets AI agents fetch individual doc pages as markdown by appending .md to any docs URL. Pages can opt out by setting llm_exclude in frontmatter to a string explaining why, which is served in place of the raw content. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
Contributor
📖 Docs PR preview links
|
Contributor
|
This looks fantastic! Great addition; thanks. |
encyclopedia/index.mdx now outputs encyclopedia.md instead of encyclopedia/index.md, matching how Docusaurus routes index pages. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Tells agents in llms.txt that they can append .md to any page URL to fetch raw Markdown source. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Adds Vercel rewrites that serve raw markdown when a request includes Accept: text/markdown. Agents can request any docs URL with this header and get the .md source instead of the rendered HTML page. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Replace vercel.json rewrites (which run after static file matching) with Vercel Routing Middleware that runs before. When a request includes Accept: text/markdown, the middleware rewrites to the .md file. Otherwise it passes through to the normal static HTML. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Set framework: null with explicit build config so Vercel treats this as a generic project that supports Routing Middleware, while still building Docusaurus correctly. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Try .js extension for middleware detection. Revert framework: null since the Docusaurus preset shouldn't block middleware. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
markdown-pagesDocusaurus plugin that copies raw.mdx/.mdsource files to the build output at their corresponding URL paths with a.mdextension (e.g.,docs.temporal.io/security.mdserves the raw markdown for the security page).slugandidfrontmatter overrides for URL path resolution.llm_excludein frontmatter to a string explaining why the page isn't available as markdown. Agents receive that explanation instead of a 404.Test plan
docusaurus buildsucceeds, generating 583 markdown files.mdfiles are placed at correct URL paths (including slug overrides)docusaurus servereturns raw markdown at.mdURLsllm_excludefrontmatter on a page (e.g., the serverless demo page)🤖 Generated with Claude Code
┆Attachments: EDU-6365 feat: serve raw markdown at .md URLs for AI agents