Skip to content

fix: serve spec-compliant .html.md versions of rendered pages#134

Open
keplervital wants to merge 1 commit into
AnswerDotAI:mainfrom
keplervital:kepler/spec-compliant-html-md-urls
Open

fix: serve spec-compliant .html.md versions of rendered pages#134
keplervital wants to merge 1 commit into
AnswerDotAI:mainfrom
keplervital:kepler/spec-compliant-html-md-urls

Conversation

@keplervital

@keplervital keplervital commented Jul 2, 2026

Copy link
Copy Markdown

Fixes #135.

The proposal on the homepage says:

pages on websites that have information that might be useful for LLMs to read provide a clean markdown version of those pages at the same URL as the original page, but with .md appended. (URLs without file names should append index.html.md instead.)

llmstxt.org itself only follows this for the notebook-built pages. The others 404:

Page Expected URL Today
/ /index.html.md 404 (/index.md exists)
/nbdev.html /nbdev.html.md 404 (/nbdev.md exists)
/domains.html /domains.html.md 404 (/domains-commonmark.md exists)
/ed.html /ed.html.md 404 (/ed-commonmark.md exists)
/intro.html /intro.html.md 200
/core.html /core.html.md 200

The .html.md files only appear for executed notebooks (via keep-md). The qmd and md pages just get the commonmark render, named {stem}.md, or {stem}-commonmark.md when the source itself is a .md file.

This PR adds a small Quarto post-render script that copies each generated markdown output to its .html.md name, plus the one line in _quarto.yml to enable it.

A few deliberate choices:

  • Nothing is removed or renamed. I noticed llms.txt was pointed at /index.md on purpose back in 0536f0c, so /index.md and /ed-commonmark.md keep working exactly as before. The script only adds the .html.md counterparts next to them.
  • The script skips files that already exist, so the richer keep-md versions of intro.html.md and core.html.md are never overwritten.
  • It is a .ts file rather than Python because nbdev_docs copies sources through _proc before rendering, and that copy drops _-prefixed files and strips the .py extension from Python files. Quarto runs .ts post-render scripts with its bundled Deno, so there are no new dependencies.

Verified locally with nbdev-docs (same as the deploy workflow): _docs now contains index.html.md, nbdev.html.md, domains.html.md and ed.html.md, each byte-identical to its .md source, and all previously working URLs still return 200 when serving _docs.

Known limitation: llmstxt-js.html is a raw HTML resource that never goes through the renderer, so it still has no markdown version.

The proposal says markdown versions of pages live at the page URL with
.md appended, and that URLs without a file name should use index.html.md.
The notebook-built pages (intro, core) already comply via keep-md, but
index, nbdev, domains and ed only get a commonmark render named {stem}.md
(or {stem}-commonmark.md when the source itself is a .md file), so
index.html.md, nbdev.html.md, domains.html.md and ed.html.md were 404.

Add a Quarto post-render script that copies each generated markdown
output to its .html.md name. Existing URLs such as index.md and
ed-commonmark.md are kept as is.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

llmstxt.org does not serve the spec's .html.md URLs for its own pages

1 participant