Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
37 changes: 37 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -299,6 +299,43 @@ DocsKit.configure { |c| c.page_markdown_action = false }
to your `get "docs/:doc"` route) to enable the `.md` URLs. Sites that don't
re-run simply have no `.md` route match — HTML rendering is untouched.

## AI-readable docs (llms.txt)

Every site serves the two [llmstxt.org](https://llmstxt.org) artifacts, built
from the **registry** with zero authoring:

```bash
curl https://your-docs.example/llms.txt # the index
curl https://your-docs.example/llms-full.txt # every page, concatenated
```

`/llms.txt` is the index an agent fetches first: an H1 brand, an optional
one-line summary blockquote, one `##` section per nav group, and a
`- [Title](…/page.md)` link to each authored page's Markdown twin. `/llms-full.txt`
concatenates every page's Markdown (the same twin as `.md`) into one document,
separated by `---`. Both are `text/plain`, HTTP-cached (they revalidate on the
registry's content plus the gem version), and derived from the same registry the
sidebar uses — an unwritten page never appears, so there are no dead links.

Set the summary blockquote with the `tagline` knob (default `nil` → the line is
omitted):

```ruby
DocsKit.configure { |c| c.tagline = "The one-line description agents see." }
```

The controller ships in the gem (`DocsKit::LlmsController`); the **routes live in
your app** so you keep full control over path, auth, and omission. The install
generator scaffolds them:

```ruby
get "/llms.txt" => "docs_kit/llms#index", as: :llms
get "/llms-full.txt" => "docs_kit/llms#full", as: :llms_full
```

**Existing sites:** re-run `bin/rails g docs_kit:install` (it adds the two routes
idempotently), or paste the two lines above into `config/routes.rb`.

## API docs — one request, every client tab

An endpoint example is a request shown in several clients (curl, JavaScript,
Expand Down
76 changes: 76 additions & 0 deletions app/controllers/docs_kit/llms_controller.rb
Original file line number Diff line number Diff line change
@@ -0,0 +1,76 @@
# frozen_string_literal: true

module DocsKit
# Serves the two AI-readable artifacts (llmstxt.org) from the registry, with
# zero authoring — the host app wires the routes (the engine is glue-only, no
# routes of its own), so a site keeps full control over path, auth, and
# omission:
#
# # config/routes.rb
# get "/llms.txt" => "docs_kit/llms#index"
# get "/llms-full.txt" => "docs_kit/llms#full"
#
# #index → the llms.txt index (brand, tagline, nav-grouped links to each page's
# `.md` twin). #full → llms-full.txt (every page's Markdown concatenated). Both
# are text/plain and HTTP-cached: the response revalidates on the registry's
# own content plus DocsKit::VERSION, so a page/gem change busts the cache while
# an unchanged registry serves a 304.
#
# All the text shaping lives in DocsKit::LlmsText (pure, Rails-free). This
# controller only threads the Rails view context: #full renders each page to
# Markdown via DocsKit::MarkdownExport (which needs url helpers/CSRF), then
# hands the [title, markdown] pairs to LlmsText.full.
class LlmsController < ActionController::Base
# #full renders each page's full HTML through this controller's view context
# (DocsKit::MarkdownExport), and DocsUI::Shell's <head> calls csrf_meta_tags —
# which needs protect_against_forgery? registered as a view helper. A gem's
# bare ActionController::Base subclass doesn't inherit the host app's
# default_protect_from_forgery, so declare it here. :null_session fits these
# GET-only, sessionless, public text endpoints (no token to verify).
protect_from_forgery with: :null_session

def index
body = DocsKit::LlmsText.index(docs_config, base_url: request.base_url)
render_text(body) if stale_llms?(body)
end

def full
pairs = DocsKit::LlmsText.pages(docs_config).map do |page|
[page.title, render_page_markdown(page)]
end
body = DocsKit::LlmsText.full(docs_config, pairs)
render_text(body) if stale_llms?(body)
end

private

# NOT named #config — ActionController::Base#config is the Rails config
# object, and RequestForgeryProtection delegates allow_forgery_protection/
# csrf_token_storage_strategy to it (`delegate ..., to: :config`). Shadowing
# #config with DocsKit.configuration would route those to the wrong object and
# blow up csrf_meta_tags when #full renders a page's <head>.
def docs_config = DocsKit.configuration

# text/plain (llms.txt is plain text, not markdown — agent tooling fetches it
# as-is). UTF-8 because page titles/taglines may carry non-ASCII.
def render_text(body)
render plain: body, content_type: "text/plain; charset=utf-8"
end

# Revalidate on the rendered body itself (so any registry/config/page change
# busts it) plus the gem version as the etag salt. In development, always
# re-render; production sites deploy immutably so the version etag is stable.
def stale_llms?(body)
stale?(etag: [DocsKit::VERSION, body], public: true)
end

# A page's Markdown twin, rendered through this controller's view context so
# url helpers/CSRF resolve and relative links absolutize to portable URLs —
# the same path DocsKit::Controller#render_page takes for a `.md` request.
def render_page_markdown(page)
DocsKit::MarkdownExport.new(
page.view_class.new, view_context:, base_url: request.base_url
).to_md
end
end
end
3 changes: 3 additions & 0 deletions docs/config/initializers/docs_kit.rb
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,9 @@
DocsKit.configure do |c|
c.brand = "docs-kit"
c.title_suffix = "docs-kit"
# The one-line summary in /llms.txt (the llmstxt.org blockquote). Default nil
# omits the line; set it so AI agents get a crisp description of the site.
c.tagline = "Shared Phlex/daisyUI chrome for documentation sites — one shell, sidebar, code kit, and page kit across every docs site."
c.themes = %w[dark light synthwave retro cyberpunk dracula night nord sunset]

# Code blocks carry a light theme by default and swap to a dark theme when the
Expand Down
8 changes: 7 additions & 1 deletion docs/config/routes.rb
Original file line number Diff line number Diff line change
@@ -1,6 +1,12 @@
Rails.application.routes.draw do
root "landings#show"
get "docs/:doc" => "docs#show", as: :doc
get "docs/:doc(.:format)" => "docs#show", as: :doc

# AI-readable docs (llmstxt.org) — served from the registry by the gem's
# DocsKit::LlmsController, zero authoring. /llms.txt is the index; /llms-full.txt
# concatenates every page's Markdown twin.
get "/llms.txt" => "docs_kit/llms#index", as: :llms
get "/llms-full.txt" => "docs_kit/llms#full", as: :llms_full
# Define your application routes per the DSL in https://guides.rubyonrails.org/routing.html

# Reveal health status on /up that returns 200 if the app boots with no exceptions, otherwise 500.
Expand Down
7 changes: 7 additions & 0 deletions lib/docs_kit/configuration.rb
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,12 @@ class Configuration
# The brand text shown in the topbar and sidebar header.
attr_accessor :brand

# A one-line site summary, rendered as the llms.txt blockquote
# (`> {tagline}`) under the H1. Defaults to nil → the blockquote line is
# omitted, so a site that never sets it still gets a valid llms.txt. Purely
# for the AI-readable index (DocsKit::LlmsText); the chrome never shows it.
attr_accessor :tagline

# The href the topbar brand link points at. Defaults to "/" (site root). A
# site whose docs live under a subpath sets its own (e.g. "/docs") so the
# brand link is a one-line config change, not a Shell subclass.
Expand Down Expand Up @@ -165,6 +171,7 @@ class Configuration

def initialize
@brand = "Docs"
@tagline = nil
@brand_href = "/"
@title_suffix = nil
@themes = %w[dark light]
Expand Down
85 changes: 85 additions & 0 deletions lib/docs_kit/llms_text.rb
Original file line number Diff line number Diff line change
@@ -0,0 +1,85 @@
# frozen_string_literal: true

module DocsKit
# Builds the two AI-readable artifacts a docs-kit site serves, straight from
# the registry — zero authoring:
#
# /llms.txt — the llmstxt.org index: H1 brand, `> tagline` blockquote,
# one `## {group}` section per nav group, and a
# `- [title](abs .md url)` line per authored page.
# /llms-full.txt — every page's Markdown twin concatenated, `# {title}` +
# body, separated by `---`.
#
# It's a pure text builder: given a DocsKit::Configuration and (for the full
# form) already-rendered `[title, markdown]` pairs, it produces strings with no
# Rails. The controller owns the Rails view context — it renders each page to
# Markdown (DocsKit::MarkdownExport) and hands the pairs to .full — so all the
# shaping is unit-testable without booting Rails.
#
# The enumeration source is DocsKit::Registry v2: each registry in
# `config.nav_registries` responds to #nav_items ({ group => [NavItem] },
# authored pages only) for the index and #all (entries with #view_class) for
# the authored page list. An unwritten page (no resolvable view_class) is
# excluded from both, so neither artifact ever links or concatenates a page
# that doesn't exist yet.
module LlmsText
module_function

# The llms.txt index string. base_url absolutizes each page's `.md` href so
# agent tooling fetches a portable URL; omit it for relative links.
#
# Blocks (H1, the tagline blockquote, and one per section) are separated by a
# blank line; within a section the `## heading` and its `- [..]` bullets are a
# single tight list (no blank lines between bullets), per the llmstxt.org
# convention.
def index(config, base_url: nil)
blocks = ["# #{config.brand}"]
tagline = config.tagline
blocks << "> #{tagline}" if tagline && !tagline.to_s.empty?

groups(config).each do |group, links|
section = ["## #{group}", *links.map { |link| link_line(link, base_url) }]
blocks << section.join("\n")
end

blocks.join("\n\n")
end

# The authored pages across every registry, in config/registry order — each
# responds to #title / #href / #view_class. The controller renders these to
# Markdown for .full.
def pages(config)
config.nav_registries.values.flat_map { |registry| registry.all.select(&:view_class) }
end

# The llms-full.txt body: each [title, markdown] pair as `# {title}` + body,
# separated by a `---` rule. Empty pairs → "".
def full(_config, title_markdown_pairs)
title_markdown_pairs.map { |title, markdown| "# #{title}\n\n#{markdown}" }.join("\n\n---\n\n")
end

# { group => [links] } across every registry's #nav_items, in config order.
# A registry with no authored pages contributes nothing, so no empty section
# is ever emitted.
def groups(config)
config.nav_registries.values.each_with_object({}) do |registry, acc|
registry.nav_items.each { |group, links| (acc[group] ||= []).concat(links) }
end
end

# `- [label](absolute .md url)`. The `.md` suffix targets the page's Markdown
# twin (DocsKit::Controller#render_page).
def link_line(link, base_url)
"- [#{link.label}](#{md_url(link.href, base_url)})"
end

# href + ".md", absolutized against base_url when given. base_url has no
# trailing slash concerns here (hrefs are root-relative like "/docs/x").
def md_url(href, base_url)
path = "#{href}.md"
return path unless base_url

"#{base_url.chomp('/')}#{path}"
end
end
end
7 changes: 7 additions & 0 deletions lib/generators/docs_kit/install/install_generator.rb
Original file line number Diff line number Diff line change
Expand Up @@ -72,6 +72,13 @@ def add_routes
# pin html and defeat the .md route.
route %(get "docs/:doc(.:format)" => "docs#show", as: :doc)
route %(root "landings#show")

# AI-readable docs (llmstxt.org), served from the registry by the gem's
# DocsKit::LlmsController — zero authoring. /llms.txt is the index;
# /llms-full.txt concatenates every page's Markdown twin. Thor's `route`
# skips a line already present, so re-running the generator is idempotent.
route %(get "/llms.txt" => "docs_kit/llms#index", as: :llms)
route %(get "/llms-full.txt" => "docs_kit/llms#full", as: :llms_full)
end

def create_css_build
Expand Down
4 changes: 4 additions & 0 deletions lib/generators/docs_kit/install/templates/docs_kit.rb.erb
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,10 @@ Rails.application.config.to_prepare do
c.themes = %w[dark light synthwave retro cyberpunk dracula night nord sunset]
c.code_theme = "Rouge::Themes::Monokai"

# The one-line summary in /llms.txt (the llmstxt.org blockquote agents read
# first). Default nil omits the line; set it to describe your docs:
# c.tagline = "What this documentation covers, in one sentence."

# The topbar brand link points at "/" by default. Point it elsewhere if your
# docs live under a subpath:
# c.brand_href = "/docs"
Expand Down
12 changes: 12 additions & 0 deletions spec/docs_kit/configuration_spec.rb
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,18 @@
end
end

describe "#tagline" do
it "defaults to nil (the llms.txt blockquote line is omitted)" do
expect(described_class.new.tagline).to be_nil
end

it "is overridable so a site sets its llms.txt summary" do
DocsKit.configure { |c| c.tagline = "The shared Phlex chrome for docs sites." }

expect(DocsKit.configuration.tagline).to eq("The shared Phlex chrome for docs sites.")
end
end

describe "#page_markdown_action" do
it "defaults to true (every page shows the 'Markdown' affordance)" do
expect(described_class.new.page_markdown_action).to be(true)
Expand Down
56 changes: 56 additions & 0 deletions spec/docs_kit/llms_controller_spec.rb
Original file line number Diff line number Diff line change
@@ -0,0 +1,56 @@
# frozen_string_literal: true

# The controller subclasses ActionController::Base, so it can't load in the
# standalone suite (no Rails request stack). Its text shaping is covered by
# spec/docs_kit/llms_text_spec.rb; here we prove the SHIPPED FILE is where Rails
# will autoload DocsKit::LlmsController from, and that the thin controller wires
# the builder + the Rails seams the way #full needs. The end-to-end constant
# load + request behavior is dogfooded against the docs/ app (see the PR).
# rubocop:disable RSpec/DescribeClass -- the class is Rails-only, can't constantize here
RSpec.describe "DocsKit::LlmsController (source wiring)" do
# app/controllers/docs_kit/llms_controller.rb → DocsKit::LlmsController under
# Rails' default inflector (docs_kit → DocsKit, llms_controller → LlmsController).
# The engine opts app/ out of the gem's zeitwerk loader (the superclass is
# Rails-only), so Rails' own autoloader owns this constant from this path.
let(:path) do
File.expand_path("../../app/controllers/docs_kit/llms_controller.rb", __dir__)
end
let(:source) { File.read(path) }

it "ships at the path Rails autoloads DocsKit::LlmsController from" do
expect(File.exist?(path)).to be(true)
end

it "declares DocsKit::LlmsController < ActionController::Base" do
expect(source).to include("module DocsKit")
expect(source).to include("class LlmsController < ActionController::Base")
end

it "exposes the two llmstxt actions" do
expect(source).to match(/def index\b/)
expect(source).to match(/def full\b/)
end

it "renders text/plain; charset=utf-8 (llms.txt is plain text, not markdown)" do
expect(source).to include('content_type: "text/plain; charset=utf-8"')
end

it "builds every artifact through the pure DocsKit::LlmsText builder" do
expect(source).to include("DocsKit::LlmsText.index")
expect(source).to include("DocsKit::LlmsText.pages")
expect(source).to include("DocsKit::LlmsText.full")
end

it "HTTP-caches on the gem version + body via stale?/etag" do
expect(source).to include("stale?(etag: [DocsKit::VERSION, body]")
end

it "does not shadow ActionController::Base#config (forgery delegates to it)" do
# RequestForgeryProtection delegates allow_forgery_protection to #config, so
# a `def config` on the controller breaks csrf_meta_tags when #full renders a
# page's <head>. Guard the regression: the DocsKit config reader is #docs_config.
expect(source).not_to match(/^\s*def config\b/)
expect(source).to include("def docs_config = DocsKit.configuration")
end
end
# rubocop:enable RSpec/DescribeClass
Loading
Loading