Implement RFC 0008: Language Reference document, packaging, and export (R-34/R-35/R-36)

CHANGELOG.md

@@ -6,6 +6,55 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/).
 
 ## [Unreleased]
 
+### Added
+
+- **Template Language Reference (RFC 0008).** New hand-written `transon/resources/LANGUAGE.md` —
+  the author-facing, cross-cutting language semantics (evaluation model, scoping,
+  the `NO_CONTENT` model, error taxonomy, `expr`/`call` machinery, composition
+  patterns; **no per-entity sections**) — canonical **and** packaged in one file
+  (ships as-is in the wheel + sdist; `docs/LANGUAGE.md` is a pointer) and served by a new versioned
+  export `transon.reference.get_language_reference()` (`REFERENCE_VERSION` `1.0`;
+  `{reference_version, engine_version, format, content, sections}` with a
+  deterministic flat `##`-heading split, stable slug ids, and sections-concatenation
+  parity) plus a `python -m transon.reference` CLI. Section ids are pinned in
+  `tests/test_reference.py`; packaging parity is tested via `importlib.resources`.
+  (Roadmap R-34, R-35, R-36)
+
+### Fixed
+
+- **`map` `items` mode validates its result shape.** An `items` template that
+  evaluates to a non-list (a dict, a string, a scalar) now raises a located
+  `DefinitionError` (``` `items` must evaluate to a list for `map` rule ```)
+  instead of accidentally iterating dict keys / string characters or leaking a
+  raw `TypeError`. Templates relying on the accidental iteration must wrap the
+  value in a list. (Found in review of Roadmap R-34)
+- **`transform(..., copy_output=True)` preserves `NO_CONTENT` identity.** When
+  the caller opts into the raw sentinel (`no_content=Transformer.NO_CONTENT`),
+  the result is no longer routed through `copy.deepcopy`, which used to return a
+  fresh `NoContent` instance and break `result is Transformer.NO_CONTENT`; a
+  caller-supplied `no_content` substitute is likewise returned as-is (it cannot
+  alias the input, so there is nothing for `copy_output` to protect). `NoContent`
+  also defines `__copy__`/`__deepcopy__` returning itself, so a sentinel kept
+  *inside* a copied container (e.g. a literal template list holding a missing
+  lookup) preserves identity too.
+
+### Changed
+
+- **`get_all_docs()['doc']` content (docs-site coordination; shape unchanged).**
+  The `Transformer` class docstring — exported as the `doc` field and rendered by the
+  docs site — is consolidated per RFC 0008's ownership principle: its language
+  sections ("Templates", "How evaluation works", the language half of "What you can
+  do") moved into the Language Reference (`transon/resources/LANGUAGE.md`); the pitch/install/comparison sections are owned
+  solely by `README.md`; what remains is the embedder-facing narrative (Python API
+  usage + extending). Symmetrically, per-rule docstrings **grew richer**: spec §4's
+  per-rule facts (edge cases, `NO_CONTENT` treatment, error conditions) folded into
+  the registration docs, so per-rule doc content in `get_all_docs()` and
+  `get_editor_metadata()['docs']` is longer (doc text is contractually opaque — no
+  shape change). `SPECIFICATION.md` deliberately retains its full §2/§4/§11 statement
+  (the engine contract stays one complete document; the duplication is banner-flagged
+  in-document and aligned by review). Docs-site counterpart work is D-20.
+  (Roadmap R-34)
+
 ## [0.1.8] - 2026-07-16
 
 ### Added

docs/DOCS_SITE_ROADMAP.md

@@ -2,7 +2,7 @@
 
 > **Status of this document**: living backlog for the **content** of the docs site /
 > playground at <https://transon-org.github.io/> (separate from the engine roadmap in
-> [`docs/ROADMAP.md`](ROADMAP.md), which tracks engine semantics `R-01…R-22`). Every
+> [`docs/ROADMAP.md`](ROADMAP.md), which tracks engine semantics as `R-xx` items). Every
 > entry follows the same format: problem → impact of not fixing → options (with a
 > recommendation when one is clearly better).
 >
@@ -41,6 +41,7 @@
 | D-04 | Broken JSON in the headline intro example | A. Correctness | high | done |
 | D-06 | Accessor docstrings omit `filter` scope | A. Correctness | medium | done |
 | D-05 | Intro is stale vs. v0.0.11 capabilities | B. Freshness | high | done |
+| D-20 | Migrate to the engine Language Reference (RFC 0008 release) | B. Freshness | medium | accepted |
 | D-11 | `file` rule has zero examples | C. Completeness | high | done |
 | D-12 | Parameters rendered with no example | C. Completeness | medium | done |
 | D-08 | Operators and functions are not discoverable | C. Completeness | medium | done |
@@ -199,6 +200,27 @@ the project today.
 3. Add a dedicated "Features"/"Highlights" section sourced from a new docstring or a
    new corpus-backed block, leaving the existing intro intact.
 
+### D-20. Migrate to the engine Language Reference (RFC 0008 release)
+
+**Status**: accepted · **Severity**: medium ·
+**Source**: engine [`proposals/0008-language-reference-export.md`](proposals/0008-language-reference-export.md) (Sequencing) — **hard dependency of that engine release (atomic)**
+
+The engine's RFC 0008 release relocates content the site renders today: the `Transformer`
+class docstring (`get_all_docs()['doc']`) shrinks to an embedder-facing narrative (its language
+sections move to the new Language Reference (`transon/resources/LANGUAGE.md`), its pitch to `README.md`), and rule docstrings
+grow richer (spec §4's per-rule facts fold in). In the same release window the site must:
+
+1. render `LANGUAGE.md` as a language-guide section/page (source: `get_language_reference()`
+   sections or the packaged file);
+2. build its landing/pitch from `README.md` at build time instead of the class docstring;
+3. repurpose the slimmed `doc` field as an "Embedding" (Python API) page.
+
+Per-rule pages need no site work — they render whatever docstring text arrives, now richer.
+
+**Impact if not fixed**: shipping against the new engine release without this leaves a slim
+embedder intro as the landing and no language narrative anywhere on the site — a regression of
+D-05/D-10/D-18.
+
 ---
 
 ## Theme C — Completeness (gaps in what the page documents)

docs/LANGUAGE.md

@@ -0,0 +1,9 @@
+# Transon — Template Language Reference (pointer)
+
+The Language Reference lives at
+[`transon/resources/LANGUAGE.md`](../transon/resources/LANGUAGE.md) — canonical,
+hand-edited, and packaged, so the file that ships in the wheel/sdist **is** the file
+you edit (single copy, no sync step; same rule as the per-rule docs living in
+`transon/rules.py`). It is served offline by
+`transon.reference.get_language_reference()` (`python -m transon.reference`) and
+rendered on the [docs site](https://transon-org.github.io/).

docs/ROADMAP.md

@@ -62,6 +62,9 @@
 | [R-31](#r-31-normalize-exports-to-one-flat-example-corpus-name-references) | Normalize exports to one flat example corpus (name references) | medium | done |
 | [R-32](#r-32-bounded-per-level-recursion-budget-for-self-include-walks) | Bounded per-level recursion budget for self-`include` walks | medium | done |
 | [R-33](#r-33-grow-the-built-in-function-library) | Grow the built-in function library (string / numeric / collection helpers) | medium | done |
+| [R-34](#r-34-language-reference-document-languagemd) | Language Reference document (`LANGUAGE.md`) | medium | done |
+| [R-35](#r-35-package-the-language-reference-as-package-data) | Package the Language Reference as package data | low | done |
+| [R-36](#r-36-get_language_reference-versioned-export) | `get_language_reference()` versioned export | medium | done |
 
 ---
 
@@ -1022,6 +1025,96 @@ failures to `TransformationError`); `split` rule in `transon/rules.py`; total `i
 
 ---
 
+## Theme G — Author-facing Language Reference (RFC 0008)
+
+> Engine-side counterpart of the `transon-authoring` authority-ladder gap (skill contract
+> rung 2): no author-facing, pinnable, offline-servable language document exists. Design and
+> the full ownership principle (structure in the catalog, per-entity behavior in registration
+> docs, cross-cutting semantics in `LANGUAGE.md`) are recorded in
+> [`proposals/0008-language-reference-export.md`](proposals/0008-language-reference-export.md);
+> accepted 2026-07-18. R-34/R-35/R-36 plus the docstring/README consolidation ship
+> **atomically in one release** (RFC Sequencing); the docs-site counterpart is D-20 in
+> [`DOCS_SITE_ROADMAP.md`](DOCS_SITE_ROADMAP.md).
+
+### R-34. Language Reference document (`LANGUAGE.md`)
+
+**Status**: done · **Severity**: medium ·
+**Source**: [`proposals/0008-language-reference-export.md`](proposals/0008-language-reference-export.md) (Deliverable 1)
+
+A new hand-written Language Reference (`transon/resources/LANGUAGE.md` — canonical and
+packaged in one file; `docs/LANGUAGE.md` is a pointer): the reference for authors (human or
+agent), carrying **cross-cutting semantics only** — the marker, context/scoping, the
+`NO_CONTENT` propagation model, the error taxonomy, `expr`/`call` machinery, composition
+patterns. **No per-entity sections**: per-rule/operator/function prose stays in the
+registration docs, so the document changes only when the language model changes, not when the
+catalog grows. Assembled by relocation, never duplication: spec §2/§11 cross-cutting content
+moves in; spec §4's per-rule facts move into the rule docstrings (which grow richer); the
+`Transformer` class docstring shrinks to embedder-facing content; `README.md` becomes the sole
+owner of the pitch. Drift protection: a pinned section-id test (no catalog-coverage check).
+
+**Impact if not done**: the cross-cutting narrative stays scattered across three
+hand-maintained copies (spec §2/§11, the class docstring, README overlap) with no
+author-scoped, pinnable document — the `transon-authoring` authority-ladder gap stays open and
+repair loops keep rediscovering semantics.
+
+**Shipped**: `transon/resources/LANGUAGE.md` (7 pinned sections: preamble, templates-and-the-marker,
+context-and-scoping, the-no_content-model, error-model, expressions-and-calls,
+composition-patterns). Consolidation: spec §4's per-rule facts folded into
+`transon/rules.py` docstrings (accessors' scope errors, `attr` error split, `map`/
+`filter`/`zip`/`join`/`file` edge cases, `expr`/`call` mode errors + the sanctioned
+reference pointer); the spec **retains its full §2/§4/§11 statement** — deliberate,
+banner-flagged duplication (decision 2026-07-18) so the engine contract stays one
+complete document; `Transformer` class docstring slimmed to the embedder narrative
+(pitch owned by README); section-id pin in `tests/test_reference.py`. Changelog entry
+under Unreleased.
+
+### R-35. Package the Language Reference as package data
+
+**Status**: done · **Severity**: low ·
+**Source**: [`proposals/0008-language-reference-export.md`](proposals/0008-language-reference-export.md) (Deliverable 2)
+
+Ship `LANGUAGE.md` in the wheel and sdist (e.g. `transon/resources/LANGUAGE.md`) so an
+installed `transon==<version>` serves its own language reference offline — the property
+`get_editor_metadata()` already has for the catalog. Single-copy refinement: the packaged file
+**is** the canonical, hand-edited source (`docs/LANGUAGE.md` is a pointer) — no build mapping,
+no mirror to sync. Acceptance: an `importlib.resources` test asserts the packaged bytes (UTF-8,
+line-endings normalized to `\n`) equal `get_language_reference()['content']`.
+
+**Impact if not done**: the `transon-authoring` harnesses mount no repo checkout, so an
+unpackaged reference is invisible to the primary consumer.
+
+**Shipped**: `transon/resources/LANGUAGE.md` is the canonical, hand-edited, single copy;
+hatchling picks it up in both wheel and sdist with no config change — verified by building
+both. `tests/test_reference.py` loads it through `importlib.resources` and asserts it equals
+`get_language_reference()['content']`. `docs/LANGUAGE.md` reduced to a pointer.
+
+### R-36. `get_language_reference()` versioned export
+
+**Status**: done · **Severity**: medium ·
+**Source**: [`proposals/0008-language-reference-export.md`](proposals/0008-language-reference-export.md) (Deliverable 3)
+
+`transon.reference.get_language_reference()` → `{reference_version, engine_version, format,
+content, sections}` with deterministic flat `##`-heading splitting (stable slug ids, preamble
+rule, sections-concatenation parity with `content`), a `METADATA_VERSION`-style version policy
+(minor = additive, major = breaking; consumers fail their drift check loudly on an unsupported
+major), and a `python -m transon.reference` CLI. Engine-global (base `Transformer` only, no
+`cls=` parameter); language facts only, no consumer-specific shapes. `transon-authoring` then
+pins/syncs/drift-checks it like the metadata snapshot (that half lives in its repo, out of
+scope here).
+
+**Impact if not done**: consumers can only ship the raw file — no targeted section lookup, no
+version pin, no drift check; a 700-line context dump instead of one section as the unit of
+consumption.
+
+**Shipped**: `transon/reference.py::get_language_reference()` — `REFERENCE_VERSION`
+`'1.0'`, fence-aware deterministic `##` splitting with GitHub-style slug ids and
+collision suffixes, preamble rule, engine-version degradation to `None` when not
+installed; `python -m transon.reference` CLI. Spec §5.2 documents the export. Tests:
+shape, section pin, concatenation parity, splitter unit cases in
+`tests/test_reference.py`.
+
+---
+
 ## Suggested sequencing
 
 1. **No-decision fixes** (can start immediately): ~~R-03~~ (done), ~~R-18~~ (done), ~~R-19~~ (done), ~~R-21~~ (done).