diff --git a/CHANGELOG.md b/CHANGELOG.md
index 4187b88..3b54e22 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -6,6 +6,55 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/).
## [Unreleased]
+### Added
+
+- **Template Language Reference (RFC 0008).** New hand-written `transon/resources/LANGUAGE.md` —
+ the author-facing, cross-cutting language semantics (evaluation model, scoping,
+ the `NO_CONTENT` model, error taxonomy, `expr`/`call` machinery, composition
+ patterns; **no per-entity sections**) — canonical **and** packaged in one file
+ (ships as-is in the wheel + sdist; `docs/LANGUAGE.md` is a pointer) and served by a new versioned
+ export `transon.reference.get_language_reference()` (`REFERENCE_VERSION` `1.0`;
+ `{reference_version, engine_version, format, content, sections}` with a
+ deterministic flat `##`-heading split, stable slug ids, and sections-concatenation
+ parity) plus a `python -m transon.reference` CLI. Section ids are pinned in
+ `tests/test_reference.py`; packaging parity is tested via `importlib.resources`.
+ (Roadmap R-34, R-35, R-36)
+
+### Fixed
+
+- **`map` `items` mode validates its result shape.** An `items` template that
+ evaluates to a non-list (a dict, a string, a scalar) now raises a located
+ `DefinitionError` (``` `items` must evaluate to a list for `map` rule ```)
+ instead of accidentally iterating dict keys / string characters or leaking a
+ raw `TypeError`. Templates relying on the accidental iteration must wrap the
+ value in a list. (Found in review of Roadmap R-34)
+- **`transform(..., copy_output=True)` preserves `NO_CONTENT` identity.** When
+ the caller opts into the raw sentinel (`no_content=Transformer.NO_CONTENT`),
+ the result is no longer routed through `copy.deepcopy`, which used to return a
+ fresh `NoContent` instance and break `result is Transformer.NO_CONTENT`; a
+ caller-supplied `no_content` substitute is likewise returned as-is (it cannot
+ alias the input, so there is nothing for `copy_output` to protect). `NoContent`
+ also defines `__copy__`/`__deepcopy__` returning itself, so a sentinel kept
+ *inside* a copied container (e.g. a literal template list holding a missing
+ lookup) preserves identity too.
+
+### Changed
+
+- **`get_all_docs()['doc']` content (docs-site coordination; shape unchanged).**
+ The `Transformer` class docstring — exported as the `doc` field and rendered by the
+ docs site — is consolidated per RFC 0008's ownership principle: its language
+ sections ("Templates", "How evaluation works", the language half of "What you can
+ do") moved into the Language Reference (`transon/resources/LANGUAGE.md`); the pitch/install/comparison sections are owned
+ solely by `README.md`; what remains is the embedder-facing narrative (Python API
+ usage + extending). Symmetrically, per-rule docstrings **grew richer**: spec §4's
+ per-rule facts (edge cases, `NO_CONTENT` treatment, error conditions) folded into
+ the registration docs, so per-rule doc content in `get_all_docs()` and
+ `get_editor_metadata()['docs']` is longer (doc text is contractually opaque — no
+ shape change). `SPECIFICATION.md` deliberately retains its full §2/§4/§11 statement
+ (the engine contract stays one complete document; the duplication is banner-flagged
+ in-document and aligned by review). Docs-site counterpart work is D-20.
+ (Roadmap R-34)
+
## [0.1.8] - 2026-07-16
### Added
diff --git a/docs/DOCS_SITE_ROADMAP.md b/docs/DOCS_SITE_ROADMAP.md
index ed1d747..07bf443 100644
--- a/docs/DOCS_SITE_ROADMAP.md
+++ b/docs/DOCS_SITE_ROADMAP.md
@@ -2,7 +2,7 @@
> **Status of this document**: living backlog for the **content** of the docs site /
> playground at (separate from the engine roadmap in
-> [`docs/ROADMAP.md`](ROADMAP.md), which tracks engine semantics `R-01…R-22`). Every
+> [`docs/ROADMAP.md`](ROADMAP.md), which tracks engine semantics as `R-xx` items). Every
> entry follows the same format: problem → impact of not fixing → options (with a
> recommendation when one is clearly better).
>
@@ -41,6 +41,7 @@
| D-04 | Broken JSON in the headline intro example | A. Correctness | high | done |
| D-06 | Accessor docstrings omit `filter` scope | A. Correctness | medium | done |
| D-05 | Intro is stale vs. v0.0.11 capabilities | B. Freshness | high | done |
+| D-20 | Migrate to the engine Language Reference (RFC 0008 release) | B. Freshness | medium | accepted |
| D-11 | `file` rule has zero examples | C. Completeness | high | done |
| D-12 | Parameters rendered with no example | C. Completeness | medium | done |
| D-08 | Operators and functions are not discoverable | C. Completeness | medium | done |
@@ -199,6 +200,27 @@ the project today.
3. Add a dedicated "Features"/"Highlights" section sourced from a new docstring or a
new corpus-backed block, leaving the existing intro intact.
+### D-20. Migrate to the engine Language Reference (RFC 0008 release)
+
+**Status**: accepted · **Severity**: medium ·
+**Source**: engine [`proposals/0008-language-reference-export.md`](proposals/0008-language-reference-export.md) (Sequencing) — **hard dependency of that engine release (atomic)**
+
+The engine's RFC 0008 release relocates content the site renders today: the `Transformer`
+class docstring (`get_all_docs()['doc']`) shrinks to an embedder-facing narrative (its language
+sections move to the new Language Reference (`transon/resources/LANGUAGE.md`), its pitch to `README.md`), and rule docstrings
+grow richer (spec §4's per-rule facts fold in). In the same release window the site must:
+
+1. render `LANGUAGE.md` as a language-guide section/page (source: `get_language_reference()`
+ sections or the packaged file);
+2. build its landing/pitch from `README.md` at build time instead of the class docstring;
+3. repurpose the slimmed `doc` field as an "Embedding" (Python API) page.
+
+Per-rule pages need no site work — they render whatever docstring text arrives, now richer.
+
+**Impact if not fixed**: shipping against the new engine release without this leaves a slim
+embedder intro as the landing and no language narrative anywhere on the site — a regression of
+D-05/D-10/D-18.
+
---
## Theme C — Completeness (gaps in what the page documents)
diff --git a/docs/LANGUAGE.md b/docs/LANGUAGE.md
new file mode 100644
index 0000000..806c4aa
--- /dev/null
+++ b/docs/LANGUAGE.md
@@ -0,0 +1,9 @@
+# Transon — Template Language Reference (pointer)
+
+The Language Reference lives at
+[`transon/resources/LANGUAGE.md`](../transon/resources/LANGUAGE.md) — canonical,
+hand-edited, and packaged, so the file that ships in the wheel/sdist **is** the file
+you edit (single copy, no sync step; same rule as the per-rule docs living in
+`transon/rules.py`). It is served offline by
+`transon.reference.get_language_reference()` (`python -m transon.reference`) and
+rendered on the [docs site](https://transon-org.github.io/).
diff --git a/docs/ROADMAP.md b/docs/ROADMAP.md
index 85d5d44..b142ff6 100644
--- a/docs/ROADMAP.md
+++ b/docs/ROADMAP.md
@@ -62,6 +62,9 @@
| [R-31](#r-31-normalize-exports-to-one-flat-example-corpus-name-references) | Normalize exports to one flat example corpus (name references) | medium | done |
| [R-32](#r-32-bounded-per-level-recursion-budget-for-self-include-walks) | Bounded per-level recursion budget for self-`include` walks | medium | done |
| [R-33](#r-33-grow-the-built-in-function-library) | Grow the built-in function library (string / numeric / collection helpers) | medium | done |
+| [R-34](#r-34-language-reference-document-languagemd) | Language Reference document (`LANGUAGE.md`) | medium | done |
+| [R-35](#r-35-package-the-language-reference-as-package-data) | Package the Language Reference as package data | low | done |
+| [R-36](#r-36-get_language_reference-versioned-export) | `get_language_reference()` versioned export | medium | done |
---
@@ -1022,6 +1025,96 @@ failures to `TransformationError`); `split` rule in `transon/rules.py`; total `i
---
+## Theme G — Author-facing Language Reference (RFC 0008)
+
+> Engine-side counterpart of the `transon-authoring` authority-ladder gap (skill contract
+> rung 2): no author-facing, pinnable, offline-servable language document exists. Design and
+> the full ownership principle (structure in the catalog, per-entity behavior in registration
+> docs, cross-cutting semantics in `LANGUAGE.md`) are recorded in
+> [`proposals/0008-language-reference-export.md`](proposals/0008-language-reference-export.md);
+> accepted 2026-07-18. R-34/R-35/R-36 plus the docstring/README consolidation ship
+> **atomically in one release** (RFC Sequencing); the docs-site counterpart is D-20 in
+> [`DOCS_SITE_ROADMAP.md`](DOCS_SITE_ROADMAP.md).
+
+### R-34. Language Reference document (`LANGUAGE.md`)
+
+**Status**: done · **Severity**: medium ·
+**Source**: [`proposals/0008-language-reference-export.md`](proposals/0008-language-reference-export.md) (Deliverable 1)
+
+A new hand-written Language Reference (`transon/resources/LANGUAGE.md` — canonical and
+packaged in one file; `docs/LANGUAGE.md` is a pointer): the reference for authors (human or
+agent), carrying **cross-cutting semantics only** — the marker, context/scoping, the
+`NO_CONTENT` propagation model, the error taxonomy, `expr`/`call` machinery, composition
+patterns. **No per-entity sections**: per-rule/operator/function prose stays in the
+registration docs, so the document changes only when the language model changes, not when the
+catalog grows. Assembled by relocation, never duplication: spec §2/§11 cross-cutting content
+moves in; spec §4's per-rule facts move into the rule docstrings (which grow richer); the
+`Transformer` class docstring shrinks to embedder-facing content; `README.md` becomes the sole
+owner of the pitch. Drift protection: a pinned section-id test (no catalog-coverage check).
+
+**Impact if not done**: the cross-cutting narrative stays scattered across three
+hand-maintained copies (spec §2/§11, the class docstring, README overlap) with no
+author-scoped, pinnable document — the `transon-authoring` authority-ladder gap stays open and
+repair loops keep rediscovering semantics.
+
+**Shipped**: `transon/resources/LANGUAGE.md` (7 pinned sections: preamble, templates-and-the-marker,
+context-and-scoping, the-no_content-model, error-model, expressions-and-calls,
+composition-patterns). Consolidation: spec §4's per-rule facts folded into
+`transon/rules.py` docstrings (accessors' scope errors, `attr` error split, `map`/
+`filter`/`zip`/`join`/`file` edge cases, `expr`/`call` mode errors + the sanctioned
+reference pointer); the spec **retains its full §2/§4/§11 statement** — deliberate,
+banner-flagged duplication (decision 2026-07-18) so the engine contract stays one
+complete document; `Transformer` class docstring slimmed to the embedder narrative
+(pitch owned by README); section-id pin in `tests/test_reference.py`. Changelog entry
+under Unreleased.
+
+### R-35. Package the Language Reference as package data
+
+**Status**: done · **Severity**: low ·
+**Source**: [`proposals/0008-language-reference-export.md`](proposals/0008-language-reference-export.md) (Deliverable 2)
+
+Ship `LANGUAGE.md` in the wheel and sdist (e.g. `transon/resources/LANGUAGE.md`) so an
+installed `transon==` serves its own language reference offline — the property
+`get_editor_metadata()` already has for the catalog. Single-copy refinement: the packaged file
+**is** the canonical, hand-edited source (`docs/LANGUAGE.md` is a pointer) — no build mapping,
+no mirror to sync. Acceptance: an `importlib.resources` test asserts the packaged bytes (UTF-8,
+line-endings normalized to `\n`) equal `get_language_reference()['content']`.
+
+**Impact if not done**: the `transon-authoring` harnesses mount no repo checkout, so an
+unpackaged reference is invisible to the primary consumer.
+
+**Shipped**: `transon/resources/LANGUAGE.md` is the canonical, hand-edited, single copy;
+hatchling picks it up in both wheel and sdist with no config change — verified by building
+both. `tests/test_reference.py` loads it through `importlib.resources` and asserts it equals
+`get_language_reference()['content']`. `docs/LANGUAGE.md` reduced to a pointer.
+
+### R-36. `get_language_reference()` versioned export
+
+**Status**: done · **Severity**: medium ·
+**Source**: [`proposals/0008-language-reference-export.md`](proposals/0008-language-reference-export.md) (Deliverable 3)
+
+`transon.reference.get_language_reference()` → `{reference_version, engine_version, format,
+content, sections}` with deterministic flat `##`-heading splitting (stable slug ids, preamble
+rule, sections-concatenation parity with `content`), a `METADATA_VERSION`-style version policy
+(minor = additive, major = breaking; consumers fail their drift check loudly on an unsupported
+major), and a `python -m transon.reference` CLI. Engine-global (base `Transformer` only, no
+`cls=` parameter); language facts only, no consumer-specific shapes. `transon-authoring` then
+pins/syncs/drift-checks it like the metadata snapshot (that half lives in its repo, out of
+scope here).
+
+**Impact if not done**: consumers can only ship the raw file — no targeted section lookup, no
+version pin, no drift check; a 700-line context dump instead of one section as the unit of
+consumption.
+
+**Shipped**: `transon/reference.py::get_language_reference()` — `REFERENCE_VERSION`
+`'1.0'`, fence-aware deterministic `##` splitting with GitHub-style slug ids and
+collision suffixes, preamble rule, engine-version degradation to `None` when not
+installed; `python -m transon.reference` CLI. Spec §5.2 documents the export. Tests:
+shape, section pin, concatenation parity, splitter unit cases in
+`tests/test_reference.py`.
+
+---
+
## Suggested sequencing
1. **No-decision fixes** (can start immediately): ~~R-03~~ (done), ~~R-18~~ (done), ~~R-19~~ (done), ~~R-21~~ (done).
diff --git a/docs/SPECIFICATION.md b/docs/SPECIFICATION.md
index 59e4756..220cb99 100644
--- a/docs/SPECIFICATION.md
+++ b/docs/SPECIFICATION.md
@@ -31,6 +31,9 @@ producing JSON *output*. It is inspired by XSLT and JsonLogic.
| `transon/functions.py` | Functions for the `call` rule (registered via `register_function`) |
| `transon/docs.py` | Documentation generator: harvests docstrings + test cases into JSON |
| `transon/metadata.py` | Editor-metadata export (`get_editor_metadata`) for the visual editor (§5.1) |
+| `transon/reference.py` | Language Reference export (`get_language_reference`) — serves the packaged `LANGUAGE.md` (§5.2) |
+| `transon/resources/LANGUAGE.md` | **Template Language Reference** — author-facing, cross-cutting semantics; canonical, hand-edited, and packaged (single copy — ships as-is in the wheel/sdist) |
+| `docs/LANGUAGE.md` | Pointer at the canonical reference above (kept for `docs/` discoverability) |
| `transon/tests/` | **Example corpus**: table-driven test cases that double as documentation |
| `tests/` | Plain pytest tests for engine mechanics (errors, extension, docs generation) |
| `.github/workflows/dev.yml` | CI: pytest + coverage on Python 3.9–3.13 (uv) |
@@ -43,6 +46,14 @@ Packaging is uv / PEP 621 (`pyproject.toml`, `uv.lock`). Runtime dependencies: n
## 2. Core concepts
+> **Deliberate duplication (decision 2026-07-18).** This section is the complete
+> normative statement of the language semantics *inside the engine contract*; the
+> author-facing packaged form is [`transon/resources/LANGUAGE.md`](../transon/resources/LANGUAGE.md)
+> (served by `transon.reference.get_language_reference()`, §5.2), and per-entity
+> behavior is also stated in the registration docs. The spec deliberately keeps its
+> own full copy so it remains a single, complete document — when changing behavior,
+> update all three surfaces in the same change.
+
### 2.1 Templates and the marker
A template is any JSON value. The engine walks it recursively (`Transformer.walk`):
@@ -156,7 +167,7 @@ of a value (distinct from JSON `null`/Python `None`). Semantics:
| Exception | Meaning | Raised when |
|---|---|---|
-| `DefinitionError` | The template is malformed | Unknown rule/operator/function name; missing required rule parameter (`Transformer.require` or `Transformer.validate()`); unknown rule parameters; ambiguous or incomplete mutually-exclusive parameter groups (`validate()`); `attr` with neither `name` nor `names`; `map` with no valid parameter combination; reserved variable name (`this`, `item`, `key`, `value`, `index`) used with `set`/`get`; iteration accessors (`item`, `key`, `value`, `index`) or `parent` used outside their valid scope; `expr`/`call` with a non-list or empty `values` parameter; non-list `chain.funcs` (`validate()`); `include` with no configured `template_loader` (default loader) |
+| `DefinitionError` | The template is malformed | Unknown rule/operator/function name; missing required rule parameter (`Transformer.require` or `Transformer.validate()`); unknown rule parameters; ambiguous or incomplete mutually-exclusive parameter groups (`validate()`); `attr` with neither `name` nor `names`; `map` with no valid parameter combination; reserved variable name (`this`, `item`, `key`, `value`, `index`) used with `set`/`get`; iteration accessors (`item`, `key`, `value`, `index`) or `parent` used outside their valid scope; `expr`/`call` with a non-list or empty `values` parameter; `map` `items` template evaluating to a non-list; non-list `chain.funcs` (`validate()`); `include` with no configured `template_loader` (default loader) |
| `TransformationError` | The template is valid but input data is incompatible | `map`/`filter` over a non-iterable (not list/dict); `join` over mixed-type items; `split` on a non-string/non-array input or with an invalid `sep`; `attr` lookup with an incompatible index type; `zip` over non-iterable items; `expr` operator applied to incompatible operand types; `call` with incompatible argument types or a function that rejects its arguments (e.g. empty `min`/`max`, bad epoch, invalid regex); `format` pattern referencing a missing key or index; `set`/`get` when a dynamic `name` evaluates to `NO_CONTENT`; `include` depth limit exceeded (nested include chain too deep) |
Both are exported from the package root. By default, errors are raised lazily during
@@ -175,6 +186,11 @@ value is not iterable: 'not-a-list'
at template → pipeline → chain → funcs[0] → map
```
+Engine-side plumbing contract: every message is routed through
+`format_error_message`, which appends the template location from the template-path
+`ContextVar` maintained by `walk` — new raise sites must use `t.definition_error` /
+`t.transformation_error` (or `format_error_message`) so the path is never lost.
+
---
## 3. The `Transformer` class
@@ -323,6 +339,12 @@ parameter with no descriptor defaults to a dynamic template (`ParamKind.DYNAMIC`
## 4. Built-in rule reference
+> **Deliberate duplication (decision 2026-07-18).** The same per-rule facts are
+> stated in the registration docs (`transon/rules.py` docstrings + param kwargs),
+> which every export carries and the docs site renders. The spec keeps this full
+> reference so it remains a single, complete contract — update both in the same
+> change.
+
All rules live in `transon/rules.py`. "Dynamic" parameters are walked as templates;
"constant" parameters are read verbatim.
@@ -369,7 +391,7 @@ Other lookup failures (e.g. `TypeError` indexing a string with a string) →
| Rule | Parameters | Semantics |
|---|---|---|
| `object` | exactly one of: `key`+`value` \| `fields` | `key`+`value` (dynamic): single-pair dict `{key: value}`; `{}` if either side is `NO_CONTENT`. For dynamically-named attributes. `fields`: literal mapping whose keys are emitted verbatim (including the marker `$` — the canonical literal-marker-key escape, R-14) and whose values are walked as templates; entries with a `NO_CONTENT` value are omitted. |
-| `map` | exactly one of: `item` \| `items` \| `key`+`value` | Iterates `context.this` (list or dict). `item`: one output element per input element → list. `items`: template yields a *list* of elements per input element, concatenated → list. `key`+`value`: → dict. `NO_CONTENT` results are skipped. Each iteration derives a sub-context with `this`=element plus iteration props. |
+| `map` | exactly one of: `item` \| `items` \| `key`+`value` | Iterates `context.this` (list or dict). `item`: one output element per input element → list. `items`: template yields a *list* of elements per input element, concatenated → list (a non-list result raises `DefinitionError`). `key`+`value`: → dict. `NO_CONTENT` results are skipped. Each iteration derives a sub-context with `this`=element plus iteration props. |
| `filter` | `cond` (required, dynamic) | Keeps elements where `cond` is truthy (and not `NO_CONTENT`). Preserves container type: list→list, dict→dict. |
| `zip` | `items` (required, dynamic) | Transposes iterables like Python's `zip`: each output row is a **list** (`[list(row) for row in zip(*items)]`). Non-iterable items → `TransformationError`. |
| `join` | `items` (required, dynamic), `sep` (dynamic, strings only, default `""`), `default` (optional, dynamic) | Type-homogeneous concatenation: all-strings → `sep.join`; all-lists → flatten one level; all-dicts → merged dict (later keys win). Items that evaluate to `NO_CONTENT` are omitted before concatenation. When no items remain → `NO_CONTENT` (or `default` when provided). Mixed types → `TransformationError`. `sep` must evaluate to a string when joining strings. |
@@ -482,7 +504,10 @@ does not limit pattern complexity.
## 5. Documentation pipeline
The documentation (and the playground at https://transon-org.github.io/) is **generated
-from source artifacts**; nothing is hand-maintained separately:
+from source artifacts**; the one hand-written artifact is
+[`transon/resources/LANGUAGE.md`](../transon/resources/LANGUAGE.md) (the Template Language Reference — cross-cutting
+semantics only, no per-entity sections; served packaged via §5.2, its section shape
+pinned by `tests/test_reference.py`). Everything else is harvested:
1. **Rule docs** — rule function docstrings (markdown, may embed plantuml).
2. **Parameter docs** — `**params` kwargs of `register_rule`.
@@ -600,6 +625,34 @@ into a lean structural `catalog` (consumed by the editor's generators) and an
- `docs.template_loader` makes every test case's template `include`-able by its class
name (e.g. `{"$": "include", "name": "MapListsToDict"}`).
+### 5.2 Language Reference export
+
+`transon/reference.py` provides `get_language_reference()` — a dedicated, versioned
+export (RFC 0008, R-36) serving the packaged `LANGUAGE.md` offline:
+
+- Shape: `{reference_version, engine_version, format: "markdown", content, sections}`.
+ `content` is the full document as canonical normalized text (UTF-8, line endings
+ normalized to LF); `sections` is a flat, ordered split on top-level `##` headings
+ (each section includes its own heading line; deeper headings and fenced code blocks
+ stay inside their parent; any non-empty prefix before the first `##` — whitespace
+ included — becomes a leading `{"id": "preamble"}` section; ids are GitHub-style
+ slugs, collisions suffixed `-2`, `-3`, … in document order). Concatenating
+ `sections` reproduces `content` exactly.
+- `REFERENCE_VERSION` policy (mirrors `METADATA_VERSION`): additive changes — a new
+ section, appended prose, a new optional field — bump the minor; removing/renaming a
+ section `id`, changing the `sections` shape, or dropping/renaming a top-level field
+ is breaking and bumps the major. Consumers MUST fail loudly on an unsupported major.
+- The export is **engine-global** (base `Transformer` only; no `cls=` parameter) and
+ states language facts only.
+- Packaging: `transon/resources/LANGUAGE.md` is the **canonical, hand-edited, single
+ copy** — the file that ships in the wheel and sdist is the file you edit (hatchling
+ packages the whole `transon/` tree; verified by building the distributions at
+ release; no sync step, same rule as per-rule docs living in `rules.py`).
+ `docs/LANGUAGE.md` is a pointer. `tests/test_reference.py` asserts the packaged
+ resource read via `importlib.resources` equals the export's `content`, pins the
+ section-id list, and checks the split parity.
+- `python -m transon.reference` prints this JSON.
+
---
## 6. Testing conventions
@@ -754,7 +807,7 @@ names indexes the tuple → output `{"a": 1, "b": 2}`.
## 12. Known issues & design questions
Suspected accidental behaviors and open design questions are tracked exclusively in
-[`docs/ROADMAP.md`](ROADMAP.md) (items `R-01`…`R-22`), each with impact analysis,
+[`docs/ROADMAP.md`](ROADMAP.md) (`R-xx` items), each with impact analysis,
fix options, and a decision status. This document describes current behavior only.
**Do not "fix" quirky behavior silently** — every behavior change needs an explicit
diff --git a/docs/proposals/0008-language-reference-export.md b/docs/proposals/0008-language-reference-export.md
index c99e5ca..9db259e 100644
--- a/docs/proposals/0008-language-reference-export.md
+++ b/docs/proposals/0008-language-reference-export.md
@@ -1,9 +1,16 @@
# RFC 0008 — Author-facing Language Reference: document, packaging, and export API
-- **Status:** Proposed
+- **Status:** Implemented (2026-07-18) — unreleased; ships in the next release per Sequencing, which will name the version here
- **Created:** 2026-07-16
-- **Roadmap:** R-34 (Language Reference document), R-35 (package the reference), R-36 (`get_language_reference()` export) — proposed; add rows to `docs/ROADMAP.md` on acceptance. (R-33 is held by [RFC 0007](0007-builtin-function-library.md).)
-- **Type:** New documentation artifact + packaging + a new read-only export API (`get_language_reference()`) — additive; no change to existing template semantics or existing engine APIs
+- **Amended:** 2026-07-18 — consolidation scope extended to the `Transformer` docstring and
+ `README.md`; single ownership principle (structure in the catalog, per-entity behavior in
+ registration docs, cross-cutting semantics in `LANGUAGE.md`); no per-entity sections in
+ `LANGUAGE.md`; sequencing decision; single-copy refinement — the packaged
+ `transon/resources/LANGUAGE.md` is canonical, `docs/LANGUAGE.md` is a pointer (see
+ Deliverable 2); spec-completeness decision — `SPECIFICATION.md` deliberately retains a full
+ duplicated copy of the semantics (see the sourcing rule)
+- **Roadmap:** R-34 (Language Reference document), R-35 (package the reference), R-36 (`get_language_reference()` export) — `accepted`, rows in `docs/ROADMAP.md`; docs-site counterpart is D-20 in `docs/DOCS_SITE_ROADMAP.md`. (R-33 is held by [RFC 0007](0007-builtin-function-library.md).)
+- **Type:** New documentation artifact + packaging + a new read-only export API (`get_language_reference()`) — additive; no change to existing template semantics or engine API shapes. The **content** of `get_all_docs()['doc']` shrinks to the embedder-facing narrative as part of the consolidation (shape unchanged; the docs export carries no schema version, so the change is coordinated by release note in `CHANGELOG.md`). Symmetrically, per-rule doc **content grows** in both exports (`get_all_docs()` and `get_editor_metadata()['docs']`) as §4's facts fold into the docstrings — also shape-unchanged, doc text is contractually opaque
- **Consumers:** `transon-authoring` (authority ladder rung 2; `SKILL.md`, AD-018/NFR-001/NFR-003), `transon-org.github.io` (docs site)
- **Supersedes / Superseded by:** — / —
@@ -19,64 +26,140 @@ material — repository layout (§1), the `Transformer` Python API and extension
the documentation pipeline (§5), testing conventions (§6), the add-a-rule checklist (§7),
versioning (§8), invariants (§9–10), known issues (§12). The **language** — what a template
author needs — lives only in §2 (marker, context, `NO_CONTENT`, error model), §4 (built-in rule
-reference), and §11 (data flow), interleaved with the rest.
+reference), and §11 (data flow), interleaved with the rest. That language half reaches consumers
+through **two channels**: the cross-cutting narrative (§2/§11) becomes `LANGUAGE.md`, while the
+per-entity facts (§4) travel through the registration docs and the exports that already carry
+them (see the ownership principle, Deliverable 1).
-Two consumers need the language half, by itself, offline, pinned to an engine version:
+Two consumers need the cross-cutting narrative, by itself, offline, pinned to an engine version:
-1. **`transon-authoring`** — its agents ground drafts in the metadata snapshot (examples show
- *structure*) and the verify gate (pass/fail shows *behavior*), but have nothing that explains
- *semantics*: `NO_CONTENT` propagation, `DefinitionError` vs `TransformationError`, mode
- selection, empty-list `expr` reduction. Repair loops burn their bounded budget rediscovering
+1. **`transon-authoring`** — its agents ground drafts in the metadata snapshot (which carries
+ the structural catalog **and** the per-entity descriptions and examples,
+ `get_editor_metadata()['docs']`) and the verify gate (pass/fail shows *behavior*), but have
+ nothing that explains the **cross-cutting** semantics no per-entity description can state:
+ `NO_CONTENT` propagation across containers, `DefinitionError` vs `TransformationError`,
+ scoping, empty-list `expr` reduction. Repair loops burn their bounded budget rediscovering
facts a reference section would state. The skill's harnesses mount no repo checkout, so the
document must travel inside the installed package, like the metadata snapshot does.
-2. **Human authors / the docs site** — the playground and docs site currently render generated
- example JSON; a coherent authoring narrative has no source document.
+2. **Human authors / the docs site** — the only narrative the site renders today is the
+ audience-mixed `Transformer` class docstring (see below); no author-scoped, spec-grade
+ narrative source exists.
Shipping `SPECIFICATION.md` itself is the wrong fix: it tells an authoring agent about extension
registries and subclassing (capabilities outside the authoring profile) and about open design
questions — content an author must not act on.
+The specification is also not the only pre-existing copy of the semantics. The `Transformer`
+class docstring (~225 lines, exported today as `get_all_docs()['doc']` and rendered by the docs
+site) restates marker detection, context/scoping, and `NO_CONTENT` propagation a **third** time —
+interleaved with install instructions, the project pitch (itself duplicated against `README.md`),
+the Python `transform()` API, and the extension registries. A consolidation that leaves the
+docstring untouched leaves an unmanaged drift surface flowing through an existing export, so this
+RFC's sourcing rule covers it (Deliverable 1).
+
## Deliverable 1 — the Language Reference document (R-34)
-A new `docs/LANGUAGE.md`: the Transon **template language reference**, addressed to template
-authors (human or agent), containing semantics only:
+A new `transon/resources/LANGUAGE.md` (canonical **and** packaged — see Deliverable 2; a
+`docs/LANGUAGE.md` pointer keeps `docs/` discoverability): the Transon **template language
+reference**, addressed to template authors (human or agent), containing semantics only:
- The marker: rule invocation shape, literal-marker escaping, marker inheritance across `include`.
- Context: `this`/`item`/`key`/`index`/`parent`, variable scoping (`set`/`get`), scope derivation.
-- `NO_CONTENT`: where it is produced, how each container rule treats it, top-level behavior.
+- `NO_CONTENT`: the propagation **model** — what the sentinel means, the skip-don't-emit
+ principle for containers, defaults, top-level conversion. Which exact behavior a given rule
+ applies is that rule's docstring's job (ownership principle below); the model section may
+ cite rules as illustrations.
- Error taxonomy **as an author experiences it**: what raises `DefinitionError` vs
`TransformationError`, with representative messages.
-- Per-rule reference: every built-in rule with its parameters, modes/variants, and edge-case
- behavior (empty input, missing keys, `NO_CONTENT` items) — plus operators and functions.
+- Expression & call machinery: the semantics shared across **all** operators and functions —
+ operator application (binary vs reduce, reduction over lists including the empty-list case),
+ type coercion, `call` input/output conventions, `NO_CONTENT` interaction. Boundary: these
+ facts span the whole operator/function domain, so they are cross-cutting even though they
+ attach to the `expr`/`call` rules; the `expr`/`call` docstrings state their own parameter
+ modes and **defer** application semantics to these sections (a deliberate pointer, the one
+ place a docstring links into the reference instead of owning the fact).
+- **No per-entity sections.** `LANGUAGE.md` contains no per-rule, per-operator, or per-function
+ reference — that prose lives in the registration docs and is already exported to every
+ consumer (see the ownership principle below). Entity names appear here only as illustrations
+ of cross-cutting behavior. This keeps the document **stable**: it changes when the language
+ model changes, not when the catalog grows.
- Composition patterns: chaining, `include`, the aggregate-from-primitives recipes (reduce-count,
flatten via `map`/`items`, and their empty-list caveats).
-**Sourcing rule — move, don't copy.** The language content of `SPECIFICATION.md` §2/§4/§11 is
-*relocated* into `LANGUAGE.md`; `SPECIFICATION.md` keeps the engine-contract material and links to
-the reference for semantics. One source of truth per fact — two hand-maintained descriptions of
-`NO_CONTENT` will diverge.
+**Sourcing rule — move, don't copy.** `LANGUAGE.md` is assembled by *relocating* every existing
+copy of the language semantics, not by writing a fresh parallel one. One source of truth per
+fact — two hand-maintained descriptions of `NO_CONTENT` will diverge, and today there are three:
+
+- **`SPECIFICATION.md` §2/§4/§11** — the language content is **copied** into `LANGUAGE.md`
+ (cross-cutting) and into the rule docstrings (per-entity facts, which grow richer), and the
+ spec **retains its own full normative statement** — a deliberate, recorded exception to
+ move-don't-copy (decision 2026-07-18): the spec's value as a single, complete engine
+ contract outweighs the drift risk, which is accepted and managed by review discipline
+ (banners at spec §2/§4 require updating all surfaces in the same change). §5's "nothing is
+ hand-maintained separately" claim is reworded to name `LANGUAGE.md`.
+- **The `Transformer` class docstring** — its language sections ("Templates", "How evaluation
+ works", the language half of "What you can do") are relocated. The docstring shrinks to
+ embedder-facing content only: a short orientation paragraph, Python API usage, "Extending", and
+ a pointer to the reference. `get_all_docs()['doc']` keeps its shape and now carries this
+ slimmed narrative — the docs site repurposes it for an "embedding" page.
+- **`README.md`** — becomes the sole owner of the pitch ("What is transon?", the comparisons,
+ install); the docstring's copies of those sections are dropped. The docs site builds its
+ landing from README at build time (docs-site work, see Sequencing), not from the docstring.
+
+**Ownership principle — one owner per altitude.** Every fact has exactly one home, chosen by
+its altitude; every **consumer channel** composes these sources (joined by entity name, the
+same name-join the corpus normalization R-31 established) and none restates another.
+(`SPECIFICATION.md` is the recorded exception — a contributor document, not a consumer
+channel, that deliberately restates the semantics in full; see the sourcing rule.)
+
+- **Structure** — parameter names, required-ness, modes/variants, dynamic-vs-constant kinds,
+ containers, operator/function types: the registration metadata, exported as the
+ `get_editor_metadata()` catalog. Never restated as prose tables anywhere.
+- **Per-entity behavior** — what one rule/operator/function does, its modes, its edge cases
+ (empty input, missing keys, `NO_CONTENT` handling): the registration docs — rule docstrings
+ and `doc=` kwargs. This channel already reaches every consumer: the docs site renders its
+ per-rule pages from it, the editor shows it as help text, and the authoring snapshot carries
+ it (`get_editor_metadata()['docs']`). The deeper per-rule facts currently in
+ `SPECIFICATION.md` §4 are **relocated into the docstrings**, which grow richer — the existing
+ no-`TBD` gate and docs tests keep guarding them.
+- **Cross-cutting semantics** — the evaluation model, scoping, `NO_CONTENT` propagation across
+ containers, the error taxonomy, `expr`/`call` machinery, composition patterns: `LANGUAGE.md`.
+ Facts that span entities have no docstring to live in; this narrative is exactly what no
+ export carries today.
+- **Examples** — the corpus, referenced by name from everything else (unchanged).
-**Drift protection.** A deterministic parity test (alongside `tests/test_docs.py` /
-`tests/test_metadata.py`) asserts that every rule, parameter, operator, and function in the
-`get_editor_metadata()` catalog has a matching heading/anchor in `LANGUAGE.md`, and that no
-reference section names a catalog entry that does not exist. The prose stays hand-written; the
-*coverage* is machine-checked, the same way the corpus invariants are.
+Volatility follows ownership: adding or changing a rule/function touches `rules.py` (and the
+corpus) only; `LANGUAGE.md` changes only when the language model itself changes.
+
+**API prose split.** `SPECIFICATION.md` §3 remains the *stability contract* for the Python API
+(what is stable across versions, what subclassing must preserve, registry resolution order);
+*usage* prose lives in the docstrings. Neither restates the other.
+
+**Drift protection.** `LANGUAGE.md`'s topical section ids are **pinned** in a deterministic
+test (alongside `tests/test_docs.py` / `tests/test_metadata.py`): the expected section list is
+explicit, so adding, renaming, or removing a section is a conscious act tied to the
+`reference_version` policy (Deliverable 3) — never silent. There is **no** catalog-coverage
+check against `LANGUAGE.md`: per-entity coverage is the registration docs' job, already guarded
+by the existing no-`TBD` gate and docs-shape tests, and `LANGUAGE.md` has no per-entity headings
+to drift. The prose stays hand-written.
## Deliverable 2 — ship the reference in the package (R-35)
-`LANGUAGE.md` becomes package data (e.g. `transon/resources/LANGUAGE.md`, included in the wheel
+`LANGUAGE.md` lives as package data (`transon/resources/LANGUAGE.md`, included in the wheel
and sdist), so an installed `transon==` serves its own language reference offline —
-exactly the property `get_editor_metadata()` already has for the catalog. The repo-root
-`docs/LANGUAGE.md` stays the canonical, human-edited source; the build maps it in (mirroring the
-`transon-authoring` `resources/` force-include pattern) or a release check asserts the two are
-identical.
+exactly the property `get_editor_metadata()` already has for the catalog. **Single-copy
+refinement (implementation decision)**: rather than a canonical `docs/` file mirrored into the
+package (force-include, or a copy plus identity check — both create a second copy or a build
+mode that differs between dev and installed layouts), the packaged file **is** the canonical,
+hand-edited source — the same rule as per-rule docs living in `rules.py` — and
+`docs/LANGUAGE.md` is a pointer. No sync step exists to forget.
**Acceptance (packaging parity).** A test loads the packaged `LANGUAGE.md` through
-`importlib.resources` — exercising the installed wheel/sdist layout, not just the source tree —
-decodes those bytes as UTF-8, normalizes line endings to `\n`, and asserts the result equals
-`get_language_reference()['content']` (which is UTF-8 text with `\n` newlines), following the
-shape-test pattern in `tests/test_metadata.py`. This catches a missing package-data glob or a stale
-packaged copy, neither of which the catalog-to-heading coverage test (Deliverable 1) would notice.
+`importlib.resources`, decodes those bytes as UTF-8, normalizes line endings to `\n`, and asserts
+the result equals `get_language_reference()['content']`, following the shape-test pattern in
+`tests/test_metadata.py`. With the single-copy refinement there is no second copy to drift; the
+wheel/sdist inclusion itself is verified by building the distributions at release — which the
+section-pin test (Deliverable 1) would not notice.
## Deliverable 3 — `get_language_reference()` export (R-36)
@@ -91,7 +174,8 @@ A read-only export, separate from the docs API, mirroring the `get_editor_metada
"format": "markdown",
"content": "",
"sections": [
- {"id": "no-content", "title": "NO_CONTENT", "heading_level": 2, "content": "..."},
+ {"id": "preamble", "title": "", "heading_level": null, "content": "..."},
+ {"id": "the-no_content-model", "title": "The NO_CONTENT model", "heading_level": 2, "content": "..."},
...
]
}
@@ -99,13 +183,15 @@ A read-only export, separate from the docs API, mirroring the `get_editor_metada
- `sections` is a flat, ordered split of `content` by heading, each with a stable slug `id`, so
consumers can serve **targeted** section lookups (a 700-line dump into an agent context is the
- failure mode; one section is the unit of consumption). `content` is the byte-exact document for
- consumers that want the whole thing.
+ failure mode; one section is the unit of consumption). `content` is the full document as
+ canonical normalized text (UTF-8, line endings normalized to LF) for consumers that want the
+ whole thing.
- **Splitting rules** (deterministic): the split is on top-level `##` headings only, so `sections`
is flat, not a tree. Each section runs from its `##` heading up to the next `##` heading and
**includes its own heading line**; any deeper (`###`+) heading stays inside its parent section.
Content before the first `##` heading (the intro under the `#` title) becomes a leading preamble
- section — present **only when that intro is non-empty** — carrying
+ section — present **whenever that prefix is non-empty**, whitespace-only prefixes included, so
+ the concatenation parity below always holds — carrying
`{"id": "preamble", "title": "", "heading_level": null, ...}`. Every other section carries
`heading_level: 2` and the heading's text as `title`. `id` is the GitHub-style slug of the heading
text (the preamble's is the literal `"preamble"`); a collision gets a `-2`, `-3`, … suffix in
@@ -121,24 +207,43 @@ A read-only export, separate from the docs API, mirroring the `get_editor_metada
`python -m transon.metadata`).
- The export states **language facts only** — no consumer-specific shapes (no skill procedure, no
editor widgets), same line as metadata-contract §2.8.
+- The export is **engine-global**: it documents the built-in language of the base `Transformer`
+ only. Unlike `get_all_docs(cls=...)`, there is no class parameter — rules registered on
+ subclasses are outside the reference.
Consumers then pin it like they pin the metadata: `transon-authoring` bundles the export at its
engine pin, syncs it in `scripts/sync_metadata.py`, drift-checks it in `scripts/check_snapshot.py`,
and serves it via a new CLI lookup (`spec search` / `spec --section`) — that half is a SPEC-first
change in the `transon-authoring` repo, out of scope here.
+## Sequencing
+
+All three deliverables **plus** the docstring/README consolidation land atomically in one
+release: at no point is a language fact absent from every published surface, and the docs site
+migrates against a single version. The docs-site counterpart (render `LANGUAGE.md`, build the
+landing from `README.md`, absorb the slimmed `doc` field as the embedding page) is **D-20** in
+`docs/DOCS_SITE_ROADMAP.md` — it is a **dependency** of this
+release, not an option, because the slimmed docstring removes the narrative the site renders
+today. The `CHANGELOG.md` entry for the release names the `get_all_docs()['doc']` content change
+explicitly.
+
## Non-goals
- **No behavior change**: no template semantics move; this is documentation + packaging + a
read-only accessor.
-- **No second source of truth**: language sections are moved out of `SPECIFICATION.md`, not
- duplicated; the parity test guards catalog coverage, not prose duplication.
+- **No second source of truth among consumer surfaces**: the `Transformer` docstring and
+ `README.md` content is moved, never duplicated, and every export composes one owner per
+ fact; the section-pin test guards the reference's shape, the no-`TBD` gate guards the
+ registration docs. The one sanctioned duplication is `SPECIFICATION.md` (decision
+ 2026-07-18): it keeps a full copy of the language semantics so the engine contract stays a
+ single, complete document, aligned by review discipline.
- **No generated prose**: rule *examples* stay generated and normalized in the corpus (§5, R-31);
- `LANGUAGE.md` references behavior, it does not re-serialize examples. (Whether the docs site
- later renders `LANGUAGE.md` is a `DOCS_SITE_ROADMAP.md` concern.)
+ `LANGUAGE.md` references behavior, it does not re-serialize examples. (How the docs site
+ *renders* `LANGUAGE.md` is docs-site work — D-20, see Sequencing — but rendering
+ it is a release dependency, not a later option.)
- **No audience fan-out**: one author-facing reference. Separate authoring/running/validating
- documents multiply drift surfaces; "running" (embedding the engine, `transform()` API) stays in
- `SPECIFICATION.md` §3.
+ documents multiply drift surfaces; "running" (embedding the engine, the `transform()` API)
+ stays with the code — usage in docstrings, stability contract in `SPECIFICATION.md` §3.
## Cross-repo provenance
@@ -147,6 +252,12 @@ change in the `transon-authoring` repo, out of scope here.
`scripts/check_snapshot.py`, `resources/metadata-snapshot.*`) is the pattern this export plugs
into.
- Secondary consumer: `transon-blockly` docs/help surfaces may link section ids for rule help
- text, but the editor contract (`metadata-contract.md`) is unchanged by this RFC.
+ text, but the editor contract (`metadata-contract.md`) is unchanged by this RFC. Rule
+ docstrings grow richer as §4's per-rule facts fold in; the contract treats doc text as opaque,
+ so no shape change — how much of it the editor displays (tooltip vs. help panel) is
+ editor-owned presentation.
+- Docs site (`transon-org.github.io`): **depends on this release** — renders `LANGUAGE.md`,
+ sources its landing from `README.md` at build time, and absorbs the slimmed
+ `get_all_docs()['doc']` as the embedding page (see Sequencing).
- Prior art in this repo: [RFC 0001](0001-editor-metadata-export.md) (R-24) — the
versioned-export conventions this RFC copies.
diff --git a/docs/proposals/README.md b/docs/proposals/README.md
index a6e6a65..fb29251 100644
--- a/docs/proposals/README.md
+++ b/docs/proposals/README.md
@@ -27,7 +27,7 @@ is the *authoritative* record of each work item's status via its **R-number**.
|---|---|---|
| **Proposed** | Open for a decision; no code yet (or scope not finalized). | `needs-decision` |
| **Accepted** | Decision made; awaiting implementation. | `accepted` |
-| **Implemented** | Shipped in a release (the `Status` line names the version). | `done` |
+| **Implemented** | Implementation merged (roadmap items `done`). The `Shipped` column names the release version once tagged — `—` until then. | `done` |
| **Rejected** | Decided against; kept for the record. | `rejected` |
| **Deferred** | Postponed pending a trigger stated in the RFC. | — |
| **Superseded** | Replaced by a later RFC (named in `Superseded by`). | — |
@@ -43,7 +43,7 @@ is the *authoritative* record of each work item's status via its **R-number**.
| [0005](0005-example-corpus-normalization.md) | Normalize exports to one flat example corpus | Implemented | R-31 | v0.1.6 | docs site, `transon-blockly` |
| [0006](0006-transformer-recursion-depth-budget.md) | Bounded per-level recursion budget (self-`include` depth) | Implemented | R-32 | v0.1.7 | `transon-blockly` |
| [0007](0007-builtin-function-library.md) | Grow the built-in function library | Implemented | R-33 | v0.1.8 | `transon-authoring`, `transon-blockly` |
-| [0008](0008-language-reference-export.md) | Author-facing Language Reference: doc, packaging, export | Proposed | R-34, R-35, R-36 | — | `transon-authoring`, docs site |
+| [0008](0008-language-reference-export.md) | Author-facing Language Reference: doc, packaging, export | Implemented | R-34, R-35, R-36 | — | `transon-authoring`, docs site |
## Adding a new RFC
diff --git a/tests/test_copy_output.py b/tests/test_copy_output.py
index 7f70480..b849816 100644
--- a/tests/test_copy_output.py
+++ b/tests/test_copy_output.py
@@ -64,3 +64,26 @@ def test_copy_output_keyword_only():
data = {'$': 'attr', 'name': 'missing'}
transformer = Transformer(data)
assert transformer.transform({}, 'fallback', copy_output=True) == 'fallback'
+
+
+def test_copy_output_preserves_no_content_identity():
+ transformer = Transformer({'$': 'attr', 'name': 'missing'})
+ result = transformer.transform(
+ {}, no_content=Transformer.NO_CONTENT, copy_output=True,
+ )
+ assert result is Transformer.NO_CONTENT
+
+
+def test_copy_output_preserves_nested_no_content_identity():
+ # A literal template list keeps NO_CONTENT elements (only container rules
+ # skip them), so the sentinel can sit inside a deep-copied result.
+ transformer = Transformer([{'$': 'attr', 'name': 'missing'}])
+ result = transformer.transform({}, copy_output=True)
+ assert result[0] is Transformer.NO_CONTENT
+
+
+def test_copy_output_returns_substitute_unchanged():
+ substitute = {'fallback': True}
+ transformer = Transformer({'$': 'attr', 'name': 'missing'})
+ result = transformer.transform({}, no_content=substitute, copy_output=True)
+ assert result is substitute
diff --git a/tests/test_invalid_value.py b/tests/test_invalid_value.py
index 88472a4..ba9a302 100644
--- a/tests/test_invalid_value.py
+++ b/tests/test_invalid_value.py
@@ -31,3 +31,14 @@ def test_filter_invalid_value():
transformer = Transformer(template)
with pytest.raises(TransformationError):
transformer.transform(1)
+
+
+def test_map_items_non_list_result():
+ from transon import DefinitionError
+ template = {
+ '$': 'map',
+ 'items': {'$': 'this'},
+ }
+ transformer = Transformer(template)
+ with pytest.raises(DefinitionError, match='`items` must evaluate to a list'):
+ transformer.transform([{'a': 1}])
diff --git a/tests/test_reference.py b/tests/test_reference.py
new file mode 100644
index 0000000..9494581
--- /dev/null
+++ b/tests/test_reference.py
@@ -0,0 +1,136 @@
+"""Shape, section-pin, split-parity, and packaging tests for the Language
+Reference export (RFC 0008, R-34/R-35/R-36)."""
+import importlib.resources
+
+from transon.reference import (
+ REFERENCE_VERSION,
+ _split_sections,
+ get_language_reference,
+)
+
+#: The pinned section-id list (RFC 0008 drift protection): adding, renaming, or
+#: removing a section in ``transon/resources/LANGUAGE.md`` must update this pin
+#: **and** follow the ``REFERENCE_VERSION`` policy — additive changes bump the
+#: minor, removals/renames are breaking and bump the major. Never a silent edit.
+PINNED_SECTION_IDS = [
+ 'preamble',
+ 'templates-and-the-marker',
+ 'context-and-scoping',
+ 'the-no_content-model',
+ 'error-model',
+ 'expressions-and-calls',
+ 'composition-patterns',
+]
+
+
+def test_reference_shape():
+ ref = get_language_reference()
+ assert set(ref) == {
+ 'reference_version', 'engine_version', 'format', 'content', 'sections',
+ }
+ assert ref['reference_version'] == REFERENCE_VERSION == '1.0'
+ assert ref['engine_version'] is None or isinstance(ref['engine_version'], str)
+ assert ref['format'] == 'markdown'
+ assert isinstance(ref['content'], str) and ref['content']
+ assert '\r' not in ref['content']
+ assert isinstance(ref['sections'], list) and ref['sections']
+
+
+def test_section_ids_are_pinned():
+ ref = get_language_reference()
+ assert [section['id'] for section in ref['sections']] == PINNED_SECTION_IDS
+
+
+def test_sections_concatenation_reproduces_content():
+ ref = get_language_reference()
+ joined = ''.join(section['content'] for section in ref['sections'])
+ assert joined == ref['content']
+
+
+def test_section_fields():
+ ref = get_language_reference()
+ preamble, *rest = ref['sections']
+ assert preamble['id'] == 'preamble'
+ assert preamble['title'] == ''
+ assert preamble['heading_level'] is None
+ assert preamble['content'].strip()
+ for section in rest:
+ assert section['heading_level'] == 2
+ assert section['title']
+ assert section['content'].startswith(f"## {section['title']}\n")
+ assert set(section) == {'id', 'title', 'heading_level', 'content'}
+
+
+def test_packaged_resource_is_served():
+ """Packaging parity (RFC 0008 Deliverable 2): the export serves the
+ canonical ``transon/resources/LANGUAGE.md`` via ``importlib.resources`` —
+ the single hand-edited copy, which is also what ships in the wheel/sdist."""
+ ref = get_language_reference()
+ packaged = (
+ importlib.resources.files('transon')
+ .joinpath('resources/LANGUAGE.md')
+ .read_bytes()
+ .decode('utf-8')
+ .replace('\r\n', '\n')
+ .replace('\r', '\n')
+ )
+ assert packaged == ref['content']
+
+
+def test_split_without_preamble():
+ sections = _split_sections('## Only\nbody\n')
+ assert [s['id'] for s in sections] == ['only']
+
+
+def test_split_whitespace_only_prefix_is_preserved_as_preamble():
+ """Every prefix byte belongs to the preamble — parity beats prettiness."""
+ content = '\n\n## First\nbody\n'
+ sections = _split_sections(content)
+ assert [s['id'] for s in sections] == ['preamble', 'first']
+ assert ''.join(s['content'] for s in sections) == content
+
+
+def test_split_slug_collisions_get_suffixes():
+ content = '## Dup\na\n## Dup\nb\n## Dup\nc\n'
+ sections = _split_sections(content)
+ assert [s['id'] for s in sections] == ['dup', 'dup-2', 'dup-3']
+ assert ''.join(s['content'] for s in sections) == content
+
+
+def test_split_ignores_headings_inside_code_fences():
+ content = '# T\nintro\n\n## Real\n```\n## not a heading\n```\ntail\n'
+ sections = _split_sections(content)
+ assert [s['id'] for s in sections] == ['preamble', 'real']
+ assert ''.join(s['content'] for s in sections) == content
+
+
+def test_split_ignores_headings_inside_tilde_fences():
+ content = '## Real\n~~~\n## not a heading\n~~~\ntail\n'
+ sections = _split_sections(content)
+ assert [s['id'] for s in sections] == ['real']
+ assert ''.join(s['content'] for s in sections) == content
+
+
+def test_split_fence_closes_only_on_matching_delimiter():
+ # A ``` line inside a ~~~ fence does not close it, and a longer run of the
+ # same character does; a shorter run does not.
+ content = (
+ '## A\n'
+ '~~~~\n'
+ '```\n'
+ '## still fenced\n'
+ '~~~\n'
+ '## still fenced too\n'
+ '~~~~~\n'
+ '## B\n'
+ )
+ sections = _split_sections(content)
+ assert [s['id'] for s in sections] == ['a', 'b']
+ assert ''.join(s['content'] for s in sections) == content
+
+
+def test_split_deeper_headings_stay_inside_parent():
+ content = '## Top\n### Sub\nbody\n#### Deeper\n'
+ sections = _split_sections(content)
+ assert [s['id'] for s in sections] == ['top']
+ assert '### Sub' in sections[0]['content']
diff --git a/transon/reference.py b/transon/reference.py
new file mode 100644
index 0000000..e73a416
--- /dev/null
+++ b/transon/reference.py
@@ -0,0 +1,144 @@
+"""Author-facing Language Reference export.
+
+A dedicated, versioned export — separate from the docs API — that serves the
+packaged ``LANGUAGE.md`` (the template language's **cross-cutting** semantics:
+evaluation model, scoping, ``NO_CONTENT``, error taxonomy, ``expr``/``call``
+machinery, composition patterns) to consumers that need it offline and pinned to
+an engine version, the way ``get_editor_metadata()`` serves the catalog (see
+``docs/proposals/0008-language-reference-export.md``).
+
+The export states **language facts only** — no consumer-specific shapes. It is
+**engine-global**: it documents the built-in language of the base ``Transformer``
+only; unlike ``get_all_docs(cls=...)`` there is no class parameter.
+"""
+import importlib.metadata
+import importlib.resources
+import re
+
+REFERENCE_VERSION = '1.0'
+
+#: Characters kept by the GitHub-style heading slugger (besides spaces → hyphens).
+_SLUG_KEEP = re.compile(r'[^0-9a-z _-]')
+
+#: A fenced-code delimiter line: up to 3 spaces of indent, then a backtick or
+#: tilde run of length >= 3 (CommonMark), with the rest of the line captured so
+#: closers (which must carry no trailing content) can be told from openers.
+_FENCE = re.compile(r'^ {0,3}(`{3,}|~{3,})(.*)$')
+
+
+def _engine_version():
+ """The installed ``transon`` distribution version, or ``None`` when unavailable.
+
+ Same degradation contract as the metadata export: the reference must be
+ usable when ``transon`` is merely importable from source and not installed
+ as a distribution.
+ """
+ try:
+ return importlib.metadata.version('transon')
+ except importlib.metadata.PackageNotFoundError:
+ return None
+
+
+def _load_content():
+ """The packaged ``LANGUAGE.md`` as UTF-8 text with ``\\n`` newlines."""
+ resource = importlib.resources.files('transon').joinpath(
+ 'resources/LANGUAGE.md'
+ )
+ text = resource.read_bytes().decode('utf-8')
+ return text.replace('\r\n', '\n').replace('\r', '\n')
+
+
+def _slugify(title):
+ """GitHub-style slug of a heading title.
+
+ Lowercase; markdown backticks dropped with the rest of the punctuation;
+ spaces become hyphens; alphanumerics, underscores, and hyphens survive.
+ """
+ slug = _SLUG_KEEP.sub('', title.lower())
+ return slug.replace(' ', '-')
+
+
+def _split_sections(content):
+ """Split ``content`` into the flat, ordered ``sections`` list.
+
+ Deterministic rules (RFC 0008 Deliverable 3): the split is on top-level
+ ``##`` headings only (``###``+ stays inside its parent; fenced code blocks
+ are opaque). Each section includes its own heading line. Content before the
+ first ``##`` heading becomes a leading ``preamble`` section, present only
+ when non-empty. Slug collisions get ``-2``, ``-3``, … suffixes in document
+ order. The concatenation of all sections reproduces ``content`` exactly.
+ """
+ lines = content.splitlines(keepends=True)
+ boundaries = []
+ fence = None
+ for index, line in enumerate(lines):
+ match = _FENCE.match(line)
+ if fence is None:
+ if match:
+ fence = match.group(1)
+ elif line.startswith('## '):
+ boundaries.append(index)
+ elif (
+ match
+ and match.group(1)[0] == fence[0]
+ and len(match.group(1)) >= len(fence)
+ and not match.group(2).strip()
+ ):
+ fence = None
+
+ sections = []
+ seen_ids = {}
+
+ def _unique(slug):
+ count = seen_ids.get(slug, 0) + 1
+ seen_ids[slug] = count
+ return slug if count == 1 else f'{slug}-{count}'
+
+ first = boundaries[0] if boundaries else len(lines)
+ preamble = ''.join(lines[:first])
+ # Any non-empty prefix — even whitespace-only — must be emitted, or the
+ # sections-concatenation parity invariant breaks.
+ if preamble:
+ sections.append({
+ 'id': _unique('preamble'),
+ 'title': '',
+ 'heading_level': None,
+ 'content': preamble,
+ })
+
+ for position, start in enumerate(boundaries):
+ end = boundaries[position + 1] if position + 1 < len(boundaries) else len(lines)
+ title = lines[start][len('## '):].strip()
+ sections.append({
+ 'id': _unique(_slugify(title)),
+ 'title': title,
+ 'heading_level': 2,
+ 'content': ''.join(lines[start:end]),
+ })
+ return sections
+
+
+def get_language_reference():
+ """Return the versioned Language Reference document.
+
+ The result carries a standalone ``reference_version`` (minor bump =
+ additive: a new section, appended prose, a new optional field; major bump =
+ breaking: a removed/renamed section ``id``, a changed ``sections`` shape, a
+ dropped/renamed top-level field — consumers MUST fail loudly on an
+ unsupported major), the ``engine_version``, the byte-exact ``content``, and
+ ``sections`` — a flat, ordered split of ``content`` so consumers can serve
+ targeted per-section lookups instead of the whole document.
+ """
+ content = _load_content()
+ return {
+ 'reference_version': REFERENCE_VERSION,
+ 'engine_version': _engine_version(),
+ 'format': 'markdown',
+ 'content': content,
+ 'sections': _split_sections(content),
+ }
+
+
+if __name__ == '__main__': # pragma: no cover
+ import json
+ print(json.dumps(get_language_reference(), indent=4))
diff --git a/transon/resources/LANGUAGE.md b/transon/resources/LANGUAGE.md
new file mode 100644
index 0000000..372a31d
--- /dev/null
+++ b/transon/resources/LANGUAGE.md
@@ -0,0 +1,207 @@
+# Transon — Template Language Reference
+
+> **Audience**: template authors (human or agent). This document is the **cross-cutting**
+> semantics of the Transon template language: the evaluation model, scoping, the
+> `NO_CONTENT` propagation model, the error taxonomy, the `expr`/`call` machinery, and
+> composition patterns. It deliberately contains **no per-rule reference**: what each
+> individual rule, operator, or function does — its parameters, modes, and edge cases —
+> lives in that entry's own documentation, exported by the engine
+> (`transon.docs.get_all_docs()` / `transon.metadata.get_editor_metadata()`) and rendered
+> on the [docs site](https://transon-org.github.io/). Entity names appear here only as
+> illustrations. Executable examples live in the example corpus shipped with those same
+> exports.
+
+## Templates and the marker
+
+A template is any JSON value. The engine walks it top-down and rebuilds it node by node;
+each node is handled by its JSON type:
+
+- a **list** → walk every element, return a new list;
+- a **dict containing the marker key** (default `"$"`) → a **rule invocation** (see below);
+- a **dict without the marker** → walk every value, return a new dict with the same keys;
+- any **scalar** (string, number, boolean, `null`) → copied through unchanged.
+
+A template that contains no markers is therefore reproduced as a deep copy of itself —
+rules are the only thing that injects data.
+
+A dict is a rule *only* when it contains the marker key; the marker's value names the
+rule and the sibling keys are the rule's parameters:
+
+```json
+{"$": "attr", "name": "x"}
+```
+
+Rule parameters are themselves templates and are walked recursively, so rules nest
+arbitrarily — this is why even arithmetic is expressed as nested rules rather than a
+string mini-language. A handful of parameters are documented as **constant** (for
+example an operator name): those are read verbatim, never walked.
+
+The marker is configurable per transformation (`marker=` on the transformer). To emit a
+literal dict that really contains the marker key — data that would otherwise be read as
+a rule invocation — use the `object` rule's `fields` mode, whose keys are emitted
+verbatim while its values stay templates; its single-pair `key`/`value` mode also
+produces one literal key.
+
+When one template `include`s another, the sub-template inherits the parent's marker by
+default, so a template tree written against the default marker stays consistent across
+`include` boundaries; the loader may pin a different marker explicitly.
+
+## Context and scoping
+
+Evaluation carries a **context** — a linked chain of scopes. Each context holds:
+
+- `this` — the current value (in the root context: the transformation input);
+- iteration properties — `item`, `index` over lists; `key`, `value`, `index` over
+ dicts — present only inside scopes derived by the iterating rules (`map`, `filter`);
+- user variables — arbitrary names written by `set`, read by `get`;
+- a link to the **parent** scope it was derived from.
+
+Rules that carry a value into a sub-template derive a **child scope**: each `map`/
+`filter` iteration derives one per element (exposing the iteration properties), and
+each `chain` step after the first derives one whose `this` is the previous step's
+result. The context accessor rules (`this`, `parent`, `item`, `key`, `value`, `index`)
+read these slots; each is valid only where its slot exists, and using one outside its
+valid scope is a template mistake (`DefinitionError`).
+
+Variables flow **downward only**. The rules of visibility, in decreasing surprise:
+
+| Where a `set` runs | Its variable is visible to |
+|---|---|
+| any scope | descendant scopes derived *after* the `set` |
+| directly at a key/element of a literal dict/list | later-evaluated siblings in that dict/list (they share one scope; dict key / list index order matters) |
+| the **first** func of a `chain` | the caller's scope — later `chain` funcs *and* later siblings outside the `chain` |
+| a later `chain` func, a `map`/`filter` iteration | only that derived scope and its descendants |
+| anywhere | **never** the parent scope after the derived scope ends; **never** an `include`d sub-template (that is a separate transformation — only the value crosses the boundary) |
+
+Refactoring pitfall: wrapping a step in `chain`, reordering dict keys, or moving a
+`set` can change visibility with no error — consult the table.
+
+The names `this`, `item`, `key`, `value`, and `index` are **reserved** and cannot be
+used as variable names with `set`/`get` (violation raises `DefinitionError`).
+
+## The NO_CONTENT model
+
+`NO_CONTENT` is the language's "no value" sentinel — distinct from JSON `null`. `null`
+is a value you can store and emit; `NO_CONTENT` means *there is nothing here*, and the
+language is built so that missing data disappears from the output instead of blowing up
+or leaving `null` holes.
+
+- **Where it comes from**: lookups that miss (an absent attribute or path, an undefined
+ variable), aggregations left with nothing (a `join` whose items all vanished), rules
+ that never produce a value (`file`), and sub-transforms that themselves produced
+ nothing (`include`).
+- **Skip, don't emit**: container rules *omit* a `NO_CONTENT` piece rather than emitting
+ `null` — as illustrations: `map` drops the item, `object` omits the entry, `filter`
+ excludes the element, `join` leaves the item out. The exact treatment each rule
+ applies is stated in that rule's documentation.
+- **Defaults stop propagation**: rules that can miss accept an optional `default`
+ template, evaluated *instead of* producing `NO_CONTENT` — the tool for "this value,
+ or X if missing" at the point of lookup.
+- **Absorption**: looking further into a missing value stays missing — a deep `attr`
+ path over an absent branch yields `NO_CONTENT`, it does not raise.
+- **Falsiness**: `NO_CONTENT` is falsy, so logical operators can express fallbacks —
+ e.g. a `chain` ending in `expr` `or` with a substitute value. Rules that *test* for
+ absence use identity, not truthiness: `false`, `0`, and `""` are values, not absence.
+- **The top level**: `transform()` never returns the raw sentinel by default — a
+ template that evaluates to `NO_CONTENT` returns `None` (configurable via the
+ `no_content` argument).
+
+## Error model
+
+Template failures are typed by *whose mistake they are*:
+
+| Exception | Meaning | Representative causes |
+|---|---|---|
+| `DefinitionError` | The **template** is malformed — fix the template | unknown rule/operator/function name; a missing required parameter; unknown parameters; ambiguous or incomplete mutually-exclusive parameter groups; a reserved variable name used with `set`/`get`; a context accessor used outside its valid scope; a structural parameter with the wrong JSON shape |
+| `TransformationError` | The template is valid but the **input data** does not fit it | iterating a non-iterable; joining mixed-type items; an operator applied to incompatible operand types; a function rejecting its arguments; a format pattern referencing a missing key; an `include` chain exceeding the depth limit |
+
+By default errors are raised **lazily**, when the failing node is actually walked — a
+typo in a branch the data never reaches will not surface. Opt in to **static
+validation** (`validate=True`, or calling `validate()`) to check the template's
+structure up front, with no input data: unknown rules, unknown or missing parameters,
+ambiguous parameter combinations, and invalid literal operator/function names all raise
+`DefinitionError` immediately.
+
+Both error types carry the **template location** where the failure occurred — a path of
+dict keys, list indices, rule names, and parameter names:
+
+```
+value is not iterable: 'not-a-list'
+ at template → pipeline → chain → funcs[0] → map
+```
+
+Which specific conditions each rule raises is part of that rule's documentation.
+
+## Expressions and calls
+
+Operators (the `expr` rule) and functions (the `call` rule) share one application
+model; these semantics hold across **every** operator and function, so they live here.
+The catalog of what exists — each operator's types and each function's signature — is
+in the `expr` `op` / `call` `name` parameter docs and the engine exports.
+
+**Operator application** (`expr`) has three modes:
+
+- no value parameter → **unary**: `op(this)`;
+- `value` → **binary**: `op(this, value)` — the current value is the left operand;
+- `values` → **reduction**: `reduce(op, values)` pairwise over the evaluated list —
+ and the current value is **ignored**; include `{"$": "this"}` as a list item if the
+ reduction should involve it. `values` must be a non-empty list — an empty reduction
+ has no seed and raises `DefinitionError` (see the empty-collection caveat under
+ composition patterns).
+
+**Type behavior** follows Python semantics: `+` concatenates strings and lists as well
+as adding numbers; comparisons work on like types; the logical operators use
+truthiness and return an *operand*, not necessarily a boolean. An operator applied to
+incompatible operand types raises `TransformationError`.
+
+**Function application** (`call`) mirrors the modes: no parameter → `fn(this)`;
+`value` → `fn(value)`; `values` → `fn(*values)` (multi-argument call). The current
+value is ignored whenever a parameter is given. Built-in functions convert their
+documented failure modes into `TransformationError` — a well-formed template never
+leaks a raw Python exception from a bad argument. A few functions are documented as
+**total** (they accept any well-formed JSON value and never raise); totality is stated
+per function in the catalog.
+
+**`NO_CONTENT` in expressions**: operators and functions do not skip it. A
+`NO_CONTENT` operand is simply falsy (useful in `and`/`or` fallbacks) or an
+incompatible argument (an error) — stop the propagation earlier with a `default` if
+the operand may be missing.
+
+## Composition patterns
+
+The language has no aggregate primitives beyond what composition provides — a handful
+of rules cover everything *because* they compose. The canonical shapes:
+
+- **Pipeline**: `chain` walks its steps in order, each result becoming `this` for the
+ next — the backbone for "extract, then reshape, then format" templates.
+- **Reshape**: `map` (over a list or dict) with a nested `object`/`attr` template body;
+ `filter` before it to drop elements; `zip` to transpose parallel lists.
+- **Compute**: nested `expr` rules — arithmetic is a tree of rules, not a string.
+- **Reuse**: `include` runs a named sub-template against the current value. Only the
+ value crosses the boundary (no variables, no iteration properties); the marker is
+ inherited by default; nested includes are depth-limited.
+- **Aggregate from primitives**: a count is `length` (or a `map` to `1`s reduced with
+ `add`); a flatten is the `flatten` function (or `map` in its `items` mode). **Mind
+ the empty collection**: an `expr` `values` reduction over an empty list is a
+ `DefinitionError` (no seed), an empty `join` yields `NO_CONTENT` (use its `default`),
+ while `sum` of an empty array is `0` — pick the primitive whose empty-case behavior
+ matches the intent.
+
+A worked end-to-end flow — pairing two parallel lists into a dict:
+
+```json
+{
+ "$": "chain",
+ "funcs": [
+ {"$": "zip", "items": [{"$": "attr", "name": "keys"},
+ {"$": "attr", "name": "values"}]},
+ {"$": "map", "key": {"$": "attr", "name": 0},
+ "value": {"$": "attr", "name": 1}}
+ ]
+}
+```
+
+Input `{"keys": ["a","b"], "values": [1,2]}` → the root context's `this` is the input →
+`zip` produces `[["a",1], ["b",2]]` → `chain` derives a context with that as `this` →
+`map` iterates, each pair becoming `this`/`item` in a per-element scope → `attr` with
+numeric names indexes each pair → output `{"a": 1, "b": 2}`.
diff --git a/transon/rules.py b/transon/rules.py
index 3bb5b22..3fa6c15 100644
--- a/transon/rules.py
+++ b/transon/rules.py
@@ -79,6 +79,8 @@ def rule_this(_t: Transformer, _template, context: Context):
def rule_parent(t: Transformer, _template, context: Context):
"""
Returns the value stored in previous context.
+ Using `parent` in the root context (where no previous scope exists) raises
+ `DefinitionError`.
"""
if context.parent is None:
t.definition_error('`parent` is not available in the root context')
@@ -90,6 +92,7 @@ def rule_item(_t: Transformer, _template, context: Context):
"""
Works inside `map`/`filter` when iterating over lists.
Returns current item.
+ Using it outside such an iteration scope raises `DefinitionError`.
"""
return context.item
@@ -99,6 +102,7 @@ def rule_key(_t: Transformer, _template, context: Context):
"""
Works inside `map`/`filter` when iterating over dicts.
Returns the key of current element.
+ Using it outside such an iteration scope raises `DefinitionError`.
"""
return context.key
@@ -108,6 +112,7 @@ def rule_index(_t: Transformer, _template, context: Context):
"""
Works inside `map`/`filter`.
Returns 0-based index of iteration.
+ Using it outside an iteration scope raises `DefinitionError`.
"""
return context.index
@@ -117,6 +122,7 @@ def rule_value(_t: Transformer, _template, context: Context):
"""
Works inside `map`/`filter` when iterating over dicts.
Returns the value of current element.
+ Using it outside such an iteration scope raises `DefinitionError`.
"""
return context.value
@@ -136,24 +142,13 @@ def rule_set(t: Transformer, template, context: Context):
Returns `context.this` unchanged (pass-through), so it can sit inside a
`chain` without altering the piped value.
- **Scoping** — where a `set` is visible depends on which context object it
- runs in:
-
- - **Descendant scopes**: visible in any context derived *after* the `set`
- (child scopes resolve ancestor variables through the parent chain; the first
- `set` in a child materializes inherited variables for write isolation).
- - **Later siblings, same scope**: when a `set` runs directly at a literal-dict
- key or list element, later-evaluated siblings in that dict/list share the
- same context and can `get` the variable. Earlier siblings cannot — visibility
- follows dict key / list index order.
- - **First `chain` func**: runs in the caller's context, so a `set` there is
- visible to later `chain` funcs and to sibling templates outside the `chain`.
- - **Later `chain` funcs, `map`/`filter` items, etc.**: run in derived contexts;
- their `set` values stay inside that scope and do not escape to parents or
- already-evaluated siblings.
-
- `include` starts a separate transformation — variables do not cross that
- boundary.
+ Visibility follows the language's downward-only scoping model (Language
+ Reference, "Context and scoping"): the variable is visible to scopes derived
+ *after* the `set` and to later-evaluated siblings sharing the same scope —
+ never to parent scopes once a derived scope ends, and never across an
+ `include` boundary (a separate transformation). Note the refactoring
+ pitfall: wrapping a step in `chain`, reordering dict keys, or moving a `set`
+ can change visibility with no error.
"""
t_name = t.require(template, 'name')
name = t.walk_param(t_name, context, 'name')
@@ -212,9 +207,14 @@ def rule_attr(t: Transformer, template, context: Context):
"""
Returns values of attribute or item from current value in context.
Can search in deeply nested structures with path.
- If attribute is not present returns no value.
- If the dynamic name or any path segment evaluates to `NO_CONTENT`, returns
- no value (uniformly for all container types).
+
+ - A missing key or an index out of range → no value (`NO_CONTENT`), or the
+ `default` when provided. Looking further into a missing value stays
+ missing — a deep path over an absent branch does not raise.
+ - If the dynamic name or any path segment evaluates to `NO_CONTENT` → no
+ value (uniformly for all container types).
+ - Any other lookup failure (e.g. indexing a string with a string) raises
+ `TransformationError`.
Parameters are mutually exclusive.
"""
@@ -311,6 +311,20 @@ def _iter_contexts(t: Transformer, context: Context):
def rule_map(t: Transformer, template, context: Context):
"""
Iterates over `list` or `dict` and produces new `dict` or `list` with items based on template.
+
+ Each iteration derives a sub-context whose `this` is the element, with the
+ iteration accessors available (`item`/`index` over lists; `key`/`value`/`index`
+ over dicts). Modes (mutually exclusive):
+
+ - `item`: one output element per input element → list.
+ - `items`: the template yields a *list* of elements per input element; the
+ lists are concatenated → list (the flatten-while-mapping mode).
+ - `key`+`value`: one output entry per input element → dict.
+
+ Results that evaluate to `NO_CONTENT` are skipped — the item (or the whole
+ key/value pair) is omitted from the output rather than emitted as `null`.
+ Iterating a value that is neither a list nor a dict raises
+ `TransformationError`.
"""
if 'item' in template:
t_item = template['item']
@@ -325,7 +339,12 @@ def rule_map(t: Transformer, template, context: Context):
t_items = template['items']
result = []
for sub_context in _iter_contexts(t, context):
- for item in t.walk_param(t_items, sub_context, 'items'):
+ items = t.walk_param(t_items, sub_context, 'items')
+ if not isinstance(items, list):
+ t.definition_error(
+ '`items` must evaluate to a list for `map` rule'
+ )
+ for item in items:
if item is t.NO_CONTENT:
continue
result.append(item)
@@ -358,6 +377,12 @@ def rule_map(t: Transformer, template, context: Context):
def rule_filter(t: Transformer, template, context: Context):
"""
Iterates over `list` or `dict` and filters out items regarding condition calculation.
+
+ Keeps the elements whose `cond` evaluates truthy; a condition that evaluates
+ to `NO_CONTENT` excludes the element. The container type is preserved:
+ list → list, dict → dict (entries keep their original keys and values).
+ Filtering a value that is neither a list nor a dict raises
+ `TransformationError`.
"""
t_cond = t.require(template, 'cond')
@@ -395,6 +420,8 @@ def rule_zip(t: Transformer, template, context: Context):
"""
Transposes iterables like Python's `zip`: rows become columns and columns become rows.
Each output row is a **list** (JSON-friendly), not a Python tuple.
+ The result is as long as the shortest input. A non-iterable item raises
+ `TransformationError`.
"""
t_items = t.require(template, 'items')
@@ -416,7 +443,8 @@ def rule_zip(t: Transformer, template, context: Context):
def rule_file(t: Transformer, template, context: Context):
"""
Writes a file using `write_file` delegate (a parameter to `Transformer` constructor).
- This rule produces no result.
+ This rule always produces no result (`NO_CONTENT`) — so a `map` over `file`
+ yields `[]`, and a template that only writes files transforms to `None`.
File will not be written if `name` or `content` returns no result.
"""
def write_file(_name, _content):
@@ -458,8 +486,13 @@ def _is_dict(x):
)
def rule_join(t: Transformer, template, context: Context):
"""
- Joins (concatenates) together several dicts, lists of strings.
- If items to be concatenated have different types an exception will be thrown.
+ Type-homogeneous concatenation — all items must share one JSON type:
+
+ - all **strings** → joined with `sep` (default empty string);
+ - all **lists** → flattened one level into a single list;
+ - all **dicts** → merged into a single dict (later keys win);
+ - mixed types → `TransformationError`.
+
Items that evaluate to `NO_CONTENT` are omitted before concatenation.
When no items remain, returns `NO_CONTENT` unless `default` is provided.
"""
@@ -654,8 +687,13 @@ def rule_expr(t: Transformer, template, context: Context):
3. **Multiple `values` were specified:**
Result is calculated by applying operation to all values in pairs (reduction).
In this case current context value is ignored.
+ `values` must be a non-empty list — an empty reduction has no seed and
+ raises `DefinitionError`.
- Parameters are mutually exclusive.
+ Parameters are mutually exclusive. An operator applied to incompatible
+ operand types raises `TransformationError`. The application semantics shared
+ by all operators (modes, reduction, type coercion, `NO_CONTENT` interaction)
+ are specified in the Language Reference ("Expressions and calls").
"""
op_code = t.require(template, 'op')
op = t.get_operator(op_code)
@@ -751,10 +789,15 @@ def rule_call(t: Transformer, template, context: Context):
Runs conversion function with single parameter - provided value.
Current context is ignored in this case.
3. **Multiple `values` were specified:**
- Runs conversion function with multiple parameters.
+ Runs conversion function with multiple parameters (`values` must be a list;
+ a non-list raises `DefinitionError`).
Current context is ignored in this case.
- Parameters are mutually exclusive.
+ Parameters are mutually exclusive. Built-in functions convert their
+ documented failure modes into `TransformationError`. The application
+ semantics shared by all functions (modes, argument passing, totality,
+ `NO_CONTENT` interaction) are specified in the Language Reference
+ ("Expressions and calls").
"""
name = t.require(template, 'name')
function = t.get_function(name)
@@ -807,6 +850,8 @@ def rule_format(t: Transformer, template, context: Context):
Returns no value when the formatting value (or any unpacked list element or dict
key/value) is `NO_CONTENT`, unless `default` is provided.
+ A pattern referencing a key or index the value does not supply raises
+ `TransformationError`.
"""
t_pattern = t.require(template, 'pattern')
pattern = t.walk_param(t_pattern, context, 'pattern')
diff --git a/transon/transformers.py b/transon/transformers.py
index c6838cb..769d4cd 100644
--- a/transon/transformers.py
+++ b/transon/transformers.py
@@ -141,6 +141,15 @@ def __getitem__(self, _):
def __bool__(self):
return False
+ # The sentinel is compared by identity everywhere, so copying must never
+ # mint a new instance (e.g. `copy_output=True` deep-copying a literal
+ # template list that kept a NO_CONTENT element).
+ def __copy__(self):
+ return self
+
+ def __deepcopy__(self, _memo):
+ return self
+
FileWriterType = Callable[[str, Any], None]
# `include` always calls the loader as ``loader(name, context=IncludeContext)``; the
@@ -248,42 +257,17 @@ def arm(*, _variants, _constants=None, _containers=None, **slots) -> ArmSpec:
class Transformer:
"""
- ## What is transon?
-
- `transon` reshapes one JSON document into another using a template that is
- itself plain JSON. Instead of writing imperative glue code to walk and rebuild
- data, you describe the *shape* of the output once and let the engine fill it in
- from the input — there is no separate template language and no string-embedded
- DSL to learn.
-
- It is **inspired by** [XSLT](https://en.wikipedia.org/wiki/XSLT) (declarative,
- tree-to-tree transformation) and [JsonLogic](https://jsonlogic.com/) (logic
- expressed as data), applying those ideas to JSON-to-JSON transformation.
-
- **Design principles:** templates are always valid JSON; rules are composable and
- nest arbitrarily (so even arithmetic is expressed as nested rules, with no DSL);
- and a single configurable `$` marker is what distinguishes a rule from literal
- data.
-
- **Compared to alternatives:**
+ The engine's entry point: construct a `Transformer` from a template (plain
+ JSON in which a dict carrying the marker key, default `$`, is a rule
+ invocation) and apply it to input data. The **template language** itself —
+ the evaluation model, scoping, the `NO_CONTENT` model, the error taxonomy,
+ composition patterns — is specified in the
+ [Language Reference](https://github.com/transon-org/transon/blob/main/transon/resources/LANGUAGE.md)
+ (also served by `transon.reference.get_language_reference()`); what the
+ project is and how it compares to alternatives is in the
+ [README](https://github.com/transon-org/transon#readme).
- - [jsonnet](https://github.com/google/jsonnet) and
- [jsonata](https://github.com/jsonata-js/jsonata) define their own
- domain-specific languages; `transon` templates stay valid JSON.
- - [jolt](https://github.com/bazaarvoice/jolt) is also JSON-to-JSON, but drives
- transformations with fixed operation specs; `transon` rules compose and nest.
- - [json-templates](https://github.com/datavis-tech/json-templates) does simple
- placeholder substitution; `transon` adds rules, expressions, and functions.
-
- ## Install & get started
-
- Install from [PyPI](https://pypi.org/project/transon/):
-
- ```shell
- pip install transon
- ```
-
- Run your first transform:
+ ## Usage
```python
from transon import Transformer
@@ -292,157 +276,28 @@ class Transformer:
Transformer(template).transform(["a", "b"]) # => {"items": ["a", "b"]}
```
- **Links:** [GitHub](https://github.com/transon-org/transon) ·
- [Specification](https://github.com/transon-org/transon/blob/main/docs/SPECIFICATION.md) ·
- [Changelog](https://github.com/transon-org/transon/blob/main/CHANGELOG.md)
-
- ## Usage
-
- `transon` is a homogeneous JSON-to-JSON template engine: templates are themselves
- plain JSON, and the shape of the output is defined entirely by the template.
- Input data is interpolated into the template's placeholders.
-
- ```plantuml
- @startuml
- skinparam shadowing false
- skinparam rectangle {
- BackgroundColor #FEFEFE
- BorderColor #333333
- }
- rectangle "JSON Template" as T
- rectangle "JSON Input" as I
- rectangle "transon" as E
- rectangle "JSON Output" as O
- T -down-> E
- I -right-> E
- E -right-> O
- @enduml
- ```
-
- ## Templates
-
- Template could be any JSON structure. It will be reflected as-is in output, except for rule structures.
- Rules are JSON objects with special attribute named `$` (this is called marker and can be changed).
- If the rule has nested template the same applies to it as well.
-
- Example template:
-
- ```json
- {
- "test": {
- "$": "map",
- "item": [
- {
- "x": {
- "$": "item"
- }
- }
- ]
- }
- }
- ```
-
- At the top level output will just copy template `{"test": ...}`.
- Then the `map` rule will be applied to the input executing sub-template, defined by `item` attribute,
- for each item in input collection.
- Let's assume that our input is `[1, 2, 3]`.
- Inner template contains another rule `{"$": "item"}` which points to value of items of the input.
-
- So the final result will be:
-
- ```json
- {
- "test": [
- [{"x": 1}],
- [{"x": 2}],
- [{"x": 3}]
- ]
- }
- ```
-
- Note that each item preserves its template definition (including list around object).
-
- ## How evaluation works
-
- A handful of rules cover everything because they compose, and they compose because
- the engine evaluates every template the same way. Understanding these four
- mechanics lets you predict how any template behaves without trial and error.
-
- **1. Recursive tree walk.** The engine walks the template top-down and rebuilds it
- node by node. Each node is handled by its JSON type:
-
- - a **list** → walk every element, return a new list;
- - a **dict containing the marker key** (`$`) → a *rule invocation* (see below);
- - a **dict without the marker** → walk every value, return a new dict with the same
- keys;
- - any **scalar** (string, number, boolean, `null`) → copied through unchanged.
-
- So a template that contains no markers is returned as a deep copy of itself; rules
- are the only thing that injects data.
-
- **2. Marker-based rule detection.** A dict is a rule *only* when it contains the
- marker key; its value names the rule (`{"$": "map", ...}`) and the sibling keys are
- the rule's parameters. Those parameters are themselves templates and are walked
- recursively, so rules nest arbitrarily — this is why even arithmetic is just nested
- `expr` rules rather than a string mini-language. (To emit a literal dict that really
- does contain a `$` key, use the `object` rule's `fields` mode, or change the marker
- with `Transformer(template, marker="@")`.)
-
- **3. Context and scope.** Evaluation carries a *context* whose `this` is the current
- value (the transformation input at the root). Iterating rules — `map` and `filter` —
- *derive* a child context for each element, exposing the accessors `item`, `index`
- (lists) and `key`, `value`, `index` (dicts); those accessors are only valid inside
- that derived scope. Variables (`set`/`get`) flow **downward only**: a value `set` in
- a scope is visible to its descendants and to later-evaluated siblings, but not to
- parent or earlier-evaluated scopes. This downward-only rule is what makes nested
- transformations predictable.
-
- **4. `NO_CONTENT` skip propagation.** Missing lookups don't blow up: `attr`/`get`
- over an absent key/variable produce the `NO_CONTENT` sentinel (distinct from
- `null`). Container rules then *skip* it rather than emitting `null` — `map` drops
- the item, `object`/`filter`/`file` omit the entry, `join` leaves it out — so
- optional data simply disappears from the output. A `default` parameter (on `attr`,
- `get`, `format`, `include`, `join`) substitutes a value instead, and a top-level
- `NO_CONTENT` becomes `None` (configurable via `transform(data, no_content=...)`).
-
- These four behaviours are the whole evaluation model; the
- [specification](https://github.com/transon-org/transon/blob/main/docs/SPECIFICATION.md)
- (§2) documents the exhaustive details (scoping edge cases, every `NO_CONTENT`
- producer/consumer, the error model).
-
- ## What you can do
-
- Beyond simple interpolation, `transon` offers:
-
- - **Static validation** — `Transformer(template, validate=True)` (or calling
- `.validate()`) checks the template's structure up front, without any input data,
- raising `DefinitionError` on malformed rules.
- - **Defaults for missing values** — `attr`, `get`, `join`, `format`, and `include`
- accept a `default` template, used when the looked-up value is absent.
- - **A "no value" model** — rules can produce `NO_CONTENT`; container rules such as
- `map`, `object`, `filter`, and `file` skip it instead of emitting `null`.
- `transform(data, no_content=...)` controls what a top-level `NO_CONTENT` becomes
- (defaults to `None`).
- - **Literal keys** — the `object` rule's `fields` mode builds dicts with literal keys,
- including a key equal to the marker (`$`).
- - **Configurable marker** — `Transformer(template, marker="@")` if `$` collides with
- your data.
- - **Safe output** — `transform(data, copy_output=True)` deep-copies the result so it
- shares no mutable structure with the input (which is never mutated regardless).
- - **A clear error model** — `DefinitionError` signals a malformed template,
- `TransformationError` signals data that does not fit; both messages include the
- template path where the problem occurred (`at template → …`). See the
- **Error model** examples below for the literal messages each one produces.
- - **I/O delegates** — the `file` rule writes through a `file_writer` callback and the
- `include` rule loads sub-templates through a `template_loader` callback.
-
- The full set of built-in rules is documented below under **Rules**; see the
- specification for exhaustive semantics.
+ Constructor options:
+
+ - **`validate=True`** (or calling `.validate()`) — static template check up
+ front, without input data; malformed rules raise `DefinitionError`.
+ - **`marker="@"`** — change the rule marker if `$` collides with your data.
+ - **`file_writer`** — callback the `file` rule writes through.
+ - **`template_loader`** — callback the `include` rule loads sub-templates
+ through; it receives an `IncludeContext` and constructs the sub-transformer
+ (see the `IncludeContext` docstring).
+ - **`max_include_depth`** — nested-`include` depth limit (default 50).
+
+ `transform(data, no_content=None, *, copy_output=False)` returns the output:
+ `no_content` chooses what a top-level `NO_CONTENT` becomes (default `None`);
+ `copy_output=True` deep-copies the result once so it shares no mutable
+ structure with the input (which is never mutated regardless — but pass-through
+ rules return references into it). Failures raise `DefinitionError` (malformed
+ template) or `TransformationError` (data does not fit) — catch these two;
+ messages include the template path (`at template → …`).
## Extending
- All rules are pluggable. Built-in rules are documented below under **Rules**.
- However, you can easily add your own rules with their own attributes.
+ All rules are pluggable, and you can add your own with their own attributes:
```python
@Transformer.register_rule('my_rule')
@@ -450,7 +305,8 @@ def my_rule(t: Transformer, template, context: Context):
...
```
- You can also inherit `Transformer` class and add rules to subclass to avoid functionality collision.
+ You can also inherit `Transformer` and register rules on the subclass to
+ avoid functionality collision:
```python
class Transformer1(Transformer):
@@ -468,11 +324,11 @@ def my_rule2(t: Transformer, template, context: Context):
...
```
- Note that `my_rule` can be used with both transformers but may behave differently.
-
- In the same fashion you can also add additional operators for expressions (`expr`) calculations
- and functions (`call`) using decorators
- `register_operator` and `register_function`.
+ Note that `my_rule` can be used with both transformers but may behave
+ differently. In the same fashion you can add operators for `expr` and
+ functions for `call` with the `register_operator` and `register_function`
+ decorators. Subclass registrations never affect the base class; lookups
+ resolve through the MRO (the stability contract is `SPECIFICATION.md` §3).
"""
DEFAULT_MARKER = '$'
@@ -881,7 +737,12 @@ def transform(self, data, no_content=None, *, copy_output: bool = False):
"""
context = Context(this=data)
result = self.walk(self.template, context)
- if result is self.NO_CONTENT and no_content is not self.NO_CONTENT:
+ if result is self.NO_CONTENT:
+ # Never deepcopy this branch: the sentinel is compared by identity
+ # everywhere, and a substitute is caller-owned (it cannot alias the
+ # input, so `copy_output` has nothing to protect).
+ if no_content is self.NO_CONTENT:
+ return result
return no_content
if copy_output:
return copy.deepcopy(result)