From ab580f9180fe52e72fe99d9ba291b3f00218dcc8 Mon Sep 17 00:00:00 2001 From: Eugene Chernyshov Date: Sat, 18 Jul 2026 01:23:36 +0300 Subject: [PATCH 1/8] docs: accept RFC 0008 (Language Reference); add R-34/R-35/R-36 and D-20 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Amend RFC 0008 through review and accept it: - Consolidation scope extended to the Transformer class docstring and README.md (sourcing rule covers all three copies of the semantics). - Single ownership principle: structure in the catalog, per-entity behavior in registration docs (spec §4 facts fold into rule docstrings), cross-cutting semantics in LANGUAGE.md, examples in the corpus. LANGUAGE.md has no per-entity sections, so it stays stable as the catalog grows. - Drift protection switched from catalog-coverage to a pinned section-id test; get_all_docs()['doc'] keeps shape, content shrinks to the embedder narrative; atomic one-release sequencing. Roadmap: new Theme G with R-34 (LANGUAGE.md document), R-35 (package as package data), R-36 (get_language_reference() versioned export), all accepted. Docs-site counterpart D-20 (render LANGUAGE.md, README-built landing, embedding page) recorded as a hard dependency of the release. Also de-stale the engine-roadmap note in DOCS_SITE_ROADMAP.md (R-01…R-22 → R-xx). check_roadmap.py: consistent. Co-Authored-By: Claude Fable 5 --- docs/DOCS_SITE_ROADMAP.md | 24 ++- docs/ROADMAP.md | 69 ++++++++ .../0008-language-reference-export.md | 164 ++++++++++++++---- docs/proposals/README.md | 2 +- 4 files changed, 223 insertions(+), 36 deletions(-) diff --git a/docs/DOCS_SITE_ROADMAP.md b/docs/DOCS_SITE_ROADMAP.md index ed1d747..6eed54d 100644 --- a/docs/DOCS_SITE_ROADMAP.md +++ b/docs/DOCS_SITE_ROADMAP.md @@ -2,7 +2,7 @@ > **Status of this document**: living backlog for the **content** of the docs site / > playground at (separate from the engine roadmap in -> [`docs/ROADMAP.md`](ROADMAP.md), which tracks engine semantics `R-01…R-22`). Every +> [`docs/ROADMAP.md`](ROADMAP.md), which tracks engine semantics as `R-xx` items). Every > entry follows the same format: problem → impact of not fixing → options (with a > recommendation when one is clearly better). > @@ -41,6 +41,7 @@ | D-04 | Broken JSON in the headline intro example | A. Correctness | high | done | | D-06 | Accessor docstrings omit `filter` scope | A. Correctness | medium | done | | D-05 | Intro is stale vs. v0.0.11 capabilities | B. Freshness | high | done | +| D-20 | Migrate to the engine Language Reference (RFC 0008 release) | B. Freshness | medium | accepted | | D-11 | `file` rule has zero examples | C. Completeness | high | done | | D-12 | Parameters rendered with no example | C. Completeness | medium | done | | D-08 | Operators and functions are not discoverable | C. Completeness | medium | done | @@ -199,6 +200,27 @@ the project today. 3. Add a dedicated "Features"/"Highlights" section sourced from a new docstring or a new corpus-backed block, leaving the existing intro intact. +### D-20. Migrate to the engine Language Reference (RFC 0008 release) + +**Status**: accepted · **Severity**: medium · +**Source**: engine [`proposals/0008-language-reference-export.md`](proposals/0008-language-reference-export.md) (Sequencing) — **hard dependency of that engine release (atomic)** + +The engine's RFC 0008 release relocates content the site renders today: the `Transformer` +class docstring (`get_all_docs()['doc']`) shrinks to an embedder-facing narrative (its language +sections move to the new `docs/LANGUAGE.md`, its pitch to `README.md`), and rule docstrings +grow richer (spec §4's per-rule facts fold in). In the same release window the site must: + +1. render `LANGUAGE.md` as a language-guide section/page (source: `get_language_reference()` + sections or the packaged file); +2. build its landing/pitch from `README.md` at build time instead of the class docstring; +3. repurpose the slimmed `doc` field as an "Embedding" (Python API) page. + +Per-rule pages need no site work — they render whatever docstring text arrives, now richer. + +**Impact if not fixed**: shipping against the new engine release without this leaves a slim +embedder intro as the landing and no language narrative anywhere on the site — a regression of +D-05/D-10/D-18. + --- ## Theme C — Completeness (gaps in what the page documents) diff --git a/docs/ROADMAP.md b/docs/ROADMAP.md index 85d5d44..de781b6 100644 --- a/docs/ROADMAP.md +++ b/docs/ROADMAP.md @@ -62,6 +62,9 @@ | [R-31](#r-31-normalize-exports-to-one-flat-example-corpus-name-references) | Normalize exports to one flat example corpus (name references) | medium | done | | [R-32](#r-32-bounded-per-level-recursion-budget-for-self-include-walks) | Bounded per-level recursion budget for self-`include` walks | medium | done | | [R-33](#r-33-grow-the-built-in-function-library) | Grow the built-in function library (string / numeric / collection helpers) | medium | done | +| [R-34](#r-34-language-reference-document-languagemd) | Language Reference document (`LANGUAGE.md`) | medium | accepted | +| [R-35](#r-35-package-the-language-reference-as-package-data) | Package the Language Reference as package data | low | accepted | +| [R-36](#r-36-get_language_reference-versioned-export) | `get_language_reference()` versioned export | medium | accepted | --- @@ -1022,6 +1025,72 @@ failures to `TransformationError`); `split` rule in `transon/rules.py`; total `i --- +## Theme G — Author-facing Language Reference (RFC 0008) + +> Engine-side counterpart of the `transon-authoring` authority-ladder gap (skill contract +> rung 2): no author-facing, pinnable, offline-servable language document exists. Design and +> the full ownership principle (structure in the catalog, per-entity behavior in registration +> docs, cross-cutting semantics in `LANGUAGE.md`) are recorded in +> [`proposals/0008-language-reference-export.md`](proposals/0008-language-reference-export.md); +> accepted 2026-07-18. R-34/R-35/R-36 plus the docstring/README consolidation ship +> **atomically in one release** (RFC Sequencing); the docs-site counterpart is D-20 in +> [`DOCS_SITE_ROADMAP.md`](DOCS_SITE_ROADMAP.md). + +### R-34. Language Reference document (`LANGUAGE.md`) + +**Status**: accepted · **Severity**: medium · +**Source**: [`proposals/0008-language-reference-export.md`](proposals/0008-language-reference-export.md) (Deliverable 1) + +A new hand-written `docs/LANGUAGE.md`: the template-language reference for authors (human or +agent), carrying **cross-cutting semantics only** — the marker, context/scoping, the +`NO_CONTENT` propagation model, the error taxonomy, `expr`/`call` machinery, composition +patterns. **No per-entity sections**: per-rule/operator/function prose stays in the +registration docs, so the document changes only when the language model changes, not when the +catalog grows. Assembled by relocation, never duplication: spec §2/§11 cross-cutting content +moves in; spec §4's per-rule facts move into the rule docstrings (which grow richer); the +`Transformer` class docstring shrinks to embedder-facing content; `README.md` becomes the sole +owner of the pitch. Drift protection: a pinned section-id test (no catalog-coverage check). + +**Impact if not done**: the cross-cutting narrative stays scattered across three +hand-maintained copies (spec §2/§11, the class docstring, README overlap) with no +author-scoped, pinnable document — the `transon-authoring` authority-ladder gap stays open and +repair loops keep rediscovering semantics. + +### R-35. Package the Language Reference as package data + +**Status**: accepted · **Severity**: low · +**Source**: [`proposals/0008-language-reference-export.md`](proposals/0008-language-reference-export.md) (Deliverable 2) + +Ship `LANGUAGE.md` in the wheel and sdist (e.g. `transon/resources/LANGUAGE.md`) so an +installed `transon==` serves its own language reference offline — the property +`get_editor_metadata()` already has for the catalog. The repo-root `docs/LANGUAGE.md` stays the +canonical, human-edited source; the build maps it in or a release check asserts the two are +identical. Acceptance: an `importlib.resources` test asserts the packaged bytes (UTF-8, +line-endings normalized to `\n`) equal `get_language_reference()['content']`. + +**Impact if not done**: the `transon-authoring` harnesses mount no repo checkout, so an +unpackaged reference is invisible to the primary consumer. + +### R-36. `get_language_reference()` versioned export + +**Status**: accepted · **Severity**: medium · +**Source**: [`proposals/0008-language-reference-export.md`](proposals/0008-language-reference-export.md) (Deliverable 3) + +`transon.reference.get_language_reference()` → `{reference_version, engine_version, format, +content, sections}` with deterministic flat `##`-heading splitting (stable slug ids, preamble +rule, sections-concatenation parity with `content`), a `METADATA_VERSION`-style version policy +(minor = additive, major = breaking; consumers fail their drift check loudly on an unsupported +major), and a `python -m transon.reference` CLI. Engine-global (base `Transformer` only, no +`cls=` parameter); language facts only, no consumer-specific shapes. `transon-authoring` then +pins/syncs/drift-checks it like the metadata snapshot (that half lives in its repo, out of +scope here). + +**Impact if not done**: consumers can only ship the raw file — no targeted section lookup, no +version pin, no drift check; a 700-line context dump instead of one section as the unit of +consumption. + +--- + ## Suggested sequencing 1. **No-decision fixes** (can start immediately): ~~R-03~~ (done), ~~R-18~~ (done), ~~R-19~~ (done), ~~R-21~~ (done). diff --git a/docs/proposals/0008-language-reference-export.md b/docs/proposals/0008-language-reference-export.md index c99e5ca..6347caf 100644 --- a/docs/proposals/0008-language-reference-export.md +++ b/docs/proposals/0008-language-reference-export.md @@ -1,9 +1,13 @@ # RFC 0008 — Author-facing Language Reference: document, packaging, and export API -- **Status:** Proposed +- **Status:** Accepted (2026-07-18) — awaiting implementation - **Created:** 2026-07-16 -- **Roadmap:** R-34 (Language Reference document), R-35 (package the reference), R-36 (`get_language_reference()` export) — proposed; add rows to `docs/ROADMAP.md` on acceptance. (R-33 is held by [RFC 0007](0007-builtin-function-library.md).) -- **Type:** New documentation artifact + packaging + a new read-only export API (`get_language_reference()`) — additive; no change to existing template semantics or existing engine APIs +- **Amended:** 2026-07-18 — consolidation scope extended to the `Transformer` docstring and + `README.md`; single ownership principle (structure in the catalog, per-entity behavior in + registration docs, cross-cutting semantics in `LANGUAGE.md`); no per-entity sections in + `LANGUAGE.md`; sequencing decision (see below) +- **Roadmap:** R-34 (Language Reference document), R-35 (package the reference), R-36 (`get_language_reference()` export) — `accepted`, rows in `docs/ROADMAP.md`; docs-site counterpart is D-20 in `docs/DOCS_SITE_ROADMAP.md`. (R-33 is held by [RFC 0007](0007-builtin-function-library.md).) +- **Type:** New documentation artifact + packaging + a new read-only export API (`get_language_reference()`) — additive; no change to existing template semantics or engine API shapes. The **content** of `get_all_docs()['doc']` shrinks to the embedder-facing narrative as part of the consolidation (shape unchanged; the docs export carries no schema version, so the change is coordinated by release note in `CHANGELOG.md`). Symmetrically, per-rule doc **content grows** in both exports (`get_all_docs()` and `get_editor_metadata()['docs']`) as §4's facts fold into the docstrings — also shape-unchanged, doc text is contractually opaque - **Consumers:** `transon-authoring` (authority ladder rung 2; `SKILL.md`, AD-018/NFR-001/NFR-003), `transon-org.github.io` (docs site) - **Supersedes / Superseded by:** — / — @@ -19,23 +23,37 @@ material — repository layout (§1), the `Transformer` Python API and extension the documentation pipeline (§5), testing conventions (§6), the add-a-rule checklist (§7), versioning (§8), invariants (§9–10), known issues (§12). The **language** — what a template author needs — lives only in §2 (marker, context, `NO_CONTENT`, error model), §4 (built-in rule -reference), and §11 (data flow), interleaved with the rest. - -Two consumers need the language half, by itself, offline, pinned to an engine version: - -1. **`transon-authoring`** — its agents ground drafts in the metadata snapshot (examples show - *structure*) and the verify gate (pass/fail shows *behavior*), but have nothing that explains - *semantics*: `NO_CONTENT` propagation, `DefinitionError` vs `TransformationError`, mode - selection, empty-list `expr` reduction. Repair loops burn their bounded budget rediscovering +reference), and §11 (data flow), interleaved with the rest. That language half reaches consumers +through **two channels**: the cross-cutting narrative (§2/§11) becomes `LANGUAGE.md`, while the +per-entity facts (§4) travel through the registration docs and the exports that already carry +them (see the ownership principle, Deliverable 1). + +Two consumers need the cross-cutting narrative, by itself, offline, pinned to an engine version: + +1. **`transon-authoring`** — its agents ground drafts in the metadata snapshot (which carries + the structural catalog **and** the per-entity descriptions and examples, + `get_editor_metadata()['docs']`) and the verify gate (pass/fail shows *behavior*), but have + nothing that explains the **cross-cutting** semantics no per-entity description can state: + `NO_CONTENT` propagation across containers, `DefinitionError` vs `TransformationError`, + scoping, empty-list `expr` reduction. Repair loops burn their bounded budget rediscovering facts a reference section would state. The skill's harnesses mount no repo checkout, so the document must travel inside the installed package, like the metadata snapshot does. -2. **Human authors / the docs site** — the playground and docs site currently render generated - example JSON; a coherent authoring narrative has no source document. +2. **Human authors / the docs site** — the only narrative the site renders today is the + audience-mixed `Transformer` class docstring (see below); no author-scoped, spec-grade + narrative source exists. Shipping `SPECIFICATION.md` itself is the wrong fix: it tells an authoring agent about extension registries and subclassing (capabilities outside the authoring profile) and about open design questions — content an author must not act on. +The specification is also not the only pre-existing copy of the semantics. The `Transformer` +class docstring (~225 lines, exported today as `get_all_docs()['doc']` and rendered by the docs +site) restates marker detection, context/scoping, and `NO_CONTENT` propagation a **third** time — +interleaved with install instructions, the project pitch (itself duplicated against `README.md`), +the Python `transform()` API, and the extension registries. A consolidation that leaves the +docstring untouched leaves an unmanaged drift surface flowing through an existing export, so this +RFC's sourcing rule covers it (Deliverable 1). + ## Deliverable 1 — the Language Reference document (R-34) A new `docs/LANGUAGE.md`: the Transon **template language reference**, addressed to template @@ -43,24 +61,79 @@ authors (human or agent), containing semantics only: - The marker: rule invocation shape, literal-marker escaping, marker inheritance across `include`. - Context: `this`/`item`/`key`/`index`/`parent`, variable scoping (`set`/`get`), scope derivation. -- `NO_CONTENT`: where it is produced, how each container rule treats it, top-level behavior. +- `NO_CONTENT`: the propagation **model** — what the sentinel means, the skip-don't-emit + principle for containers, defaults, top-level conversion. Which exact behavior a given rule + applies is that rule's docstring's job (ownership principle below); the model section may + cite rules as illustrations. - Error taxonomy **as an author experiences it**: what raises `DefinitionError` vs `TransformationError`, with representative messages. -- Per-rule reference: every built-in rule with its parameters, modes/variants, and edge-case - behavior (empty input, missing keys, `NO_CONTENT` items) — plus operators and functions. +- Expression & call machinery: the semantics shared across **all** operators and functions — + operator application (binary vs reduce, reduction over lists including the empty-list case), + type coercion, `call` input/output conventions, `NO_CONTENT` interaction. Boundary: these + facts span the whole operator/function domain, so they are cross-cutting even though they + attach to the `expr`/`call` rules; the `expr`/`call` docstrings state their own parameter + modes and **defer** application semantics to these sections (a deliberate pointer, the one + place a docstring links into the reference instead of owning the fact). +- **No per-entity sections.** `LANGUAGE.md` contains no per-rule, per-operator, or per-function + reference — that prose lives in the registration docs and is already exported to every + consumer (see the ownership principle below). Entity names appear here only as illustrations + of cross-cutting behavior. This keeps the document **stable**: it changes when the language + model changes, not when the catalog grows. - Composition patterns: chaining, `include`, the aggregate-from-primitives recipes (reduce-count, flatten via `map`/`items`, and their empty-list caveats). -**Sourcing rule — move, don't copy.** The language content of `SPECIFICATION.md` §2/§4/§11 is -*relocated* into `LANGUAGE.md`; `SPECIFICATION.md` keeps the engine-contract material and links to -the reference for semantics. One source of truth per fact — two hand-maintained descriptions of -`NO_CONTENT` will diverge. - -**Drift protection.** A deterministic parity test (alongside `tests/test_docs.py` / -`tests/test_metadata.py`) asserts that every rule, parameter, operator, and function in the -`get_editor_metadata()` catalog has a matching heading/anchor in `LANGUAGE.md`, and that no -reference section names a catalog entry that does not exist. The prose stays hand-written; the -*coverage* is machine-checked, the same way the corpus invariants are. +**Sourcing rule — move, don't copy.** `LANGUAGE.md` is assembled by *relocating* every existing +copy of the language semantics, not by writing a fresh parallel one. One source of truth per +fact — two hand-maintained descriptions of `NO_CONTENT` will diverge, and today there are three: + +- **`SPECIFICATION.md` §2/§11** — the cross-cutting language content is relocated into + `LANGUAGE.md`; **§4's per-rule facts are relocated into the rule docstrings** (see the + ownership principle), and §4 shrinks to a pointer at the generated reference. The spec keeps + the engine-contract material and links out for semantics. §5's "nothing is hand-maintained + separately" claim is reworded to name `LANGUAGE.md` as the one hand-written artifact. +- **The `Transformer` class docstring** — its language sections ("Templates", "How evaluation + works", the language half of "What you can do") are relocated. The docstring shrinks to + embedder-facing content only: a short orientation paragraph, Python API usage, "Extending", and + a pointer to the reference. `get_all_docs()['doc']` keeps its shape and now carries this + slimmed narrative — the docs site repurposes it for an "embedding" page. +- **`README.md`** — becomes the sole owner of the pitch ("What is transon?", the comparisons, + install); the docstring's copies of those sections are dropped. The docs site builds its + landing from README at build time (docs-site work, see Sequencing), not from the docstring. + +**Ownership principle — one owner per altitude.** Every fact has exactly one home, chosen by +its altitude; every channel composes these sources (joined by entity name, the same name-join +the corpus normalization R-31 established) and none restates another: + +- **Structure** — parameter names, required-ness, modes/variants, dynamic-vs-constant kinds, + containers, operator/function types: the registration metadata, exported as the + `get_editor_metadata()` catalog. Never restated as prose tables anywhere. +- **Per-entity behavior** — what one rule/operator/function does, its modes, its edge cases + (empty input, missing keys, `NO_CONTENT` handling): the registration docs — rule docstrings + and `doc=` kwargs. This channel already reaches every consumer: the docs site renders its + per-rule pages from it, the editor shows it as help text, and the authoring snapshot carries + it (`get_editor_metadata()['docs']`). The deeper per-rule facts currently in + `SPECIFICATION.md` §4 are **relocated into the docstrings**, which grow richer — the existing + no-`TBD` gate and docs tests keep guarding them. +- **Cross-cutting semantics** — the evaluation model, scoping, `NO_CONTENT` propagation across + containers, the error taxonomy, `expr`/`call` machinery, composition patterns: `LANGUAGE.md`. + Facts that span entities have no docstring to live in; this narrative is exactly what no + export carries today. +- **Examples** — the corpus, referenced by name from everything else (unchanged). + +Volatility follows ownership: adding or changing a rule/function touches `rules.py` (and the +corpus) only; `LANGUAGE.md` changes only when the language model itself changes. + +**API prose split.** `SPECIFICATION.md` §3 remains the *stability contract* for the Python API +(what is stable across versions, what subclassing must preserve, registry resolution order); +*usage* prose lives in the docstrings. Neither restates the other. + +**Drift protection.** `LANGUAGE.md`'s topical section ids are **pinned** in a deterministic +test (alongside `tests/test_docs.py` / `tests/test_metadata.py`): the expected section list is +explicit, so adding, renaming, or removing a section is a conscious act tied to the +`reference_version` policy (Deliverable 3) — never silent. There is **no** catalog-coverage +check against `LANGUAGE.md`: per-entity coverage is the registration docs' job, already guarded +by the existing no-`TBD` gate and docs-shape tests, and `LANGUAGE.md` has no per-entity headings +to drift. The prose stays hand-written. ## Deliverable 2 — ship the reference in the package (R-35) @@ -76,7 +149,7 @@ identical. decodes those bytes as UTF-8, normalizes line endings to `\n`, and asserts the result equals `get_language_reference()['content']` (which is UTF-8 text with `\n` newlines), following the shape-test pattern in `tests/test_metadata.py`. This catches a missing package-data glob or a stale -packaged copy, neither of which the catalog-to-heading coverage test (Deliverable 1) would notice. +packaged copy, neither of which the section-pin test (Deliverable 1) would notice. ## Deliverable 3 — `get_language_reference()` export (R-36) @@ -121,24 +194,41 @@ A read-only export, separate from the docs API, mirroring the `get_editor_metada `python -m transon.metadata`). - The export states **language facts only** — no consumer-specific shapes (no skill procedure, no editor widgets), same line as metadata-contract §2.8. +- The export is **engine-global**: it documents the built-in language of the base `Transformer` + only. Unlike `get_all_docs(cls=...)`, there is no class parameter — rules registered on + subclasses are outside the reference. Consumers then pin it like they pin the metadata: `transon-authoring` bundles the export at its engine pin, syncs it in `scripts/sync_metadata.py`, drift-checks it in `scripts/check_snapshot.py`, and serves it via a new CLI lookup (`spec search` / `spec --section`) — that half is a SPEC-first change in the `transon-authoring` repo, out of scope here. +## Sequencing + +All three deliverables **plus** the docstring/README consolidation land atomically in one +release: at no point is a language fact absent from every published surface, and the docs site +migrates against a single version. The docs-site counterpart (render `LANGUAGE.md`, build the +landing from `README.md`, absorb the slimmed `doc` field as the embedding page) is **D-20** in +`docs/DOCS_SITE_ROADMAP.md` — it is a **dependency** of this +release, not an option, because the slimmed docstring removes the narrative the site renders +today. The `CHANGELOG.md` entry for the release names the `get_all_docs()['doc']` content change +explicitly. + ## Non-goals - **No behavior change**: no template semantics move; this is documentation + packaging + a read-only accessor. -- **No second source of truth**: language sections are moved out of `SPECIFICATION.md`, not - duplicated; the parity test guards catalog coverage, not prose duplication. +- **No second source of truth**: content is moved out of `SPECIFICATION.md` (§2/§11 to the + reference, §4 to the docstrings), the `Transformer` docstring, and `README.md` — never + duplicated; the section-pin test guards the reference's shape, the no-`TBD` gate guards the + registration docs. - **No generated prose**: rule *examples* stay generated and normalized in the corpus (§5, R-31); - `LANGUAGE.md` references behavior, it does not re-serialize examples. (Whether the docs site - later renders `LANGUAGE.md` is a `DOCS_SITE_ROADMAP.md` concern.) + `LANGUAGE.md` references behavior, it does not re-serialize examples. (How the docs site + *renders* `LANGUAGE.md` is docs-site work — D-20, see Sequencing — but rendering + it is a release dependency, not a later option.) - **No audience fan-out**: one author-facing reference. Separate authoring/running/validating - documents multiply drift surfaces; "running" (embedding the engine, `transform()` API) stays in - `SPECIFICATION.md` §3. + documents multiply drift surfaces; "running" (embedding the engine, the `transform()` API) + stays with the code — usage in docstrings, stability contract in `SPECIFICATION.md` §3. ## Cross-repo provenance @@ -147,6 +237,12 @@ change in the `transon-authoring` repo, out of scope here. `scripts/check_snapshot.py`, `resources/metadata-snapshot.*`) is the pattern this export plugs into. - Secondary consumer: `transon-blockly` docs/help surfaces may link section ids for rule help - text, but the editor contract (`metadata-contract.md`) is unchanged by this RFC. + text, but the editor contract (`metadata-contract.md`) is unchanged by this RFC. Rule + docstrings grow richer as §4's per-rule facts fold in; the contract treats doc text as opaque, + so no shape change — how much of it the editor displays (tooltip vs. help panel) is + editor-owned presentation. +- Docs site (`transon-org.github.io`): **depends on this release** — renders `LANGUAGE.md`, + sources its landing from `README.md` at build time, and absorbs the slimmed + `get_all_docs()['doc']` as the embedding page (see Sequencing). - Prior art in this repo: [RFC 0001](0001-editor-metadata-export.md) (R-24) — the versioned-export conventions this RFC copies. diff --git a/docs/proposals/README.md b/docs/proposals/README.md index a6e6a65..8f737ba 100644 --- a/docs/proposals/README.md +++ b/docs/proposals/README.md @@ -43,7 +43,7 @@ is the *authoritative* record of each work item's status via its **R-number**. | [0005](0005-example-corpus-normalization.md) | Normalize exports to one flat example corpus | Implemented | R-31 | v0.1.6 | docs site, `transon-blockly` | | [0006](0006-transformer-recursion-depth-budget.md) | Bounded per-level recursion budget (self-`include` depth) | Implemented | R-32 | v0.1.7 | `transon-blockly` | | [0007](0007-builtin-function-library.md) | Grow the built-in function library | Implemented | R-33 | v0.1.8 | `transon-authoring`, `transon-blockly` | -| [0008](0008-language-reference-export.md) | Author-facing Language Reference: doc, packaging, export | Proposed | R-34, R-35, R-36 | — | `transon-authoring`, docs site | +| [0008](0008-language-reference-export.md) | Author-facing Language Reference: doc, packaging, export | Accepted | R-34, R-35, R-36 | — | `transon-authoring`, docs site | ## Adding a new RFC From fcf07534b23249b560e476f6a9dfd0079fad67c9 Mon Sep 17 00:00:00 2001 From: Eugene Chernyshov Date: Sat, 18 Jul 2026 01:40:09 +0300 Subject: [PATCH 2/8] =?UTF-8?q?feat:=20implement=20RFC=200008=20=E2=80=94?= =?UTF-8?q?=20Language=20Reference=20document,=20packaging,=20export=20(R-?= =?UTF-8?q?34/35/36)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit R-34: new hand-written docs/LANGUAGE.md — the author-facing, cross-cutting template-language semantics (evaluation model, scoping, NO_CONTENT model, error taxonomy, expr/call machinery, composition patterns; no per-entity sections). Consolidation per the RFC's ownership principle: - spec §2 reduced to the engine-internal view; §4 collapsed to a pointer plus the Recursion budget invariant; §11 points at the reference - spec §4's per-rule facts folded into the rules.py docstrings (richer): accessor scope errors, attr error split, map/filter modes, zip/join/ file/format edge cases, expr/call empty-values errors + the sanctioned pointer to the reference's machinery sections - Transformer class docstring slimmed to the embedder narrative (usage, constructor options, extending); pitch/install/comparison owned by README R-35: packaged copy transon/resources/LANGUAGE.md ships in wheel and sdist (hatchling default pickup, verified by building both); identity with the canonical docs/LANGUAGE.md is test-enforced. R-36: transon/reference.py get_language_reference() — REFERENCE_VERSION 1.0, fence-aware deterministic heading split, GitHub-style slug ids with collision suffixes, preamble rule, python -m transon.reference CLI; spec §5.2 documents the export. Tests: tests/test_reference.py pins the section-id list (drift protection), asserts sections-concatenation parity, packaging parity via importlib.resources, and unit-tests the splitter. 392 passed, 99% cov; python -m transon.docs reports no TBD; check_roadmap.py consistent. Bookkeeping: CHANGELOG entry under Unreleased (names the get_all_docs doc-field content change per Sequencing); R-34/35/36 done with Shipped notes; RFC 0008 and proposals index moved to Implemented (unreleased). Docs-site counterpart is D-20 (separate repo). Co-Authored-By: Claude Fable 5 --- CHANGELOG.md | 29 ++ docs/LANGUAGE.md | 207 +++++++++ docs/ROADMAP.md | 34 +- docs/SPECIFICATION.md | 397 +++++------------- .../0008-language-reference-export.md | 2 +- docs/proposals/README.md | 2 +- tests/test_reference.py | 118 ++++++ transon/reference.py | 129 ++++++ transon/resources/LANGUAGE.md | 207 +++++++++ transon/rules.py | 94 +++-- transon/transformers.py | 225 ++-------- 11 files changed, 928 insertions(+), 516 deletions(-) create mode 100644 docs/LANGUAGE.md create mode 100644 tests/test_reference.py create mode 100644 transon/reference.py create mode 100644 transon/resources/LANGUAGE.md diff --git a/CHANGELOG.md b/CHANGELOG.md index 4187b88..2635332 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -6,6 +6,35 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/). ## [Unreleased] +### Added + +- **Template Language Reference (RFC 0008).** New hand-written `docs/LANGUAGE.md` — + the author-facing, cross-cutting language semantics (evaluation model, scoping, + the `NO_CONTENT` model, error taxonomy, `expr`/`call` machinery, composition + patterns; **no per-entity sections**) — shipped as package data + (`transon/resources/LANGUAGE.md`, wheel + sdist) and served by a new versioned + export `transon.reference.get_language_reference()` (`REFERENCE_VERSION` `1.0`; + `{reference_version, engine_version, format, content, sections}` with a + deterministic flat `##`-heading split, stable slug ids, and sections-concatenation + parity) plus a `python -m transon.reference` CLI. Section ids are pinned in + `tests/test_reference.py`; packaging parity is tested via `importlib.resources`. + (Roadmap R-34, R-35, R-36) + +### Changed + +- **`get_all_docs()['doc']` content (docs-site coordination; shape unchanged).** + The `Transformer` class docstring — exported as the `doc` field and rendered by the + docs site — is consolidated per RFC 0008's ownership principle: its language + sections ("Templates", "How evaluation works", the language half of "What you can + do") moved into `docs/LANGUAGE.md`; the pitch/install/comparison sections are owned + solely by `README.md`; what remains is the embedder-facing narrative (Python API + usage + extending). Symmetrically, per-rule docstrings **grew richer**: spec §4's + per-rule facts (edge cases, `NO_CONTENT` treatment, error conditions) folded into + the registration docs, so per-rule doc content in `get_all_docs()` and + `get_editor_metadata()['docs']` is longer (doc text is contractually opaque — no + shape change). `SPECIFICATION.md` §2/§4/§11 now hold the engine-internal view and + link out to the reference. Docs-site counterpart work is D-20. (Roadmap R-34) + ## [0.1.8] - 2026-07-16 ### Added diff --git a/docs/LANGUAGE.md b/docs/LANGUAGE.md new file mode 100644 index 0000000..372a31d --- /dev/null +++ b/docs/LANGUAGE.md @@ -0,0 +1,207 @@ +# Transon — Template Language Reference + +> **Audience**: template authors (human or agent). This document is the **cross-cutting** +> semantics of the Transon template language: the evaluation model, scoping, the +> `NO_CONTENT` propagation model, the error taxonomy, the `expr`/`call` machinery, and +> composition patterns. It deliberately contains **no per-rule reference**: what each +> individual rule, operator, or function does — its parameters, modes, and edge cases — +> lives in that entry's own documentation, exported by the engine +> (`transon.docs.get_all_docs()` / `transon.metadata.get_editor_metadata()`) and rendered +> on the [docs site](https://transon-org.github.io/). Entity names appear here only as +> illustrations. Executable examples live in the example corpus shipped with those same +> exports. + +## Templates and the marker + +A template is any JSON value. The engine walks it top-down and rebuilds it node by node; +each node is handled by its JSON type: + +- a **list** → walk every element, return a new list; +- a **dict containing the marker key** (default `"$"`) → a **rule invocation** (see below); +- a **dict without the marker** → walk every value, return a new dict with the same keys; +- any **scalar** (string, number, boolean, `null`) → copied through unchanged. + +A template that contains no markers is therefore reproduced as a deep copy of itself — +rules are the only thing that injects data. + +A dict is a rule *only* when it contains the marker key; the marker's value names the +rule and the sibling keys are the rule's parameters: + +```json +{"$": "attr", "name": "x"} +``` + +Rule parameters are themselves templates and are walked recursively, so rules nest +arbitrarily — this is why even arithmetic is expressed as nested rules rather than a +string mini-language. A handful of parameters are documented as **constant** (for +example an operator name): those are read verbatim, never walked. + +The marker is configurable per transformation (`marker=` on the transformer). To emit a +literal dict that really contains the marker key — data that would otherwise be read as +a rule invocation — use the `object` rule's `fields` mode, whose keys are emitted +verbatim while its values stay templates; its single-pair `key`/`value` mode also +produces one literal key. + +When one template `include`s another, the sub-template inherits the parent's marker by +default, so a template tree written against the default marker stays consistent across +`include` boundaries; the loader may pin a different marker explicitly. + +## Context and scoping + +Evaluation carries a **context** — a linked chain of scopes. Each context holds: + +- `this` — the current value (in the root context: the transformation input); +- iteration properties — `item`, `index` over lists; `key`, `value`, `index` over + dicts — present only inside scopes derived by the iterating rules (`map`, `filter`); +- user variables — arbitrary names written by `set`, read by `get`; +- a link to the **parent** scope it was derived from. + +Rules that carry a value into a sub-template derive a **child scope**: each `map`/ +`filter` iteration derives one per element (exposing the iteration properties), and +each `chain` step after the first derives one whose `this` is the previous step's +result. The context accessor rules (`this`, `parent`, `item`, `key`, `value`, `index`) +read these slots; each is valid only where its slot exists, and using one outside its +valid scope is a template mistake (`DefinitionError`). + +Variables flow **downward only**. The rules of visibility, in decreasing surprise: + +| Where a `set` runs | Its variable is visible to | +|---|---| +| any scope | descendant scopes derived *after* the `set` | +| directly at a key/element of a literal dict/list | later-evaluated siblings in that dict/list (they share one scope; dict key / list index order matters) | +| the **first** func of a `chain` | the caller's scope — later `chain` funcs *and* later siblings outside the `chain` | +| a later `chain` func, a `map`/`filter` iteration | only that derived scope and its descendants | +| anywhere | **never** the parent scope after the derived scope ends; **never** an `include`d sub-template (that is a separate transformation — only the value crosses the boundary) | + +Refactoring pitfall: wrapping a step in `chain`, reordering dict keys, or moving a +`set` can change visibility with no error — consult the table. + +The names `this`, `item`, `key`, `value`, and `index` are **reserved** and cannot be +used as variable names with `set`/`get` (violation raises `DefinitionError`). + +## The NO_CONTENT model + +`NO_CONTENT` is the language's "no value" sentinel — distinct from JSON `null`. `null` +is a value you can store and emit; `NO_CONTENT` means *there is nothing here*, and the +language is built so that missing data disappears from the output instead of blowing up +or leaving `null` holes. + +- **Where it comes from**: lookups that miss (an absent attribute or path, an undefined + variable), aggregations left with nothing (a `join` whose items all vanished), rules + that never produce a value (`file`), and sub-transforms that themselves produced + nothing (`include`). +- **Skip, don't emit**: container rules *omit* a `NO_CONTENT` piece rather than emitting + `null` — as illustrations: `map` drops the item, `object` omits the entry, `filter` + excludes the element, `join` leaves the item out. The exact treatment each rule + applies is stated in that rule's documentation. +- **Defaults stop propagation**: rules that can miss accept an optional `default` + template, evaluated *instead of* producing `NO_CONTENT` — the tool for "this value, + or X if missing" at the point of lookup. +- **Absorption**: looking further into a missing value stays missing — a deep `attr` + path over an absent branch yields `NO_CONTENT`, it does not raise. +- **Falsiness**: `NO_CONTENT` is falsy, so logical operators can express fallbacks — + e.g. a `chain` ending in `expr` `or` with a substitute value. Rules that *test* for + absence use identity, not truthiness: `false`, `0`, and `""` are values, not absence. +- **The top level**: `transform()` never returns the raw sentinel by default — a + template that evaluates to `NO_CONTENT` returns `None` (configurable via the + `no_content` argument). + +## Error model + +Template failures are typed by *whose mistake they are*: + +| Exception | Meaning | Representative causes | +|---|---|---| +| `DefinitionError` | The **template** is malformed — fix the template | unknown rule/operator/function name; a missing required parameter; unknown parameters; ambiguous or incomplete mutually-exclusive parameter groups; a reserved variable name used with `set`/`get`; a context accessor used outside its valid scope; a structural parameter with the wrong JSON shape | +| `TransformationError` | The template is valid but the **input data** does not fit it | iterating a non-iterable; joining mixed-type items; an operator applied to incompatible operand types; a function rejecting its arguments; a format pattern referencing a missing key; an `include` chain exceeding the depth limit | + +By default errors are raised **lazily**, when the failing node is actually walked — a +typo in a branch the data never reaches will not surface. Opt in to **static +validation** (`validate=True`, or calling `validate()`) to check the template's +structure up front, with no input data: unknown rules, unknown or missing parameters, +ambiguous parameter combinations, and invalid literal operator/function names all raise +`DefinitionError` immediately. + +Both error types carry the **template location** where the failure occurred — a path of +dict keys, list indices, rule names, and parameter names: + +``` +value is not iterable: 'not-a-list' + at template → pipeline → chain → funcs[0] → map +``` + +Which specific conditions each rule raises is part of that rule's documentation. + +## Expressions and calls + +Operators (the `expr` rule) and functions (the `call` rule) share one application +model; these semantics hold across **every** operator and function, so they live here. +The catalog of what exists — each operator's types and each function's signature — is +in the `expr` `op` / `call` `name` parameter docs and the engine exports. + +**Operator application** (`expr`) has three modes: + +- no value parameter → **unary**: `op(this)`; +- `value` → **binary**: `op(this, value)` — the current value is the left operand; +- `values` → **reduction**: `reduce(op, values)` pairwise over the evaluated list — + and the current value is **ignored**; include `{"$": "this"}` as a list item if the + reduction should involve it. `values` must be a non-empty list — an empty reduction + has no seed and raises `DefinitionError` (see the empty-collection caveat under + composition patterns). + +**Type behavior** follows Python semantics: `+` concatenates strings and lists as well +as adding numbers; comparisons work on like types; the logical operators use +truthiness and return an *operand*, not necessarily a boolean. An operator applied to +incompatible operand types raises `TransformationError`. + +**Function application** (`call`) mirrors the modes: no parameter → `fn(this)`; +`value` → `fn(value)`; `values` → `fn(*values)` (multi-argument call). The current +value is ignored whenever a parameter is given. Built-in functions convert their +documented failure modes into `TransformationError` — a well-formed template never +leaks a raw Python exception from a bad argument. A few functions are documented as +**total** (they accept any well-formed JSON value and never raise); totality is stated +per function in the catalog. + +**`NO_CONTENT` in expressions**: operators and functions do not skip it. A +`NO_CONTENT` operand is simply falsy (useful in `and`/`or` fallbacks) or an +incompatible argument (an error) — stop the propagation earlier with a `default` if +the operand may be missing. + +## Composition patterns + +The language has no aggregate primitives beyond what composition provides — a handful +of rules cover everything *because* they compose. The canonical shapes: + +- **Pipeline**: `chain` walks its steps in order, each result becoming `this` for the + next — the backbone for "extract, then reshape, then format" templates. +- **Reshape**: `map` (over a list or dict) with a nested `object`/`attr` template body; + `filter` before it to drop elements; `zip` to transpose parallel lists. +- **Compute**: nested `expr` rules — arithmetic is a tree of rules, not a string. +- **Reuse**: `include` runs a named sub-template against the current value. Only the + value crosses the boundary (no variables, no iteration properties); the marker is + inherited by default; nested includes are depth-limited. +- **Aggregate from primitives**: a count is `length` (or a `map` to `1`s reduced with + `add`); a flatten is the `flatten` function (or `map` in its `items` mode). **Mind + the empty collection**: an `expr` `values` reduction over an empty list is a + `DefinitionError` (no seed), an empty `join` yields `NO_CONTENT` (use its `default`), + while `sum` of an empty array is `0` — pick the primitive whose empty-case behavior + matches the intent. + +A worked end-to-end flow — pairing two parallel lists into a dict: + +```json +{ + "$": "chain", + "funcs": [ + {"$": "zip", "items": [{"$": "attr", "name": "keys"}, + {"$": "attr", "name": "values"}]}, + {"$": "map", "key": {"$": "attr", "name": 0}, + "value": {"$": "attr", "name": 1}} + ] +} +``` + +Input `{"keys": ["a","b"], "values": [1,2]}` → the root context's `this` is the input → +`zip` produces `[["a",1], ["b",2]]` → `chain` derives a context with that as `this` → +`map` iterates, each pair becoming `this`/`item` in a per-element scope → `attr` with +numeric names indexes each pair → output `{"a": 1, "b": 2}`. diff --git a/docs/ROADMAP.md b/docs/ROADMAP.md index de781b6..f21272b 100644 --- a/docs/ROADMAP.md +++ b/docs/ROADMAP.md @@ -62,9 +62,9 @@ | [R-31](#r-31-normalize-exports-to-one-flat-example-corpus-name-references) | Normalize exports to one flat example corpus (name references) | medium | done | | [R-32](#r-32-bounded-per-level-recursion-budget-for-self-include-walks) | Bounded per-level recursion budget for self-`include` walks | medium | done | | [R-33](#r-33-grow-the-built-in-function-library) | Grow the built-in function library (string / numeric / collection helpers) | medium | done | -| [R-34](#r-34-language-reference-document-languagemd) | Language Reference document (`LANGUAGE.md`) | medium | accepted | -| [R-35](#r-35-package-the-language-reference-as-package-data) | Package the Language Reference as package data | low | accepted | -| [R-36](#r-36-get_language_reference-versioned-export) | `get_language_reference()` versioned export | medium | accepted | +| [R-34](#r-34-language-reference-document-languagemd) | Language Reference document (`LANGUAGE.md`) | medium | done | +| [R-35](#r-35-package-the-language-reference-as-package-data) | Package the Language Reference as package data | low | done | +| [R-36](#r-36-get_language_reference-versioned-export) | `get_language_reference()` versioned export | medium | done | --- @@ -1038,7 +1038,7 @@ failures to `TransformationError`); `split` rule in `transon/rules.py`; total `i ### R-34. Language Reference document (`LANGUAGE.md`) -**Status**: accepted · **Severity**: medium · +**Status**: done · **Severity**: medium · **Source**: [`proposals/0008-language-reference-export.md`](proposals/0008-language-reference-export.md) (Deliverable 1) A new hand-written `docs/LANGUAGE.md`: the template-language reference for authors (human or @@ -1056,9 +1056,19 @@ hand-maintained copies (spec §2/§11, the class docstring, README overlap) with author-scoped, pinnable document — the `transon-authoring` authority-ladder gap stays open and repair loops keep rediscovering semantics. +**Shipped**: `docs/LANGUAGE.md` (7 pinned sections: preamble, templates-and-the-marker, +context-and-scoping, the-no_content-model, error-model, expressions-and-calls, +composition-patterns). Consolidation: spec §2 reduced to the engine-internal view, §4 +to a pointer + the Recursion budget invariant (per-rule facts folded into +`transon/rules.py` docstrings — accessors' scope errors, `attr` error split, `map`/ +`filter`/`zip`/`join`/`file` edge cases, `expr`/`call` mode errors + the sanctioned +reference pointer), §11 to a pointer; `Transformer` class docstring slimmed to the +embedder narrative (pitch owned by README); section-id pin in +`tests/test_reference.py`. Changelog entry under Unreleased. + ### R-35. Package the Language Reference as package data -**Status**: accepted · **Severity**: low · +**Status**: done · **Severity**: low · **Source**: [`proposals/0008-language-reference-export.md`](proposals/0008-language-reference-export.md) (Deliverable 2) Ship `LANGUAGE.md` in the wheel and sdist (e.g. `transon/resources/LANGUAGE.md`) so an @@ -1071,9 +1081,14 @@ line-endings normalized to `\n`) equal `get_language_reference()['content']`. **Impact if not done**: the `transon-authoring` harnesses mount no repo checkout, so an unpackaged reference is invisible to the primary consumer. +**Shipped**: `transon/resources/LANGUAGE.md` (committed copy; hatchling picks it up in +both wheel and sdist with no config change — verified by building both). +`tests/test_reference.py` loads it through `importlib.resources` and asserts it equals +both `get_language_reference()['content']` and the canonical `docs/LANGUAGE.md`. + ### R-36. `get_language_reference()` versioned export -**Status**: accepted · **Severity**: medium · +**Status**: done · **Severity**: medium · **Source**: [`proposals/0008-language-reference-export.md`](proposals/0008-language-reference-export.md) (Deliverable 3) `transon.reference.get_language_reference()` → `{reference_version, engine_version, format, @@ -1089,6 +1104,13 @@ scope here). version pin, no drift check; a 700-line context dump instead of one section as the unit of consumption. +**Shipped**: `transon/reference.py::get_language_reference()` — `REFERENCE_VERSION` +`'1.0'`, fence-aware deterministic `##` splitting with GitHub-style slug ids and +collision suffixes, preamble rule, engine-version degradation to `None` when not +installed; `python -m transon.reference` CLI. Spec §5.2 documents the export. Tests: +shape, section pin, concatenation parity, splitter unit cases in +`tests/test_reference.py`. + --- ## Suggested sequencing diff --git a/docs/SPECIFICATION.md b/docs/SPECIFICATION.md index 59e4756..c7ae060 100644 --- a/docs/SPECIFICATION.md +++ b/docs/SPECIFICATION.md @@ -31,6 +31,9 @@ producing JSON *output*. It is inspired by XSLT and JsonLogic. | `transon/functions.py` | Functions for the `call` rule (registered via `register_function`) | | `transon/docs.py` | Documentation generator: harvests docstrings + test cases into JSON | | `transon/metadata.py` | Editor-metadata export (`get_editor_metadata`) for the visual editor (§5.1) | +| `transon/reference.py` | Language Reference export (`get_language_reference`) — serves the packaged `LANGUAGE.md` (§5.2) | +| `transon/resources/LANGUAGE.md` | Packaged copy of `docs/LANGUAGE.md` (ships in the wheel/sdist; a test asserts identity) | +| `docs/LANGUAGE.md` | **Template Language Reference** — author-facing, cross-cutting semantics (canonical, hand-edited) | | `transon/tests/` | **Example corpus**: table-driven test cases that double as documentation | | `tests/` | Plain pytest tests for engine mechanics (errors, extension, docs generation) | | `.github/workflows/dev.yml` | CI: pytest + coverage on Python 3.9–3.13 (uv) | @@ -43,137 +46,68 @@ Packaging is uv / PEP 621 (`pyproject.toml`, `uv.lock`). Runtime dependencies: n ## 2. Core concepts -### 2.1 Templates and the marker +> **Author-facing semantics live in [`docs/LANGUAGE.md`](LANGUAGE.md)** — the Template +> Language Reference (evaluation model, scoping, the `NO_CONTENT` model, the error +> taxonomy, `expr`/`call` machinery, composition patterns), also served by +> `transon.reference.get_language_reference()` (§5.2). This section keeps only the +> **engine-internal** view: which code implements those semantics and the +> implementation invariants a contributor must preserve. -A template is any JSON value. The engine walks it recursively (`Transformer.walk`): +### 2.1 Template walk -- **list** → walk each element, return a new list (`walk_list`). -- **dict containing the marker key** (default `"$"`) → this is a **rule invocation**; - dispatch to the registered rule named by the marker's value (`walk_rule`). -- **dict without the marker** → walk each value, return a new dict with the same keys (`walk_dict`). -- **anything else** (scalar) → returned as-is (`walk_scalar`). - -The marker is configurable per `Transformer` instance (`marker=` constructor kwarg). -To emit a literal dict that contains the marker key, use the `object` rule in `fields` -mode (`{"$": "object", "fields": {"$": ...}}`): the keys of `fields` are emitted -verbatim while the values are walked as templates (R-14). The single-pair form -(`{"$": "object", "key": "$", "value": ...}`) also works for one literal key. - -A rule invocation dict carries its parameters as sibling keys of the marker: - -```json -{"$": "attr", "name": "x"} -``` - -Rule parameters are themselves templates (walked recursively), except where a rule -explicitly requires a constant (e.g. `expr.op`, `call.name`, `chain.funcs` list -structure). +`Transformer.walk` dispatches by JSON type: **list** → `walk_list`, **dict with the +marker key** → `walk_rule` (registry dispatch), **dict without** → `walk_dict`, +**scalar** → `walk_scalar`. The marker is per-instance (`marker=` constructor kwarg). +Rule parameters are walked recursively except parameters registered as constants +(`ParamKind.CONSTANT`, e.g. `expr.op`, `call.name`). The literal-marker escape is the +`object` rule's `fields` mode (R-14). ### 2.2 Context -`Context` (in `transformers.py`) is a linked chain of scopes. Each context holds: - -- `this` — the current value (root context: the transformation input). -- Iteration properties — `item`, `index` (lists), `key`, `value`, `index` (dicts); - only present inside `map`/`filter` iterations. -- User variables — arbitrary names written by the `set` rule, read by `get`. -- `parent` — the context this one was derived from. - -`context.derive(**props)` creates a child linked via `parent`: it stores only the -new props (e.g. `this`, iteration slots). Reads of user variables walk the parent -chain; the first `set` in a derived scope materializes inherited variables into the -local dict so writes stay isolated. `set` writes into the data dict of the exact -context object it executes in. - -#### Variable scoping (`set` / `get`) - -Template authors should treat these rules as the contract (R-15): - -| Where `set` runs | Visible to | -|---|---| -| Descendant scopes (`derive()` after the `set`) | Yes — visible via parent-chain lookup; first `set` in a child snapshots inherited vars | -| Later sibling keys/items in the same literal dict/list | Yes — siblings share one context object; order matters | -| Earlier sibling keys/items in the same literal dict/list | No — already evaluated | -| First func of a `chain` | Caller's scope, later `chain` funcs, and later siblings outside the `chain` | -| Later `chain` func, `map`/`filter` iteration, etc. | Only that derived scope and its descendants | -| Parent scope after a derived scope ends | No | -| `include` sub-template | No — separate `transform()` | - -Consequences (all verified against the implementation): - -- Variables flow **downward only**: a `derive()` performed *after* a `set` carries the - variable; the parent's own dict is untouched by sets in derived contexts. So a `set` - inside a `map` item or inside a non-first `chain` step is invisible once that scope - ends. -- `walk_dict`/`walk_list` pass the **same context object** to all siblings, so a `set` - executed directly at one key of a literal dict is visible to later-evaluated sibling - keys (insertion order). The first func of a `chain` also runs in the caller's - context, so a `set` there escapes into the caller's scope; subsequent funcs run in - derived contexts and their sets do not. -- `get` walks the current context's data dict, then the parent chain for undefined - names — so ancestor variables are visible without eager copying on `derive()`. The - first `set` in a derived scope materializes inherited variables locally. - -Refactoring pitfall: wrapping a step in `chain`, reordering dict keys, or moving a -`set` can change visibility with no error — consult the table above. - -The names `this`, `item`, `key`, `value`, `index` are **reserved** -(`Context.RESERVED_NAMES`): `Context.__contains__`, `__getitem__`, `__setitem__` -check against them, so they cannot be used as variable names with `set`/`get` -(violation raises `DefinitionError`). +`Context` (in `transformers.py`) is a linked chain of scopes holding `this`, the +iteration slots, user variables, and a `parent` link. Implementation invariants +(R-15/R-22 — the observable scoping model built on them is specified in the Language +Reference, "Context and scoping"): + +- `context.derive(**props)` stores **only** the new props (copy-on-write); reads of + user variables walk the parent chain; the **first** `set` in a derived scope + materializes inherited variables locally so writes stay isolated. +- `set` writes into the data dict of the exact context object it executes in; + `walk_dict`/`walk_list` pass the **same** context object to all siblings, and the + first func of a `chain` runs in the caller's context — these two facts produce the + documented sibling-order and first-chain-func visibility. +- The names `this`, `item`, `key`, `value`, `index` are **reserved** + (`Context.RESERVED_NAMES`): `__contains__`/`__getitem__`/`__setitem__` check against + them and raise `DefinitionError`. ### 2.3 NO_CONTENT — the "no value" sentinel `Transformer.NO_CONTENT` is a singleton `NoContent` instance representing the absence -of a value (distinct from JSON `null`/Python `None`). Semantics: - -- **Producers**: `attr` (missing key/index), `get` (undefined variable), `file` (always), - `include` (when the included template yields `NO_CONTENT`), `join` (when there are no - items to join). Optional `default` on `attr`, `get`, `format`, `include`, and `join` - returns a substitute instead of `NO_CONTENT`. -- **Falsiness**: `NoContent` is falsy in boolean context (`bool()`, `expr` `and`/`or`), - so logical operators can fall back from `NO_CONTENT` (e.g. `chain` of `join` then - `expr` `or` with a fallback value). Rules that test for absence still use identity - (`is NO_CONTENT`), not truthiness. -- **Absorption**: `NoContent.__getitem__` returns `self`, so further `attr` lookups on - a missing value stay `NO_CONTENT` instead of raising. -- **Consumers (skip/filter behavior)**: - - `map`: items (or key/value pairs) that evaluate to `NO_CONTENT` are omitted. - - `object`: in `key`/`value` mode returns `{}` if key or value is `NO_CONTENT`; - in `fields` mode omits each entry whose value is `NO_CONTENT`. - - `file`: skips writing if name or content is `NO_CONTENT`. - - `filter`: a condition evaluating to `NO_CONTENT` excludes the element. - - `join`: items that evaluate to `NO_CONTENT` are omitted before concatenation; when - no items remain the result is `NO_CONTENT` unless `default` is provided. - - `split`: when `context.this` is `NO_CONTENT`, returns `NO_CONTENT` (passthrough). -- **`format`**: returns `NO_CONTENT` when the formatting value (or any unpacked list - element or dict key/value) is `NO_CONTENT`, unless `default` is provided. -- **Top-level `transform()`**: by default maps a top-level `NO_CONTENT` result to - `None` via the `no_content` parameter (see §3.1). Pass `Transformer.NO_CONTENT` - as `no_content` to receive the raw sentinel. +of a value (distinct from JSON `null`/Python `None`). Implementation facts: -### 2.4 Error model - -| Exception | Meaning | Raised when | -|---|---|---| -| `DefinitionError` | The template is malformed | Unknown rule/operator/function name; missing required rule parameter (`Transformer.require` or `Transformer.validate()`); unknown rule parameters; ambiguous or incomplete mutually-exclusive parameter groups (`validate()`); `attr` with neither `name` nor `names`; `map` with no valid parameter combination; reserved variable name (`this`, `item`, `key`, `value`, `index`) used with `set`/`get`; iteration accessors (`item`, `key`, `value`, `index`) or `parent` used outside their valid scope; `expr`/`call` with a non-list or empty `values` parameter; non-list `chain.funcs` (`validate()`); `include` with no configured `template_loader` (default loader) | -| `TransformationError` | The template is valid but input data is incompatible | `map`/`filter` over a non-iterable (not list/dict); `join` over mixed-type items; `split` on a non-string/non-array input or with an invalid `sep`; `attr` lookup with an incompatible index type; `zip` over non-iterable items; `expr` operator applied to incompatible operand types; `call` with incompatible argument types or a function that rejects its arguments (e.g. empty `min`/`max`, bad epoch, invalid regex); `format` pattern referencing a missing key or index; `set`/`get` when a dynamic `name` evaluates to `NO_CONTENT`; `include` depth limit exceeded (nested include chain too deep) | +- `NoContent.__bool__` is `False` (falsiness powers `expr` `and`/`or` fallbacks) and + `NoContent.__getitem__` returns `self` (absorption for deep `attr` paths). Rules + that test for absence use **identity** (`is NO_CONTENT`), never truthiness. +- The skip-don't-emit propagation model is specified in the Language Reference ("The + NO_CONTENT model"); each rule's exact treatment is stated in that rule's docstring + and is part of the public contract (§10). +- `transform()` substitutes a top-level `NO_CONTENT` via its `no_content` parameter + (default `None`, §3.1); inside evaluation rules always see the raw sentinel. -Both are exported from the package root. By default, errors are raised lazily during -`transform()` — there is no automatic validation. Opt in with `Transformer.validate()` -or `Transformer(..., validate=True)` for a static walk that raises `DefinitionError` -for unknown rules, unknown rule parameters, missing required parameters, ambiguous -mutually-exclusive parameter combinations, and invalid literal operator/function names -(see §3.4). +### 2.4 Error model -`DefinitionError` and `TransformationError` messages include the template location -where the failure occurred (a path of dict keys, list indices, rule names, and rule -parameter names), for example: +`DefinitionError` (malformed template) and `TransformationError` (valid template, +incompatible data) are both defined in `transformers.py` and exported from the package +root. The taxonomy as an author experiences it is in the Language Reference ("Error +model"); which conditions each rule raises is in that rule's docstring. Engine-side +contract: -``` -value is not iterable: 'not-a-list' - at template → pipeline → chain → funcs[0] → map -``` +- Errors are raised **lazily** during `transform()`; static checking is the opt-in + `validate()` walk (§3.4). +- Every message is routed through `format_error_message`, which appends the template + location (`at template → …`) from the template-path `ContextVar` maintained by + `walk` — new raise sites must use `t.definition_error` / `t.transformation_error` + (or `format_error_message`) so the path is never lost. --- @@ -323,166 +257,37 @@ parameter with no descriptor defaults to a dynamic template (`ParamKind.DYNAMIC` ## 4. Built-in rule reference -All rules live in `transon/rules.py`. "Dynamic" parameters are walked as templates; -"constant" parameters are read verbatim. - -### 4.1 Context accessors (no parameters) - -| Rule | Returns | Valid scope | -|---|---|---| -| `this` | `context.this` — current value | anywhere | -| `parent` | `context.parent.this` — value of previous scope | any non-root scope | -| `item` | current list element | inside `map`/`filter` over a list | -| `index` | 0-based iteration index | inside `map`/`filter` | -| `key` | current dict key | inside `map`/`filter` over a dict | -| `value` | current dict value | inside `map`/`filter` over a dict | - -Accessing `parent` in the root context or an iteration property outside its scope -raises `DefinitionError`. - -### 4.2 Variables - -| Rule | Parameters | Semantics | -|---|---|---| -| `set` | `name` (dynamic) | Stores `context.this` under `name` in the *current* context; returns `context.this` (pass-through, usable as a tap inside `chain`). Raises `TransformationError` if `name` evaluates to `NO_CONTENT`. Scoping: see §2.2 (variable scoping table). | -| `get` | `name` (dynamic), `default` (optional, dynamic) | Returns the stored value from the *current* context (ancestor variables resolved via parent-chain lookup). Returns `NO_CONTENT` if undefined. Raises `TransformationError` if `name` evaluates to `NO_CONTENT`. When `default` is provided, returns its evaluation instead of `NO_CONTENT`. Scoping: see §2.2. | - -Examples for each scoping case live in `transon/tests/test_set.py` (docs-site examples). - -### 4.3 Data access — `attr` - -Parameters (mutually exclusive; one required, else `DefinitionError`): - -- `name` (dynamic): single key or numeric index into `context.this`. -- `names` (dynamic): list of keys/indexes, applied sequentially (deep path). -- `default` (optional, dynamic): returned instead of `NO_CONTENT` when the lookup misses. - -Missing key (`KeyError`) or index out of range (`IndexError`) → `NO_CONTENT` (or -`default` when provided). -When `name` or any path segment in `names` evaluates to `NO_CONTENT` → `NO_CONTENT` -(uniform regardless of container type). -Other lookup failures (e.g. `TypeError` indexing a string with a string) → -`TransformationError`. - -### 4.4 Structure builders - -| Rule | Parameters | Semantics | -|---|---|---| -| `object` | exactly one of: `key`+`value` \| `fields` | `key`+`value` (dynamic): single-pair dict `{key: value}`; `{}` if either side is `NO_CONTENT`. For dynamically-named attributes. `fields`: literal mapping whose keys are emitted verbatim (including the marker `$` — the canonical literal-marker-key escape, R-14) and whose values are walked as templates; entries with a `NO_CONTENT` value are omitted. | -| `map` | exactly one of: `item` \| `items` \| `key`+`value` | Iterates `context.this` (list or dict). `item`: one output element per input element → list. `items`: template yields a *list* of elements per input element, concatenated → list. `key`+`value`: → dict. `NO_CONTENT` results are skipped. Each iteration derives a sub-context with `this`=element plus iteration props. | -| `filter` | `cond` (required, dynamic) | Keeps elements where `cond` is truthy (and not `NO_CONTENT`). Preserves container type: list→list, dict→dict. | -| `zip` | `items` (required, dynamic) | Transposes iterables like Python's `zip`: each output row is a **list** (`[list(row) for row in zip(*items)]`). Non-iterable items → `TransformationError`. | -| `join` | `items` (required, dynamic), `sep` (dynamic, strings only, default `""`), `default` (optional, dynamic) | Type-homogeneous concatenation: all-strings → `sep.join`; all-lists → flatten one level; all-dicts → merged dict (later keys win). Items that evaluate to `NO_CONTENT` are omitted before concatenation. When no items remain → `NO_CONTENT` (or `default` when provided). Mixed types → `TransformationError`. `sep` must evaluate to a string when joining strings. | -| `split` | `sep` (required, dynamic) | Inverse of string/list `join`. Input `NO_CONTENT` → `NO_CONTENT`. String input: `sep` must be a non-empty string → list of strings (empty `sep` → `TransformationError`). Array input: result is a list of lists; `sep` is a single non-array element (split on `==`) or a non-empty array (split on each contiguous subsequence occurrence). Empty-array `sep` → `TransformationError`. Because an array `sep` means subsequence, you cannot split on a separator *element* that is itself an array. Other input types → `TransformationError`. | -| `chain` | `funcs` (required; list of templates) | Function composition: walks each template in order, each result becomes `this` of a derived context for the next. `chain(f1, f2, f3)(x) == f3(f2(f1(x)))`. | - -### 4.5 Computation - -| Rule | Parameters | Semantics | -|---|---|---| -| `expr` | `op` (required, constant), optionally `value` (dynamic) or `values` (dynamic) | No param → unary `op(this)`. `value` → binary `op(this, value)`. `values` → `reduce(op, values)` (**`this` is ignored**). | -| `call` | `name` (required, constant), optionally `value` or `values` (dynamic) | No param → `fn(this)`. `value` → `fn(value)`. `values` → `fn(*values)`. `this` is ignored when params given. | -| `format` | `pattern` (required, dynamic), `value` (optional, dynamic; defaults to `this`), `default` (optional, dynamic) | Python `str.format`. `pattern` must evaluate to a string. Returns `NO_CONTENT` when the formatting value (or any unpacked list element or dict key/value) is `NO_CONTENT`, unless `default` is provided. List value → positional unpack `pattern.format(*v)`; dict value → keyword unpack `pattern.format(**v)`; otherwise single argument. | -| `switch` | `key` (required, dynamic), `cases` (required, `mapping`), `default` (optional, dynamic) | **Lazy** equality dispatch: evaluates `key`, then walks **only** the matching entry of `cases` (a literal-keyed mapping of templates). No match — including `key` → `NO_CONTENT` — evaluates `default` if present, else `NO_CONTENT`. Non-selected cases are never walked. | -| `cond` | `cases` (required, `arms`), `default` (optional, dynamic) | **Lazy** ordered conditional (subsumes `if`/`else`): `cases` is an ordered list of `{when, then}` arms. Walks each `when` in order; the first truthy one selects its `then` (the only `then` walked). A `when` of `NO_CONTENT` is falsy. No match → `default` if present, else `NO_CONTENT`. | - -### 4.6 Side effects & composition - -| Rule | Parameters | Semantics | -|---|---|---| -| `file` | `name`, `content` (both required, dynamic) | Calls the configured `file_writer(name, content)`. Skipped if either is `NO_CONTENT`. Always returns `NO_CONTENT` (so `map` over `file` yields `[]`). | -| `include` | `name` (required, dynamic), `default` (optional, dynamic) | Loads a sub-`Transformer` via the configured `template_loader` and runs it against `context.this`. Variables/context do **not** cross the boundary — only the value. The `template_loader` is always called as `loader(name, context=...)` and handed an `IncludeContext` (parent loader, marker, depth guard, include-stack); it **constructs** the sub-`Transformer` itself (e.g. via `context.transformer(template)`). The sub-`Transformer` therefore inherits the parent transformer's marker by default (so a sub-template using the default marker stays consistent across the boundary; pass an explicit `marker` to `context.transformer` to pin a different one), the parent's `template_loader` propagates so recursive/self-`include`ing templates re-resolve without per-host patching, and the loaded instance is never mutated. Sub-result `NO_CONTENT` is propagated as this transformer's `NO_CONTENT` (or `default` when provided). Nested includes are tracked by name; exceeding `max_include_depth` (constructor parameter, default 50) raises `TransformationError` with the include chain in the message. | - -> **Recursion budget.** Walking one level of template nesting consumes a bounded, small number of -> Python call-stack frames — one core recursion frame per node (no `walk`/`_walk`-style doubling). -> Because the generated `transon-blockly` editor codec self-`include`s once per document node -> (AD-030), its reachable nesting depth is governed by this budget, not by `max_include_depth`. The -> engine therefore transforms self-`include`ing templates at nesting depths well past the editor's -> deepest generator (`G_encode`, depth 41) within CPython's default recursion limit (1000). -> Exceeding `max_include_depth` MUST surface as the `include` depth-limit `TransformationError`, -> never a raw `RecursionError`. (Roadmap R-32.) - -### 4.7 Built-in operators (`expr`) - -Each operator has a mnemonic and a code-style alias. Most map to the Python -`operator` module; `in` is a total membership predicate: - -| Mnemonic | Alias | Python impl | Note | -|---|---|---|---| -| `lt le eq ne ge gt` | `< <= == != >= >` | `operator.lt` … | comparisons | -| `add sub mul div mod` | `+ - * / %` | `operator.add` …, `truediv` for `div` | `+` also concatenates strings/lists | -| `and or not` | `&& \|\| !` | logical `and`/`or`/`not` | Python truthiness; returns operands, not always `bool` | -| `in` | `in` | membership | **Total** (never raises). Binary `op(a, b)` = "`a` is a member of `b`": array → element membership; string → substring (`a` must be a string, else `false`); object → key presence (`a` must be a string, else `false`); any other container → `false`. | - -### 4.8 Built-in functions (`call`) - -Existing conversion family: `str`, `int`, `float` (Python builtins), `type` (JSON type -name — **total** over well-formed JSON), and `bool` (Python truthiness — **total**). - -Every other built-in is a wrapper that converts documented failure modes into -`TransformationError` itself (`rule_call` only catches `TypeError`). Multi-argument -forms use `call` `values`. - -#### Strings - -| name | args | notes | -|---|---|---| -| `upper` `lower` `capitalize` | unary | non-string → `TransformationError` | -| `replace` | `[s, old, new]` | total on strings | -| `removeprefix` `removesuffix` | `[s, fix]` | prefix/suffix removal (not character-set strip) | -| `strip` `lstrip` `rstrip` | unary or `[s, chars]` | character-set trimming | - -#### Strings and arrays - -| name | args | notes | -|---|---|---| -| `slice` | `[x, start]` or `[x, start, stop]` | `x` string or array; Python slice semantics (negative indices, out-of-range clamps). Non-int index or other `x` type → `TransformationError`. | -| `reversed` | unary | array → reversed array; string → reversed string; other → `TransformationError` | - -#### Epoch dates (UTC only) - -| name | args | notes | -|---|---|---| -| `from_epoch` | unary or `[n, fmt]` | Epoch **seconds** (int/float) → string. Default format is fixed ISO-8601 `YYYY-MM-DDThh:mm:ssZ`. Fractional seconds are **truncated**. Non-numeric / NaN / inf / out-of-range → `TransformationError`. | -| `to_epoch` | `[s]` or `[s, fmt]` | String → epoch seconds (int). Default accepts the same fixed ISO form. Parse failure → `TransformationError`. | - -Shared format whitelist (locale-free, deterministic): `%Y %m %d %H %M %S %j %z %%` -plus literal text. Any other directive (including `%a %b %c %x %X %p %Z`) → -`TransformationError`. - -#### Collections and numerics - -| name | args | notes | -|---|---|---| -| `length` | unary | string / array / object → int; other → `TransformationError` | -| `flatten` | unary | one level; non-array input or non-array element → `TransformationError` | -| `sum` | unary | array of numbers; `sum([]) == 0`; booleans and non-numeric elements → `TransformationError` | -| `min` `max` | `[array]` or `[array, default]` | empty without `default` → `TransformationError` | -| `sorted` | unary | homogeneous scalars only; mixed types → `TransformationError` | -| `unique` | unary | first-occurrence order; dict/list elements → `TransformationError` | -| `abs` `floor` `ceil` | unary | numbers only (`math.floor`/`ceil`); non-number → `TransformationError` | -| `round` | unary or `[x, ndigits]` | numbers only | - -#### Encoding, hashing, regex - -| name | args | notes | -|---|---|---| -| `b64encode` | unary | str → UTF-8 → standard-alphabet base64 str | -| `b64decode` | unary | invalid base64 or non-UTF-8 payload → `TransformationError` | -| `uuid5` | `[namespace, name]` | Deterministic UUID; `namespace` is `dns`/`url`/`oid`/`x500` or a UUID string. Random `uuid4` is deliberately absent (determinism). | -| `regex_match` | `[s, pattern]` | On match: array of capture groups (`groups()`; unmatched optionals → `null`); with no groups: `[full match]`. On no match: `null`. Condition use: `bool(regex_match(...))`. Invalid pattern → `TransformationError`. | -| `regex_replace` | `[s, pattern, repl]` | str; invalid pattern → `TransformationError` | - -**Regex dialect** is Python `re` (the same engine on CPython and the Pyodide reference -host). ReDoS exposure is a **host** responsibility (e.g. per-case timeouts); the engine -does not limit pattern complexity. +Per-rule behavior is **not** specified here (ownership principle, RFC 0008): each +rule's semantics — parameters, modes, edge cases, `NO_CONTENT` treatment, error +conditions — live in its registration docs in `transon/rules.py` (docstrings + +`register_rule` param kwargs), which every export carries (`get_all_docs()`, +`get_editor_metadata()['docs']`) and the docs site renders. Operator and function +catalogs live the same way in `transon/operators.py` / `transon/functions.py` and the +`expr` `op` / `call` `name` parameter docs. Cross-cutting semantics (evaluation model, +scoping, `NO_CONTENT`, errors, `expr`/`call` machinery) are in +[`docs/LANGUAGE.md`](LANGUAGE.md). + +### Recursion budget + +Walking one level of template nesting consumes a bounded, small number of Python +call-stack frames — one core recursion frame per node (no `walk`/`_walk`-style +doubling). Because the generated `transon-blockly` editor codec self-`include`s once +per document node (AD-030), its reachable nesting depth is governed by this budget, +not by `max_include_depth`. The engine therefore transforms self-`include`ing +templates at nesting depths well past the editor's deepest generator (`G_encode`, +depth 41) within CPython's default recursion limit (1000). Exceeding +`max_include_depth` MUST surface as the `include` depth-limit `TransformationError`, +never a raw `RecursionError`. (Roadmap R-32.) --- ## 5. Documentation pipeline The documentation (and the playground at https://transon-org.github.io/) is **generated -from source artifacts**; nothing is hand-maintained separately: +from source artifacts**; the one hand-written artifact is +[`docs/LANGUAGE.md`](LANGUAGE.md) (the Template Language Reference — cross-cutting +semantics only, no per-entity sections; served packaged via §5.2, its section shape +pinned by `tests/test_reference.py`). Everything else is harvested: 1. **Rule docs** — rule function docstrings (markdown, may embed plantuml). 2. **Parameter docs** — `**params` kwargs of `register_rule`. @@ -600,6 +405,30 @@ into a lean structural `catalog` (consumed by the editor's generators) and an - `docs.template_loader` makes every test case's template `include`-able by its class name (e.g. `{"$": "include", "name": "MapListsToDict"}`). +### 5.2 Language Reference export + +`transon/reference.py` provides `get_language_reference()` — a dedicated, versioned +export (RFC 0008, R-36) serving the packaged `LANGUAGE.md` offline: + +- Shape: `{reference_version, engine_version, format: "markdown", content, sections}`. + `content` is the byte-exact document (UTF-8, `\n` newlines); `sections` is a flat, + ordered split on top-level `##` headings (each section includes its own heading + line; deeper headings stay inside their parent; a non-empty intro before the first + `##` becomes a leading `{"id": "preamble"}` section; ids are GitHub-style slugs, + collisions suffixed `-2`, `-3`, … in document order). Concatenating `sections` + reproduces `content` exactly. +- `REFERENCE_VERSION` policy (mirrors `METADATA_VERSION`): additive changes — a new + section, appended prose, a new optional field — bump the minor; removing/renaming a + section `id`, changing the `sections` shape, or dropping/renaming a top-level field + is breaking and bumps the major. Consumers MUST fail loudly on an unsupported major. +- The export is **engine-global** (base `Transformer` only; no `cls=` parameter) and + states language facts only. +- Packaging: `transon/resources/LANGUAGE.md` ships in the wheel and sdist; + `docs/LANGUAGE.md` is the canonical hand-edited source and + `tests/test_reference.py` asserts the two are identical, pins the section-id list, + and checks the split parity. +- `python -m transon.reference` prints this JSON. + --- ## 6. Testing conventions @@ -724,37 +553,21 @@ tagged example cases. (no `walk`/`_walk`-style per-node doubling), so self-`include`ing templates reach depths well past `G_encode` (41) before the host stack overflows; over-depth surfaces as the `include` depth-limit `TransformationError`, never a raw `RecursionError` - (§4.6 "Recursion budget", Roadmap R-32). + (§4 "Recursion budget", Roadmap R-32). --- ## 11. Engine data flow (reference example) -Template: - -```json -{ - "$": "chain", - "funcs": [ - {"$": "zip", "items": [{"$": "attr", "name": "keys"}, - {"$": "attr", "name": "values"}]}, - {"$": "map", "key": {"$": "attr", "name": 0}, - "value": {"$": "attr", "name": 1}} - ] -} -``` - -Input `{"keys": ["a","b"], "values": [1,2]}` → root context `this=input` → -`zip` produces `[["a",1], ["b",2]]` → chain derives context with that as `this` → -`map` iterates, each pair becomes `this`/`item` in a sub-context → `attr` with numeric -names indexes the tuple → output `{"a": 1, "b": 2}`. +Relocated to the Language Reference: [`docs/LANGUAGE.md`](LANGUAGE.md), "Composition +patterns" (the `zip` + `map` worked end-to-end flow). --- ## 12. Known issues & design questions Suspected accidental behaviors and open design questions are tracked exclusively in -[`docs/ROADMAP.md`](ROADMAP.md) (items `R-01`…`R-22`), each with impact analysis, +[`docs/ROADMAP.md`](ROADMAP.md) (`R-xx` items), each with impact analysis, fix options, and a decision status. This document describes current behavior only. **Do not "fix" quirky behavior silently** — every behavior change needs an explicit diff --git a/docs/proposals/0008-language-reference-export.md b/docs/proposals/0008-language-reference-export.md index 6347caf..53363c2 100644 --- a/docs/proposals/0008-language-reference-export.md +++ b/docs/proposals/0008-language-reference-export.md @@ -1,6 +1,6 @@ # RFC 0008 — Author-facing Language Reference: document, packaging, and export API -- **Status:** Accepted (2026-07-18) — awaiting implementation +- **Status:** Implemented (2026-07-18, in `main` — unreleased; ships in the next release per Sequencing) - **Created:** 2026-07-16 - **Amended:** 2026-07-18 — consolidation scope extended to the `Transformer` docstring and `README.md`; single ownership principle (structure in the catalog, per-entity behavior in diff --git a/docs/proposals/README.md b/docs/proposals/README.md index 8f737ba..caeced6 100644 --- a/docs/proposals/README.md +++ b/docs/proposals/README.md @@ -43,7 +43,7 @@ is the *authoritative* record of each work item's status via its **R-number**. | [0005](0005-example-corpus-normalization.md) | Normalize exports to one flat example corpus | Implemented | R-31 | v0.1.6 | docs site, `transon-blockly` | | [0006](0006-transformer-recursion-depth-budget.md) | Bounded per-level recursion budget (self-`include` depth) | Implemented | R-32 | v0.1.7 | `transon-blockly` | | [0007](0007-builtin-function-library.md) | Grow the built-in function library | Implemented | R-33 | v0.1.8 | `transon-authoring`, `transon-blockly` | -| [0008](0008-language-reference-export.md) | Author-facing Language Reference: doc, packaging, export | Accepted | R-34, R-35, R-36 | — | `transon-authoring`, docs site | +| [0008](0008-language-reference-export.md) | Author-facing Language Reference: doc, packaging, export | Implemented | R-34, R-35, R-36 | unreleased | `transon-authoring`, docs site | ## Adding a new RFC diff --git a/tests/test_reference.py b/tests/test_reference.py new file mode 100644 index 0000000..669abd8 --- /dev/null +++ b/tests/test_reference.py @@ -0,0 +1,118 @@ +"""Shape, section-pin, split-parity, and packaging tests for the Language +Reference export (RFC 0008, R-34/R-35/R-36).""" +import importlib.resources +from pathlib import Path + +from transon.reference import ( + REFERENCE_VERSION, + _split_sections, + get_language_reference, +) + +#: The pinned section-id list (RFC 0008 drift protection): adding, renaming, or +#: removing a section in ``docs/LANGUAGE.md`` must update this pin **and** follow +#: the ``REFERENCE_VERSION`` policy — additive changes bump the minor, removals/ +#: renames are breaking and bump the major. Never a silent edit. +PINNED_SECTION_IDS = [ + 'preamble', + 'templates-and-the-marker', + 'context-and-scoping', + 'the-no_content-model', + 'error-model', + 'expressions-and-calls', + 'composition-patterns', +] + +REPO_ROOT = Path(__file__).resolve().parent.parent + + +def test_reference_shape(): + ref = get_language_reference() + assert set(ref) == { + 'reference_version', 'engine_version', 'format', 'content', 'sections', + } + assert ref['reference_version'] == REFERENCE_VERSION == '1.0' + assert ref['engine_version'] is None or isinstance(ref['engine_version'], str) + assert ref['format'] == 'markdown' + assert isinstance(ref['content'], str) and ref['content'] + assert '\r' not in ref['content'] + assert isinstance(ref['sections'], list) and ref['sections'] + + +def test_section_ids_are_pinned(): + ref = get_language_reference() + assert [section['id'] for section in ref['sections']] == PINNED_SECTION_IDS + + +def test_sections_concatenation_reproduces_content(): + ref = get_language_reference() + joined = ''.join(section['content'] for section in ref['sections']) + assert joined == ref['content'] + + +def test_section_fields(): + ref = get_language_reference() + preamble, *rest = ref['sections'] + assert preamble['id'] == 'preamble' + assert preamble['title'] == '' + assert preamble['heading_level'] is None + assert preamble['content'].strip() + for section in rest: + assert section['heading_level'] == 2 + assert section['title'] + assert section['content'].startswith(f"## {section['title']}\n") + assert set(section) == {'id', 'title', 'heading_level', 'content'} + + +def test_packaged_copy_is_served_and_matches_canonical(): + """Packaging parity (RFC 0008 Deliverable 2): the export serves the packaged + ``transon/resources/LANGUAGE.md`` via ``importlib.resources``, and that copy + is byte-identical (modulo line endings) to the canonical, hand-edited + ``docs/LANGUAGE.md``.""" + ref = get_language_reference() + packaged = ( + importlib.resources.files('transon') + .joinpath('resources/LANGUAGE.md') + .read_bytes() + .decode('utf-8') + .replace('\r\n', '\n') + .replace('\r', '\n') + ) + assert packaged == ref['content'] + canonical = (REPO_ROOT / 'docs' / 'LANGUAGE.md').read_text(encoding='utf-8') + canonical = canonical.replace('\r\n', '\n').replace('\r', '\n') + assert canonical == ref['content'], ( + 'docs/LANGUAGE.md and transon/resources/LANGUAGE.md have diverged — ' + 'copy the canonical docs/LANGUAGE.md over the packaged resource' + ) + + +def test_split_without_preamble(): + sections = _split_sections('## Only\nbody\n') + assert [s['id'] for s in sections] == ['only'] + + +def test_split_blank_intro_emits_no_preamble(): + sections = _split_sections('\n\n## First\nbody\n') + assert [s['id'] for s in sections] == ['first'] + + +def test_split_slug_collisions_get_suffixes(): + content = '## Dup\na\n## Dup\nb\n## Dup\nc\n' + sections = _split_sections(content) + assert [s['id'] for s in sections] == ['dup', 'dup-2', 'dup-3'] + assert ''.join(s['content'] for s in sections) == content + + +def test_split_ignores_headings_inside_code_fences(): + content = '# T\nintro\n\n## Real\n```\n## not a heading\n```\ntail\n' + sections = _split_sections(content) + assert [s['id'] for s in sections] == ['preamble', 'real'] + assert ''.join(s['content'] for s in sections) == content + + +def test_split_deeper_headings_stay_inside_parent(): + content = '## Top\n### Sub\nbody\n#### Deeper\n' + sections = _split_sections(content) + assert [s['id'] for s in sections] == ['top'] + assert '### Sub' in sections[0]['content'] diff --git a/transon/reference.py b/transon/reference.py new file mode 100644 index 0000000..1d0c969 --- /dev/null +++ b/transon/reference.py @@ -0,0 +1,129 @@ +"""Author-facing Language Reference export. + +A dedicated, versioned export — separate from the docs API — that serves the +packaged ``LANGUAGE.md`` (the template language's **cross-cutting** semantics: +evaluation model, scoping, ``NO_CONTENT``, error taxonomy, ``expr``/``call`` +machinery, composition patterns) to consumers that need it offline and pinned to +an engine version, the way ``get_editor_metadata()`` serves the catalog (see +``docs/proposals/0008-language-reference-export.md``). + +The export states **language facts only** — no consumer-specific shapes. It is +**engine-global**: it documents the built-in language of the base ``Transformer`` +only; unlike ``get_all_docs(cls=...)`` there is no class parameter. +""" +import importlib.metadata +import importlib.resources +import re + +REFERENCE_VERSION = '1.0' + +#: Characters kept by the GitHub-style heading slugger (besides spaces → hyphens). +_SLUG_KEEP = re.compile(r'[^0-9a-z _-]') + + +def _engine_version(): + """The installed ``transon`` distribution version, or ``None`` when unavailable. + + Same degradation contract as the metadata export: the reference must be + usable when ``transon`` is merely importable from source and not installed + as a distribution. + """ + try: + return importlib.metadata.version('transon') + except importlib.metadata.PackageNotFoundError: + return None + + +def _load_content(): + """The packaged ``LANGUAGE.md`` as UTF-8 text with ``\\n`` newlines.""" + resource = importlib.resources.files('transon').joinpath( + 'resources/LANGUAGE.md' + ) + text = resource.read_bytes().decode('utf-8') + return text.replace('\r\n', '\n').replace('\r', '\n') + + +def _slugify(title): + """GitHub-style slug of a heading title. + + Lowercase; markdown backticks dropped with the rest of the punctuation; + spaces become hyphens; alphanumerics, underscores, and hyphens survive. + """ + slug = _SLUG_KEEP.sub('', title.lower()) + return slug.replace(' ', '-') + + +def _split_sections(content): + """Split ``content`` into the flat, ordered ``sections`` list. + + Deterministic rules (RFC 0008 Deliverable 3): the split is on top-level + ``##`` headings only (``###``+ stays inside its parent; fenced code blocks + are opaque). Each section includes its own heading line. Content before the + first ``##`` heading becomes a leading ``preamble`` section, present only + when non-empty. Slug collisions get ``-2``, ``-3``, … suffixes in document + order. The concatenation of all sections reproduces ``content`` exactly. + """ + lines = content.splitlines(keepends=True) + boundaries = [] + in_fence = False + for index, line in enumerate(lines): + if line.lstrip().startswith('```'): + in_fence = not in_fence + continue + if not in_fence and line.startswith('## '): + boundaries.append(index) + + sections = [] + seen_ids = {} + + def _unique(slug): + count = seen_ids.get(slug, 0) + 1 + seen_ids[slug] = count + return slug if count == 1 else f'{slug}-{count}' + + first = boundaries[0] if boundaries else len(lines) + preamble = ''.join(lines[:first]) + if preamble.strip(): + sections.append({ + 'id': _unique('preamble'), + 'title': '', + 'heading_level': None, + 'content': preamble, + }) + + for position, start in enumerate(boundaries): + end = boundaries[position + 1] if position + 1 < len(boundaries) else len(lines) + title = lines[start][len('## '):].strip() + sections.append({ + 'id': _unique(_slugify(title)), + 'title': title, + 'heading_level': 2, + 'content': ''.join(lines[start:end]), + }) + return sections + + +def get_language_reference(): + """Return the versioned Language Reference document. + + The result carries a standalone ``reference_version`` (minor bump = + additive: a new section, appended prose, a new optional field; major bump = + breaking: a removed/renamed section ``id``, a changed ``sections`` shape, a + dropped/renamed top-level field — consumers MUST fail loudly on an + unsupported major), the ``engine_version``, the byte-exact ``content``, and + ``sections`` — a flat, ordered split of ``content`` so consumers can serve + targeted per-section lookups instead of the whole document. + """ + content = _load_content() + return { + 'reference_version': REFERENCE_VERSION, + 'engine_version': _engine_version(), + 'format': 'markdown', + 'content': content, + 'sections': _split_sections(content), + } + + +if __name__ == '__main__': # pragma: no cover + import json + print(json.dumps(get_language_reference(), indent=4)) diff --git a/transon/resources/LANGUAGE.md b/transon/resources/LANGUAGE.md new file mode 100644 index 0000000..372a31d --- /dev/null +++ b/transon/resources/LANGUAGE.md @@ -0,0 +1,207 @@ +# Transon — Template Language Reference + +> **Audience**: template authors (human or agent). This document is the **cross-cutting** +> semantics of the Transon template language: the evaluation model, scoping, the +> `NO_CONTENT` propagation model, the error taxonomy, the `expr`/`call` machinery, and +> composition patterns. It deliberately contains **no per-rule reference**: what each +> individual rule, operator, or function does — its parameters, modes, and edge cases — +> lives in that entry's own documentation, exported by the engine +> (`transon.docs.get_all_docs()` / `transon.metadata.get_editor_metadata()`) and rendered +> on the [docs site](https://transon-org.github.io/). Entity names appear here only as +> illustrations. Executable examples live in the example corpus shipped with those same +> exports. + +## Templates and the marker + +A template is any JSON value. The engine walks it top-down and rebuilds it node by node; +each node is handled by its JSON type: + +- a **list** → walk every element, return a new list; +- a **dict containing the marker key** (default `"$"`) → a **rule invocation** (see below); +- a **dict without the marker** → walk every value, return a new dict with the same keys; +- any **scalar** (string, number, boolean, `null`) → copied through unchanged. + +A template that contains no markers is therefore reproduced as a deep copy of itself — +rules are the only thing that injects data. + +A dict is a rule *only* when it contains the marker key; the marker's value names the +rule and the sibling keys are the rule's parameters: + +```json +{"$": "attr", "name": "x"} +``` + +Rule parameters are themselves templates and are walked recursively, so rules nest +arbitrarily — this is why even arithmetic is expressed as nested rules rather than a +string mini-language. A handful of parameters are documented as **constant** (for +example an operator name): those are read verbatim, never walked. + +The marker is configurable per transformation (`marker=` on the transformer). To emit a +literal dict that really contains the marker key — data that would otherwise be read as +a rule invocation — use the `object` rule's `fields` mode, whose keys are emitted +verbatim while its values stay templates; its single-pair `key`/`value` mode also +produces one literal key. + +When one template `include`s another, the sub-template inherits the parent's marker by +default, so a template tree written against the default marker stays consistent across +`include` boundaries; the loader may pin a different marker explicitly. + +## Context and scoping + +Evaluation carries a **context** — a linked chain of scopes. Each context holds: + +- `this` — the current value (in the root context: the transformation input); +- iteration properties — `item`, `index` over lists; `key`, `value`, `index` over + dicts — present only inside scopes derived by the iterating rules (`map`, `filter`); +- user variables — arbitrary names written by `set`, read by `get`; +- a link to the **parent** scope it was derived from. + +Rules that carry a value into a sub-template derive a **child scope**: each `map`/ +`filter` iteration derives one per element (exposing the iteration properties), and +each `chain` step after the first derives one whose `this` is the previous step's +result. The context accessor rules (`this`, `parent`, `item`, `key`, `value`, `index`) +read these slots; each is valid only where its slot exists, and using one outside its +valid scope is a template mistake (`DefinitionError`). + +Variables flow **downward only**. The rules of visibility, in decreasing surprise: + +| Where a `set` runs | Its variable is visible to | +|---|---| +| any scope | descendant scopes derived *after* the `set` | +| directly at a key/element of a literal dict/list | later-evaluated siblings in that dict/list (they share one scope; dict key / list index order matters) | +| the **first** func of a `chain` | the caller's scope — later `chain` funcs *and* later siblings outside the `chain` | +| a later `chain` func, a `map`/`filter` iteration | only that derived scope and its descendants | +| anywhere | **never** the parent scope after the derived scope ends; **never** an `include`d sub-template (that is a separate transformation — only the value crosses the boundary) | + +Refactoring pitfall: wrapping a step in `chain`, reordering dict keys, or moving a +`set` can change visibility with no error — consult the table. + +The names `this`, `item`, `key`, `value`, and `index` are **reserved** and cannot be +used as variable names with `set`/`get` (violation raises `DefinitionError`). + +## The NO_CONTENT model + +`NO_CONTENT` is the language's "no value" sentinel — distinct from JSON `null`. `null` +is a value you can store and emit; `NO_CONTENT` means *there is nothing here*, and the +language is built so that missing data disappears from the output instead of blowing up +or leaving `null` holes. + +- **Where it comes from**: lookups that miss (an absent attribute or path, an undefined + variable), aggregations left with nothing (a `join` whose items all vanished), rules + that never produce a value (`file`), and sub-transforms that themselves produced + nothing (`include`). +- **Skip, don't emit**: container rules *omit* a `NO_CONTENT` piece rather than emitting + `null` — as illustrations: `map` drops the item, `object` omits the entry, `filter` + excludes the element, `join` leaves the item out. The exact treatment each rule + applies is stated in that rule's documentation. +- **Defaults stop propagation**: rules that can miss accept an optional `default` + template, evaluated *instead of* producing `NO_CONTENT` — the tool for "this value, + or X if missing" at the point of lookup. +- **Absorption**: looking further into a missing value stays missing — a deep `attr` + path over an absent branch yields `NO_CONTENT`, it does not raise. +- **Falsiness**: `NO_CONTENT` is falsy, so logical operators can express fallbacks — + e.g. a `chain` ending in `expr` `or` with a substitute value. Rules that *test* for + absence use identity, not truthiness: `false`, `0`, and `""` are values, not absence. +- **The top level**: `transform()` never returns the raw sentinel by default — a + template that evaluates to `NO_CONTENT` returns `None` (configurable via the + `no_content` argument). + +## Error model + +Template failures are typed by *whose mistake they are*: + +| Exception | Meaning | Representative causes | +|---|---|---| +| `DefinitionError` | The **template** is malformed — fix the template | unknown rule/operator/function name; a missing required parameter; unknown parameters; ambiguous or incomplete mutually-exclusive parameter groups; a reserved variable name used with `set`/`get`; a context accessor used outside its valid scope; a structural parameter with the wrong JSON shape | +| `TransformationError` | The template is valid but the **input data** does not fit it | iterating a non-iterable; joining mixed-type items; an operator applied to incompatible operand types; a function rejecting its arguments; a format pattern referencing a missing key; an `include` chain exceeding the depth limit | + +By default errors are raised **lazily**, when the failing node is actually walked — a +typo in a branch the data never reaches will not surface. Opt in to **static +validation** (`validate=True`, or calling `validate()`) to check the template's +structure up front, with no input data: unknown rules, unknown or missing parameters, +ambiguous parameter combinations, and invalid literal operator/function names all raise +`DefinitionError` immediately. + +Both error types carry the **template location** where the failure occurred — a path of +dict keys, list indices, rule names, and parameter names: + +``` +value is not iterable: 'not-a-list' + at template → pipeline → chain → funcs[0] → map +``` + +Which specific conditions each rule raises is part of that rule's documentation. + +## Expressions and calls + +Operators (the `expr` rule) and functions (the `call` rule) share one application +model; these semantics hold across **every** operator and function, so they live here. +The catalog of what exists — each operator's types and each function's signature — is +in the `expr` `op` / `call` `name` parameter docs and the engine exports. + +**Operator application** (`expr`) has three modes: + +- no value parameter → **unary**: `op(this)`; +- `value` → **binary**: `op(this, value)` — the current value is the left operand; +- `values` → **reduction**: `reduce(op, values)` pairwise over the evaluated list — + and the current value is **ignored**; include `{"$": "this"}` as a list item if the + reduction should involve it. `values` must be a non-empty list — an empty reduction + has no seed and raises `DefinitionError` (see the empty-collection caveat under + composition patterns). + +**Type behavior** follows Python semantics: `+` concatenates strings and lists as well +as adding numbers; comparisons work on like types; the logical operators use +truthiness and return an *operand*, not necessarily a boolean. An operator applied to +incompatible operand types raises `TransformationError`. + +**Function application** (`call`) mirrors the modes: no parameter → `fn(this)`; +`value` → `fn(value)`; `values` → `fn(*values)` (multi-argument call). The current +value is ignored whenever a parameter is given. Built-in functions convert their +documented failure modes into `TransformationError` — a well-formed template never +leaks a raw Python exception from a bad argument. A few functions are documented as +**total** (they accept any well-formed JSON value and never raise); totality is stated +per function in the catalog. + +**`NO_CONTENT` in expressions**: operators and functions do not skip it. A +`NO_CONTENT` operand is simply falsy (useful in `and`/`or` fallbacks) or an +incompatible argument (an error) — stop the propagation earlier with a `default` if +the operand may be missing. + +## Composition patterns + +The language has no aggregate primitives beyond what composition provides — a handful +of rules cover everything *because* they compose. The canonical shapes: + +- **Pipeline**: `chain` walks its steps in order, each result becoming `this` for the + next — the backbone for "extract, then reshape, then format" templates. +- **Reshape**: `map` (over a list or dict) with a nested `object`/`attr` template body; + `filter` before it to drop elements; `zip` to transpose parallel lists. +- **Compute**: nested `expr` rules — arithmetic is a tree of rules, not a string. +- **Reuse**: `include` runs a named sub-template against the current value. Only the + value crosses the boundary (no variables, no iteration properties); the marker is + inherited by default; nested includes are depth-limited. +- **Aggregate from primitives**: a count is `length` (or a `map` to `1`s reduced with + `add`); a flatten is the `flatten` function (or `map` in its `items` mode). **Mind + the empty collection**: an `expr` `values` reduction over an empty list is a + `DefinitionError` (no seed), an empty `join` yields `NO_CONTENT` (use its `default`), + while `sum` of an empty array is `0` — pick the primitive whose empty-case behavior + matches the intent. + +A worked end-to-end flow — pairing two parallel lists into a dict: + +```json +{ + "$": "chain", + "funcs": [ + {"$": "zip", "items": [{"$": "attr", "name": "keys"}, + {"$": "attr", "name": "values"}]}, + {"$": "map", "key": {"$": "attr", "name": 0}, + "value": {"$": "attr", "name": 1}} + ] +} +``` + +Input `{"keys": ["a","b"], "values": [1,2]}` → the root context's `this` is the input → +`zip` produces `[["a",1], ["b",2]]` → `chain` derives a context with that as `this` → +`map` iterates, each pair becoming `this`/`item` in a per-element scope → `attr` with +numeric names indexes each pair → output `{"a": 1, "b": 2}`. diff --git a/transon/rules.py b/transon/rules.py index 3bb5b22..2533966 100644 --- a/transon/rules.py +++ b/transon/rules.py @@ -79,6 +79,8 @@ def rule_this(_t: Transformer, _template, context: Context): def rule_parent(t: Transformer, _template, context: Context): """ Returns the value stored in previous context. + Using `parent` in the root context (where no previous scope exists) raises + `DefinitionError`. """ if context.parent is None: t.definition_error('`parent` is not available in the root context') @@ -90,6 +92,7 @@ def rule_item(_t: Transformer, _template, context: Context): """ Works inside `map`/`filter` when iterating over lists. Returns current item. + Using it outside such an iteration scope raises `DefinitionError`. """ return context.item @@ -99,6 +102,7 @@ def rule_key(_t: Transformer, _template, context: Context): """ Works inside `map`/`filter` when iterating over dicts. Returns the key of current element. + Using it outside such an iteration scope raises `DefinitionError`. """ return context.key @@ -108,6 +112,7 @@ def rule_index(_t: Transformer, _template, context: Context): """ Works inside `map`/`filter`. Returns 0-based index of iteration. + Using it outside an iteration scope raises `DefinitionError`. """ return context.index @@ -117,6 +122,7 @@ def rule_value(_t: Transformer, _template, context: Context): """ Works inside `map`/`filter` when iterating over dicts. Returns the value of current element. + Using it outside such an iteration scope raises `DefinitionError`. """ return context.value @@ -136,24 +142,13 @@ def rule_set(t: Transformer, template, context: Context): Returns `context.this` unchanged (pass-through), so it can sit inside a `chain` without altering the piped value. - **Scoping** — where a `set` is visible depends on which context object it - runs in: - - - **Descendant scopes**: visible in any context derived *after* the `set` - (child scopes resolve ancestor variables through the parent chain; the first - `set` in a child materializes inherited variables for write isolation). - - **Later siblings, same scope**: when a `set` runs directly at a literal-dict - key or list element, later-evaluated siblings in that dict/list share the - same context and can `get` the variable. Earlier siblings cannot — visibility - follows dict key / list index order. - - **First `chain` func**: runs in the caller's context, so a `set` there is - visible to later `chain` funcs and to sibling templates outside the `chain`. - - **Later `chain` funcs, `map`/`filter` items, etc.**: run in derived contexts; - their `set` values stay inside that scope and do not escape to parents or - already-evaluated siblings. - - `include` starts a separate transformation — variables do not cross that - boundary. + Visibility follows the language's downward-only scoping model (Language + Reference, "Context and scoping"): the variable is visible to scopes derived + *after* the `set` and to later-evaluated siblings sharing the same scope — + never to parent scopes once a derived scope ends, and never across an + `include` boundary (a separate transformation). Note the refactoring + pitfall: wrapping a step in `chain`, reordering dict keys, or moving a `set` + can change visibility with no error. """ t_name = t.require(template, 'name') name = t.walk_param(t_name, context, 'name') @@ -212,9 +207,14 @@ def rule_attr(t: Transformer, template, context: Context): """ Returns values of attribute or item from current value in context. Can search in deeply nested structures with path. - If attribute is not present returns no value. - If the dynamic name or any path segment evaluates to `NO_CONTENT`, returns - no value (uniformly for all container types). + + - A missing key or an index out of range → no value (`NO_CONTENT`), or the + `default` when provided. Looking further into a missing value stays + missing — a deep path over an absent branch does not raise. + - If the dynamic name or any path segment evaluates to `NO_CONTENT` → no + value (uniformly for all container types). + - Any other lookup failure (e.g. indexing a string with a string) raises + `TransformationError`. Parameters are mutually exclusive. """ @@ -311,6 +311,20 @@ def _iter_contexts(t: Transformer, context: Context): def rule_map(t: Transformer, template, context: Context): """ Iterates over `list` or `dict` and produces new `dict` or `list` with items based on template. + + Each iteration derives a sub-context whose `this` is the element, with the + iteration accessors available (`item`/`index` over lists; `key`/`value`/`index` + over dicts). Modes (mutually exclusive): + + - `item`: one output element per input element → list. + - `items`: the template yields a *list* of elements per input element; the + lists are concatenated → list (the flatten-while-mapping mode). + - `key`+`value`: one output entry per input element → dict. + + Results that evaluate to `NO_CONTENT` are skipped — the item (or the whole + key/value pair) is omitted from the output rather than emitted as `null`. + Iterating a value that is neither a list nor a dict raises + `TransformationError`. """ if 'item' in template: t_item = template['item'] @@ -358,6 +372,12 @@ def rule_map(t: Transformer, template, context: Context): def rule_filter(t: Transformer, template, context: Context): """ Iterates over `list` or `dict` and filters out items regarding condition calculation. + + Keeps the elements whose `cond` evaluates truthy; a condition that evaluates + to `NO_CONTENT` excludes the element. The container type is preserved: + list → list, dict → dict (entries keep their original keys and values). + Filtering a value that is neither a list nor a dict raises + `TransformationError`. """ t_cond = t.require(template, 'cond') @@ -395,6 +415,8 @@ def rule_zip(t: Transformer, template, context: Context): """ Transposes iterables like Python's `zip`: rows become columns and columns become rows. Each output row is a **list** (JSON-friendly), not a Python tuple. + The result is as long as the shortest input. A non-iterable item raises + `TransformationError`. """ t_items = t.require(template, 'items') @@ -416,7 +438,8 @@ def rule_zip(t: Transformer, template, context: Context): def rule_file(t: Transformer, template, context: Context): """ Writes a file using `write_file` delegate (a parameter to `Transformer` constructor). - This rule produces no result. + This rule always produces no result (`NO_CONTENT`) — so a `map` over `file` + yields `[]`, and a template that only writes files transforms to `None`. File will not be written if `name` or `content` returns no result. """ def write_file(_name, _content): @@ -458,8 +481,13 @@ def _is_dict(x): ) def rule_join(t: Transformer, template, context: Context): """ - Joins (concatenates) together several dicts, lists of strings. - If items to be concatenated have different types an exception will be thrown. + Type-homogeneous concatenation — all items must share one JSON type: + + - all **strings** → joined with `sep` (default empty string); + - all **lists** → flattened one level into a single list; + - all **dicts** → merged into a single dict (later keys win); + - mixed types → `TransformationError`. + Items that evaluate to `NO_CONTENT` are omitted before concatenation. When no items remain, returns `NO_CONTENT` unless `default` is provided. """ @@ -654,8 +682,13 @@ def rule_expr(t: Transformer, template, context: Context): 3. **Multiple `values` were specified:** Result is calculated by applying operation to all values in pairs (reduction). In this case current context value is ignored. + `values` must be a non-empty list — an empty reduction has no seed and + raises `DefinitionError`. - Parameters are mutually exclusive. + Parameters are mutually exclusive. An operator applied to incompatible + operand types raises `TransformationError`. The application semantics shared + by all operators (modes, reduction, type coercion, `NO_CONTENT` interaction) + are specified in the Language Reference ("Expressions and calls"). """ op_code = t.require(template, 'op') op = t.get_operator(op_code) @@ -751,10 +784,15 @@ def rule_call(t: Transformer, template, context: Context): Runs conversion function with single parameter - provided value. Current context is ignored in this case. 3. **Multiple `values` were specified:** - Runs conversion function with multiple parameters. + Runs conversion function with multiple parameters (`values` must be a list; + a non-list raises `DefinitionError`). Current context is ignored in this case. - Parameters are mutually exclusive. + Parameters are mutually exclusive. Built-in functions convert their + documented failure modes into `TransformationError`. The application + semantics shared by all functions (modes, argument passing, totality, + `NO_CONTENT` interaction) are specified in the Language Reference + ("Expressions and calls"). """ name = t.require(template, 'name') function = t.get_function(name) @@ -807,6 +845,8 @@ def rule_format(t: Transformer, template, context: Context): Returns no value when the formatting value (or any unpacked list element or dict key/value) is `NO_CONTENT`, unless `default` is provided. + A pattern referencing a key or index the value does not supply raises + `TransformationError`. """ t_pattern = t.require(template, 'pattern') pattern = t.walk_param(t_pattern, context, 'pattern') diff --git a/transon/transformers.py b/transon/transformers.py index c6838cb..00e174d 100644 --- a/transon/transformers.py +++ b/transon/transformers.py @@ -248,42 +248,17 @@ def arm(*, _variants, _constants=None, _containers=None, **slots) -> ArmSpec: class Transformer: """ - ## What is transon? + The engine's entry point: construct a `Transformer` from a template (plain + JSON in which a dict carrying the marker key, default `$`, is a rule + invocation) and apply it to input data. The **template language** itself — + the evaluation model, scoping, the `NO_CONTENT` model, the error taxonomy, + composition patterns — is specified in the + [Language Reference](https://github.com/transon-org/transon/blob/main/docs/LANGUAGE.md) + (also served by `transon.reference.get_language_reference()`); what the + project is and how it compares to alternatives is in the + [README](https://github.com/transon-org/transon#readme). - `transon` reshapes one JSON document into another using a template that is - itself plain JSON. Instead of writing imperative glue code to walk and rebuild - data, you describe the *shape* of the output once and let the engine fill it in - from the input — there is no separate template language and no string-embedded - DSL to learn. - - It is **inspired by** [XSLT](https://en.wikipedia.org/wiki/XSLT) (declarative, - tree-to-tree transformation) and [JsonLogic](https://jsonlogic.com/) (logic - expressed as data), applying those ideas to JSON-to-JSON transformation. - - **Design principles:** templates are always valid JSON; rules are composable and - nest arbitrarily (so even arithmetic is expressed as nested rules, with no DSL); - and a single configurable `$` marker is what distinguishes a rule from literal - data. - - **Compared to alternatives:** - - - [jsonnet](https://github.com/google/jsonnet) and - [jsonata](https://github.com/jsonata-js/jsonata) define their own - domain-specific languages; `transon` templates stay valid JSON. - - [jolt](https://github.com/bazaarvoice/jolt) is also JSON-to-JSON, but drives - transformations with fixed operation specs; `transon` rules compose and nest. - - [json-templates](https://github.com/datavis-tech/json-templates) does simple - placeholder substitution; `transon` adds rules, expressions, and functions. - - ## Install & get started - - Install from [PyPI](https://pypi.org/project/transon/): - - ```shell - pip install transon - ``` - - Run your first transform: + ## Usage ```python from transon import Transformer @@ -292,157 +267,28 @@ class Transformer: Transformer(template).transform(["a", "b"]) # => {"items": ["a", "b"]} ``` - **Links:** [GitHub](https://github.com/transon-org/transon) · - [Specification](https://github.com/transon-org/transon/blob/main/docs/SPECIFICATION.md) · - [Changelog](https://github.com/transon-org/transon/blob/main/CHANGELOG.md) - - ## Usage - - `transon` is a homogeneous JSON-to-JSON template engine: templates are themselves - plain JSON, and the shape of the output is defined entirely by the template. - Input data is interpolated into the template's placeholders. - - ```plantuml - @startuml - skinparam shadowing false - skinparam rectangle { - BackgroundColor #FEFEFE - BorderColor #333333 - } - rectangle "JSON Template" as T - rectangle "JSON Input" as I - rectangle "transon" as E - rectangle "JSON Output" as O - T -down-> E - I -right-> E - E -right-> O - @enduml - ``` - - ## Templates - - Template could be any JSON structure. It will be reflected as-is in output, except for rule structures. - Rules are JSON objects with special attribute named `$` (this is called marker and can be changed). - If the rule has nested template the same applies to it as well. - - Example template: - - ```json - { - "test": { - "$": "map", - "item": [ - { - "x": { - "$": "item" - } - } - ] - } - } - ``` - - At the top level output will just copy template `{"test": ...}`. - Then the `map` rule will be applied to the input executing sub-template, defined by `item` attribute, - for each item in input collection. - Let's assume that our input is `[1, 2, 3]`. - Inner template contains another rule `{"$": "item"}` which points to value of items of the input. - - So the final result will be: - - ```json - { - "test": [ - [{"x": 1}], - [{"x": 2}], - [{"x": 3}] - ] - } - ``` - - Note that each item preserves its template definition (including list around object). - - ## How evaluation works - - A handful of rules cover everything because they compose, and they compose because - the engine evaluates every template the same way. Understanding these four - mechanics lets you predict how any template behaves without trial and error. - - **1. Recursive tree walk.** The engine walks the template top-down and rebuilds it - node by node. Each node is handled by its JSON type: - - - a **list** → walk every element, return a new list; - - a **dict containing the marker key** (`$`) → a *rule invocation* (see below); - - a **dict without the marker** → walk every value, return a new dict with the same - keys; - - any **scalar** (string, number, boolean, `null`) → copied through unchanged. - - So a template that contains no markers is returned as a deep copy of itself; rules - are the only thing that injects data. - - **2. Marker-based rule detection.** A dict is a rule *only* when it contains the - marker key; its value names the rule (`{"$": "map", ...}`) and the sibling keys are - the rule's parameters. Those parameters are themselves templates and are walked - recursively, so rules nest arbitrarily — this is why even arithmetic is just nested - `expr` rules rather than a string mini-language. (To emit a literal dict that really - does contain a `$` key, use the `object` rule's `fields` mode, or change the marker - with `Transformer(template, marker="@")`.) - - **3. Context and scope.** Evaluation carries a *context* whose `this` is the current - value (the transformation input at the root). Iterating rules — `map` and `filter` — - *derive* a child context for each element, exposing the accessors `item`, `index` - (lists) and `key`, `value`, `index` (dicts); those accessors are only valid inside - that derived scope. Variables (`set`/`get`) flow **downward only**: a value `set` in - a scope is visible to its descendants and to later-evaluated siblings, but not to - parent or earlier-evaluated scopes. This downward-only rule is what makes nested - transformations predictable. - - **4. `NO_CONTENT` skip propagation.** Missing lookups don't blow up: `attr`/`get` - over an absent key/variable produce the `NO_CONTENT` sentinel (distinct from - `null`). Container rules then *skip* it rather than emitting `null` — `map` drops - the item, `object`/`filter`/`file` omit the entry, `join` leaves it out — so - optional data simply disappears from the output. A `default` parameter (on `attr`, - `get`, `format`, `include`, `join`) substitutes a value instead, and a top-level - `NO_CONTENT` becomes `None` (configurable via `transform(data, no_content=...)`). - - These four behaviours are the whole evaluation model; the - [specification](https://github.com/transon-org/transon/blob/main/docs/SPECIFICATION.md) - (§2) documents the exhaustive details (scoping edge cases, every `NO_CONTENT` - producer/consumer, the error model). - - ## What you can do - - Beyond simple interpolation, `transon` offers: - - - **Static validation** — `Transformer(template, validate=True)` (or calling - `.validate()`) checks the template's structure up front, without any input data, - raising `DefinitionError` on malformed rules. - - **Defaults for missing values** — `attr`, `get`, `join`, `format`, and `include` - accept a `default` template, used when the looked-up value is absent. - - **A "no value" model** — rules can produce `NO_CONTENT`; container rules such as - `map`, `object`, `filter`, and `file` skip it instead of emitting `null`. - `transform(data, no_content=...)` controls what a top-level `NO_CONTENT` becomes - (defaults to `None`). - - **Literal keys** — the `object` rule's `fields` mode builds dicts with literal keys, - including a key equal to the marker (`$`). - - **Configurable marker** — `Transformer(template, marker="@")` if `$` collides with - your data. - - **Safe output** — `transform(data, copy_output=True)` deep-copies the result so it - shares no mutable structure with the input (which is never mutated regardless). - - **A clear error model** — `DefinitionError` signals a malformed template, - `TransformationError` signals data that does not fit; both messages include the - template path where the problem occurred (`at template → …`). See the - **Error model** examples below for the literal messages each one produces. - - **I/O delegates** — the `file` rule writes through a `file_writer` callback and the - `include` rule loads sub-templates through a `template_loader` callback. - - The full set of built-in rules is documented below under **Rules**; see the - specification for exhaustive semantics. + Constructor options: + + - **`validate=True`** (or calling `.validate()`) — static template check up + front, without input data; malformed rules raise `DefinitionError`. + - **`marker="@"`** — change the rule marker if `$` collides with your data. + - **`file_writer`** — callback the `file` rule writes through. + - **`template_loader`** — callback the `include` rule loads sub-templates + through; it receives an `IncludeContext` and constructs the sub-transformer + (see :class:`IncludeContext`). + - **`max_include_depth`** — nested-`include` depth limit (default 50). + + `transform(data, no_content=None, *, copy_output=False)` returns the output: + `no_content` chooses what a top-level `NO_CONTENT` becomes (default `None`); + `copy_output=True` deep-copies the result once so it shares no mutable + structure with the input (which is never mutated regardless — but pass-through + rules return references into it). Failures raise `DefinitionError` (malformed + template) or `TransformationError` (data does not fit) — catch these two; + messages include the template path (`at template → …`). ## Extending - All rules are pluggable. Built-in rules are documented below under **Rules**. - However, you can easily add your own rules with their own attributes. + All rules are pluggable, and you can add your own with their own attributes: ```python @Transformer.register_rule('my_rule') @@ -450,7 +296,8 @@ def my_rule(t: Transformer, template, context: Context): ... ``` - You can also inherit `Transformer` class and add rules to subclass to avoid functionality collision. + You can also inherit `Transformer` and register rules on the subclass to + avoid functionality collision: ```python class Transformer1(Transformer): @@ -468,11 +315,11 @@ def my_rule2(t: Transformer, template, context: Context): ... ``` - Note that `my_rule` can be used with both transformers but may behave differently. - - In the same fashion you can also add additional operators for expressions (`expr`) calculations - and functions (`call`) using decorators - `register_operator` and `register_function`. + Note that `my_rule` can be used with both transformers but may behave + differently. In the same fashion you can add operators for `expr` and + functions for `call` with the `register_operator` and `register_function` + decorators. Subclass registrations never affect the base class; lookups + resolve through the MRO (the stability contract is `SPECIFICATION.md` §3). """ DEFAULT_MARKER = '$' From 830acfb35a7ffd20ab4a2819d36677ff3463ced6 Mon Sep 17 00:00:00 2001 From: Eugene Chernyshov Date: Sat, 18 Jul 2026 02:04:01 +0300 Subject: [PATCH 3/8] fix: address CodeRabbit review on RFC 0008 PR MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Code fixes (7 of 10 findings confirmed): - reference.py: CommonMark-correct fence tracking (backtick AND tilde fences; closers must match the opening character, be at least as long, and carry no trailing content) — a heading inside a ~~~ block is no longer treated as a section boundary; regression tests added - reference.py: whitespace-only prefixes are preserved as the preamble section (previously dropped by .strip(), silently breaking the sections-concatenation parity invariant); test updated to assert parity - rules.py: map `items` mode validates the evaluated value is a list and raises a located DefinitionError (matches the expr/call `values` precedent) instead of iterating dict keys / string chars or leaking a raw TypeError; regression test + changelog entry - transformers.py: transform(copy_output=True) no longer deepcopies the NO_CONTENT branch — sentinel identity is preserved and a caller-owned no_content substitute is returned as-is; identity tests + changelog Doc fixes: - "byte-exact" wording corrected to canonical normalized text (UTF-8, LF) in spec §5.2 and the RFC - packaging-parity claims scoped honestly to source-tree parity (wheel/ sdist inclusion verified by building distributions at release) - RFC export example now matches the implemented section ids (preamble, the-no_content-model); preamble rule wording covers whitespace-only prefixes - RFC status no longer claims "in main"; proposals index Shipped column is "—" until a release names the version Declined with reasons (posted on the PR): the two "future date" findings (2026-07-18 is the actual decision/implementation date) and walk-time mutually-exclusive-mode enforcement (explicitly rejected in the recorded R-04 decision — option 1, opt-in validate(), was chosen over walk-time enforcement precisely because the latter is breaking). 397 passed; no TBD; check_roadmap consistent. Co-Authored-By: Claude Fable 5 --- CHANGELOG.md | 15 ++++++++ docs/SPECIFICATION.md | 23 +++++++------ .../0008-language-reference-export.md | 24 +++++++------ docs/proposals/README.md | 2 +- tests/test_copy_output.py | 15 ++++++++ tests/test_invalid_value.py | 11 ++++++ tests/test_reference.py | 34 +++++++++++++++++-- transon/reference.py | 29 ++++++++++++---- transon/rules.py | 7 +++- transon/transformers.py | 7 +++- 10 files changed, 134 insertions(+), 33 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 2635332..c684331 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -20,6 +20,21 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/). `tests/test_reference.py`; packaging parity is tested via `importlib.resources`. (Roadmap R-34, R-35, R-36) +### Fixed + +- **`map` `items` mode validates its result shape.** An `items` template that + evaluates to a non-list (a dict, a string, a scalar) now raises a located + `DefinitionError` (``` `items` must evaluate to a list for `map` rule ```) + instead of accidentally iterating dict keys / string characters or leaking a + raw `TypeError`. Templates relying on the accidental iteration must wrap the + value in a list. (Found in review of Roadmap R-34) +- **`transform(..., copy_output=True)` preserves `NO_CONTENT` identity.** When + the caller opts into the raw sentinel (`no_content=Transformer.NO_CONTENT`), + the result is no longer routed through `copy.deepcopy`, which used to return a + fresh `NoContent` instance and break `result is Transformer.NO_CONTENT`; a + caller-supplied `no_content` substitute is likewise returned as-is (it cannot + alias the input, so there is nothing for `copy_output` to protect). + ### Changed - **`get_all_docs()['doc']` content (docs-site coordination; shape unchanged).** diff --git a/docs/SPECIFICATION.md b/docs/SPECIFICATION.md index c7ae060..00e3f7c 100644 --- a/docs/SPECIFICATION.md +++ b/docs/SPECIFICATION.md @@ -411,22 +411,25 @@ into a lean structural `catalog` (consumed by the editor's generators) and an export (RFC 0008, R-36) serving the packaged `LANGUAGE.md` offline: - Shape: `{reference_version, engine_version, format: "markdown", content, sections}`. - `content` is the byte-exact document (UTF-8, `\n` newlines); `sections` is a flat, - ordered split on top-level `##` headings (each section includes its own heading - line; deeper headings stay inside their parent; a non-empty intro before the first - `##` becomes a leading `{"id": "preamble"}` section; ids are GitHub-style slugs, - collisions suffixed `-2`, `-3`, … in document order). Concatenating `sections` - reproduces `content` exactly. + `content` is the full document as canonical normalized text (UTF-8, line endings + normalized to LF); `sections` is a flat, ordered split on top-level `##` headings + (each section includes its own heading line; deeper headings and fenced code blocks + stay inside their parent; any non-empty prefix before the first `##` — whitespace + included — becomes a leading `{"id": "preamble"}` section; ids are GitHub-style + slugs, collisions suffixed `-2`, `-3`, … in document order). Concatenating + `sections` reproduces `content` exactly. - `REFERENCE_VERSION` policy (mirrors `METADATA_VERSION`): additive changes — a new section, appended prose, a new optional field — bump the minor; removing/renaming a section `id`, changing the `sections` shape, or dropping/renaming a top-level field is breaking and bumps the major. Consumers MUST fail loudly on an unsupported major. - The export is **engine-global** (base `Transformer` only; no `cls=` parameter) and states language facts only. -- Packaging: `transon/resources/LANGUAGE.md` ships in the wheel and sdist; - `docs/LANGUAGE.md` is the canonical hand-edited source and - `tests/test_reference.py` asserts the two are identical, pins the section-id list, - and checks the split parity. +- Packaging: `transon/resources/LANGUAGE.md` ships in the wheel and sdist (hatchling + packages the whole `transon/` tree; verified by building the distributions at + release). `docs/LANGUAGE.md` is the canonical hand-edited source; + `tests/test_reference.py` asserts **source-tree parity** — the packaged resource + read via `importlib.resources` equals both the export's `content` and the + canonical file — pins the section-id list, and checks the split parity. - `python -m transon.reference` prints this JSON. --- diff --git a/docs/proposals/0008-language-reference-export.md b/docs/proposals/0008-language-reference-export.md index 53363c2..1b5466d 100644 --- a/docs/proposals/0008-language-reference-export.md +++ b/docs/proposals/0008-language-reference-export.md @@ -1,6 +1,6 @@ # RFC 0008 — Author-facing Language Reference: document, packaging, and export API -- **Status:** Implemented (2026-07-18, in `main` — unreleased; ships in the next release per Sequencing) +- **Status:** Implemented (2026-07-18) — unreleased; ships in the next release per Sequencing, which will name the version here - **Created:** 2026-07-16 - **Amended:** 2026-07-18 — consolidation scope extended to the `Transformer` docstring and `README.md`; single ownership principle (structure in the catalog, per-entity behavior in @@ -145,11 +145,12 @@ exactly the property `get_editor_metadata()` already has for the catalog. The re identical. **Acceptance (packaging parity).** A test loads the packaged `LANGUAGE.md` through -`importlib.resources` — exercising the installed wheel/sdist layout, not just the source tree — -decodes those bytes as UTF-8, normalizes line endings to `\n`, and asserts the result equals -`get_language_reference()['content']` (which is UTF-8 text with `\n` newlines), following the -shape-test pattern in `tests/test_metadata.py`. This catches a missing package-data glob or a stale -packaged copy, neither of which the section-pin test (Deliverable 1) would notice. +`importlib.resources`, decodes those bytes as UTF-8, normalizes line endings to `\n`, and asserts +the result equals `get_language_reference()['content']` **and** the canonical `docs/LANGUAGE.md`, +following the shape-test pattern in `tests/test_metadata.py`. In CI this proves **source-tree +parity** (a stale packaged copy fails immediately); the wheel/sdist inclusion itself is verified +by building the distributions at release — the section-pin test (Deliverable 1) would notice +neither. ## Deliverable 3 — `get_language_reference()` export (R-36) @@ -164,7 +165,8 @@ A read-only export, separate from the docs API, mirroring the `get_editor_metada "format": "markdown", "content": "", "sections": [ - {"id": "no-content", "title": "NO_CONTENT", "heading_level": 2, "content": "..."}, + {"id": "preamble", "title": "", "heading_level": null, "content": "..."}, + {"id": "the-no_content-model", "title": "The NO_CONTENT model", "heading_level": 2, "content": "..."}, ... ] } @@ -172,13 +174,15 @@ A read-only export, separate from the docs API, mirroring the `get_editor_metada - `sections` is a flat, ordered split of `content` by heading, each with a stable slug `id`, so consumers can serve **targeted** section lookups (a 700-line dump into an agent context is the - failure mode; one section is the unit of consumption). `content` is the byte-exact document for - consumers that want the whole thing. + failure mode; one section is the unit of consumption). `content` is the full document as + canonical normalized text (UTF-8, line endings normalized to LF) for consumers that want the + whole thing. - **Splitting rules** (deterministic): the split is on top-level `##` headings only, so `sections` is flat, not a tree. Each section runs from its `##` heading up to the next `##` heading and **includes its own heading line**; any deeper (`###`+) heading stays inside its parent section. Content before the first `##` heading (the intro under the `#` title) becomes a leading preamble - section — present **only when that intro is non-empty** — carrying + section — present **whenever that prefix is non-empty**, whitespace-only prefixes included, so + the concatenation parity below always holds — carrying `{"id": "preamble", "title": "", "heading_level": null, ...}`. Every other section carries `heading_level: 2` and the heading's text as `title`. `id` is the GitHub-style slug of the heading text (the preamble's is the literal `"preamble"`); a collision gets a `-2`, `-3`, … suffix in diff --git a/docs/proposals/README.md b/docs/proposals/README.md index caeced6..f471640 100644 --- a/docs/proposals/README.md +++ b/docs/proposals/README.md @@ -43,7 +43,7 @@ is the *authoritative* record of each work item's status via its **R-number**. | [0005](0005-example-corpus-normalization.md) | Normalize exports to one flat example corpus | Implemented | R-31 | v0.1.6 | docs site, `transon-blockly` | | [0006](0006-transformer-recursion-depth-budget.md) | Bounded per-level recursion budget (self-`include` depth) | Implemented | R-32 | v0.1.7 | `transon-blockly` | | [0007](0007-builtin-function-library.md) | Grow the built-in function library | Implemented | R-33 | v0.1.8 | `transon-authoring`, `transon-blockly` | -| [0008](0008-language-reference-export.md) | Author-facing Language Reference: doc, packaging, export | Implemented | R-34, R-35, R-36 | unreleased | `transon-authoring`, docs site | +| [0008](0008-language-reference-export.md) | Author-facing Language Reference: doc, packaging, export | Implemented | R-34, R-35, R-36 | — | `transon-authoring`, docs site | ## Adding a new RFC diff --git a/tests/test_copy_output.py b/tests/test_copy_output.py index 7f70480..9125f55 100644 --- a/tests/test_copy_output.py +++ b/tests/test_copy_output.py @@ -64,3 +64,18 @@ def test_copy_output_keyword_only(): data = {'$': 'attr', 'name': 'missing'} transformer = Transformer(data) assert transformer.transform({}, 'fallback', copy_output=True) == 'fallback' + + +def test_copy_output_preserves_no_content_identity(): + transformer = Transformer({'$': 'attr', 'name': 'missing'}) + result = transformer.transform( + {}, no_content=Transformer.NO_CONTENT, copy_output=True, + ) + assert result is Transformer.NO_CONTENT + + +def test_copy_output_returns_substitute_unchanged(): + substitute = {'fallback': True} + transformer = Transformer({'$': 'attr', 'name': 'missing'}) + result = transformer.transform({}, no_content=substitute, copy_output=True) + assert result is substitute diff --git a/tests/test_invalid_value.py b/tests/test_invalid_value.py index 88472a4..ba9a302 100644 --- a/tests/test_invalid_value.py +++ b/tests/test_invalid_value.py @@ -31,3 +31,14 @@ def test_filter_invalid_value(): transformer = Transformer(template) with pytest.raises(TransformationError): transformer.transform(1) + + +def test_map_items_non_list_result(): + from transon import DefinitionError + template = { + '$': 'map', + 'items': {'$': 'this'}, + } + transformer = Transformer(template) + with pytest.raises(DefinitionError, match='`items` must evaluate to a list'): + transformer.transform([{'a': 1}]) diff --git a/tests/test_reference.py b/tests/test_reference.py index 669abd8..b7f431a 100644 --- a/tests/test_reference.py +++ b/tests/test_reference.py @@ -92,9 +92,12 @@ def test_split_without_preamble(): assert [s['id'] for s in sections] == ['only'] -def test_split_blank_intro_emits_no_preamble(): - sections = _split_sections('\n\n## First\nbody\n') - assert [s['id'] for s in sections] == ['first'] +def test_split_whitespace_only_prefix_is_preserved_as_preamble(): + """Every prefix byte belongs to the preamble — parity beats prettiness.""" + content = '\n\n## First\nbody\n' + sections = _split_sections(content) + assert [s['id'] for s in sections] == ['preamble', 'first'] + assert ''.join(s['content'] for s in sections) == content def test_split_slug_collisions_get_suffixes(): @@ -111,6 +114,31 @@ def test_split_ignores_headings_inside_code_fences(): assert ''.join(s['content'] for s in sections) == content +def test_split_ignores_headings_inside_tilde_fences(): + content = '## Real\n~~~\n## not a heading\n~~~\ntail\n' + sections = _split_sections(content) + assert [s['id'] for s in sections] == ['real'] + assert ''.join(s['content'] for s in sections) == content + + +def test_split_fence_closes_only_on_matching_delimiter(): + # A ``` line inside a ~~~ fence does not close it, and a longer run of the + # same character does; a shorter run does not. + content = ( + '## A\n' + '~~~~\n' + '```\n' + '## still fenced\n' + '~~~\n' + '## still fenced too\n' + '~~~~~\n' + '## B\n' + ) + sections = _split_sections(content) + assert [s['id'] for s in sections] == ['a', 'b'] + assert ''.join(s['content'] for s in sections) == content + + def test_split_deeper_headings_stay_inside_parent(): content = '## Top\n### Sub\nbody\n#### Deeper\n' sections = _split_sections(content) diff --git a/transon/reference.py b/transon/reference.py index 1d0c969..e73a416 100644 --- a/transon/reference.py +++ b/transon/reference.py @@ -20,6 +20,11 @@ #: Characters kept by the GitHub-style heading slugger (besides spaces → hyphens). _SLUG_KEEP = re.compile(r'[^0-9a-z _-]') +#: A fenced-code delimiter line: up to 3 spaces of indent, then a backtick or +#: tilde run of length >= 3 (CommonMark), with the rest of the line captured so +#: closers (which must carry no trailing content) can be told from openers. +_FENCE = re.compile(r'^ {0,3}(`{3,}|~{3,})(.*)$') + def _engine_version(): """The installed ``transon`` distribution version, or ``None`` when unavailable. @@ -65,13 +70,21 @@ def _split_sections(content): """ lines = content.splitlines(keepends=True) boundaries = [] - in_fence = False + fence = None for index, line in enumerate(lines): - if line.lstrip().startswith('```'): - in_fence = not in_fence - continue - if not in_fence and line.startswith('## '): - boundaries.append(index) + match = _FENCE.match(line) + if fence is None: + if match: + fence = match.group(1) + elif line.startswith('## '): + boundaries.append(index) + elif ( + match + and match.group(1)[0] == fence[0] + and len(match.group(1)) >= len(fence) + and not match.group(2).strip() + ): + fence = None sections = [] seen_ids = {} @@ -83,7 +96,9 @@ def _unique(slug): first = boundaries[0] if boundaries else len(lines) preamble = ''.join(lines[:first]) - if preamble.strip(): + # Any non-empty prefix — even whitespace-only — must be emitted, or the + # sections-concatenation parity invariant breaks. + if preamble: sections.append({ 'id': _unique('preamble'), 'title': '', diff --git a/transon/rules.py b/transon/rules.py index 2533966..3fa6c15 100644 --- a/transon/rules.py +++ b/transon/rules.py @@ -339,7 +339,12 @@ def rule_map(t: Transformer, template, context: Context): t_items = template['items'] result = [] for sub_context in _iter_contexts(t, context): - for item in t.walk_param(t_items, sub_context, 'items'): + items = t.walk_param(t_items, sub_context, 'items') + if not isinstance(items, list): + t.definition_error( + '`items` must evaluate to a list for `map` rule' + ) + for item in items: if item is t.NO_CONTENT: continue result.append(item) diff --git a/transon/transformers.py b/transon/transformers.py index 00e174d..b3cb700 100644 --- a/transon/transformers.py +++ b/transon/transformers.py @@ -728,7 +728,12 @@ def transform(self, data, no_content=None, *, copy_output: bool = False): """ context = Context(this=data) result = self.walk(self.template, context) - if result is self.NO_CONTENT and no_content is not self.NO_CONTENT: + if result is self.NO_CONTENT: + # Never deepcopy this branch: the sentinel is compared by identity + # everywhere, and a substitute is caller-owned (it cannot alias the + # input, so `copy_output` has nothing to protect). + if no_content is self.NO_CONTENT: + return result return no_content if copy_output: return copy.deepcopy(result) From 58397f7a6fe5893d0e5c36e0a3fad65ae2f695b6 Mon Sep 17 00:00:00 2001 From: Eugene Chernyshov Date: Sat, 18 Jul 2026 02:11:29 +0300 Subject: [PATCH 4/8] fix: preserve NO_CONTENT identity inside deep-copied containers MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit NoContent now defines __copy__/__deepcopy__ returning itself, so a sentinel kept inside a copied container (a literal template list holding a missing lookup) survives transform(copy_output=True) with identity intact — the earlier fix only covered the top-level branch. Regression test + changelog note extended. (CodeRabbit follow-up on PR 4) Co-Authored-By: Claude Fable 5 --- CHANGELOG.md | 5 ++++- tests/test_copy_output.py | 8 ++++++++ transon/transformers.py | 9 +++++++++ 3 files changed, 21 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index c684331..cd836d1 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -33,7 +33,10 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/). the result is no longer routed through `copy.deepcopy`, which used to return a fresh `NoContent` instance and break `result is Transformer.NO_CONTENT`; a caller-supplied `no_content` substitute is likewise returned as-is (it cannot - alias the input, so there is nothing for `copy_output` to protect). + alias the input, so there is nothing for `copy_output` to protect). `NoContent` + also defines `__copy__`/`__deepcopy__` returning itself, so a sentinel kept + *inside* a copied container (e.g. a literal template list holding a missing + lookup) preserves identity too. ### Changed diff --git a/tests/test_copy_output.py b/tests/test_copy_output.py index 9125f55..b849816 100644 --- a/tests/test_copy_output.py +++ b/tests/test_copy_output.py @@ -74,6 +74,14 @@ def test_copy_output_preserves_no_content_identity(): assert result is Transformer.NO_CONTENT +def test_copy_output_preserves_nested_no_content_identity(): + # A literal template list keeps NO_CONTENT elements (only container rules + # skip them), so the sentinel can sit inside a deep-copied result. + transformer = Transformer([{'$': 'attr', 'name': 'missing'}]) + result = transformer.transform({}, copy_output=True) + assert result[0] is Transformer.NO_CONTENT + + def test_copy_output_returns_substitute_unchanged(): substitute = {'fallback': True} transformer = Transformer({'$': 'attr', 'name': 'missing'}) diff --git a/transon/transformers.py b/transon/transformers.py index b3cb700..1a31e73 100644 --- a/transon/transformers.py +++ b/transon/transformers.py @@ -141,6 +141,15 @@ def __getitem__(self, _): def __bool__(self): return False + # The sentinel is compared by identity everywhere, so copying must never + # mint a new instance (e.g. `copy_output=True` deep-copying a literal + # template list that kept a NO_CONTENT element). + def __copy__(self): + return self + + def __deepcopy__(self, _memo): + return self + FileWriterType = Callable[[str, Any], None] # `include` always calls the loader as ``loader(name, context=IncludeContext)``; the From 7a97ad99abaceb4a329a0ea6dfe8fec7018e992f Mon Sep 17 00:00:00 2001 From: Eugene Chernyshov Date: Sat, 18 Jul 2026 02:20:26 +0300 Subject: [PATCH 5/8] =?UTF-8?q?refactor:=20single=20canonical=20LANGUAGE.m?= =?UTF-8?q?d=20=E2=80=94=20packaged=20file=20is=20the=20source?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Invert the R-35 ownership: transon/resources/LANGUAGE.md is now the canonical, hand-edited, single copy (the file that ships in the wheel/ sdist is the file you edit — same rule as per-rule docs living in rules.py); docs/LANGUAGE.md is reduced to a pointer for docs/ discoverability. Drops the mirror + identity test (no sync step exists to forget) while keeping the packaging-parity, section-pin, and split tests. RFC 0008 Deliverable 2, spec (layout table, section banners, 5.2), roadmap Shipped notes, CHANGELOG, D-20, and the class-docstring link all repointed at the canonical path. 398 passed; check_roadmap consistent. Co-Authored-By: Claude Fable 5 --- CHANGELOG.md | 8 +- docs/DOCS_SITE_ROADMAP.md | 2 +- docs/LANGUAGE.md | 216 +----------------- docs/ROADMAP.md | 19 +- docs/SPECIFICATION.md | 23 +- .../0008-language-reference-export.md | 30 +-- tests/test_reference.py | 24 +- transon/transformers.py | 2 +- 8 files changed, 61 insertions(+), 263 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index cd836d1..a9bed1a 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -8,11 +8,11 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/). ### Added -- **Template Language Reference (RFC 0008).** New hand-written `docs/LANGUAGE.md` — +- **Template Language Reference (RFC 0008).** New hand-written `transon/resources/LANGUAGE.md` — the author-facing, cross-cutting language semantics (evaluation model, scoping, the `NO_CONTENT` model, error taxonomy, `expr`/`call` machinery, composition - patterns; **no per-entity sections**) — shipped as package data - (`transon/resources/LANGUAGE.md`, wheel + sdist) and served by a new versioned + patterns; **no per-entity sections**) — canonical **and** packaged in one file + (ships as-is in the wheel + sdist; `docs/LANGUAGE.md` is a pointer) and served by a new versioned export `transon.reference.get_language_reference()` (`REFERENCE_VERSION` `1.0`; `{reference_version, engine_version, format, content, sections}` with a deterministic flat `##`-heading split, stable slug ids, and sections-concatenation @@ -44,7 +44,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/). The `Transformer` class docstring — exported as the `doc` field and rendered by the docs site — is consolidated per RFC 0008's ownership principle: its language sections ("Templates", "How evaluation works", the language half of "What you can - do") moved into `docs/LANGUAGE.md`; the pitch/install/comparison sections are owned + do") moved into the Language Reference (`transon/resources/LANGUAGE.md`); the pitch/install/comparison sections are owned solely by `README.md`; what remains is the embedder-facing narrative (Python API usage + extending). Symmetrically, per-rule docstrings **grew richer**: spec §4's per-rule facts (edge cases, `NO_CONTENT` treatment, error conditions) folded into diff --git a/docs/DOCS_SITE_ROADMAP.md b/docs/DOCS_SITE_ROADMAP.md index 6eed54d..07bf443 100644 --- a/docs/DOCS_SITE_ROADMAP.md +++ b/docs/DOCS_SITE_ROADMAP.md @@ -207,7 +207,7 @@ the project today. The engine's RFC 0008 release relocates content the site renders today: the `Transformer` class docstring (`get_all_docs()['doc']`) shrinks to an embedder-facing narrative (its language -sections move to the new `docs/LANGUAGE.md`, its pitch to `README.md`), and rule docstrings +sections move to the new Language Reference (`transon/resources/LANGUAGE.md`), its pitch to `README.md`), and rule docstrings grow richer (spec §4's per-rule facts fold in). In the same release window the site must: 1. render `LANGUAGE.md` as a language-guide section/page (source: `get_language_reference()` diff --git a/docs/LANGUAGE.md b/docs/LANGUAGE.md index 372a31d..806c4aa 100644 --- a/docs/LANGUAGE.md +++ b/docs/LANGUAGE.md @@ -1,207 +1,9 @@ -# Transon — Template Language Reference - -> **Audience**: template authors (human or agent). This document is the **cross-cutting** -> semantics of the Transon template language: the evaluation model, scoping, the -> `NO_CONTENT` propagation model, the error taxonomy, the `expr`/`call` machinery, and -> composition patterns. It deliberately contains **no per-rule reference**: what each -> individual rule, operator, or function does — its parameters, modes, and edge cases — -> lives in that entry's own documentation, exported by the engine -> (`transon.docs.get_all_docs()` / `transon.metadata.get_editor_metadata()`) and rendered -> on the [docs site](https://transon-org.github.io/). Entity names appear here only as -> illustrations. Executable examples live in the example corpus shipped with those same -> exports. - -## Templates and the marker - -A template is any JSON value. The engine walks it top-down and rebuilds it node by node; -each node is handled by its JSON type: - -- a **list** → walk every element, return a new list; -- a **dict containing the marker key** (default `"$"`) → a **rule invocation** (see below); -- a **dict without the marker** → walk every value, return a new dict with the same keys; -- any **scalar** (string, number, boolean, `null`) → copied through unchanged. - -A template that contains no markers is therefore reproduced as a deep copy of itself — -rules are the only thing that injects data. - -A dict is a rule *only* when it contains the marker key; the marker's value names the -rule and the sibling keys are the rule's parameters: - -```json -{"$": "attr", "name": "x"} -``` - -Rule parameters are themselves templates and are walked recursively, so rules nest -arbitrarily — this is why even arithmetic is expressed as nested rules rather than a -string mini-language. A handful of parameters are documented as **constant** (for -example an operator name): those are read verbatim, never walked. - -The marker is configurable per transformation (`marker=` on the transformer). To emit a -literal dict that really contains the marker key — data that would otherwise be read as -a rule invocation — use the `object` rule's `fields` mode, whose keys are emitted -verbatim while its values stay templates; its single-pair `key`/`value` mode also -produces one literal key. - -When one template `include`s another, the sub-template inherits the parent's marker by -default, so a template tree written against the default marker stays consistent across -`include` boundaries; the loader may pin a different marker explicitly. - -## Context and scoping - -Evaluation carries a **context** — a linked chain of scopes. Each context holds: - -- `this` — the current value (in the root context: the transformation input); -- iteration properties — `item`, `index` over lists; `key`, `value`, `index` over - dicts — present only inside scopes derived by the iterating rules (`map`, `filter`); -- user variables — arbitrary names written by `set`, read by `get`; -- a link to the **parent** scope it was derived from. - -Rules that carry a value into a sub-template derive a **child scope**: each `map`/ -`filter` iteration derives one per element (exposing the iteration properties), and -each `chain` step after the first derives one whose `this` is the previous step's -result. The context accessor rules (`this`, `parent`, `item`, `key`, `value`, `index`) -read these slots; each is valid only where its slot exists, and using one outside its -valid scope is a template mistake (`DefinitionError`). - -Variables flow **downward only**. The rules of visibility, in decreasing surprise: - -| Where a `set` runs | Its variable is visible to | -|---|---| -| any scope | descendant scopes derived *after* the `set` | -| directly at a key/element of a literal dict/list | later-evaluated siblings in that dict/list (they share one scope; dict key / list index order matters) | -| the **first** func of a `chain` | the caller's scope — later `chain` funcs *and* later siblings outside the `chain` | -| a later `chain` func, a `map`/`filter` iteration | only that derived scope and its descendants | -| anywhere | **never** the parent scope after the derived scope ends; **never** an `include`d sub-template (that is a separate transformation — only the value crosses the boundary) | - -Refactoring pitfall: wrapping a step in `chain`, reordering dict keys, or moving a -`set` can change visibility with no error — consult the table. - -The names `this`, `item`, `key`, `value`, and `index` are **reserved** and cannot be -used as variable names with `set`/`get` (violation raises `DefinitionError`). - -## The NO_CONTENT model - -`NO_CONTENT` is the language's "no value" sentinel — distinct from JSON `null`. `null` -is a value you can store and emit; `NO_CONTENT` means *there is nothing here*, and the -language is built so that missing data disappears from the output instead of blowing up -or leaving `null` holes. - -- **Where it comes from**: lookups that miss (an absent attribute or path, an undefined - variable), aggregations left with nothing (a `join` whose items all vanished), rules - that never produce a value (`file`), and sub-transforms that themselves produced - nothing (`include`). -- **Skip, don't emit**: container rules *omit* a `NO_CONTENT` piece rather than emitting - `null` — as illustrations: `map` drops the item, `object` omits the entry, `filter` - excludes the element, `join` leaves the item out. The exact treatment each rule - applies is stated in that rule's documentation. -- **Defaults stop propagation**: rules that can miss accept an optional `default` - template, evaluated *instead of* producing `NO_CONTENT` — the tool for "this value, - or X if missing" at the point of lookup. -- **Absorption**: looking further into a missing value stays missing — a deep `attr` - path over an absent branch yields `NO_CONTENT`, it does not raise. -- **Falsiness**: `NO_CONTENT` is falsy, so logical operators can express fallbacks — - e.g. a `chain` ending in `expr` `or` with a substitute value. Rules that *test* for - absence use identity, not truthiness: `false`, `0`, and `""` are values, not absence. -- **The top level**: `transform()` never returns the raw sentinel by default — a - template that evaluates to `NO_CONTENT` returns `None` (configurable via the - `no_content` argument). - -## Error model - -Template failures are typed by *whose mistake they are*: - -| Exception | Meaning | Representative causes | -|---|---|---| -| `DefinitionError` | The **template** is malformed — fix the template | unknown rule/operator/function name; a missing required parameter; unknown parameters; ambiguous or incomplete mutually-exclusive parameter groups; a reserved variable name used with `set`/`get`; a context accessor used outside its valid scope; a structural parameter with the wrong JSON shape | -| `TransformationError` | The template is valid but the **input data** does not fit it | iterating a non-iterable; joining mixed-type items; an operator applied to incompatible operand types; a function rejecting its arguments; a format pattern referencing a missing key; an `include` chain exceeding the depth limit | - -By default errors are raised **lazily**, when the failing node is actually walked — a -typo in a branch the data never reaches will not surface. Opt in to **static -validation** (`validate=True`, or calling `validate()`) to check the template's -structure up front, with no input data: unknown rules, unknown or missing parameters, -ambiguous parameter combinations, and invalid literal operator/function names all raise -`DefinitionError` immediately. - -Both error types carry the **template location** where the failure occurred — a path of -dict keys, list indices, rule names, and parameter names: - -``` -value is not iterable: 'not-a-list' - at template → pipeline → chain → funcs[0] → map -``` - -Which specific conditions each rule raises is part of that rule's documentation. - -## Expressions and calls - -Operators (the `expr` rule) and functions (the `call` rule) share one application -model; these semantics hold across **every** operator and function, so they live here. -The catalog of what exists — each operator's types and each function's signature — is -in the `expr` `op` / `call` `name` parameter docs and the engine exports. - -**Operator application** (`expr`) has three modes: - -- no value parameter → **unary**: `op(this)`; -- `value` → **binary**: `op(this, value)` — the current value is the left operand; -- `values` → **reduction**: `reduce(op, values)` pairwise over the evaluated list — - and the current value is **ignored**; include `{"$": "this"}` as a list item if the - reduction should involve it. `values` must be a non-empty list — an empty reduction - has no seed and raises `DefinitionError` (see the empty-collection caveat under - composition patterns). - -**Type behavior** follows Python semantics: `+` concatenates strings and lists as well -as adding numbers; comparisons work on like types; the logical operators use -truthiness and return an *operand*, not necessarily a boolean. An operator applied to -incompatible operand types raises `TransformationError`. - -**Function application** (`call`) mirrors the modes: no parameter → `fn(this)`; -`value` → `fn(value)`; `values` → `fn(*values)` (multi-argument call). The current -value is ignored whenever a parameter is given. Built-in functions convert their -documented failure modes into `TransformationError` — a well-formed template never -leaks a raw Python exception from a bad argument. A few functions are documented as -**total** (they accept any well-formed JSON value and never raise); totality is stated -per function in the catalog. - -**`NO_CONTENT` in expressions**: operators and functions do not skip it. A -`NO_CONTENT` operand is simply falsy (useful in `and`/`or` fallbacks) or an -incompatible argument (an error) — stop the propagation earlier with a `default` if -the operand may be missing. - -## Composition patterns - -The language has no aggregate primitives beyond what composition provides — a handful -of rules cover everything *because* they compose. The canonical shapes: - -- **Pipeline**: `chain` walks its steps in order, each result becoming `this` for the - next — the backbone for "extract, then reshape, then format" templates. -- **Reshape**: `map` (over a list or dict) with a nested `object`/`attr` template body; - `filter` before it to drop elements; `zip` to transpose parallel lists. -- **Compute**: nested `expr` rules — arithmetic is a tree of rules, not a string. -- **Reuse**: `include` runs a named sub-template against the current value. Only the - value crosses the boundary (no variables, no iteration properties); the marker is - inherited by default; nested includes are depth-limited. -- **Aggregate from primitives**: a count is `length` (or a `map` to `1`s reduced with - `add`); a flatten is the `flatten` function (or `map` in its `items` mode). **Mind - the empty collection**: an `expr` `values` reduction over an empty list is a - `DefinitionError` (no seed), an empty `join` yields `NO_CONTENT` (use its `default`), - while `sum` of an empty array is `0` — pick the primitive whose empty-case behavior - matches the intent. - -A worked end-to-end flow — pairing two parallel lists into a dict: - -```json -{ - "$": "chain", - "funcs": [ - {"$": "zip", "items": [{"$": "attr", "name": "keys"}, - {"$": "attr", "name": "values"}]}, - {"$": "map", "key": {"$": "attr", "name": 0}, - "value": {"$": "attr", "name": 1}} - ] -} -``` - -Input `{"keys": ["a","b"], "values": [1,2]}` → the root context's `this` is the input → -`zip` produces `[["a",1], ["b",2]]` → `chain` derives a context with that as `this` → -`map` iterates, each pair becoming `this`/`item` in a per-element scope → `attr` with -numeric names indexes each pair → output `{"a": 1, "b": 2}`. +# Transon — Template Language Reference (pointer) + +The Language Reference lives at +[`transon/resources/LANGUAGE.md`](../transon/resources/LANGUAGE.md) — canonical, +hand-edited, and packaged, so the file that ships in the wheel/sdist **is** the file +you edit (single copy, no sync step; same rule as the per-rule docs living in +`transon/rules.py`). It is served offline by +`transon.reference.get_language_reference()` (`python -m transon.reference`) and +rendered on the [docs site](https://transon-org.github.io/). diff --git a/docs/ROADMAP.md b/docs/ROADMAP.md index f21272b..76ff3c7 100644 --- a/docs/ROADMAP.md +++ b/docs/ROADMAP.md @@ -1041,7 +1041,8 @@ failures to `TransformationError`); `split` rule in `transon/rules.py`; total `i **Status**: done · **Severity**: medium · **Source**: [`proposals/0008-language-reference-export.md`](proposals/0008-language-reference-export.md) (Deliverable 1) -A new hand-written `docs/LANGUAGE.md`: the template-language reference for authors (human or +A new hand-written Language Reference (`transon/resources/LANGUAGE.md` — canonical and +packaged in one file; `docs/LANGUAGE.md` is a pointer): the reference for authors (human or agent), carrying **cross-cutting semantics only** — the marker, context/scoping, the `NO_CONTENT` propagation model, the error taxonomy, `expr`/`call` machinery, composition patterns. **No per-entity sections**: per-rule/operator/function prose stays in the @@ -1056,7 +1057,7 @@ hand-maintained copies (spec §2/§11, the class docstring, README overlap) with author-scoped, pinnable document — the `transon-authoring` authority-ladder gap stays open and repair loops keep rediscovering semantics. -**Shipped**: `docs/LANGUAGE.md` (7 pinned sections: preamble, templates-and-the-marker, +**Shipped**: `transon/resources/LANGUAGE.md` (7 pinned sections: preamble, templates-and-the-marker, context-and-scoping, the-no_content-model, error-model, expressions-and-calls, composition-patterns). Consolidation: spec §2 reduced to the engine-internal view, §4 to a pointer + the Recursion budget invariant (per-rule facts folded into @@ -1073,18 +1074,18 @@ embedder narrative (pitch owned by README); section-id pin in Ship `LANGUAGE.md` in the wheel and sdist (e.g. `transon/resources/LANGUAGE.md`) so an installed `transon==` serves its own language reference offline — the property -`get_editor_metadata()` already has for the catalog. The repo-root `docs/LANGUAGE.md` stays the -canonical, human-edited source; the build maps it in or a release check asserts the two are -identical. Acceptance: an `importlib.resources` test asserts the packaged bytes (UTF-8, +`get_editor_metadata()` already has for the catalog. Single-copy refinement: the packaged file +**is** the canonical, hand-edited source (`docs/LANGUAGE.md` is a pointer) — no build mapping, +no mirror to sync. Acceptance: an `importlib.resources` test asserts the packaged bytes (UTF-8, line-endings normalized to `\n`) equal `get_language_reference()['content']`. **Impact if not done**: the `transon-authoring` harnesses mount no repo checkout, so an unpackaged reference is invisible to the primary consumer. -**Shipped**: `transon/resources/LANGUAGE.md` (committed copy; hatchling picks it up in -both wheel and sdist with no config change — verified by building both). -`tests/test_reference.py` loads it through `importlib.resources` and asserts it equals -both `get_language_reference()['content']` and the canonical `docs/LANGUAGE.md`. +**Shipped**: `transon/resources/LANGUAGE.md` is the canonical, hand-edited, single copy; +hatchling picks it up in both wheel and sdist with no config change — verified by building +both. `tests/test_reference.py` loads it through `importlib.resources` and asserts it equals +`get_language_reference()['content']`. `docs/LANGUAGE.md` reduced to a pointer. ### R-36. `get_language_reference()` versioned export diff --git a/docs/SPECIFICATION.md b/docs/SPECIFICATION.md index 00e3f7c..7dd2125 100644 --- a/docs/SPECIFICATION.md +++ b/docs/SPECIFICATION.md @@ -32,8 +32,8 @@ producing JSON *output*. It is inspired by XSLT and JsonLogic. | `transon/docs.py` | Documentation generator: harvests docstrings + test cases into JSON | | `transon/metadata.py` | Editor-metadata export (`get_editor_metadata`) for the visual editor (§5.1) | | `transon/reference.py` | Language Reference export (`get_language_reference`) — serves the packaged `LANGUAGE.md` (§5.2) | -| `transon/resources/LANGUAGE.md` | Packaged copy of `docs/LANGUAGE.md` (ships in the wheel/sdist; a test asserts identity) | -| `docs/LANGUAGE.md` | **Template Language Reference** — author-facing, cross-cutting semantics (canonical, hand-edited) | +| `transon/resources/LANGUAGE.md` | **Template Language Reference** — author-facing, cross-cutting semantics; canonical, hand-edited, and packaged (single copy — ships as-is in the wheel/sdist) | +| `docs/LANGUAGE.md` | Pointer at the canonical reference above (kept for `docs/` discoverability) | | `transon/tests/` | **Example corpus**: table-driven test cases that double as documentation | | `tests/` | Plain pytest tests for engine mechanics (errors, extension, docs generation) | | `.github/workflows/dev.yml` | CI: pytest + coverage on Python 3.9–3.13 (uv) | @@ -46,7 +46,7 @@ Packaging is uv / PEP 621 (`pyproject.toml`, `uv.lock`). Runtime dependencies: n ## 2. Core concepts -> **Author-facing semantics live in [`docs/LANGUAGE.md`](LANGUAGE.md)** — the Template +> **Author-facing semantics live in [`transon/resources/LANGUAGE.md`](../transon/resources/LANGUAGE.md)** — the Template > Language Reference (evaluation model, scoping, the `NO_CONTENT` model, the error > taxonomy, `expr`/`call` machinery, composition patterns), also served by > `transon.reference.get_language_reference()` (§5.2). This section keeps only the @@ -265,7 +265,7 @@ conditions — live in its registration docs in `transon/rules.py` (docstrings + catalogs live the same way in `transon/operators.py` / `transon/functions.py` and the `expr` `op` / `call` `name` parameter docs. Cross-cutting semantics (evaluation model, scoping, `NO_CONTENT`, errors, `expr`/`call` machinery) are in -[`docs/LANGUAGE.md`](LANGUAGE.md). +[`transon/resources/LANGUAGE.md`](../transon/resources/LANGUAGE.md). ### Recursion budget @@ -285,7 +285,7 @@ never a raw `RecursionError`. (Roadmap R-32.) The documentation (and the playground at https://transon-org.github.io/) is **generated from source artifacts**; the one hand-written artifact is -[`docs/LANGUAGE.md`](LANGUAGE.md) (the Template Language Reference — cross-cutting +[`transon/resources/LANGUAGE.md`](../transon/resources/LANGUAGE.md) (the Template Language Reference — cross-cutting semantics only, no per-entity sections; served packaged via §5.2, its section shape pinned by `tests/test_reference.py`). Everything else is harvested: @@ -424,12 +424,13 @@ export (RFC 0008, R-36) serving the packaged `LANGUAGE.md` offline: is breaking and bumps the major. Consumers MUST fail loudly on an unsupported major. - The export is **engine-global** (base `Transformer` only; no `cls=` parameter) and states language facts only. -- Packaging: `transon/resources/LANGUAGE.md` ships in the wheel and sdist (hatchling +- Packaging: `transon/resources/LANGUAGE.md` is the **canonical, hand-edited, single + copy** — the file that ships in the wheel and sdist is the file you edit (hatchling packages the whole `transon/` tree; verified by building the distributions at - release). `docs/LANGUAGE.md` is the canonical hand-edited source; - `tests/test_reference.py` asserts **source-tree parity** — the packaged resource - read via `importlib.resources` equals both the export's `content` and the - canonical file — pins the section-id list, and checks the split parity. + release; no sync step, same rule as per-rule docs living in `rules.py`). + `docs/LANGUAGE.md` is a pointer. `tests/test_reference.py` asserts the packaged + resource read via `importlib.resources` equals the export's `content`, pins the + section-id list, and checks the split parity. - `python -m transon.reference` prints this JSON. --- @@ -562,7 +563,7 @@ tagged example cases. ## 11. Engine data flow (reference example) -Relocated to the Language Reference: [`docs/LANGUAGE.md`](LANGUAGE.md), "Composition +Relocated to the Language Reference: [`transon/resources/LANGUAGE.md`](../transon/resources/LANGUAGE.md), "Composition patterns" (the `zip` + `map` worked end-to-end flow). --- diff --git a/docs/proposals/0008-language-reference-export.md b/docs/proposals/0008-language-reference-export.md index 1b5466d..4005c01 100644 --- a/docs/proposals/0008-language-reference-export.md +++ b/docs/proposals/0008-language-reference-export.md @@ -5,7 +5,9 @@ - **Amended:** 2026-07-18 — consolidation scope extended to the `Transformer` docstring and `README.md`; single ownership principle (structure in the catalog, per-entity behavior in registration docs, cross-cutting semantics in `LANGUAGE.md`); no per-entity sections in - `LANGUAGE.md`; sequencing decision (see below) + `LANGUAGE.md`; sequencing decision; single-copy refinement — the packaged + `transon/resources/LANGUAGE.md` is canonical, `docs/LANGUAGE.md` is a pointer (see + Deliverable 2) - **Roadmap:** R-34 (Language Reference document), R-35 (package the reference), R-36 (`get_language_reference()` export) — `accepted`, rows in `docs/ROADMAP.md`; docs-site counterpart is D-20 in `docs/DOCS_SITE_ROADMAP.md`. (R-33 is held by [RFC 0007](0007-builtin-function-library.md).) - **Type:** New documentation artifact + packaging + a new read-only export API (`get_language_reference()`) — additive; no change to existing template semantics or engine API shapes. The **content** of `get_all_docs()['doc']` shrinks to the embedder-facing narrative as part of the consolidation (shape unchanged; the docs export carries no schema version, so the change is coordinated by release note in `CHANGELOG.md`). Symmetrically, per-rule doc **content grows** in both exports (`get_all_docs()` and `get_editor_metadata()['docs']`) as §4's facts fold into the docstrings — also shape-unchanged, doc text is contractually opaque - **Consumers:** `transon-authoring` (authority ladder rung 2; `SKILL.md`, AD-018/NFR-001/NFR-003), `transon-org.github.io` (docs site) @@ -56,8 +58,9 @@ RFC's sourcing rule covers it (Deliverable 1). ## Deliverable 1 — the Language Reference document (R-34) -A new `docs/LANGUAGE.md`: the Transon **template language reference**, addressed to template -authors (human or agent), containing semantics only: +A new `transon/resources/LANGUAGE.md` (canonical **and** packaged — see Deliverable 2; a +`docs/LANGUAGE.md` pointer keeps `docs/` discoverability): the Transon **template language +reference**, addressed to template authors (human or agent), containing semantics only: - The marker: rule invocation shape, literal-marker escaping, marker inheritance across `include`. - Context: `this`/`item`/`key`/`index`/`parent`, variable scoping (`set`/`get`), scope derivation. @@ -137,20 +140,21 @@ to drift. The prose stays hand-written. ## Deliverable 2 — ship the reference in the package (R-35) -`LANGUAGE.md` becomes package data (e.g. `transon/resources/LANGUAGE.md`, included in the wheel +`LANGUAGE.md` lives as package data (`transon/resources/LANGUAGE.md`, included in the wheel and sdist), so an installed `transon==` serves its own language reference offline — -exactly the property `get_editor_metadata()` already has for the catalog. The repo-root -`docs/LANGUAGE.md` stays the canonical, human-edited source; the build maps it in (mirroring the -`transon-authoring` `resources/` force-include pattern) or a release check asserts the two are -identical. +exactly the property `get_editor_metadata()` already has for the catalog. **Single-copy +refinement (implementation decision)**: rather than a canonical `docs/` file mirrored into the +package (force-include, or a copy plus identity check — both create a second copy or a build +mode that differs between dev and installed layouts), the packaged file **is** the canonical, +hand-edited source — the same rule as per-rule docs living in `rules.py` — and +`docs/LANGUAGE.md` is a pointer. No sync step exists to forget. **Acceptance (packaging parity).** A test loads the packaged `LANGUAGE.md` through `importlib.resources`, decodes those bytes as UTF-8, normalizes line endings to `\n`, and asserts -the result equals `get_language_reference()['content']` **and** the canonical `docs/LANGUAGE.md`, -following the shape-test pattern in `tests/test_metadata.py`. In CI this proves **source-tree -parity** (a stale packaged copy fails immediately); the wheel/sdist inclusion itself is verified -by building the distributions at release — the section-pin test (Deliverable 1) would notice -neither. +the result equals `get_language_reference()['content']`, following the shape-test pattern in +`tests/test_metadata.py`. With the single-copy refinement there is no second copy to drift; the +wheel/sdist inclusion itself is verified by building the distributions at release — which the +section-pin test (Deliverable 1) would not notice. ## Deliverable 3 — `get_language_reference()` export (R-36) diff --git a/tests/test_reference.py b/tests/test_reference.py index b7f431a..9494581 100644 --- a/tests/test_reference.py +++ b/tests/test_reference.py @@ -1,7 +1,6 @@ """Shape, section-pin, split-parity, and packaging tests for the Language Reference export (RFC 0008, R-34/R-35/R-36).""" import importlib.resources -from pathlib import Path from transon.reference import ( REFERENCE_VERSION, @@ -10,9 +9,9 @@ ) #: The pinned section-id list (RFC 0008 drift protection): adding, renaming, or -#: removing a section in ``docs/LANGUAGE.md`` must update this pin **and** follow -#: the ``REFERENCE_VERSION`` policy — additive changes bump the minor, removals/ -#: renames are breaking and bump the major. Never a silent edit. +#: removing a section in ``transon/resources/LANGUAGE.md`` must update this pin +#: **and** follow the ``REFERENCE_VERSION`` policy — additive changes bump the +#: minor, removals/renames are breaking and bump the major. Never a silent edit. PINNED_SECTION_IDS = [ 'preamble', 'templates-and-the-marker', @@ -23,8 +22,6 @@ 'composition-patterns', ] -REPO_ROOT = Path(__file__).resolve().parent.parent - def test_reference_shape(): ref = get_language_reference() @@ -64,11 +61,10 @@ def test_section_fields(): assert set(section) == {'id', 'title', 'heading_level', 'content'} -def test_packaged_copy_is_served_and_matches_canonical(): - """Packaging parity (RFC 0008 Deliverable 2): the export serves the packaged - ``transon/resources/LANGUAGE.md`` via ``importlib.resources``, and that copy - is byte-identical (modulo line endings) to the canonical, hand-edited - ``docs/LANGUAGE.md``.""" +def test_packaged_resource_is_served(): + """Packaging parity (RFC 0008 Deliverable 2): the export serves the + canonical ``transon/resources/LANGUAGE.md`` via ``importlib.resources`` — + the single hand-edited copy, which is also what ships in the wheel/sdist.""" ref = get_language_reference() packaged = ( importlib.resources.files('transon') @@ -79,12 +75,6 @@ def test_packaged_copy_is_served_and_matches_canonical(): .replace('\r', '\n') ) assert packaged == ref['content'] - canonical = (REPO_ROOT / 'docs' / 'LANGUAGE.md').read_text(encoding='utf-8') - canonical = canonical.replace('\r\n', '\n').replace('\r', '\n') - assert canonical == ref['content'], ( - 'docs/LANGUAGE.md and transon/resources/LANGUAGE.md have diverged — ' - 'copy the canonical docs/LANGUAGE.md over the packaged resource' - ) def test_split_without_preamble(): diff --git a/transon/transformers.py b/transon/transformers.py index 1a31e73..809d4d4 100644 --- a/transon/transformers.py +++ b/transon/transformers.py @@ -262,7 +262,7 @@ class Transformer: invocation) and apply it to input data. The **template language** itself — the evaluation model, scoping, the `NO_CONTENT` model, the error taxonomy, composition patterns — is specified in the - [Language Reference](https://github.com/transon-org/transon/blob/main/docs/LANGUAGE.md) + [Language Reference](https://github.com/transon-org/transon/blob/main/transon/resources/LANGUAGE.md) (also served by `transon.reference.get_language_reference()`); what the project is and how it compares to alternatives is in the [README](https://github.com/transon-org/transon#readme). From c6eb9f919f3b7475b057cbcc22d91a5ac430a6d8 Mon Sep 17 00:00:00 2001 From: Eugene Chernyshov Date: Sat, 18 Jul 2026 02:33:20 +0300 Subject: [PATCH 6/8] =?UTF-8?q?docs:=20spec=20keeps=20its=20full=20semanti?= =?UTF-8?q?cs=20=E2=80=94=20deliberate=20duplication=20accepted?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Decision (2026-07-18): SPECIFICATION.md remains the single, complete engine contract. Restore the original section 2 (core concepts incl. the scoping table and full NO_CONTENT/error-model statements), section 4 (full per-rule/operator/function reference, 4.1-4.8 incl. the Recursion budget), and section 11 (data-flow example) from the pre-PR spec — keeping the additions made since (error-plumbing contract note, the new map.items non-list fact in 2.4/4.4). The duplication with LANGUAGE.md and the registration docs is deliberate and banner-flagged at sections 2 and 4: update all surfaces in the same change. RFC 0008 amended accordingly (sourcing rule records the exception; ownership principle scoped to consumer channels; non-goal reworded; Amended header updated); roadmap R-34 Shipped note and CHANGELOG aligned. 398 passed; check_roadmap consistent. Co-Authored-By: Claude Fable 5 --- CHANGELOG.md | 6 +- docs/ROADMAP.md | 13 +- docs/SPECIFICATION.md | 386 ++++++++++++++---- .../0008-language-reference-export.md | 31 +- 4 files changed, 341 insertions(+), 95 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index a9bed1a..3b54e22 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -50,8 +50,10 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/). per-rule facts (edge cases, `NO_CONTENT` treatment, error conditions) folded into the registration docs, so per-rule doc content in `get_all_docs()` and `get_editor_metadata()['docs']` is longer (doc text is contractually opaque — no - shape change). `SPECIFICATION.md` §2/§4/§11 now hold the engine-internal view and - link out to the reference. Docs-site counterpart work is D-20. (Roadmap R-34) + shape change). `SPECIFICATION.md` deliberately retains its full §2/§4/§11 statement + (the engine contract stays one complete document; the duplication is banner-flagged + in-document and aligned by review). Docs-site counterpart work is D-20. + (Roadmap R-34) ## [0.1.8] - 2026-07-16 diff --git a/docs/ROADMAP.md b/docs/ROADMAP.md index 76ff3c7..b142ff6 100644 --- a/docs/ROADMAP.md +++ b/docs/ROADMAP.md @@ -1059,13 +1059,14 @@ repair loops keep rediscovering semantics. **Shipped**: `transon/resources/LANGUAGE.md` (7 pinned sections: preamble, templates-and-the-marker, context-and-scoping, the-no_content-model, error-model, expressions-and-calls, -composition-patterns). Consolidation: spec §2 reduced to the engine-internal view, §4 -to a pointer + the Recursion budget invariant (per-rule facts folded into -`transon/rules.py` docstrings — accessors' scope errors, `attr` error split, `map`/ +composition-patterns). Consolidation: spec §4's per-rule facts folded into +`transon/rules.py` docstrings (accessors' scope errors, `attr` error split, `map`/ `filter`/`zip`/`join`/`file` edge cases, `expr`/`call` mode errors + the sanctioned -reference pointer), §11 to a pointer; `Transformer` class docstring slimmed to the -embedder narrative (pitch owned by README); section-id pin in -`tests/test_reference.py`. Changelog entry under Unreleased. +reference pointer); the spec **retains its full §2/§4/§11 statement** — deliberate, +banner-flagged duplication (decision 2026-07-18) so the engine contract stays one +complete document; `Transformer` class docstring slimmed to the embedder narrative +(pitch owned by README); section-id pin in `tests/test_reference.py`. Changelog entry +under Unreleased. ### R-35. Package the Language Reference as package data diff --git a/docs/SPECIFICATION.md b/docs/SPECIFICATION.md index 7dd2125..220cb99 100644 --- a/docs/SPECIFICATION.md +++ b/docs/SPECIFICATION.md @@ -46,68 +46,150 @@ Packaging is uv / PEP 621 (`pyproject.toml`, `uv.lock`). Runtime dependencies: n ## 2. Core concepts -> **Author-facing semantics live in [`transon/resources/LANGUAGE.md`](../transon/resources/LANGUAGE.md)** — the Template -> Language Reference (evaluation model, scoping, the `NO_CONTENT` model, the error -> taxonomy, `expr`/`call` machinery, composition patterns), also served by -> `transon.reference.get_language_reference()` (§5.2). This section keeps only the -> **engine-internal** view: which code implements those semantics and the -> implementation invariants a contributor must preserve. - -### 2.1 Template walk - -`Transformer.walk` dispatches by JSON type: **list** → `walk_list`, **dict with the -marker key** → `walk_rule` (registry dispatch), **dict without** → `walk_dict`, -**scalar** → `walk_scalar`. The marker is per-instance (`marker=` constructor kwarg). -Rule parameters are walked recursively except parameters registered as constants -(`ParamKind.CONSTANT`, e.g. `expr.op`, `call.name`). The literal-marker escape is the -`object` rule's `fields` mode (R-14). +> **Deliberate duplication (decision 2026-07-18).** This section is the complete +> normative statement of the language semantics *inside the engine contract*; the +> author-facing packaged form is [`transon/resources/LANGUAGE.md`](../transon/resources/LANGUAGE.md) +> (served by `transon.reference.get_language_reference()`, §5.2), and per-entity +> behavior is also stated in the registration docs. The spec deliberately keeps its +> own full copy so it remains a single, complete document — when changing behavior, +> update all three surfaces in the same change. + +### 2.1 Templates and the marker + +A template is any JSON value. The engine walks it recursively (`Transformer.walk`): + +- **list** → walk each element, return a new list (`walk_list`). +- **dict containing the marker key** (default `"$"`) → this is a **rule invocation**; + dispatch to the registered rule named by the marker's value (`walk_rule`). +- **dict without the marker** → walk each value, return a new dict with the same keys (`walk_dict`). +- **anything else** (scalar) → returned as-is (`walk_scalar`). + +The marker is configurable per `Transformer` instance (`marker=` constructor kwarg). +To emit a literal dict that contains the marker key, use the `object` rule in `fields` +mode (`{"$": "object", "fields": {"$": ...}}`): the keys of `fields` are emitted +verbatim while the values are walked as templates (R-14). The single-pair form +(`{"$": "object", "key": "$", "value": ...}`) also works for one literal key. + +A rule invocation dict carries its parameters as sibling keys of the marker: + +```json +{"$": "attr", "name": "x"} +``` + +Rule parameters are themselves templates (walked recursively), except where a rule +explicitly requires a constant (e.g. `expr.op`, `call.name`, `chain.funcs` list +structure). ### 2.2 Context -`Context` (in `transformers.py`) is a linked chain of scopes holding `this`, the -iteration slots, user variables, and a `parent` link. Implementation invariants -(R-15/R-22 — the observable scoping model built on them is specified in the Language -Reference, "Context and scoping"): - -- `context.derive(**props)` stores **only** the new props (copy-on-write); reads of - user variables walk the parent chain; the **first** `set` in a derived scope - materializes inherited variables locally so writes stay isolated. -- `set` writes into the data dict of the exact context object it executes in; - `walk_dict`/`walk_list` pass the **same** context object to all siblings, and the - first func of a `chain` runs in the caller's context — these two facts produce the - documented sibling-order and first-chain-func visibility. -- The names `this`, `item`, `key`, `value`, `index` are **reserved** - (`Context.RESERVED_NAMES`): `__contains__`/`__getitem__`/`__setitem__` check against - them and raise `DefinitionError`. +`Context` (in `transformers.py`) is a linked chain of scopes. Each context holds: + +- `this` — the current value (root context: the transformation input). +- Iteration properties — `item`, `index` (lists), `key`, `value`, `index` (dicts); + only present inside `map`/`filter` iterations. +- User variables — arbitrary names written by the `set` rule, read by `get`. +- `parent` — the context this one was derived from. + +`context.derive(**props)` creates a child linked via `parent`: it stores only the +new props (e.g. `this`, iteration slots). Reads of user variables walk the parent +chain; the first `set` in a derived scope materializes inherited variables into the +local dict so writes stay isolated. `set` writes into the data dict of the exact +context object it executes in. + +#### Variable scoping (`set` / `get`) + +Template authors should treat these rules as the contract (R-15): + +| Where `set` runs | Visible to | +|---|---| +| Descendant scopes (`derive()` after the `set`) | Yes — visible via parent-chain lookup; first `set` in a child snapshots inherited vars | +| Later sibling keys/items in the same literal dict/list | Yes — siblings share one context object; order matters | +| Earlier sibling keys/items in the same literal dict/list | No — already evaluated | +| First func of a `chain` | Caller's scope, later `chain` funcs, and later siblings outside the `chain` | +| Later `chain` func, `map`/`filter` iteration, etc. | Only that derived scope and its descendants | +| Parent scope after a derived scope ends | No | +| `include` sub-template | No — separate `transform()` | + +Consequences (all verified against the implementation): + +- Variables flow **downward only**: a `derive()` performed *after* a `set` carries the + variable; the parent's own dict is untouched by sets in derived contexts. So a `set` + inside a `map` item or inside a non-first `chain` step is invisible once that scope + ends. +- `walk_dict`/`walk_list` pass the **same context object** to all siblings, so a `set` + executed directly at one key of a literal dict is visible to later-evaluated sibling + keys (insertion order). The first func of a `chain` also runs in the caller's + context, so a `set` there escapes into the caller's scope; subsequent funcs run in + derived contexts and their sets do not. +- `get` walks the current context's data dict, then the parent chain for undefined + names — so ancestor variables are visible without eager copying on `derive()`. The + first `set` in a derived scope materializes inherited variables locally. + +Refactoring pitfall: wrapping a step in `chain`, reordering dict keys, or moving a +`set` can change visibility with no error — consult the table above. + +The names `this`, `item`, `key`, `value`, `index` are **reserved** +(`Context.RESERVED_NAMES`): `Context.__contains__`, `__getitem__`, `__setitem__` +check against them, so they cannot be used as variable names with `set`/`get` +(violation raises `DefinitionError`). ### 2.3 NO_CONTENT — the "no value" sentinel `Transformer.NO_CONTENT` is a singleton `NoContent` instance representing the absence -of a value (distinct from JSON `null`/Python `None`). Implementation facts: - -- `NoContent.__bool__` is `False` (falsiness powers `expr` `and`/`or` fallbacks) and - `NoContent.__getitem__` returns `self` (absorption for deep `attr` paths). Rules - that test for absence use **identity** (`is NO_CONTENT`), never truthiness. -- The skip-don't-emit propagation model is specified in the Language Reference ("The - NO_CONTENT model"); each rule's exact treatment is stated in that rule's docstring - and is part of the public contract (§10). -- `transform()` substitutes a top-level `NO_CONTENT` via its `no_content` parameter - (default `None`, §3.1); inside evaluation rules always see the raw sentinel. +of a value (distinct from JSON `null`/Python `None`). Semantics: + +- **Producers**: `attr` (missing key/index), `get` (undefined variable), `file` (always), + `include` (when the included template yields `NO_CONTENT`), `join` (when there are no + items to join). Optional `default` on `attr`, `get`, `format`, `include`, and `join` + returns a substitute instead of `NO_CONTENT`. +- **Falsiness**: `NoContent` is falsy in boolean context (`bool()`, `expr` `and`/`or`), + so logical operators can fall back from `NO_CONTENT` (e.g. `chain` of `join` then + `expr` `or` with a fallback value). Rules that test for absence still use identity + (`is NO_CONTENT`), not truthiness. +- **Absorption**: `NoContent.__getitem__` returns `self`, so further `attr` lookups on + a missing value stay `NO_CONTENT` instead of raising. +- **Consumers (skip/filter behavior)**: + - `map`: items (or key/value pairs) that evaluate to `NO_CONTENT` are omitted. + - `object`: in `key`/`value` mode returns `{}` if key or value is `NO_CONTENT`; + in `fields` mode omits each entry whose value is `NO_CONTENT`. + - `file`: skips writing if name or content is `NO_CONTENT`. + - `filter`: a condition evaluating to `NO_CONTENT` excludes the element. + - `join`: items that evaluate to `NO_CONTENT` are omitted before concatenation; when + no items remain the result is `NO_CONTENT` unless `default` is provided. + - `split`: when `context.this` is `NO_CONTENT`, returns `NO_CONTENT` (passthrough). +- **`format`**: returns `NO_CONTENT` when the formatting value (or any unpacked list + element or dict key/value) is `NO_CONTENT`, unless `default` is provided. +- **Top-level `transform()`**: by default maps a top-level `NO_CONTENT` result to + `None` via the `no_content` parameter (see §3.1). Pass `Transformer.NO_CONTENT` + as `no_content` to receive the raw sentinel. ### 2.4 Error model -`DefinitionError` (malformed template) and `TransformationError` (valid template, -incompatible data) are both defined in `transformers.py` and exported from the package -root. The taxonomy as an author experiences it is in the Language Reference ("Error -model"); which conditions each rule raises is in that rule's docstring. Engine-side -contract: +| Exception | Meaning | Raised when | +|---|---|---| +| `DefinitionError` | The template is malformed | Unknown rule/operator/function name; missing required rule parameter (`Transformer.require` or `Transformer.validate()`); unknown rule parameters; ambiguous or incomplete mutually-exclusive parameter groups (`validate()`); `attr` with neither `name` nor `names`; `map` with no valid parameter combination; reserved variable name (`this`, `item`, `key`, `value`, `index`) used with `set`/`get`; iteration accessors (`item`, `key`, `value`, `index`) or `parent` used outside their valid scope; `expr`/`call` with a non-list or empty `values` parameter; `map` `items` template evaluating to a non-list; non-list `chain.funcs` (`validate()`); `include` with no configured `template_loader` (default loader) | +| `TransformationError` | The template is valid but input data is incompatible | `map`/`filter` over a non-iterable (not list/dict); `join` over mixed-type items; `split` on a non-string/non-array input or with an invalid `sep`; `attr` lookup with an incompatible index type; `zip` over non-iterable items; `expr` operator applied to incompatible operand types; `call` with incompatible argument types or a function that rejects its arguments (e.g. empty `min`/`max`, bad epoch, invalid regex); `format` pattern referencing a missing key or index; `set`/`get` when a dynamic `name` evaluates to `NO_CONTENT`; `include` depth limit exceeded (nested include chain too deep) | + +Both are exported from the package root. By default, errors are raised lazily during +`transform()` — there is no automatic validation. Opt in with `Transformer.validate()` +or `Transformer(..., validate=True)` for a static walk that raises `DefinitionError` +for unknown rules, unknown rule parameters, missing required parameters, ambiguous +mutually-exclusive parameter combinations, and invalid literal operator/function names +(see §3.4). -- Errors are raised **lazily** during `transform()`; static checking is the opt-in - `validate()` walk (§3.4). -- Every message is routed through `format_error_message`, which appends the template - location (`at template → …`) from the template-path `ContextVar` maintained by - `walk` — new raise sites must use `t.definition_error` / `t.transformation_error` - (or `format_error_message`) so the path is never lost. +`DefinitionError` and `TransformationError` messages include the template location +where the failure occurred (a path of dict keys, list indices, rule names, and rule +parameter names), for example: + +``` +value is not iterable: 'not-a-list' + at template → pipeline → chain → funcs[0] → map +``` + +Engine-side plumbing contract: every message is routed through +`format_error_message`, which appends the template location from the template-path +`ContextVar` maintained by `walk` — new raise sites must use `t.definition_error` / +`t.transformation_error` (or `format_error_message`) so the path is never lost. --- @@ -257,27 +339,165 @@ parameter with no descriptor defaults to a dynamic template (`ParamKind.DYNAMIC` ## 4. Built-in rule reference -Per-rule behavior is **not** specified here (ownership principle, RFC 0008): each -rule's semantics — parameters, modes, edge cases, `NO_CONTENT` treatment, error -conditions — live in its registration docs in `transon/rules.py` (docstrings + -`register_rule` param kwargs), which every export carries (`get_all_docs()`, -`get_editor_metadata()['docs']`) and the docs site renders. Operator and function -catalogs live the same way in `transon/operators.py` / `transon/functions.py` and the -`expr` `op` / `call` `name` parameter docs. Cross-cutting semantics (evaluation model, -scoping, `NO_CONTENT`, errors, `expr`/`call` machinery) are in -[`transon/resources/LANGUAGE.md`](../transon/resources/LANGUAGE.md). - -### Recursion budget - -Walking one level of template nesting consumes a bounded, small number of Python -call-stack frames — one core recursion frame per node (no `walk`/`_walk`-style -doubling). Because the generated `transon-blockly` editor codec self-`include`s once -per document node (AD-030), its reachable nesting depth is governed by this budget, -not by `max_include_depth`. The engine therefore transforms self-`include`ing -templates at nesting depths well past the editor's deepest generator (`G_encode`, -depth 41) within CPython's default recursion limit (1000). Exceeding -`max_include_depth` MUST surface as the `include` depth-limit `TransformationError`, -never a raw `RecursionError`. (Roadmap R-32.) +> **Deliberate duplication (decision 2026-07-18).** The same per-rule facts are +> stated in the registration docs (`transon/rules.py` docstrings + param kwargs), +> which every export carries and the docs site renders. The spec keeps this full +> reference so it remains a single, complete contract — update both in the same +> change. + +All rules live in `transon/rules.py`. "Dynamic" parameters are walked as templates; +"constant" parameters are read verbatim. + +### 4.1 Context accessors (no parameters) + +| Rule | Returns | Valid scope | +|---|---|---| +| `this` | `context.this` — current value | anywhere | +| `parent` | `context.parent.this` — value of previous scope | any non-root scope | +| `item` | current list element | inside `map`/`filter` over a list | +| `index` | 0-based iteration index | inside `map`/`filter` | +| `key` | current dict key | inside `map`/`filter` over a dict | +| `value` | current dict value | inside `map`/`filter` over a dict | + +Accessing `parent` in the root context or an iteration property outside its scope +raises `DefinitionError`. + +### 4.2 Variables + +| Rule | Parameters | Semantics | +|---|---|---| +| `set` | `name` (dynamic) | Stores `context.this` under `name` in the *current* context; returns `context.this` (pass-through, usable as a tap inside `chain`). Raises `TransformationError` if `name` evaluates to `NO_CONTENT`. Scoping: see §2.2 (variable scoping table). | +| `get` | `name` (dynamic), `default` (optional, dynamic) | Returns the stored value from the *current* context (ancestor variables resolved via parent-chain lookup). Returns `NO_CONTENT` if undefined. Raises `TransformationError` if `name` evaluates to `NO_CONTENT`. When `default` is provided, returns its evaluation instead of `NO_CONTENT`. Scoping: see §2.2. | + +Examples for each scoping case live in `transon/tests/test_set.py` (docs-site examples). + +### 4.3 Data access — `attr` + +Parameters (mutually exclusive; one required, else `DefinitionError`): + +- `name` (dynamic): single key or numeric index into `context.this`. +- `names` (dynamic): list of keys/indexes, applied sequentially (deep path). +- `default` (optional, dynamic): returned instead of `NO_CONTENT` when the lookup misses. + +Missing key (`KeyError`) or index out of range (`IndexError`) → `NO_CONTENT` (or +`default` when provided). +When `name` or any path segment in `names` evaluates to `NO_CONTENT` → `NO_CONTENT` +(uniform regardless of container type). +Other lookup failures (e.g. `TypeError` indexing a string with a string) → +`TransformationError`. + +### 4.4 Structure builders + +| Rule | Parameters | Semantics | +|---|---|---| +| `object` | exactly one of: `key`+`value` \| `fields` | `key`+`value` (dynamic): single-pair dict `{key: value}`; `{}` if either side is `NO_CONTENT`. For dynamically-named attributes. `fields`: literal mapping whose keys are emitted verbatim (including the marker `$` — the canonical literal-marker-key escape, R-14) and whose values are walked as templates; entries with a `NO_CONTENT` value are omitted. | +| `map` | exactly one of: `item` \| `items` \| `key`+`value` | Iterates `context.this` (list or dict). `item`: one output element per input element → list. `items`: template yields a *list* of elements per input element, concatenated → list (a non-list result raises `DefinitionError`). `key`+`value`: → dict. `NO_CONTENT` results are skipped. Each iteration derives a sub-context with `this`=element plus iteration props. | +| `filter` | `cond` (required, dynamic) | Keeps elements where `cond` is truthy (and not `NO_CONTENT`). Preserves container type: list→list, dict→dict. | +| `zip` | `items` (required, dynamic) | Transposes iterables like Python's `zip`: each output row is a **list** (`[list(row) for row in zip(*items)]`). Non-iterable items → `TransformationError`. | +| `join` | `items` (required, dynamic), `sep` (dynamic, strings only, default `""`), `default` (optional, dynamic) | Type-homogeneous concatenation: all-strings → `sep.join`; all-lists → flatten one level; all-dicts → merged dict (later keys win). Items that evaluate to `NO_CONTENT` are omitted before concatenation. When no items remain → `NO_CONTENT` (or `default` when provided). Mixed types → `TransformationError`. `sep` must evaluate to a string when joining strings. | +| `split` | `sep` (required, dynamic) | Inverse of string/list `join`. Input `NO_CONTENT` → `NO_CONTENT`. String input: `sep` must be a non-empty string → list of strings (empty `sep` → `TransformationError`). Array input: result is a list of lists; `sep` is a single non-array element (split on `==`) or a non-empty array (split on each contiguous subsequence occurrence). Empty-array `sep` → `TransformationError`. Because an array `sep` means subsequence, you cannot split on a separator *element* that is itself an array. Other input types → `TransformationError`. | +| `chain` | `funcs` (required; list of templates) | Function composition: walks each template in order, each result becomes `this` of a derived context for the next. `chain(f1, f2, f3)(x) == f3(f2(f1(x)))`. | + +### 4.5 Computation + +| Rule | Parameters | Semantics | +|---|---|---| +| `expr` | `op` (required, constant), optionally `value` (dynamic) or `values` (dynamic) | No param → unary `op(this)`. `value` → binary `op(this, value)`. `values` → `reduce(op, values)` (**`this` is ignored**). | +| `call` | `name` (required, constant), optionally `value` or `values` (dynamic) | No param → `fn(this)`. `value` → `fn(value)`. `values` → `fn(*values)`. `this` is ignored when params given. | +| `format` | `pattern` (required, dynamic), `value` (optional, dynamic; defaults to `this`), `default` (optional, dynamic) | Python `str.format`. `pattern` must evaluate to a string. Returns `NO_CONTENT` when the formatting value (or any unpacked list element or dict key/value) is `NO_CONTENT`, unless `default` is provided. List value → positional unpack `pattern.format(*v)`; dict value → keyword unpack `pattern.format(**v)`; otherwise single argument. | +| `switch` | `key` (required, dynamic), `cases` (required, `mapping`), `default` (optional, dynamic) | **Lazy** equality dispatch: evaluates `key`, then walks **only** the matching entry of `cases` (a literal-keyed mapping of templates). No match — including `key` → `NO_CONTENT` — evaluates `default` if present, else `NO_CONTENT`. Non-selected cases are never walked. | +| `cond` | `cases` (required, `arms`), `default` (optional, dynamic) | **Lazy** ordered conditional (subsumes `if`/`else`): `cases` is an ordered list of `{when, then}` arms. Walks each `when` in order; the first truthy one selects its `then` (the only `then` walked). A `when` of `NO_CONTENT` is falsy. No match → `default` if present, else `NO_CONTENT`. | + +### 4.6 Side effects & composition + +| Rule | Parameters | Semantics | +|---|---|---| +| `file` | `name`, `content` (both required, dynamic) | Calls the configured `file_writer(name, content)`. Skipped if either is `NO_CONTENT`. Always returns `NO_CONTENT` (so `map` over `file` yields `[]`). | +| `include` | `name` (required, dynamic), `default` (optional, dynamic) | Loads a sub-`Transformer` via the configured `template_loader` and runs it against `context.this`. Variables/context do **not** cross the boundary — only the value. The `template_loader` is always called as `loader(name, context=...)` and handed an `IncludeContext` (parent loader, marker, depth guard, include-stack); it **constructs** the sub-`Transformer` itself (e.g. via `context.transformer(template)`). The sub-`Transformer` therefore inherits the parent transformer's marker by default (so a sub-template using the default marker stays consistent across the boundary; pass an explicit `marker` to `context.transformer` to pin a different one), the parent's `template_loader` propagates so recursive/self-`include`ing templates re-resolve without per-host patching, and the loaded instance is never mutated. Sub-result `NO_CONTENT` is propagated as this transformer's `NO_CONTENT` (or `default` when provided). Nested includes are tracked by name; exceeding `max_include_depth` (constructor parameter, default 50) raises `TransformationError` with the include chain in the message. | + +> **Recursion budget.** Walking one level of template nesting consumes a bounded, small number of +> Python call-stack frames — one core recursion frame per node (no `walk`/`_walk`-style doubling). +> Because the generated `transon-blockly` editor codec self-`include`s once per document node +> (AD-030), its reachable nesting depth is governed by this budget, not by `max_include_depth`. The +> engine therefore transforms self-`include`ing templates at nesting depths well past the editor's +> deepest generator (`G_encode`, depth 41) within CPython's default recursion limit (1000). +> Exceeding `max_include_depth` MUST surface as the `include` depth-limit `TransformationError`, +> never a raw `RecursionError`. (Roadmap R-32.) + +### 4.7 Built-in operators (`expr`) + +Each operator has a mnemonic and a code-style alias. Most map to the Python +`operator` module; `in` is a total membership predicate: + +| Mnemonic | Alias | Python impl | Note | +|---|---|---|---| +| `lt le eq ne ge gt` | `< <= == != >= >` | `operator.lt` … | comparisons | +| `add sub mul div mod` | `+ - * / %` | `operator.add` …, `truediv` for `div` | `+` also concatenates strings/lists | +| `and or not` | `&& \|\| !` | logical `and`/`or`/`not` | Python truthiness; returns operands, not always `bool` | +| `in` | `in` | membership | **Total** (never raises). Binary `op(a, b)` = "`a` is a member of `b`": array → element membership; string → substring (`a` must be a string, else `false`); object → key presence (`a` must be a string, else `false`); any other container → `false`. | + +### 4.8 Built-in functions (`call`) + +Existing conversion family: `str`, `int`, `float` (Python builtins), `type` (JSON type +name — **total** over well-formed JSON), and `bool` (Python truthiness — **total**). + +Every other built-in is a wrapper that converts documented failure modes into +`TransformationError` itself (`rule_call` only catches `TypeError`). Multi-argument +forms use `call` `values`. + +#### Strings + +| name | args | notes | +|---|---|---| +| `upper` `lower` `capitalize` | unary | non-string → `TransformationError` | +| `replace` | `[s, old, new]` | total on strings | +| `removeprefix` `removesuffix` | `[s, fix]` | prefix/suffix removal (not character-set strip) | +| `strip` `lstrip` `rstrip` | unary or `[s, chars]` | character-set trimming | + +#### Strings and arrays + +| name | args | notes | +|---|---|---| +| `slice` | `[x, start]` or `[x, start, stop]` | `x` string or array; Python slice semantics (negative indices, out-of-range clamps). Non-int index or other `x` type → `TransformationError`. | +| `reversed` | unary | array → reversed array; string → reversed string; other → `TransformationError` | + +#### Epoch dates (UTC only) + +| name | args | notes | +|---|---|---| +| `from_epoch` | unary or `[n, fmt]` | Epoch **seconds** (int/float) → string. Default format is fixed ISO-8601 `YYYY-MM-DDThh:mm:ssZ`. Fractional seconds are **truncated**. Non-numeric / NaN / inf / out-of-range → `TransformationError`. | +| `to_epoch` | `[s]` or `[s, fmt]` | String → epoch seconds (int). Default accepts the same fixed ISO form. Parse failure → `TransformationError`. | + +Shared format whitelist (locale-free, deterministic): `%Y %m %d %H %M %S %j %z %%` +plus literal text. Any other directive (including `%a %b %c %x %X %p %Z`) → +`TransformationError`. + +#### Collections and numerics + +| name | args | notes | +|---|---|---| +| `length` | unary | string / array / object → int; other → `TransformationError` | +| `flatten` | unary | one level; non-array input or non-array element → `TransformationError` | +| `sum` | unary | array of numbers; `sum([]) == 0`; booleans and non-numeric elements → `TransformationError` | +| `min` `max` | `[array]` or `[array, default]` | empty without `default` → `TransformationError` | +| `sorted` | unary | homogeneous scalars only; mixed types → `TransformationError` | +| `unique` | unary | first-occurrence order; dict/list elements → `TransformationError` | +| `abs` `floor` `ceil` | unary | numbers only (`math.floor`/`ceil`); non-number → `TransformationError` | +| `round` | unary or `[x, ndigits]` | numbers only | + +#### Encoding, hashing, regex + +| name | args | notes | +|---|---|---| +| `b64encode` | unary | str → UTF-8 → standard-alphabet base64 str | +| `b64decode` | unary | invalid base64 or non-UTF-8 payload → `TransformationError` | +| `uuid5` | `[namespace, name]` | Deterministic UUID; `namespace` is `dns`/`url`/`oid`/`x500` or a UUID string. Random `uuid4` is deliberately absent (determinism). | +| `regex_match` | `[s, pattern]` | On match: array of capture groups (`groups()`; unmatched optionals → `null`); with no groups: `[full match]`. On no match: `null`. Condition use: `bool(regex_match(...))`. Invalid pattern → `TransformationError`. | +| `regex_replace` | `[s, pattern, repl]` | str; invalid pattern → `TransformationError` | + +**Regex dialect** is Python `re` (the same engine on CPython and the Pyodide reference +host). ReDoS exposure is a **host** responsibility (e.g. per-case timeouts); the engine +does not limit pattern complexity. --- @@ -557,14 +777,30 @@ tagged example cases. (no `walk`/`_walk`-style per-node doubling), so self-`include`ing templates reach depths well past `G_encode` (41) before the host stack overflows; over-depth surfaces as the `include` depth-limit `TransformationError`, never a raw `RecursionError` - (§4 "Recursion budget", Roadmap R-32). + (§4.6 "Recursion budget", Roadmap R-32). --- ## 11. Engine data flow (reference example) -Relocated to the Language Reference: [`transon/resources/LANGUAGE.md`](../transon/resources/LANGUAGE.md), "Composition -patterns" (the `zip` + `map` worked end-to-end flow). +Template: + +```json +{ + "$": "chain", + "funcs": [ + {"$": "zip", "items": [{"$": "attr", "name": "keys"}, + {"$": "attr", "name": "values"}]}, + {"$": "map", "key": {"$": "attr", "name": 0}, + "value": {"$": "attr", "name": 1}} + ] +} +``` + +Input `{"keys": ["a","b"], "values": [1,2]}` → root context `this=input` → +`zip` produces `[["a",1], ["b",2]]` → chain derives context with that as `this` → +`map` iterates, each pair becomes `this`/`item` in a sub-context → `attr` with numeric +names indexes the tuple → output `{"a": 1, "b": 2}`. --- diff --git a/docs/proposals/0008-language-reference-export.md b/docs/proposals/0008-language-reference-export.md index 4005c01..9db259e 100644 --- a/docs/proposals/0008-language-reference-export.md +++ b/docs/proposals/0008-language-reference-export.md @@ -7,7 +7,8 @@ registration docs, cross-cutting semantics in `LANGUAGE.md`); no per-entity sections in `LANGUAGE.md`; sequencing decision; single-copy refinement — the packaged `transon/resources/LANGUAGE.md` is canonical, `docs/LANGUAGE.md` is a pointer (see - Deliverable 2) + Deliverable 2); spec-completeness decision — `SPECIFICATION.md` deliberately retains a full + duplicated copy of the semantics (see the sourcing rule) - **Roadmap:** R-34 (Language Reference document), R-35 (package the reference), R-36 (`get_language_reference()` export) — `accepted`, rows in `docs/ROADMAP.md`; docs-site counterpart is D-20 in `docs/DOCS_SITE_ROADMAP.md`. (R-33 is held by [RFC 0007](0007-builtin-function-library.md).) - **Type:** New documentation artifact + packaging + a new read-only export API (`get_language_reference()`) — additive; no change to existing template semantics or engine API shapes. The **content** of `get_all_docs()['doc']` shrinks to the embedder-facing narrative as part of the consolidation (shape unchanged; the docs export carries no schema version, so the change is coordinated by release note in `CHANGELOG.md`). Symmetrically, per-rule doc **content grows** in both exports (`get_all_docs()` and `get_editor_metadata()['docs']`) as §4's facts fold into the docstrings — also shape-unchanged, doc text is contractually opaque - **Consumers:** `transon-authoring` (authority ladder rung 2; `SKILL.md`, AD-018/NFR-001/NFR-003), `transon-org.github.io` (docs site) @@ -89,11 +90,13 @@ reference**, addressed to template authors (human or agent), containing semantic copy of the language semantics, not by writing a fresh parallel one. One source of truth per fact — two hand-maintained descriptions of `NO_CONTENT` will diverge, and today there are three: -- **`SPECIFICATION.md` §2/§11** — the cross-cutting language content is relocated into - `LANGUAGE.md`; **§4's per-rule facts are relocated into the rule docstrings** (see the - ownership principle), and §4 shrinks to a pointer at the generated reference. The spec keeps - the engine-contract material and links out for semantics. §5's "nothing is hand-maintained - separately" claim is reworded to name `LANGUAGE.md` as the one hand-written artifact. +- **`SPECIFICATION.md` §2/§4/§11** — the language content is **copied** into `LANGUAGE.md` + (cross-cutting) and into the rule docstrings (per-entity facts, which grow richer), and the + spec **retains its own full normative statement** — a deliberate, recorded exception to + move-don't-copy (decision 2026-07-18): the spec's value as a single, complete engine + contract outweighs the drift risk, which is accepted and managed by review discipline + (banners at spec §2/§4 require updating all surfaces in the same change). §5's "nothing is + hand-maintained separately" claim is reworded to name `LANGUAGE.md`. - **The `Transformer` class docstring** — its language sections ("Templates", "How evaluation works", the language half of "What you can do") are relocated. The docstring shrinks to embedder-facing content only: a short orientation paragraph, Python API usage, "Extending", and @@ -104,8 +107,10 @@ fact — two hand-maintained descriptions of `NO_CONTENT` will diverge, and toda landing from README at build time (docs-site work, see Sequencing), not from the docstring. **Ownership principle — one owner per altitude.** Every fact has exactly one home, chosen by -its altitude; every channel composes these sources (joined by entity name, the same name-join -the corpus normalization R-31 established) and none restates another: +its altitude; every **consumer channel** composes these sources (joined by entity name, the +same name-join the corpus normalization R-31 established) and none restates another. +(`SPECIFICATION.md` is the recorded exception — a contributor document, not a consumer +channel, that deliberately restates the semantics in full; see the sourcing rule.) - **Structure** — parameter names, required-ness, modes/variants, dynamic-vs-constant kinds, containers, operator/function types: the registration metadata, exported as the @@ -226,10 +231,12 @@ explicitly. - **No behavior change**: no template semantics move; this is documentation + packaging + a read-only accessor. -- **No second source of truth**: content is moved out of `SPECIFICATION.md` (§2/§11 to the - reference, §4 to the docstrings), the `Transformer` docstring, and `README.md` — never - duplicated; the section-pin test guards the reference's shape, the no-`TBD` gate guards the - registration docs. +- **No second source of truth among consumer surfaces**: the `Transformer` docstring and + `README.md` content is moved, never duplicated, and every export composes one owner per + fact; the section-pin test guards the reference's shape, the no-`TBD` gate guards the + registration docs. The one sanctioned duplication is `SPECIFICATION.md` (decision + 2026-07-18): it keeps a full copy of the language semantics so the engine contract stays a + single, complete document, aligned by review discipline. - **No generated prose**: rule *examples* stay generated and normalized in the corpus (§5, R-31); `LANGUAGE.md` references behavior, it does not re-serialize examples. (How the docs site *renders* `LANGUAGE.md` is docs-site work — D-20, see Sequencing — but rendering From 3e45c154a416e3922aec4f1e077c091766188980 Mon Sep 17 00:00:00 2001 From: Eugene Chernyshov Date: Sat, 18 Jul 2026 02:56:00 +0300 Subject: [PATCH 7/8] docs: markdown-safe cross-reference in the class docstring The doc field is rendered as markdown by the docs site; a Sphinx-style :class: role would show literally. Co-Authored-By: Claude Fable 5 --- transon/transformers.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/transon/transformers.py b/transon/transformers.py index 809d4d4..769d4cd 100644 --- a/transon/transformers.py +++ b/transon/transformers.py @@ -284,7 +284,7 @@ class Transformer: - **`file_writer`** — callback the `file` rule writes through. - **`template_loader`** — callback the `include` rule loads sub-templates through; it receives an `IncludeContext` and constructs the sub-transformer - (see :class:`IncludeContext`). + (see the `IncludeContext` docstring). - **`max_include_depth`** — nested-`include` depth limit (default 50). `transform(data, no_content=None, *, copy_output=False)` returns the output: From c728598bef0a62db109173c01e3f1ec767326349 Mon Sep 17 00:00:00 2001 From: Eugene Chernyshov Date: Sat, 18 Jul 2026 03:00:34 +0300 Subject: [PATCH 8/8] docs: define Implemented lifecycle state for merged-but-unreleased RFCs The lifecycle table said Implemented means shipped in a named release, which contradicted RFC 0008 being Implemented with roadmap items done but no tag yet (the roadmap term mapping requires Implemented <-> done). Implemented now means implementation merged; the Shipped column names the version once the release is tagged. Co-Authored-By: Claude Fable 5 --- docs/proposals/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/proposals/README.md b/docs/proposals/README.md index f471640..fb29251 100644 --- a/docs/proposals/README.md +++ b/docs/proposals/README.md @@ -27,7 +27,7 @@ is the *authoritative* record of each work item's status via its **R-number**. |---|---|---| | **Proposed** | Open for a decision; no code yet (or scope not finalized). | `needs-decision` | | **Accepted** | Decision made; awaiting implementation. | `accepted` | -| **Implemented** | Shipped in a release (the `Status` line names the version). | `done` | +| **Implemented** | Implementation merged (roadmap items `done`). The `Shipped` column names the release version once tagged — `—` until then. | `done` | | **Rejected** | Decided against; kept for the record. | `rejected` | | **Deferred** | Postponed pending a trigger stated in the RFC. | — | | **Superseded** | Replaced by a later RFC (named in `Superseded by`). | — |