diff --git a/README.md b/README.md index e61ee6a..c424111 100644 --- a/README.md +++ b/README.md @@ -45,18 +45,25 @@ Split a big document into a directory: $ yamlx split dashboard.yml --out-dir dashboard/ $ tree dashboard/ dashboard/ -├── root.yml # entry document; the rest hang off !include tags +├── root.yml # entry document; the rest hang off !include tags ├── definition.yml └── definition/ ├── calculated-fields.yml ├── filter-groups.yml ├── sheets.yml └── sheets/ - ├── kpi-overview.yml # one file per list element, named by its identity field - ├── trading-summary.yml - └── risk-rating.yml + ├── kpi-overview.yml # one folder + details file per sheet + └── kpi-overview/ + ├── visuals.yml # one file per subcomponent category + ├── filtercontrols.yml + └── layouts.yml ``` +With the default `quicksight` preset, each dashboard/analysis sheet becomes a +folder and each subcomponent category its own file, and a dataset's custom SQL is +lifted into a readable `.sql` sidecar — see +**[Configuring yamlx for different YAML types](docs/guides/configuring-for-yaml-types.md)**. + Reassemble it (to stdout, or `--out` a file): ```console diff --git a/docs/adr/0002-preset-semantic-explosion.md b/docs/adr/0002-preset-semantic-explosion.md new file mode 100644 index 0000000..0ef4405 --- /dev/null +++ b/docs/adr/0002-preset-semantic-explosion.md @@ -0,0 +1,133 @@ +# ADR-0002: Preset-based semantic explosion + +- Status: Accepted +- Date: 2026-07-12 +- Deciders: NickMoignard +- Supersedes parts of [ADR-0001](0001-split-join-architecture.md) §4 (the `profile` + abstraction). + +## Context + +ADR-0001 splits a document **by size and depth** alone. That produces a +structurally-correct tree but not always a *meaningful* one. On real QuickSight +assets it fell short in two concrete ways: + +- A dashboard/analysis **sheet** is one array element. At the default depth it stays + a single large file (one observed sheet was ~1,000 lines) with all its visuals, + filter controls, and layouts inline — the opposite of a reviewable diff. +- A dataset's **SQL** lives at `DataSet.PhysicalTableMap..CustomSql.SqlQuery` + as one `\r\n`-escaped double-quoted scalar. Size/depth splitting can never lift it + out (it is a scalar, not a subtree), so it stays unreadable. + +We want the `quicksight` preset to explode these *semantically* — each sheet a +folder, each subcomponent category its own file, SQL in a readable `.sql` file — +without weakening the prime invariant `join(split(x)) ≡ x`. + +## Decision + +### 1. `preset` supersedes `profile` + +`internal/profile` (identity keys only) becomes `internal/preset`. A `Preset` +bundles a name, split-option **defaults** (`MaxLines`, `MaxDepth`), identity keys, +and a list of **rules**. Presets remain built-in Go values; a config file and flags +select and tune them (decision 4). We deliberately did **not** make rules +user-definable in config yet: the rule surface is small and easy to author +incorrectly (a bad rule can defeat the round-trip), so it stays in code where it is +tested. The schema is designed so it *could* be exposed later. + +### 2. Path-anchored rule engine + +The split walk threads the **path** from the document root to each node (mapping +keys, and a marker for sequence elements). Before the size/depth gate, it consults +the preset's rules, matched segment-for-segment against that path: a literal key, +`*` (any map key), or `[]` (any sequence element). Two actions: + +- **ForceExtract** — extract a complex node regardless of size or depth. Optional + **StopRecurse** writes it whole (one file, no descent), so a subcomponent category + becomes a single file rather than a file per element. +- **Sidecar** — lift a *scalar's* text into a standalone non-YAML file (decision 3). + +Rules are **root-anchored**, so they self-scope: a `Definition.Sheets[]…` rule only +matches a dashboard/analysis, a `DataSet.…` rule only a dataset. No explicit +asset-type detection is needed, and the preset is inert (beyond naming) on unrelated +documents. + +The depth gate moved from the walk's early-return into the per-child check, so a +forced node deeper than `--max-depth` still extracts (a sheet category sits at depth +4, under the default depth 3) while unforced deep nodes stay inline exactly as +before. The recursion still self-limits: a subtree is only reached by having +extracted its parent. + +The `quicksight` preset forces `Definition`, `Definition.Sheets`, and each sheet +element open; force-extracts the six sheet categories (`Visuals`, `FilterControls`, +`ParameterControls`, `Layouts`, `SheetControlLayouts`, `TextBoxes`) whole; and forces +the `DataSet.PhysicalTableMap.*.CustomSql` chain open so the walk reaches the SQL. + +### 3. Sidecar extraction with a local fidelity guard + +A `Sidecar` rule writes a scalar's **decoded** text to a file with the rule's +extension (e.g. `.sql`), replacing it with an `!include`. On `join`, an include whose +extension is not `.yml`/`.yaml` is read back as raw bytes and spliced in as a plain +string scalar. `split.IsRawInclude` is the single source of truth for that +raw-vs-YAML decision, used by both split (which rejects a YAML-extension sidecar) and +join, so they cannot disagree. + +Extracting a scalar risks the invariant, because `join` must reconstruct the scalar's +exact quoting/style and `verify` compares re-rendered forms. So extraction is +**guarded**: before committing a sidecar, `canReconstruct` builds the exact node +`join` will produce (`yamltree.RawScalar` — a default-style `!!str`) and extracts +**only when it re-renders identically** to the original. Otherwise the scalar stays +inline. The guard is local, per-scalar, with no global fallback — the invariant is +never weakened; some scalars simply stay inline. + +The guard compares *standalone* renders but extraction happens *nested*. This is +sound because of two `gopkg.in/yaml.v3` properties, and **both must hold**: scalar +style selection is content-only (not indentation-dependent), and line-wrapping is +disabled (`yamltree.Save` never sets a width). Adding `SetWidth` would silently break +the guard's soundness — a long scalar could wrap differently at depth while matching +standalone. This is documented at `canReconstruct` and must not regress. + +### 4. Config selects and tunes; flags win + +The config file gains a `split:` section (`preset`, `max_lines`, `max_depth`, +`id_keys`) with matching `YAMLX_SPLIT_*` environment variables. Split options resolve +in the order **flags > environment > config file > preset > engine defaults**. Rules +are intrinsic to the preset — not tunable by config or flags — except `--no-sidecar`, +which drops a preset's sidecar rules for a structural-only split. The flag is +`--preset`; the old `--profile` remains a hidden, deprecated alias. + +### 5. `verify` exercises the real preset + +`verify` now splits with the `quicksight` preset (rules and sidecars included) +instead of the bare engine defaults, so it gates sidecar fidelity too, not just the +size/depth engine. The rules are root-anchored, so this is harmless on non-QuickSight +documents. + +## Package layout delta + +- `internal/preset` replaces `internal/profile`: `Preset` (defaults + IDKeys + + `Rules`), `Rule`/`Action`/`Step`, and the built-in `QuickSight`/`Generic` presets. +- `internal/split` gains path threading, `ForceExtract`/`Sidecar` handling, the + `canReconstruct` guard, and `IsRawInclude`. +- `internal/yamltree` gains `SaveRaw` and `RawScalar`. +- `internal/config` gains the `split:` section. + +## Consequences + +- The `quicksight` preset now yields a readable tree: a folder per sheet, one file + per subcomponent category, and SQL in `.sql` files — all still lossless. +- Byte-exact round-trip is preserved by construction: forced extraction reuses the + existing `!include`/anchor machinery, and sidecars only extract when provably + reversible. +- Domain knowledge stays confined to the preset; the engine remains generic and + evaluates whatever rules a preset supplies. +- A latent coupling exists between the sidecar guard and yaml.v3's disabled wrapping + (documented; guarded by the `canReconstruct` comment). + +## Open questions (deferred, not blocking) + +- **User-definable rules** in the config file (path → action/extension), rather than + built-in Go presets only. +- **More sidecars**: calculated-field `Expression` strings and TextBox/visual-title + rich-text markup to their own files. SQL only, for now. +- **Nested identity paths** (e.g. `metadata.name`) for naming — still direct-key only. diff --git a/docs/guides/configuring-for-yaml-types.md b/docs/guides/configuring-for-yaml-types.md index 47e6b9d..1c7653c 100644 --- a/docs/guides/configuring-for-yaml-types.md +++ b/docs/guides/configuring-for-yaml-types.md @@ -1,18 +1,21 @@ # Configuring yamlx for different YAML types -`yamlx` has no domain knowledge baked into its engine. A handful of `split` flags -tell it **where** to cut files and **how** to name them, and the right settings -depend on the shape of your document. This guide explains the flags, how to reason -about them, and how to configure `yamlx` for several common YAML types. +`yamlx`'s **engine** has no domain knowledge — a handful of `split` flags tell it +**where** to cut files and **how** to name them, and the right settings depend on +the shape of your document. Domain knowledge lives in **presets**, which layer +semantic rules on top of the generic size/depth split (see the QuickSight worked +example). This guide explains the flags, how to reason about them, and how to +configure `yamlx` for several common YAML types. -There are only four flags to learn: +The core tuning flags: | Flag | Controls | Default | | --- | --- | --- | | `--max-lines` | The size threshold for extracting a value into its own file. | `40` | | `--max-depth` | How deep extraction reaches before values stay inline. | `3` | -| `--preset` | A named set of split defaults (thresholds + identity keys) for a document shape. | `quicksight` | +| `--preset` | A named set of split defaults (thresholds, identity keys, and semantic rules) for a document shape. | `quicksight` | | `--id-key` | Explicit identity keys, in precedence order (overrides the preset). | — | +| `--no-sidecar` | Disable a preset's sidecar extraction (keep, e.g., SQL inline instead of writing `.sql` files). | off | Whatever you choose, the round-trip is always safe to check: @@ -43,10 +46,11 @@ Think of the two flags as answering two questions: to break the document into more, smaller files; raise it to keep more inline. A value of `0` extracts every list and mapping regardless of size. - **`--max-depth` — "how far down should the split reach?"** This is the main lever - for controlling file count. At the default `3`, a QuickSight dashboard yields one - file per sheet with each sheet's visuals kept *inline*; raise it to `4`+ to also - break every visual into its own file (dozens more files); lower it to `1` to only - split the top-level values. + for controlling file count for *size-based* splitting: raise it to reach deeper, + lower it to `1` to only split the top-level values. A **preset's rules can force + structure beyond this** — the `quicksight` preset always folders sheets and their + subcomponents regardless of `--max-depth` (see the worked example) — but for + everything the rules don't touch, `--max-depth` is the reach. A good rule of thumb: start with the defaults, run `yamlx split`, look at the tree, and adjust. If you see one enormous file you wanted broken up, raise @@ -83,23 +87,40 @@ $ yamlx split app.yml --out-dir app/ --id-key name --id-key id ## Worked examples -### AWS QuickSight dashboard (the default) +### AWS QuickSight (the default) -QuickSight `describe-dashboard-definition` output is what `yamlx` is tuned for out -of the box: +QuickSight is what `yamlx` is tuned for out of the box, and the `quicksight` preset +does more than size/depth splitting: it applies **semantic rules** that give each +asset shape a stable, readable layout. The rules are anchored to QuickSight's own +paths, so they only fire on the shape they match and are inert elsewhere. + +**Dashboards and analyses** — each sheet becomes its own folder with a details file, +and each subcomponent category (`Visuals`, `FilterControls`, `ParameterControls`, +`Layouts`, `SheetControlLayouts`, `TextBoxes`) is written whole to its own file, +regardless of size: ```console -$ yamlx split dashboard.yml --out-dir dashboard/ +$ yamlx split analysis.yml --out-dir analysis/ +# => analysis/definition/sheets/customer-analysis.yml +# analysis/definition/sheets/customer-analysis/visuals.yml +# analysis/definition/sheets/customer-analysis/filtercontrols.yml +# ... ``` -The defaults (`--max-lines 40 --max-depth 3 --preset quicksight`) produce one -readable file per sheet — `kpi-overview.yml`, `trading-summary.yml`, … — with each -sheet's visuals inline. To break out every visual too, go deeper: +**Datasets** — the custom SQL buried in `PhysicalTableMap..CustomSql.SqlQuery` +is lifted into a readable `.sql` sidecar file (decoded, so real newlines instead of +escaped `\r\n`): ```console -$ yamlx split dashboard.yml --out-dir dashboard/ --max-depth 5 +$ yamlx split dataset.yml --out-dir dataset/ +# => dataset/.../customsql/sqlquery.sql (the SQL, readable) +# dataset/.../customsql.yml (SqlQuery: !include customsql/sqlquery.sql) ``` +Sidecar extraction is **guarded**: `yamlx` only lifts a scalar into a `.sql` file +when it can prove the value round-trips back exactly, so the invariant always holds. +To keep everything as YAML (no `.sql` files), pass `--no-sidecar`. + ### A config with a list of named things Many documents contain a top-level list whose elements each have a `name`: