docs: ADR-0002 + config guide for preset-based semantic explosion (#17)

README.md

@@ -45,18 +45,25 @@ Split a big document into a directory:
 $ yamlx split dashboard.yml --out-dir dashboard/
 $ tree dashboard/
 dashboard/
-├── root.yml                     # entry document; the rest hang off !include tags
+├── root.yml                      # entry document; the rest hang off !include tags
 ├── definition.yml
 └── definition/
     ├── calculated-fields.yml
     ├── filter-groups.yml
     ├── sheets.yml
     └── sheets/
-        ├── kpi-overview.yml     # one file per list element, named by its identity field
-        ├── trading-summary.yml
-        └── risk-rating.yml
+        ├── kpi-overview.yml      # one folder + details file per sheet
+        └── kpi-overview/
+            ├── visuals.yml       # one file per subcomponent category
+            ├── filtercontrols.yml
+            └── layouts.yml
 ```
 
+With the default `quicksight` preset, each dashboard/analysis sheet becomes a
+folder and each subcomponent category its own file, and a dataset's custom SQL is
+lifted into a readable `.sql` sidecar — see
+**[Configuring yamlx for different YAML types](docs/guides/configuring-for-yaml-types.md)**.
+
 Reassemble it (to stdout, or `--out` a file):
 
 ```console

docs/adr/0002-preset-semantic-explosion.md

@@ -0,0 +1,133 @@
+# ADR-0002: Preset-based semantic explosion
+
+- Status: Accepted
+- Date: 2026-07-12
+- Deciders: NickMoignard
+- Supersedes parts of [ADR-0001](0001-split-join-architecture.md) §4 (the `profile`
+  abstraction).
+
+## Context
+
+ADR-0001 splits a document **by size and depth** alone. That produces a
+structurally-correct tree but not always a *meaningful* one. On real QuickSight
+assets it fell short in two concrete ways:
+
+- A dashboard/analysis **sheet** is one array element. At the default depth it stays
+  a single large file (one observed sheet was ~1,000 lines) with all its visuals,
+  filter controls, and layouts inline — the opposite of a reviewable diff.
+- A dataset's **SQL** lives at `DataSet.PhysicalTableMap.<uuid>.CustomSql.SqlQuery`
+  as one `\r\n`-escaped double-quoted scalar. Size/depth splitting can never lift it
+  out (it is a scalar, not a subtree), so it stays unreadable.
+
+We want the `quicksight` preset to explode these *semantically* — each sheet a
+folder, each subcomponent category its own file, SQL in a readable `.sql` file —
+without weakening the prime invariant `join(split(x)) ≡ x`.
+
+## Decision
+
+### 1. `preset` supersedes `profile`
+
+`internal/profile` (identity keys only) becomes `internal/preset`. A `Preset`
+bundles a name, split-option **defaults** (`MaxLines`, `MaxDepth`), identity keys,
+and a list of **rules**. Presets remain built-in Go values; a config file and flags
+select and tune them (decision 4). We deliberately did **not** make rules
+user-definable in config yet: the rule surface is small and easy to author
+incorrectly (a bad rule can defeat the round-trip), so it stays in code where it is
+tested. The schema is designed so it *could* be exposed later.
+
+### 2. Path-anchored rule engine
+
+The split walk threads the **path** from the document root to each node (mapping
+keys, and a marker for sequence elements). Before the size/depth gate, it consults
+the preset's rules, matched segment-for-segment against that path: a literal key,
+`*` (any map key), or `[]` (any sequence element). Two actions:
+
+- **ForceExtract** — extract a complex node regardless of size or depth. Optional
+  **StopRecurse** writes it whole (one file, no descent), so a subcomponent category
+  becomes a single file rather than a file per element.
+- **Sidecar** — lift a *scalar's* text into a standalone non-YAML file (decision 3).
+
+Rules are **root-anchored**, so they self-scope: a `Definition.Sheets[]…` rule only
+matches a dashboard/analysis, a `DataSet.…` rule only a dataset. No explicit
+asset-type detection is needed, and the preset is inert (beyond naming) on unrelated
+documents.
+
+The depth gate moved from the walk's early-return into the per-child check, so a
+forced node deeper than `--max-depth` still extracts (a sheet category sits at depth
+4, under the default depth 3) while unforced deep nodes stay inline exactly as
+before. The recursion still self-limits: a subtree is only reached by having
+extracted its parent.
+
+The `quicksight` preset forces `Definition`, `Definition.Sheets`, and each sheet
+element open; force-extracts the six sheet categories (`Visuals`, `FilterControls`,
+`ParameterControls`, `Layouts`, `SheetControlLayouts`, `TextBoxes`) whole; and forces
+the `DataSet.PhysicalTableMap.*.CustomSql` chain open so the walk reaches the SQL.
+
+### 3. Sidecar extraction with a local fidelity guard
+
+A `Sidecar` rule writes a scalar's **decoded** text to a file with the rule's
+extension (e.g. `.sql`), replacing it with an `!include`. On `join`, an include whose
+extension is not `.yml`/`.yaml` is read back as raw bytes and spliced in as a plain
+string scalar. `split.IsRawInclude` is the single source of truth for that
+raw-vs-YAML decision, used by both split (which rejects a YAML-extension sidecar) and
+join, so they cannot disagree.
+
+Extracting a scalar risks the invariant, because `join` must reconstruct the scalar's
+exact quoting/style and `verify` compares re-rendered forms. So extraction is
+**guarded**: before committing a sidecar, `canReconstruct` builds the exact node
+`join` will produce (`yamltree.RawScalar` — a default-style `!!str`) and extracts
+**only when it re-renders identically** to the original. Otherwise the scalar stays
+inline. The guard is local, per-scalar, with no global fallback — the invariant is
+never weakened; some scalars simply stay inline.
+
+The guard compares *standalone* renders but extraction happens *nested*. This is
+sound because of two `gopkg.in/yaml.v3` properties, and **both must hold**: scalar
+style selection is content-only (not indentation-dependent), and line-wrapping is
+disabled (`yamltree.Save` never sets a width). Adding `SetWidth` would silently break
+the guard's soundness — a long scalar could wrap differently at depth while matching
+standalone. This is documented at `canReconstruct` and must not regress.
+
+### 4. Config selects and tunes; flags win
+
+The config file gains a `split:` section (`preset`, `max_lines`, `max_depth`,
+`id_keys`) with matching `YAMLX_SPLIT_*` environment variables. Split options resolve
+in the order **flags > environment > config file > preset > engine defaults**. Rules
+are intrinsic to the preset — not tunable by config or flags — except `--no-sidecar`,
+which drops a preset's sidecar rules for a structural-only split. The flag is
+`--preset`; the old `--profile` remains a hidden, deprecated alias.
+
+### 5. `verify` exercises the real preset
+
+`verify` now splits with the `quicksight` preset (rules and sidecars included)
+instead of the bare engine defaults, so it gates sidecar fidelity too, not just the
+size/depth engine. The rules are root-anchored, so this is harmless on non-QuickSight
+documents.
+
+## Package layout delta
+
+- `internal/preset` replaces `internal/profile`: `Preset` (defaults + IDKeys +
+  `Rules`), `Rule`/`Action`/`Step`, and the built-in `QuickSight`/`Generic` presets.
+- `internal/split` gains path threading, `ForceExtract`/`Sidecar` handling, the
+  `canReconstruct` guard, and `IsRawInclude`.
+- `internal/yamltree` gains `SaveRaw` and `RawScalar`.
+- `internal/config` gains the `split:` section.
+
+## Consequences
+
+- The `quicksight` preset now yields a readable tree: a folder per sheet, one file
+  per subcomponent category, and SQL in `.sql` files — all still lossless.
+- Byte-exact round-trip is preserved by construction: forced extraction reuses the
+  existing `!include`/anchor machinery, and sidecars only extract when provably
+  reversible.
+- Domain knowledge stays confined to the preset; the engine remains generic and
+  evaluates whatever rules a preset supplies.
+- A latent coupling exists between the sidecar guard and yaml.v3's disabled wrapping
+  (documented; guarded by the `canReconstruct` comment).
+
+## Open questions (deferred, not blocking)
+
+- **User-definable rules** in the config file (path → action/extension), rather than
+  built-in Go presets only.
+- **More sidecars**: calculated-field `Expression` strings and TextBox/visual-title
+  rich-text markup to their own files. SQL only, for now.
+- **Nested identity paths** (e.g. `metadata.name`) for naming — still direct-key only.

docs/guides/configuring-for-yaml-types.md

@@ -1,18 +1,21 @@
 # Configuring yamlx for different YAML types
 
-`yamlx` has no domain knowledge baked into its engine. A handful of `split` flags
-tell it **where** to cut files and **how** to name them, and the right settings
-depend on the shape of your document. This guide explains the flags, how to reason
-about them, and how to configure `yamlx` for several common YAML types.
+`yamlx`'s **engine** has no domain knowledge — a handful of `split` flags tell it
+**where** to cut files and **how** to name them, and the right settings depend on
+the shape of your document. Domain knowledge lives in **presets**, which layer
+semantic rules on top of the generic size/depth split (see the QuickSight worked
+example). This guide explains the flags, how to reason about them, and how to
+configure `yamlx` for several common YAML types.
 
-There are only four flags to learn:
+The core tuning flags:
 
 | Flag | Controls | Default |
 | --- | --- | --- |
 | `--max-lines` | The size threshold for extracting a value into its own file. | `40` |
 | `--max-depth` | How deep extraction reaches before values stay inline. | `3` |
-| `--preset` | A named set of split defaults (thresholds + identity keys) for a document shape. | `quicksight` |
+| `--preset` | A named set of split defaults (thresholds, identity keys, and semantic rules) for a document shape. | `quicksight` |
 | `--id-key` | Explicit identity keys, in precedence order (overrides the preset). | — |
+| `--no-sidecar` | Disable a preset's sidecar extraction (keep, e.g., SQL inline instead of writing `.sql` files). | off |
 
 Whatever you choose, the round-trip is always safe to check:
 
@@ -43,10 +46,11 @@ Think of the two flags as answering two questions:
   to break the document into more, smaller files; raise it to keep more inline. A
   value of `0` extracts every list and mapping regardless of size.
 - **`--max-depth` — "how far down should the split reach?"** This is the main lever
-  for controlling file count. At the default `3`, a QuickSight dashboard yields one
-  file per sheet with each sheet's visuals kept *inline*; raise it to `4`+ to also
-  break every visual into its own file (dozens more files); lower it to `1` to only
-  split the top-level values.
+  for controlling file count for *size-based* splitting: raise it to reach deeper,
+  lower it to `1` to only split the top-level values. A **preset's rules can force
+  structure beyond this** — the `quicksight` preset always folders sheets and their
+  subcomponents regardless of `--max-depth` (see the worked example) — but for
+  everything the rules don't touch, `--max-depth` is the reach.
 
 A good rule of thumb: start with the defaults, run `yamlx split`, look at the
 tree, and adjust. If you see one enormous file you wanted broken up, raise
@@ -83,23 +87,40 @@ $ yamlx split app.yml --out-dir app/ --id-key name --id-key id
 
 ## Worked examples
 
-### AWS QuickSight dashboard (the default)
+### AWS QuickSight (the default)
 
-QuickSight `describe-dashboard-definition` output is what `yamlx` is tuned for out
-of the box:
+QuickSight is what `yamlx` is tuned for out of the box, and the `quicksight` preset
+does more than size/depth splitting: it applies **semantic rules** that give each
+asset shape a stable, readable layout. The rules are anchored to QuickSight's own
+paths, so they only fire on the shape they match and are inert elsewhere.
+
+**Dashboards and analyses** — each sheet becomes its own folder with a details file,
+and each subcomponent category (`Visuals`, `FilterControls`, `ParameterControls`,
+`Layouts`, `SheetControlLayouts`, `TextBoxes`) is written whole to its own file,
+regardless of size:
 
 ```console
-$ yamlx split dashboard.yml --out-dir dashboard/
+$ yamlx split analysis.yml --out-dir analysis/
+# => analysis/definition/sheets/customer-analysis.yml
+#    analysis/definition/sheets/customer-analysis/visuals.yml
+#    analysis/definition/sheets/customer-analysis/filtercontrols.yml
+#    ...
 ```
 
-The defaults (`--max-lines 40 --max-depth 3 --preset quicksight`) produce one
-readable file per sheet — `kpi-overview.yml`, `trading-summary.yml`, … — with each
-sheet's visuals inline. To break out every visual too, go deeper:
+**Datasets** — the custom SQL buried in `PhysicalTableMap.<uuid>.CustomSql.SqlQuery`
+is lifted into a readable `.sql` sidecar file (decoded, so real newlines instead of
+escaped `\r\n`):
 
 ```console
-$ yamlx split dashboard.yml --out-dir dashboard/ --max-depth 5
+$ yamlx split dataset.yml --out-dir dataset/
+# => dataset/.../customsql/sqlquery.sql   (the SQL, readable)
+#    dataset/.../customsql.yml            (SqlQuery: !include customsql/sqlquery.sql)
 ```
 
+Sidecar extraction is **guarded**: `yamlx` only lifts a scalar into a `.sql` file
+when it can prove the value round-trips back exactly, so the invariant always holds.
+To keep everything as YAML (no `.sql` files), pass `--no-sidecar`.
+
 ### A config with a list of named things
 
 Many documents contain a top-level list whose elements each have a `name`: