yamlx

Break apart a large, deeply-nested YAML document into a tidy tree of small files — then losslessly reassemble it.

yamlx takes a monster YAML file (think a 16,000-line AWS QuickSight dashboard definition) and splits it into many small files nested in directories, linked by a custom !include tag. You review and edit those files with sane git diffs, then join them back into the single document your tooling expects. A verify command proves the round-trip is lossless.

describe-dashboard ─▶ yamlx split ─▶ edit files, review diffs in git
                                            │
   redeploy ◀── yamlx join ◀────────────────┘   (lossless)

The split form is a source format for humans and git — it is not valid input to tools that don't understand !include. Always yamlx join before feeding the result back to something else.

Install

# Go toolchain
go install github.com/NickMoignard/yamlx/cmd/yamlx@latest

# Homebrew
brew install NickMoignard/tap/yamlx

Or download a prebuilt archive for your platform from the latest release.

Quick start

Split a big document into a directory:

$ yamlx split dashboard.yml --out-dir dashboard/
$ tree dashboard/
dashboard/
├── root.yml                      # entry document; the rest hang off !include tags
├── definition.yml
└── definition/
    ├── calculated-fields.yml
    ├── filter-groups.yml
    ├── sheets.yml
    └── sheets/
        ├── kpi-overview.yml      # one folder + details file per sheet
        └── kpi-overview/
            ├── visuals.yml       # one file per subcomponent category
            ├── filtercontrols.yml
            └── layouts.yml

With the default quicksight preset, each dashboard/analysis sheet becomes a folder and each subcomponent category its own file, and a dataset's custom SQL is lifted into a readable .sql sidecar — see Configuring yamlx for different YAML types.

Reassemble it (to stdout, or --out a file):

$ yamlx join dashboard/ --out rebuilt.yml

Prove the round-trip loses nothing:

$ yamlx verify dashboard.yml
ok	dashboard.yml

Commands

Command	What it does
yamlx split <input.yml|-> --out-dir <dir>	Explode a document into a tree of !include-linked files.
yamlx join <dir|root.yml> [--out <file>]	Resolve the !include tags back into one document.
yamlx verify <input.yml>	Split then join in a temp dir and confirm the result matches.

split reads its document from the <input.yml> argument, or from stdin when the argument is - (or omitted while piping), so it composes with other tools:

# AWS CLI defaults to JSON — ask for YAML, then split the stream straight in.
$ aws quicksight describe-data-set \
    --aws-account-id "$ACCOUNT_ID" --region "$REGION" \
    --data-set-id "$DATASET_ID" --output yaml \
  | yamlx split - --out-dir dataset/

Tuning where it splits and how files are named

Two knobs control where files are cut, two control how list elements are named:

Flag	Default	Effect
--max-lines	40	Only extract a value whose residual rendered form exceeds this many lines.
--max-depth	3	Stop extracting past this nesting depth; deeper values stay inline.
--preset	quicksight	Named set of defaults (thresholds + identity keys) for a document shape (quicksight or generic).
--id-key	—	Mapping keys (in precedence order) to name list elements by; overrides the preset.

"Residual" means the value's size after its own large children are extracted, so a node that just wraps one big child (a single-key PhysicalTableMap relaying its custom SQL, say) never becomes a one-line file — it stays inline while its big child is lifted out. Extracted children flatten up to the nearest ancestor that has its own file, so you get dataset/customsql.yml rather than a dataset/physicaltablemap/<uuid>/… chain of wrapper files and folders.

The quicksight preset names list elements after Name, then SheetId, then VisualId. On non-QuickSight documents those keys simply don't match, so elements fall back to a zero-padded position (001.yml, 002.yml). For other document shapes, pick your own with --id-key — see Configuring yamlx for different YAML types.

You can set these defaults without repeating flags: a split: block in the config file (.yamlx.yaml) or YAMLX_SPLIT_* environment variables supply preset, max_lines, max_depth, and id_keys, resolved as flags > env > config > preset > built-in defaults. (--profile is still accepted as a hidden, deprecated alias for --preset.)

Fidelity

yamlx is built around one invariant: join(split(x)) equals x.

• Comments, key order, and quoting are preserved — including \r\n escape sequences inside quoted scalars (as in QuickSight Expression fields).
• Cross-file YAML aliases are refused, not silently broken. An anchor and its alias can't live in different files, so split fails with a clear error naming the anchor rather than emitting output that won't round-trip.
• join rejects includes that escape the output directory (../, absolute paths) and detects include cycles.

Exit codes

Code	Meaning
0	Success.
1	The input was processed but is invalid (malformed YAML, or a verify mismatch).
2	Usage error (bad flags, missing file).
>2	Unexpected internal error.

Use -o json for machine-readable output and --help on any command for the full flag set.

Build from source

$ git clone https://github.com/NickMoignard/yamlx
$ cd yamlx
$ go build ./cmd/yamlx
$ go test ./...

Contributing

Issues and pull requests are welcome — see CONTRIBUTING.md and the Code of Conduct. To report a security issue, see SECURITY.md.

License

MIT © NickMoignard