diff --git a/CHANGELOG.md b/CHANGELOG.md index c454d71..fa9411d 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,28 @@ All notable changes to `content-model-simulator` are documented here. +## [0.6.0] — 2026-05-10 + +### Added +- **`cms-sim pull-sanity`** — new CLI subcommand that converts a Sanity dataset NDJSON export (produced by `sanity dataset export`) into the same on-disk shape that `cms-sim pull` produces for Contentful spaces: `contentful-space.json` + `schemas/.js` (one per Sanity `_type`) + `data/entries.ndjson`. The output drops into `cms-sim simulate`, `cms-sim to-import`, and `cms-sim diff` unchanged. Flags: `--input` (required), `--output` (default `./sanity-export`), `--default-locale` (default `en-US`), `--include-drafts`, `--include-assets`, `--source`, `--verbose`. Zero new runtime dependencies. Output contract validated end-to-end; field-type inference (Symbol/Text/Integer/Number/Boolean/Date/Object/RichText/Link/Array) shipped. Pipeline warnings against a real Sanity dataset (link-vehicles export, 21 documents) drop from 97 (`FIELD_NOT_IN_DEFINITION` at baseline) to 0. +- **`pullSanity(options)`** — programmatic API behind the CLI, exported from `src/sanity/pull-sanity.ts`. Returns `{ schemas, documents, assets, warnings, output }`. Reuses `parseSanity` / `parseSanityString` from the existing `sanity-reader` module. +- **`inferSchemas(samplesByType, options)`** — pure schema-inference API in `src/sanity/sanity-schema-inference.ts`. Walks document samples per content type, classifies each field's value shape, and emits a `ContentTypeDefinition` plus a `warnings[]` array describing type conflicts and array-collapse fallbacks. Mappings: short strings → `Symbol`, strings >256 chars → `Text`, ISO 8601 → `Date`, integers → `Integer`, floats → `Number`, booleans → `Boolean`, `{_type: reference, _ref}` → `Link Entry`, `{_type: image, asset}` → `Link Asset`, arrays of refs/images/symbols → `Array` / `Array` / `Array`. Mixed-shape arrays collapse to `Object` with a warning. Required flag is set when the field is present and non-null in 100% of samples (configurable via `requiredThreshold`). +- **11 unit tests** for `pullSanity` covering the M1 output contract (three pieces written, `contentful-space.json` shape, custom default locale, one schema per type, Contentful schema shape with inferred fields, camelCase humanization in schema names, entries.ndjson per-line shape, draft/system filtering opt-in via flags, in-memory return shape, empty-input no-write behavior, missing-input error). +- **18 unit tests** for `inferSchemas` covering primitive shape detection (Symbol/Text by string length, Integer/Number split, Boolean, Date for ISO strings), Sanity-specific shapes (`reference`, `image`, arrays of each), disagreement resolution (majority shape wins with warning, mixed arrays collapse to Object), the `required` flag (100% presence → required; missing or null in any sample → not required), and multi-type orchestration (one schema per type, sorted by id, humanized). +- **`mapDocumentRefs(docs)`** — pure transform in `src/sanity/sanity-ref-mapper.ts` that rewrites every `{_type: 'reference', _ref}` shape (including nested in arrays/objects, with or without `_key`/`_weak` modifiers) to the Contentful Entry-link shape `{sys: {type: 'Link', linkType: 'Entry', id}}`. Returns the transformed documents plus a `linkTargets` map (`contentTypeId.fieldId` → Set of target content types) harvested from the actual cross-document refs. Image references (`{_type: 'image', asset: {_ref}}`) are left untouched here — M6 handles asset links. `pullSanity` runs `mapDocumentRefs` before writing `entries.ndjson` so the simulator's link-resolution stage populates `linkedEntryIds` and the model graph draws relationship arrows from data, not just schema validations. +- **`linkContentType` validation inference** in `inferSchemas` — when `linkTargets` is passed, single Link Entry fields get `validations: [{ linkContentType: [...sorted targets] }]` and Array fields get the same on `items.validations`. Orphan refs (target id missing from dataset) don't pollute the validation list. +- **15 unit tests** for `mapRefsToContentful` / `mapDocumentRefs` and **6 unit tests** for the linkContentType inference in `pullSanity` covering single + array refs, aggregation across multiple target types, orphan-ref handling, and end-to-end ref rewriting in `entries.ndjson`. +- **`portableTextToRichText(blocks)`** — converts an array of Sanity Portable Text blocks into a Contentful RichText document (the structured `nodeType` tree, not HTML). Maps block styles (`normal`/`h1..h6`/`blockquote`), groups consecutive `listItem` blocks into `unordered-list` / `ordered-list` with nested `list-item` + `paragraph`, converts span marks (`strong`→`bold`, `em`→`italic`, `underline`, `code`), and wraps spans in `hyperlink` nodes when their mark refs a `link` markDef with `href`. Unknown block styles, marks, annotations, and nested list levels (>1) emit explicit warnings while preserving the underlying text. Custom (non-`block`) PT items are skipped with a warning. Exported from the public API along with `isPortableText`. +- **`portableText: 'html' | 'richtext' | 'raw'`** option on `SanityReadOptions` (default `'html'` for back-compat). `'richtext'` invokes `portableTextToRichText`; `'raw'` leaves PT blocks untouched. `pullSanity` opts into `'richtext'` so its `entries.ndjson` carries Contentful-shaped RichText documents, which (a) lets `to-import` emit valid RichText payloads for top-level PT fields, and (b) makes `inferSchemas` classify those fields as `RichText` instead of `Object`. The default-`html` path keeps `cms-sim simulate --input=*.ndjson` (Sanity NDJSON detected via `isSanityNDJSON`) and existing transforms producing HTML strings as they did before. +- **`RichText` shape detection** in `inferSchemas` — objects with `nodeType === 'document'` and an array `content` are classified as `RichText` fields. Verified end-to-end against the link-vehicles Sanity export: `post.body[en]` now lands as a real RichText document with 15 structured content nodes instead of an HTML blob. +- **18 unit tests** for `portableTextToRichText` covering paragraph / heading / blockquote mapping, decorator marks, hyperlink annotations, list grouping (bullet+number), nested-list flattening warning, unknown-mark/annotation/block-style fallbacks, and non-block skip behavior. **`isPortableText`** has its own 2-test block. +- **Sanity asset linking** in `pullSanity` (`src/sanity/sanity-asset-mapper.ts`). Image references (`{_type: 'image', asset: {_ref}}`, including those nested in arrays/objects) are rewritten to the canonical Contentful Link/Asset sys shape `{sys: {type: 'Link', linkType: 'Asset', id: <_ref>}}` — matching the shape that `pull` (Contentful) emits for asset-field entries. The asset id is preserved verbatim from `_ref`, so it joins the asset metadata written to `/assets/assets.json` (mirroring pull-contentful's `assets/assets.json` shape: `{id, title, fileName, contentType, url, size}`). End-to-end against the link-vehicles dataset: `ASSET_FIELD_NOT_LINKED` warnings dropped from 13 → 0, `linkedAssetIds` is now populated on every post entry (the model graph picks up entry↔asset relationships), and `mainImage` resolves to a Link/Asset sys instead of a raw `{_type: 'image'}` object. Orphan references (asset id missing from the dataset) are still rewritten — the resulting link is dangling and the simulator's link-resolution stage surfaces it. +- **11 unit tests** for `buildAssetUrlMap` / `mapImageRefsToLinks` / `mapDocumentAssets` / `toAssetMetadata` covering single + array refs, deep nesting, orphan refs (still rewritten — dangling, expected), pass-through of primitives and Link Entry sys shapes, no input mutation, and `assets.json` shape (including the extension-derived filename fallback when `originalFilename` is empty). +- **Multi-locale detection + expansion** in `pullSanity` (`src/sanity/sanity-locale-expander.ts`). `detectLocales` scans every document's `data` for objects whose keys all match the locale-code pattern `^[a-z]{2,3}(-[A-Z]{2})?$` (e.g., `en`, `es`, `en-US`, `pt-BR`) and which have no `_type` / `sys` / `nodeType` keys — so Sanity refs, Link sys, and RichText documents are correctly skipped. `expandDocumentsByLocale` then fans each document out to N documents (one per locale), substituting the locale-specific value for every detected localized field and falling back to the base locale when missing. `inferSchemas` consumes the resulting `localizedFields` map to mark those fields `localized: true` on the inferred schema, and `pullSanity` writes the detected locales (with the user's `--default-locale` preserved as base when present) into `contentful-space.json`. The detection logic is per (content type, field) so the same field name can be localized in one CT and not in another. +- **Inference recognizes Contentful Link sys shapes post-mapping.** The shape detector in `inferSchemas` now classifies `{sys: {type: 'Link', linkType: 'Entry'}}` as `Link Entry` and `{sys: {type: 'Link', linkType: 'Asset'}}` as `Link Asset`, in addition to the original Sanity `{_type: 'reference'}` and `{_type: 'image'}` shapes. This lets `pullSanity` map refs / assets BEFORE inference (so locale expansion sees Contentful-shaped values) without losing the ability to detect Link fields. Required to keep the `linkContentType` validation derivation working after M7's pipeline reorder. +- **End-to-end on the link-vehicles export:** `contentful-space.json` now reports `baseLocale: 'en', locales: ['en', 'es']`. Each Sanity doc fans out to 2 entry-locale documents (21 → 42). `post.body` is typed `RichText, localized: true` (was `Object`), `post.title` is `Symbol, localized: true` (was `Object`), `post.subtitle` is `Symbol, localized: true`. The simulator merges the 42 documents into 21 entries with 2 locales each, and the pipeline reports `0 warnings, 0 errors` against this dataset — full close-out from the M2 baseline of 97 warnings. +- **15 unit tests** for `detectLocales` / `expandDocumentsByLocale` covering positive cases (en/es, regional codes, singleton, aggregation across docs), negative cases (refs, Link sys, RichText, arbitrary keys, empty objects), per-CT-per-field scoping, base-locale fallback when a locale value is missing, the no-locale no-op pass-through, and absence of input mutation. + ## [0.5.0] — 2026-05-06 ### Fixed diff --git a/README.md b/README.md index 989877c..02c1214 100644 --- a/README.md +++ b/README.md @@ -95,6 +95,7 @@ Interactive SVG diagram of your content types and their relationships. Zoom, pan | **Offline simulation** | 10-step pipeline: load → validate → transform → link → resolve → convert → merge → validate → stats → report. Zero network calls. | | **Contentful validation** | Catches missing required fields, unknown fields, unresolved links, field type mismatches — the same errors Contentful would reject. | | **Pull from Contentful** | `cms-sim pull` downloads your existing content model and entries (read-only CDA token). Add `--management-token` to include all field validations. Modify locally, simulate, then apply when ready. | +| **Pull from Sanity** | `cms-sim pull-sanity` converts a Sanity NDJSON export (`sanity dataset export`) into the same on-disk shape as `cms-sim pull`. Schema inference, `_ref` → Link Entry, image refs → Link Asset, Portable Text → RichText, and locale-shape `{en, es}` fan-out are all handled automatically. Offline, zero deps. | | **Mock data generator** | No data? No problem. Auto-generates realistic entries from your schemas with field-type-aware values and cross-references. | | **CMS migration preview** | Feed WordPress XML, Sanity NDJSON, or generic JSON exports alongside your Contentful schemas. See exactly how the migrated content will look. | | **CI/CD validation** | `cms-sim validate --json` for pipelines. Exit code 1 on errors. | @@ -147,6 +148,32 @@ npx cms-sim --schemas=schemas/ --input=data/export.ndjson --transforms=transform npx cms-sim scaffold --input=data/export.xml --output=my-project/ ``` +#### `cms-sim pull-sanity` — zero-config Sanity → Contentful + +When you don't want to hand-write `transforms/`, `pull-sanity` converts a Sanity dataset NDJSON export (the file `sanity dataset export` produces) into the same on-disk shape that `cms-sim pull` writes for a real Contentful space — `contentful-space.json` + `schemas/.js` + `data/entries.ndjson` + `assets/assets.json` — in one command: + +```bash +# Generate the NDJSON from your Sanity project (their CLI, not ours) +sanity dataset export production export.ndjson + +# Convert it to Contentful shape (offline, zero deps, read-only) +npx cms-sim pull-sanity --input=export.ndjson --output=pulled-sanity/ + +# Now everything downstream works the same as with cms-sim pull +npx cms-sim --schemas=pulled-sanity/schemas/ --input=pulled-sanity/data/entries.ndjson --open +npx cms-sim to-import --input=output/ --schemas=pulled-sanity/schemas/ --output=bundle/ +``` + +What it handles automatically: + +- **Schema inference** from real document samples — Symbol / Text / Integer / Number / Boolean / Date / Object / RichText / Link Entry / Link Asset / Array variants. Disagreement between samples picks the dominant shape and emits a warning. +- **`_ref` → Link Entry** rewrites for both top-level and nested arrays, plus `linkContentType` validations derived from the cross-document refs actually present in the corpus. +- **Image refs → Link Asset** alongside a generated `assets/assets.json` index (id, title, fileName, contentType, url, size). +- **Portable Text → Contentful RichText** (paragraph / heading / lists / blockquote / decorators / hyperlinks). Unknown marks, custom block types, and nested list levels emit explicit warnings while preserving the underlying text. +- **Locale detection** (`{en, es}`, `{en-US, pt-BR}`, …). Each Sanity doc fans out to one document per detected locale; the matching schema fields are marked `localized: true`. `contentful-space.json` reflects the detected locales. + +The example at [`example-sanity-pull/`](../example-sanity-pull/) is a runnable walk-through against the link-vehicles dataset. + ### 4. `contentful-migration` preview (`from-migrations`) You already have migration scripts as your source of truth. Replay them against a mock and see the resulting model — without running them against a real Contentful environment. diff --git a/package-lock.json b/package-lock.json index 9bab03a..95c385d 100644 --- a/package-lock.json +++ b/package-lock.json @@ -1,12 +1,12 @@ { "name": "content-model-simulator", - "version": "0.3.0", + "version": "0.6.0", "lockfileVersion": 3, "requires": true, "packages": { "": { "name": "content-model-simulator", - "version": "0.3.0", + "version": "0.6.0", "license": "MIT", "bin": { "cms-sim": "bin/cms-sim.js" diff --git a/package.json b/package.json index 739d065..8a55f37 100644 --- a/package.json +++ b/package.json @@ -1,6 +1,6 @@ { "name": "content-model-simulator", - "version": "0.5.0", + "version": "0.6.0", "description": "Local-only simulator for Contentful content models. Preview entries, validate schemas, and export ready-to-import bundles — without connecting to Contentful.", "type": "module", "main": "dist/index.js", diff --git a/src/cli.ts b/src/cli.ts index 5129043..477b024 100755 --- a/src/cli.ts +++ b/src/cli.ts @@ -4,6 +4,7 @@ * Usage: * cms-sim --schemas [options] Simulate locally * cms-sim pull --space-id=XXX [options] Download from Contentful + * cms-sim pull-sanity --input= [opt] Convert a Sanity NDJSON export to the same shape as `pull` * cms-sim --help */ @@ -182,6 +183,7 @@ ${c.cyan}COMMANDS:${c.reset} cms-sim scaffold [options] ${c.dim}Generate schemas + transforms from a WordPress XML export${c.reset} cms-sim from-migrations [opts] ${c.dim}Convert contentful-migration scripts → cms-sim schemas${c.reset} cms-sim pull [options] ${c.dim}Download content model from Contentful${c.reset} + cms-sim pull-sanity [options] ${c.dim}Convert a Sanity NDJSON export to the same shape as ${c.bold}pull${c.reset}${c.dim} (offline)${c.reset} cms-sim diff --old=A --new=B ${c.dim}Compare two schema directories${c.reset} cms-sim validate [options] ${c.dim}Validate schemas + data (no HTML output)${c.reset} @@ -915,6 +917,150 @@ async function pullMain(argv: string[]): Promise { console.log(''); } +// ── pull-sanity sub-command ────────────────────────────────────── + +function showPullSanityHelp(): void { + console.log(` +${c.bold}Content Model Simulator — Pull Sanity${c.reset} +Convert a Sanity dataset NDJSON export into the same on-disk shape that +${c.bold}cms-sim pull${c.reset} produces (Contentful-flavored). The result drops into +${c.bold}cms-sim simulate${c.reset}, ${c.bold}cms-sim to-import${c.reset}, and ${c.bold}cms-sim diff${c.reset} unchanged. + +${c.cyan}USAGE:${c.reset} + cms-sim pull-sanity --input= [options] + +${c.cyan}REQUIRED:${c.reset} + --input= Path to the Sanity NDJSON export (produced by ${c.bold}sanity dataset export${c.reset}) + +${c.cyan}OPTIONS:${c.reset} + --output= Output directory (default: ./sanity-export) + --default-locale= Locale tag assigned to every entry (default: en-US) + --include-drafts Include documents whose _id starts with ${c.bold}drafts.${c.reset} (default: off) + --include-assets Emit ${c.bold}sanity.imageAsset${c.reset} docs as entries too (default: kept separately) + --source=