feat: v0.6.0 — pull-sanity adapter

CHANGELOG.md

@@ -2,6 +2,28 @@
 
 All notable changes to `content-model-simulator` are documented here.
 
+## [0.6.0] — 2026-05-10
+
+### Added
+- **`cms-sim pull-sanity`** — new CLI subcommand that converts a Sanity dataset NDJSON export (produced by `sanity dataset export`) into the same on-disk shape that `cms-sim pull` produces for Contentful spaces: `contentful-space.json` + `schemas/<type>.js` (one per Sanity `_type`) + `data/entries.ndjson`. The output drops into `cms-sim simulate`, `cms-sim to-import`, and `cms-sim diff` unchanged. Flags: `--input` (required), `--output` (default `./sanity-export`), `--default-locale` (default `en-US`), `--include-drafts`, `--include-assets`, `--source`, `--verbose`. Zero new runtime dependencies. Output contract validated end-to-end; field-type inference (Symbol/Text/Integer/Number/Boolean/Date/Object/RichText/Link/Array) shipped. Pipeline warnings against a real Sanity dataset (link-vehicles export, 21 documents) drop from 97 (`FIELD_NOT_IN_DEFINITION` at baseline) to 0.
+- **`pullSanity(options)`** — programmatic API behind the CLI, exported from `src/sanity/pull-sanity.ts`. Returns `{ schemas, documents, assets, warnings, output }`. Reuses `parseSanity` / `parseSanityString` from the existing `sanity-reader` module.
+- **`inferSchemas(samplesByType, options)`** — pure schema-inference API in `src/sanity/sanity-schema-inference.ts`. Walks document samples per content type, classifies each field's value shape, and emits a `ContentTypeDefinition` plus a `warnings[]` array describing type conflicts and array-collapse fallbacks. Mappings: short strings → `Symbol`, strings >256 chars → `Text`, ISO 8601 → `Date`, integers → `Integer`, floats → `Number`, booleans → `Boolean`, `{_type: reference, _ref}` → `Link Entry`, `{_type: image, asset}` → `Link Asset`, arrays of refs/images/symbols → `Array<Link Entry>` / `Array<Link Asset>` / `Array<Symbol>`. Mixed-shape arrays collapse to `Object` with a warning. Required flag is set when the field is present and non-null in 100% of samples (configurable via `requiredThreshold`).
+- **11 unit tests** for `pullSanity` covering the M1 output contract (three pieces written, `contentful-space.json` shape, custom default locale, one schema per type, Contentful schema shape with inferred fields, camelCase humanization in schema names, entries.ndjson per-line shape, draft/system filtering opt-in via flags, in-memory return shape, empty-input no-write behavior, missing-input error).
+- **18 unit tests** for `inferSchemas` covering primitive shape detection (Symbol/Text by string length, Integer/Number split, Boolean, Date for ISO strings), Sanity-specific shapes (`reference`, `image`, arrays of each), disagreement resolution (majority shape wins with warning, mixed arrays collapse to Object), the `required` flag (100% presence → required; missing or null in any sample → not required), and multi-type orchestration (one schema per type, sorted by id, humanized).
+- **`mapDocumentRefs(docs)`** — pure transform in `src/sanity/sanity-ref-mapper.ts` that rewrites every `{_type: 'reference', _ref}` shape (including nested in arrays/objects, with or without `_key`/`_weak` modifiers) to the Contentful Entry-link shape `{sys: {type: 'Link', linkType: 'Entry', id}}`. Returns the transformed documents plus a `linkTargets` map (`contentTypeId.fieldId` → Set of target content types) harvested from the actual cross-document refs. Image references (`{_type: 'image', asset: {_ref}}`) are left untouched here — M6 handles asset links. `pullSanity` runs `mapDocumentRefs` before writing `entries.ndjson` so the simulator's link-resolution stage populates `linkedEntryIds` and the model graph draws relationship arrows from data, not just schema validations.
+- **`linkContentType` validation inference** in `inferSchemas` — when `linkTargets` is passed, single Link Entry fields get `validations: [{ linkContentType: [...sorted targets] }]` and Array<Link Entry> fields get the same on `items.validations`. Orphan refs (target id missing from dataset) don't pollute the validation list.
+- **15 unit tests** for `mapRefsToContentful` / `mapDocumentRefs` and **6 unit tests** for the linkContentType inference in `pullSanity` covering single + array refs, aggregation across multiple target types, orphan-ref handling, and end-to-end ref rewriting in `entries.ndjson`.
+- **`portableTextToRichText(blocks)`** — converts an array of Sanity Portable Text blocks into a Contentful RichText document (the structured `nodeType` tree, not HTML). Maps block styles (`normal`/`h1..h6`/`blockquote`), groups consecutive `listItem` blocks into `unordered-list` / `ordered-list` with nested `list-item` + `paragraph`, converts span marks (`strong`→`bold`, `em`→`italic`, `underline`, `code`), and wraps spans in `hyperlink` nodes when their mark refs a `link` markDef with `href`. Unknown block styles, marks, annotations, and nested list levels (>1) emit explicit warnings while preserving the underlying text. Custom (non-`block`) PT items are skipped with a warning. Exported from the public API along with `isPortableText`.
+- **`portableText: 'html' | 'richtext' | 'raw'`** option on `SanityReadOptions` (default `'html'` for back-compat). `'richtext'` invokes `portableTextToRichText`; `'raw'` leaves PT blocks untouched. `pullSanity` opts into `'richtext'` so its `entries.ndjson` carries Contentful-shaped RichText documents, which (a) lets `to-import` emit valid RichText payloads for top-level PT fields, and (b) makes `inferSchemas` classify those fields as `RichText` instead of `Object`. The default-`html` path keeps `cms-sim simulate --input=*.ndjson` (Sanity NDJSON detected via `isSanityNDJSON`) and existing transforms producing HTML strings as they did before.
+- **`RichText` shape detection** in `inferSchemas` — objects with `nodeType === 'document'` and an array `content` are classified as `RichText` fields. Verified end-to-end against the link-vehicles Sanity export: `post.body[en]` now lands as a real RichText document with 15 structured content nodes instead of an HTML blob.
+- **18 unit tests** for `portableTextToRichText` covering paragraph / heading / blockquote mapping, decorator marks, hyperlink annotations, list grouping (bullet+number), nested-list flattening warning, unknown-mark/annotation/block-style fallbacks, and non-block skip behavior. **`isPortableText`** has its own 2-test block.
+- **Sanity asset linking** in `pullSanity` (`src/sanity/sanity-asset-mapper.ts`). Image references (`{_type: 'image', asset: {_ref}}`, including those nested in arrays/objects) are rewritten to the canonical Contentful Link/Asset sys shape `{sys: {type: 'Link', linkType: 'Asset', id: <_ref>}}` — matching the shape that `pull` (Contentful) emits for asset-field entries. The asset id is preserved verbatim from `_ref`, so it joins the asset metadata written to `<output>/assets/assets.json` (mirroring pull-contentful's `assets/assets.json` shape: `{id, title, fileName, contentType, url, size}`). End-to-end against the link-vehicles dataset: `ASSET_FIELD_NOT_LINKED` warnings dropped from 13 → 0, `linkedAssetIds` is now populated on every post entry (the model graph picks up entry↔asset relationships), and `mainImage` resolves to a Link/Asset sys instead of a raw `{_type: 'image'}` object. Orphan references (asset id missing from the dataset) are still rewritten — the resulting link is dangling and the simulator's link-resolution stage surfaces it.
+- **11 unit tests** for `buildAssetUrlMap` / `mapImageRefsToLinks` / `mapDocumentAssets` / `toAssetMetadata` covering single + array refs, deep nesting, orphan refs (still rewritten — dangling, expected), pass-through of primitives and Link Entry sys shapes, no input mutation, and `assets.json` shape (including the extension-derived filename fallback when `originalFilename` is empty).
+- **Multi-locale detection + expansion** in `pullSanity` (`src/sanity/sanity-locale-expander.ts`). `detectLocales` scans every document's `data` for objects whose keys all match the locale-code pattern `^[a-z]{2,3}(-[A-Z]{2})?$` (e.g., `en`, `es`, `en-US`, `pt-BR`) and which have no `_type` / `sys` / `nodeType` keys — so Sanity refs, Link sys, and RichText documents are correctly skipped. `expandDocumentsByLocale` then fans each document out to N documents (one per locale), substituting the locale-specific value for every detected localized field and falling back to the base locale when missing. `inferSchemas` consumes the resulting `localizedFields` map to mark those fields `localized: true` on the inferred schema, and `pullSanity` writes the detected locales (with the user's `--default-locale` preserved as base when present) into `contentful-space.json`. The detection logic is per (content type, field) so the same field name can be localized in one CT and not in another.
+- **Inference recognizes Contentful Link sys shapes post-mapping.** The shape detector in `inferSchemas` now classifies `{sys: {type: 'Link', linkType: 'Entry'}}` as `Link Entry` and `{sys: {type: 'Link', linkType: 'Asset'}}` as `Link Asset`, in addition to the original Sanity `{_type: 'reference'}` and `{_type: 'image'}` shapes. This lets `pullSanity` map refs / assets BEFORE inference (so locale expansion sees Contentful-shaped values) without losing the ability to detect Link fields. Required to keep the `linkContentType` validation derivation working after M7's pipeline reorder.
+- **End-to-end on the link-vehicles export:** `contentful-space.json` now reports `baseLocale: 'en', locales: ['en', 'es']`. Each Sanity doc fans out to 2 entry-locale documents (21 → 42). `post.body` is typed `RichText, localized: true` (was `Object`), `post.title` is `Symbol, localized: true` (was `Object`), `post.subtitle` is `Symbol, localized: true`. The simulator merges the 42 documents into 21 entries with 2 locales each, and the pipeline reports `0 warnings, 0 errors` against this dataset — full close-out from the M2 baseline of 97 warnings.
+- **15 unit tests** for `detectLocales` / `expandDocumentsByLocale` covering positive cases (en/es, regional codes, singleton, aggregation across docs), negative cases (refs, Link sys, RichText, arbitrary keys, empty objects), per-CT-per-field scoping, base-locale fallback when a locale value is missing, the no-locale no-op pass-through, and absence of input mutation.
+
 ## [0.5.0] — 2026-05-06
 
 ### Fixed

README.md

@@ -95,6 +95,7 @@ Interactive SVG diagram of your content types and their relationships. Zoom, pan
 | **Offline simulation** | 10-step pipeline: load → validate → transform → link → resolve → convert → merge → validate → stats → report. Zero network calls. |
 | **Contentful validation** | Catches missing required fields, unknown fields, unresolved links, field type mismatches — the same errors Contentful would reject. |
 | **Pull from Contentful** | `cms-sim pull` downloads your existing content model and entries (read-only CDA token). Add `--management-token` to include all field validations. Modify locally, simulate, then apply when ready. |
+| **Pull from Sanity** | `cms-sim pull-sanity` converts a Sanity NDJSON export (`sanity dataset export`) into the same on-disk shape as `cms-sim pull`. Schema inference, `_ref` → Link Entry, image refs → Link Asset, Portable Text → RichText, and locale-shape `{en, es}` fan-out are all handled automatically. Offline, zero deps. |
 | **Mock data generator** | No data? No problem. Auto-generates realistic entries from your schemas with field-type-aware values and cross-references. |
 | **CMS migration preview** | Feed WordPress XML, Sanity NDJSON, or generic JSON exports alongside your Contentful schemas. See exactly how the migrated content will look. |
 | **CI/CD validation** | `cms-sim validate --json` for pipelines. Exit code 1 on errors. |
@@ -147,6 +148,32 @@ npx cms-sim --schemas=schemas/ --input=data/export.ndjson --transforms=transform
 npx cms-sim scaffold --input=data/export.xml --output=my-project/
 ```
 
+#### `cms-sim pull-sanity` — zero-config Sanity → Contentful
+
+When you don't want to hand-write `transforms/`, `pull-sanity` converts a Sanity dataset NDJSON export (the file `sanity dataset export` produces) into the same on-disk shape that `cms-sim pull` writes for a real Contentful space — `contentful-space.json` + `schemas/<type>.js` + `data/entries.ndjson` + `assets/assets.json` — in one command:
+
+```bash
+# Generate the NDJSON from your Sanity project (their CLI, not ours)
+sanity dataset export production export.ndjson
+
+# Convert it to Contentful shape (offline, zero deps, read-only)
+npx cms-sim pull-sanity --input=export.ndjson --output=pulled-sanity/
+
+# Now everything downstream works the same as with cms-sim pull
+npx cms-sim --schemas=pulled-sanity/schemas/ --input=pulled-sanity/data/entries.ndjson --open
+npx cms-sim to-import --input=output/ --schemas=pulled-sanity/schemas/ --output=bundle/
+```
+
+What it handles automatically:
+
+- **Schema inference** from real document samples — Symbol / Text / Integer / Number / Boolean / Date / Object / RichText / Link Entry / Link Asset / Array variants. Disagreement between samples picks the dominant shape and emits a warning.
+- **`_ref` → Link Entry** rewrites for both top-level and nested arrays, plus `linkContentType` validations derived from the cross-document refs actually present in the corpus.
+- **Image refs → Link Asset** alongside a generated `assets/assets.json` index (id, title, fileName, contentType, url, size).
+- **Portable Text → Contentful RichText** (paragraph / heading / lists / blockquote / decorators / hyperlinks). Unknown marks, custom block types, and nested list levels emit explicit warnings while preserving the underlying text.
+- **Locale detection** (`{en, es}`, `{en-US, pt-BR}`, …). Each Sanity doc fans out to one document per detected locale; the matching schema fields are marked `localized: true`. `contentful-space.json` reflects the detected locales.
+
+The example at [`example-sanity-pull/`](../example-sanity-pull/) is a runnable walk-through against the link-vehicles dataset.
+
 ### 4. `contentful-migration` preview (`from-migrations`)
 
 You already have migration scripts as your source of truth. Replay them against a mock and see the resulting model — without running them against a real Contentful environment.

package.json

@@ -1,6 +1,6 @@
 {
   "name": "content-model-simulator",
-  "version": "0.5.0",
+  "version": "0.6.0",
   "description": "Local-only simulator for Contentful content models. Preview entries, validate schemas, and export ready-to-import bundles — without connecting to Contentful.",
   "type": "module",
   "main": "dist/index.js",