feat: v0.6.0 — pull-sanity adapter#12
Merged
Merged
Conversation
Adds `cms-sim pull-sanity` and the public `pullSanity()` API. Reads a
Sanity dataset NDJSON export (offline, from `sanity dataset export`)
and writes the same on-disk shape as `cms-sim pull` so the rest of the
pipeline (simulate / to-import / diff) consumes it unchanged.
Output structure mirrors pull-contentful exactly:
schemas/<id>.js one .js per Sanity _type
data/entries.ndjson one JSON per line
contentful-space.json { baseLocale, locales, pulledAt, source }
Field-type inference walks document samples and maps observed value
shapes to Contentful field types (Symbol / Text / Integer / Number /
Boolean / Date / Object / Link Entry / Link Asset / Array<...>).
Mixed-shape arrays collapse to Object with a warning; presence at 100%
across samples marks a field as required.
Real-data validation against the link-vehicles Sanity export drops
pipeline warnings from 97 (FIELD_NOT_IN_DEFINITION) to 13 (all of which
are ASSET_FIELD_NOT_LINKED, deferred to the asset-linking milestone).
Zero new runtime dependencies. +29 unit tests (539 total, was 510).
…e (M4)
`mapDocumentRefs` walks every Sanity document's `data` payload (including
arrays and deeply nested objects) and rewrites `{_type: 'reference', _ref}`
shapes into the Contentful Entry-link shape:
{ sys: { type: 'Link', linkType: 'Entry', id: <_ref> } }
Modifier fields (_key, _weak, _strengthenOnPublish) are dropped — the
result is the canonical Contentful shape only. Image references
(`{_type: 'image', asset: {_ref}}`) are left untouched; M6 will handle
asset link reshaping alongside asset extraction.
The same pass harvests a `linkTargets` map (contentTypeId.fieldId →
distinct target content types). `inferSchemas` consumes it to emit real
`linkContentType` validations on Link Entry fields and on
Array<Link Entry> items — derived from what the corpus actually
references, not hand-written. Orphan refs (target id missing from the
dataset) are rewritten but don't pollute the validation list.
End-to-end against the link-vehicles Sanity dataset: post entries now
report `linkedEntryIds` populated (categoryBlog + 2 tags per post), and
the model graph draws relationship arrows from real data instead of
schema-only validations.
+21 unit tests (558 total). Pre-commit smoke against the four `file:`-
linked examples (from-scratch, sanity-migration, wp-migration,
pull-contentful cached) all pass with the expected counts.
`portableTextToRichText` converts an array of Sanity Portable Text blocks into the Contentful RichText document tree expected by contentful-import and the RichText field type. Block-style mapping: normal → paragraph h1..h6 → heading-1..heading-6 blockquote → blockquote listItem bullet → unordered-list of list-item > paragraph listItem number → ordered-list of list-item > paragraph Span marks: strong → bold, em → italic, underline, code markDef _type:link with href → hyperlink wrapping the span Unknown block styles, marks, annotations, and nested list levels (level > 1) emit explicit warnings while preserving the underlying text. Custom (non-`block`) PT items are skipped with a warning rather than guessing the wrong embedded-entry shape. The sanity-reader gained a `portableText` option: 'html' (default — backwards-compatible HTML string output) 'richtext' (used by pullSanity) 'raw' (leave PT blocks untouched) pullSanity opts into 'richtext' so its entries.ndjson now carries real Contentful RichText documents end-to-end. inferSchemas classifies those as field type `RichText` (was `Object`). Real-data check on the link-vehicles export: post.body[en] is now a 15-node RichText document (was a single HTML string). 4 example projects regression-free (5/25, 6/20, 5/18, 23/69) thanks to the HTML default preserving the simulate-from-NDJSON path. +18 unit tests for the walker (576 total).
`mapDocumentAssets` rewrites every Sanity image reference
(`{_type: 'image', asset: {_ref}}`, including those nested in arrays
and objects) into the canonical Contentful Link/Asset sys shape:
{ sys: { type: 'Link', linkType: 'Asset', id: <_ref> } }
This is the same shape that `pull` (Contentful) emits for asset fields,
which is what the simulator's validator recognizes by checking
`val.sys.linkType === 'Asset'`. The asset id is preserved verbatim
from the original `_ref` so it matches the metadata index emitted at
`<output>/assets/assets.json` (id, title, fileName, contentType, url,
size — same shape as pull-contentful's assets.json).
Orphan refs (asset id missing from the dataset) are still rewritten
as dangling links — that's a faithful representation of what the
dataset says, and the simulator's link-resolution stage surfaces them
with its own warning.
Real-data validation against the link-vehicles export:
- ASSET_FIELD_NOT_LINKED warnings dropped 13 → 0
- linkedAssetIds populated on every post entry (entry↔asset edges
show up in the model graph)
- post.mainImage → Link/Asset sys (was the raw {_type: 'image'} shape)
+11 unit tests (587 total). Pre-commit smoke against the four
`file:`-linked examples is regression-free (5/25, 6/20, 5/18, 23/69).
`detectLocales` scans every document's `data` for objects whose keys all
match the locale-code pattern `^[a-z]{2,3}(-[A-Z]{2})?$` (en, es, en-US,
pt-BR, …) and which carry no `_type` / `sys` / `nodeType` — so Sanity
refs, Contentful Link sys shapes, and RichText documents are skipped.
The result is the dataset's distinct locale set plus a per-content-type
set of localized field ids.
`expandDocumentsByLocale` then fans each Sanity doc out to N documents
(one per locale). For each localized field it substitutes that locale's
value; missing locales fall back to the base locale. Non-localized
fields pass through identically across variants.
`inferSchemas` accepts the resulting `localizedFields` map and marks the
inferred fields `localized: true` accordingly. The detector regex also
gets a sibling: the shape detector inside `inferSchemas` now classifies
`{sys: {type: 'Link', linkType: 'Entry'/'Asset'}}` directly, which was
required to keep `linkContentType` validations working after M7's
pipeline reorder (refs/assets are now mapped BEFORE inference so locale
expansion sees Contentful-shaped values, not Sanity ones).
`pullSanity` reflects the detected locales in `contentful-space.json`
(`baseLocale`, `locales[]`) and falls back to `--default-locale` when no
locale-shaped fields are present.
End-to-end against the link-vehicles dataset:
- 21 docs → 42 entry-locale documents (en + es)
- post.body now typed RichText, localized: true (was Object)
- post.title / post.subtitle / post.metaDescription now Symbol/Text,
localized: true (were Object)
- contentful-space.json: { baseLocale: 'en', locales: ['en', 'es'] }
- Simulator merges 42 docs into 21 entries with 2 locales each
- Pipeline reports 0 warnings, 0 errors — full close-out from the M2
baseline of 97 warnings
+15 unit tests (601 total). 4 `file:`-linked examples regression-free
(5/25, 6/20, 5/18, 23/69).
Adds the pull-sanity subsection to Real workflows § 3 ("WordPress /
Sanity migration planning"). Documents the zero-config Sanity → Contentful
path:
sanity dataset export production export.ndjson
cms-sim pull-sanity --input=export.ndjson --output=pulled-sanity/
cms-sim --schemas=pulled-sanity/schemas/ --input=pulled-sanity/data/entries.ndjson --open
Calls out what the adapter handles automatically (schema inference, _ref
rewriting, image refs → Link Asset + assets.json, Portable Text → RichText,
locale detection + fan-out), and points at the runnable walk-through in
`example-sanity-pull/`.
Also adds a "Pull from Sanity" row to the Core capabilities table.
Companion runnable example lives at the workspace level in
`example-sanity-pull/` (file:-linked, not part of this repo) — it pulls
the same link-vehicles Sanity NDJSON used as dogfood during M1–M7 and
ships a README walk-through with expected counters.
Bumps version to 0.6.0 and closes the [Unreleased] CHANGELOG section
with the 2026-05-10 date.
v0.6.0 ships `cms-sim pull-sanity`: a single-command path from a Sanity
dataset NDJSON export to the same on-disk shape that `cms-sim pull`
produces for Contentful spaces, so simulate / to-import / diff consume
it unchanged. Schema inference, _ref → Link Entry rewrites, image refs
→ Link Asset, Portable Text → RichText, and locale-shape `{en, es}`
fan-out are all automatic. Zero new runtime dependencies. 601 tests
passing (+91 vs 0.5.0).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
cms-sim pull-sanitysubcommand: single-command path from a Sanity dataset NDJSON export to the same Contentful-shape output thatcms-sim pullproduces.linkContentTypevalidations derived from real cross-document refs andlocalized: trueflags from detected locale shapes.What's inside
7c5281bcontentful-space.json+schemas/<id>.js+data/entries.ndjson) + schema inference from samples.1235404_ref→ Link Entry rewrites +linkContentTypevalidation derived from real cross-doc refs.9dac93035e2728assets/assets.jsonindex.4dbcdf2{en, es}shapes → one doc per locale,localized: trueschema flags).a3516631d2b8d7[Unreleased]→[0.6.0] — 2026-05-10.M3 was a no-op (slug / datetime / primitives already handled by the existing
sanity-reader). M8 dogfood was done implicitly across M1–M7 usingexample-sanity-migration/data/production.ndjson.Pre-merge state
v0.6.0already pushed (annotated, points at1d2b8d7). The tag is unreachable frommainuntil this PR merges with a merge commit (not squash), which is the project's existing pattern (v0.5.0was reached via merge commit222aabf).file:-linked examples (example-from-scratch,example-sanity-migration,example-wp-migration,example-pull-contentful): counts identical to v0.5.0 (5/25, 6/20, 5/18, 23/69). Warning counts on sanity-migration (17) and wp-migration (24) are unchanged from v0.5.0 — verified by checking out the tag and re-running.example-sanity-pull/(workspace-level, file:-linked) demonstrates the pull → simulate end-to-end (7 CTs, 21 entries merged from 42 docs × 2 locales, 0 warnings, 0 errors).Test plan
npm test— 601 passingnpm run build— cleanexample-sanity-pull/runs end-to-end cleanMerge note
Please merge commit rather than squash so the
v0.6.0tag (already on remote, pointing at1d2b8d7) stays reachable from main without retagging.