diff --git a/CHANGELOG.md b/CHANGELOG.md index fa9411d..1f5a6df 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,25 @@ All notable changes to `content-model-simulator` are documented here. +## [0.6.1] — 2026-05-10 + +### Added +- **`cms-sim pull-wordpress`** — new CLI subcommand and public `pullWordpress()` API that convert a WordPress WXR XML export (from wp-admin → Tools → Export → All content) into the same on-disk shape as `cms-sim pull` for Contentful spaces: `contentful-space.json` + `schemas/.js` + `data/entries.ndjson`. Reuses `parseWXR` for parsing and the shared `inferSchemas` / `detectLocales` / `expandDocumentsByLocale` from `src/shared/`. Flags: `--input` (required), `--output` (default `./wordpress-export`), `--default-locale`, `--force-locale`, `--include-attachments`, `--skip-types=`, `--source`, `--verbose`. Stable id prefixes per content type: `wp_` for posts/pages, `wp_author_`, `wp_category_`, `wp_tag_` — so cross-document refs have predictable targets. Default skip set drops `nav_menu_item` / `wp_navigation` / `wp_template` / `wp_template_part` / `wp_global_styles` / `wp_block` / `oembed_cache` / `customize_changeset` / `custom_css` / `revision` / `attachment` (the last opts back in via `--include-attachments` and will be reshaped into `assets/assets.json` in a follow-up commit). **Status:** skeleton + ref rewriting shipped; asset linking, Gutenberg/Classic HTML → RichText, and Polylang locale detection land in follow-up commits. +- **10 unit tests** for `pullWordpress` covering: three pieces written, `contentful-space.json` shape with WP site metadata + `sourceFormat: 'wordpress-wxr'`, `--force-locale` precedence, one schema per surviving content type sorted, Contentful schema shape with `id` / `name` / `fields[]`, prefixed-id assignment across post / page / author / category / tag, default skip set (attachments + nav_menu_item filtered), `--include-attachments` opt-in, in-memory return shape, missing-input error. +- **`mapWordpressRefs(documents)`** — pure transform in `src/wordpress/wp-ref-mapper.ts`. Rewrites every WordPress cross-document reference into a Contentful Entry-link sys shape: `post.author` (login) → `Link Entry pointing at wp_author_`; `post.categories[]` (slugs) → `Array` pointing at `wp_category_`; `post.tags[]` (slugs) → `Array` pointing at `wp_tag_`; `category.parent` (slug) → self-referential `Link Entry`. Returns the rewritten documents plus a `linkTargets` map (`contentTypeId.fieldId` → set of resolved target content types) which the shared `inferSchemas` consumes to attach real `linkContentType` validations to Link fields and `Array.items`. Orphan refs (target id missing from the corpus, e.g., a post tagged with a slug whose `` block was filtered) stay rewritten as dangling links — they just don't pollute the validation list. `pullWordpress` runs the mapper between id assignment and schema inference. **+14 unit tests** (3 in `pull-wordpress.test.ts` end-to-end + 11 in `wp-ref-mapper.test.ts`). +- **`mapWordpressAssets(documents)` + `assets/assets.json`** — extracts WordPress `attachment` documents into Contentful-shaped asset metadata (`{id, title, fileName, contentType, url, size}`) and rewrites featured-image references (`meta._thumbnail_id` post-id pointer) into `featuredImage: {sys: Link Asset}` on the parent document. Asset ids stay aligned with the attachment's prefixed entry id (`wp_`), so `assets/assets.json` and the in-entry Link sys share the same identifier. `_thumbnail_id` is stripped from `meta` after rewriting (no duplicate Symbol field on the schema). The shared `inferSchemas` shape detector picks up `featuredImage` as `Link Asset` without WP-specific knowledge. Dangling featured-image refs (target attachment missing) leave the original meta string in place rather than emit a bogus link. `--include-attachments` still re-adds attachment docs to entries (in addition to the assets index). **+19 unit tests** (13 in `wp-asset-mapper.test.ts` + 6 in `pull-wordpress.test.ts` end-to-end). +- **`INCLUDED_INTERNAL_META_KEYS` whitelist** in `wxr-reader.ts` so a curated set of `_`-prefixed `` keys (`_thumbnail_id`, `_wp_attached_file`, `_wp_attachment_metadata`, `_wp_page_template`) flow through `extractPostMeta` instead of being filtered. Necessary for featured-image resolution; non-breaking addition (only adds keys, doesn't change existing ones). +- **`mapWordpressBodies(documents)`** — converts WordPress `body` and `excerpt` fields (raw HTML, already Gutenberg-stripped by `parseWXR`) into Contentful RichText documents via the existing `htmlToRichText` walker from `transform/rich-text.ts`. Block elements (`p`, `h1..h6`, `blockquote`, `ul`/`ol`/`li`, `pre`, `table`/`tr`/`td`/`th`), inline marks (`strong`/`em`/`u`/`code`/`s`/`del`/`strike`/`sup`/`sub`), `a[href]` hyperlinks, and inline `` (as `embedded-asset-block` placeholders) all map. Empty / whitespace-only fields are dropped entirely so they don't pollute schema inference with stale `Text` fields. The shared shape detector then picks up the resulting `{nodeType: 'document'}` and classifies the field as `RichText` on the inferred schema. **+13 unit tests.** +- **Polylang multi-locale detection** in `pullWordpress`. `parseWXR` now extracts the Polylang `language` taxonomy alongside categories and tags (`` in `` blocks), and surfaces the first language slug as the document's `locale` tag. `pullWordpress` collects all distinct per-doc locales into `contentful-space.json` (`locales[]`), so a bilingual WP site exports as multi-locale automatically. **The shared `expandDocumentsByLocale` now distinguishes Sanity-style locales (per-field `{en, es}` shapes → fan out one doc into N variants) from WordPress-style locales (per-doc tag → emit one variant per doc):** the per-CT `localizedFields` map only triggers fan-out for docs whose values actually carry a locale-shape, while WP docs with a single `locale` tag emit one entry each. `--force-locale` overrides Polylang (and Sanity) for callers that want to flatten everything to one locale. **+3 unit tests for `pullWordpress` Polylang behavior**, and the existing `expandDocumentsByLocale` tests still pass (no Sanity regressions). +- **README** gains a "Pull from WordPress" row in Core capabilities and a `cms-sim pull-wordpress — zero-config WordPress → Contentful` subsection under Real workflows § 3, alongside the existing `pull-sanity` documentation. Tests badge bumped from 510 → 659. +- **`example-wp-pull/`** — new runnable example at the workspace level (file:-linked, mirrors `example-pull-contentful/` and `example-sanity-pull/`) with a synthetic 12-document WXR fixture exercising authors, nested categories, tags, Gutenberg + Classic bodies, attachments, featured images, and the default content-type filter (nav_menu term filtered). Includes a README walk-through with expected counters and intentional-behavior callouts. + +### Changed (breaking, pre-1.0.0) +- **`parseWXR` / `readWXR` now emit category and tag slugs in `data.categories` / `data.tags`, not display names.** Previously `extractInlineTerms` returned the CDATA value (e.g., `'Technology'`, `'JavaScript'`); now it returns the `nicename` attribute (e.g., `'tech'`, `'javascript'`). The slug is WordPress's canonical taxonomy identifier — it's what cross-references in `pull-wordpress` resolve against, and what any reasonable downstream tool would want. The display name is still available on the matching `` / `` document (which is parsed separately and ends up as `category.data.name` / `tag.data.name`). One inline test in `wxr-reader.test.ts` updated to assert slugs. + +### Changed +- **`inferSchemas` and locale detection promoted to `src/shared/`** (originally named `sanity-schema-inference` / `sanity-locale-expander` in v0.6.0 because Sanity was the only caller). Both modules are source-agnostic — `inferSchemas` recognizes both Sanity-shaped (`{_type: 'reference'}`, `{_type: 'image'}`) and Contentful-shaped (`{sys: Link Entry/Asset}`) values, and the locale detector inspects only field-value shape. Moved to `src/shared/schema-inference.{ts,test.ts}` and `src/shared/locale-expander.{ts,test.ts}`. Public API export `inferSanitySchemas` is replaced by the unprefixed `inferSchemas` (breaking, but pre-1.0.0 minor — see SemVer caveat in README). Imports in `pull-sanity.ts` updated. No runtime/behavior change; 601 tests still pass. + ## [0.6.0] — 2026-05-10 ### Added diff --git a/README.md b/README.md index 02c1214..5f1561d 100644 --- a/README.md +++ b/README.md @@ -2,7 +2,7 @@ [![npm](https://img.shields.io/npm/v/content-model-simulator)](https://www.npmjs.com/package/content-model-simulator) ![Node.js](https://img.shields.io/badge/node-%3E%3D22-brightgreen) -![Tests](https://img.shields.io/badge/tests-510%20passing-brightgreen) +![Tests](https://img.shields.io/badge/tests-659%20passing-brightgreen) ![Dependencies](https://img.shields.io/badge/dependencies-0-brightgreen) ![License](https://img.shields.io/badge/license-MIT-green) @@ -96,6 +96,7 @@ Interactive SVG diagram of your content types and their relationships. Zoom, pan | **Contentful validation** | Catches missing required fields, unknown fields, unresolved links, field type mismatches — the same errors Contentful would reject. | | **Pull from Contentful** | `cms-sim pull` downloads your existing content model and entries (read-only CDA token). Add `--management-token` to include all field validations. Modify locally, simulate, then apply when ready. | | **Pull from Sanity** | `cms-sim pull-sanity` converts a Sanity NDJSON export (`sanity dataset export`) into the same on-disk shape as `cms-sim pull`. Schema inference, `_ref` → Link Entry, image refs → Link Asset, Portable Text → RichText, and locale-shape `{en, es}` fan-out are all handled automatically. Offline, zero deps. | +| **Pull from WordPress** | `cms-sim pull-wordpress` converts a WordPress WXR XML export (wp-admin → Tools → Export → All content) into the same on-disk shape as `cms-sim pull`. Schema inference, taxonomies + author + parent → Link Entry, attachments → `assets/assets.json` + featured-image → Link Asset, Gutenberg / Classic HTML body → RichText, and Polylang `language` taxonomy → per-doc locale are all handled automatically. Offline, zero deps. | | **Mock data generator** | No data? No problem. Auto-generates realistic entries from your schemas with field-type-aware values and cross-references. | | **CMS migration preview** | Feed WordPress XML, Sanity NDJSON, or generic JSON exports alongside your Contentful schemas. See exactly how the migrated content will look. | | **CI/CD validation** | `cms-sim validate --json` for pipelines. Exit code 1 on errors. | @@ -174,6 +175,33 @@ What it handles automatically: The example at [`example-sanity-pull/`](../example-sanity-pull/) is a runnable walk-through against the link-vehicles dataset. +#### `cms-sim pull-wordpress` — zero-config WordPress → Contentful + +The same single-command path, but for WordPress WXR XML exports (the file wp-admin's Tools → Export → All content produces): + +```bash +# Export from WordPress +# wp-admin → Tools → Export → All content → "Download Export File" + +# Convert it to Contentful shape (offline, zero deps, read-only) +npx cms-sim pull-wordpress --input=wp-export.xml --output=pulled-wp/ + +# Now everything downstream works the same as with cms-sim pull +npx cms-sim --schemas=pulled-wp/schemas/ --input=pulled-wp/data/entries.ndjson --open +npx cms-sim to-import --input=output/ --schemas=pulled-wp/schemas/ --output=bundle/ +``` + +What it handles automatically: + +- **Default content-type filtering.** `nav_menu_item`, `wp_navigation`, `wp_template`, `wp_template_part`, `wp_global_styles`, `wp_block`, `oembed_cache`, `customize_changeset`, `custom_css`, `revision` are dropped — they have no Contentful equivalent. `--skip-types=` adds more. +- **Stable, prefixed ids per type:** `wp_` for posts/pages, `wp_author_`, `wp_category_`, `wp_tag_` — so cross-references resolve without ambiguity. +- **Reference rewriting.** `post.author` (login) → `Link Entry`; `post.categories[]` / `post.tags[]` (slugs) → `Array`; `category.parent` (slug) → self-referential `Link Entry`. `linkContentType` validations derived from actual cross-doc refs. +- **Attachments → `assets/assets.json`** with `{id, title, fileName, contentType, url, size}`. Featured-image references (`` key `_thumbnail_id`, a post-id pointer) resolve to `featuredImage: { sys: Link Asset }` on the parent document. +- **Gutenberg / Classic HTML body → Contentful RichText** via `htmlToRichText`. Block elements, inline marks, hyperlinks, and inline `` (as `embedded-asset-block` placeholders) all map. +- **Polylang multi-locale detection.** `` on items → per-doc `locale` tag. All distinct locales surfaced in `contentful-space.json`. Single-locale sites just inherit the channel ``. + +The example at [`example-wp-pull/`](../example-wp-pull/) is a runnable walk-through with a synthetic WXR fixture exercising authors, nested categories, tags, attachments, featured images, Gutenberg + Classic bodies, and the nav-menu filter. + ### 4. `contentful-migration` preview (`from-migrations`) You already have migration scripts as your source of truth. Replay them against a mock and see the resulting model — without running them against a real Contentful environment. diff --git a/package-lock.json b/package-lock.json index 95c385d..8730d00 100644 --- a/package-lock.json +++ b/package-lock.json @@ -1,12 +1,12 @@ { "name": "content-model-simulator", - "version": "0.6.0", + "version": "0.6.1", "lockfileVersion": 3, "requires": true, "packages": { "": { "name": "content-model-simulator", - "version": "0.6.0", + "version": "0.6.1", "license": "MIT", "bin": { "cms-sim": "bin/cms-sim.js" diff --git a/package.json b/package.json index 8a55f37..8899981 100644 --- a/package.json +++ b/package.json @@ -1,6 +1,6 @@ { "name": "content-model-simulator", - "version": "0.6.0", + "version": "0.6.1", "description": "Local-only simulator for Contentful content models. Preview entries, validate schemas, and export ready-to-import bundles — without connecting to Contentful.", "type": "module", "main": "dist/index.js", diff --git a/src/cli.ts b/src/cli.ts index 477b024..8dd4eab 100755 --- a/src/cli.ts +++ b/src/cli.ts @@ -184,6 +184,7 @@ ${c.cyan}COMMANDS:${c.reset} cms-sim from-migrations [opts] ${c.dim}Convert contentful-migration scripts → cms-sim schemas${c.reset} cms-sim pull [options] ${c.dim}Download content model from Contentful${c.reset} cms-sim pull-sanity [options] ${c.dim}Convert a Sanity NDJSON export to the same shape as ${c.bold}pull${c.reset}${c.dim} (offline)${c.reset} + cms-sim pull-wordpress [opts] ${c.dim}Convert a WordPress WXR export to the same shape as ${c.bold}pull${c.reset}${c.dim} (offline)${c.reset} cms-sim diff --old=A --new=B ${c.dim}Compare two schema directories${c.reset} cms-sim validate [options] ${c.dim}Validate schemas + data (no HTML output)${c.reset} @@ -1061,6 +1062,158 @@ async function pullSanityMain(argv: string[]): Promise { console.log(''); } +// ── pull-wordpress sub-command ─────────────────────────────────── + +function showPullWordpressHelp(): void { + console.log(` +${c.bold}Content Model Simulator — Pull WordPress${c.reset} +Convert a WordPress WXR XML export into the same on-disk shape that +${c.bold}cms-sim pull${c.reset} produces (Contentful-flavored). The result drops into +${c.bold}cms-sim simulate${c.reset}, ${c.bold}cms-sim to-import${c.reset}, and ${c.bold}cms-sim diff${c.reset} unchanged. + +${c.cyan}USAGE:${c.reset} + cms-sim pull-wordpress --input= [options] + +${c.cyan}REQUIRED:${c.reset} + --input= Path to the WordPress WXR XML export + (Tools → Export → All content in wp-admin) + +${c.cyan}OPTIONS:${c.reset} + --output= Output directory (default: ./wordpress-export) + --default-locale= Override the language detected from the WXR tag + --force-locale= Force-tag every entry with this locale (ignore WXR) + --include-attachments Emit attachment posts as entries (default: filtered; + commit 4 will reshape them into assets/assets.json) + --skip-types= Additional post_types to skip (merged with defaults: + nav_menu_item, wp_navigation, wp_template, …) + --source=