wheelercat-productbus-ingest

Reads a query-index.json feed of used equipment and ingests it into the Helix Product Bus, mapping source fields to the Product Bus schema.

Requirements

• Node.js >= 18 (uses the built-in fetch)
• No dependencies to install

Usage

Dry run (recommended first)

Maps the feed and shows exactly what would be sent — no API calls are made. Writes the full mapped payload to dry-run-output.json.

node index.js --dry-run

Preview just a few rows:

node index.js --dry-run --limit 5

Live ingest

Copy .env.example to .env and fill in credentials, then:

node index.js

Configuration

Set via .env (see .env.example) or environment variables:

Variable	Required	Default	Description
PRODUCT_BUS_API_KEY	live only	—	Product Bus bearer token (sitekey)
PRODUCT_BUS_ORG	live only	—	Organization identifier
PRODUCT_BUS_SITE	live only	—	Site identifier
PRODUCT_BUS_BASE_URL	no	https://api.adobecommerce.live	API base URL
SOURCE_URL	no	wheelercat used-equipment feed	Source query-index.json
SOURCE_PREFIX	no	/used-equipment	Leading source path segment to replace
TARGET_PREFIX	no	/used	Replacement Product Bus path prefix

Flags

Flag	Description
--dry-run	Map + preview, write output file, no API calls
--source <url|file>	Override source feed (URL or local JSON file)
--source-prefix <p>	Leading source path segment to replace (default /used-equipment)
--target-prefix <p>	Replacement Product Bus path prefix (default /used)
--limit <n>	Only process the first n rows
--batch-size <n>	Products per bulk request (max 50)
--concurrency <n>	Concurrent requests in flight (default 2)
--delay <ms>	Min delay between dispatching requests (default 250)
--out <file>	Dry-run output path (default dry-run-output.json)

Field mapping

Product Bus field	Source	Notes
sku	last segment of path	Unique per row; serialNum is not unique
name	title
path	path	Leading /used-equipment replaced with /used, lowercased. e.g. /used-equipment/water-trucks/foo → /used/water-trucks/foo
price	price	"$409,000" → { final: "409000", currency: "USD" }; omitted for Request-a-Quote / N/A / $0 (see price signaling below)
availability	constant	"InStock"
itemCondition	constant	"UsedCondition"
brand	first token of model	e.g. "K-tec PAN 1237 R" → "K-tec"
images	image	Query string stripped so the full asset is fetched; [{ url }] only
metadata.robots	constant	"noindex, nofollow"
custom	model, year, hours, rating, serialNum, usedHotline, location	Domain attributes, preserved in JSON responses
custom.categories	derived from path	Array of category segment(s) between the prefix and the product slug, e.g. ["wt-vibratory-plate-compactors"]
custom.priceType	derived from price	"fixed" | "quote" | "unavailable"
custom.sourcePrice	price	Original raw string (when present)

Price signaling

The schema requires price.final to be a numeric string, so "Request a Quote" cannot live in the price object. Instead every product carries a machine flag in custom so the signal is never lost and the storefront can render the right CTA:

Source price	custom.priceType	price object	custom.sourcePrice
"$409,000"	"fixed"	populated	"$409,000"
"Request a Quote for Pricing"	"quote"	omitted	"Request a Quote for Pricing"
"N/A", "$0", empty	"unavailable"	omitted	original string (if any)

Ingestion behavior

• Uses the bulk endpoint POST /{org}/sites/{site}/catalog/*.
• Batches of up to 50 products (API hard limit).
• Concurrency-limited with an inter-request delay to respect quotas.
• Retries 429 and 5xx responses with exponential backoff (honors Retry-After).
• Reports per-batch status and a final success/failure summary; exits non-zero on any failure.

Notes

• Indexing: for products to appear in an index/feed, an index must exist at or above their path (e.g. POST /index/us/en/products/index.json). This script only ingests products; create the index separately if needed.
• PUT / bulk POST fully replace product data — there is no partial merge.