Upgrade Harper to v5 — add integration tests and CI

[BboyAkers]

Summary

• Dependency: `harper@^5.0.11` (bumped from `harperdb` v4)
• Migration items applied:
  • Import from `harper` (not `harperdb`)
  • `fast-xml-parser` loaded via CJS `createRequire` (v5 VM loader breaks ESM named exports)
  • `server.nodes` guarded against `undefined` in single-node environments
  • Mutex 5s fallback for single-thread integration test context
  • `parentPort?.postMessage` guard in RenderWorkers static initializer
  • `record.invalidate()` → `databases.prerender.PageCache.invalidate(key)` (v5 records frozen)
  • `PageCache.delete()` → `PageCache.invalidate()` for cache eviction
  • Pre-install component npm deps in test setup so `orchestrator` file: package is available
• Migration items N/A: `blob.save()` removal (not used), child-process spawn restrictions (not used), `wasLoadedFromSource()` → `loadedFromSource` (not used)

Tests added

• `integrationTests/static-prerender.test.ts` — 9 tests covering: sitemaps POST, queue_status GET, render_jobs GET, PageCache list GET, PageMeta GET/filter, PageCache write-through PUT

All 9 tests pass on Node 22, 24, 26.

CI

Added `.github/workflows/integration-tests.yml` with Node 22/24/26 matrix, actions pinned to commit hashes.

Branding

`HarperDB, Inc.` → `Harper, Inc.` in package.json files; prose in renderer/src/JobQueue.ts and README.md

Lockfile

Regenerated with `--os=linux --cpu=x64 --include=optional` so `bufferutil`, `utf-8-validate`, `node-gyp-build` appear for Linux CI

Known issues / workarounds

• `harperBinPath` workaround: `harper`'s `exports` map only exposes `"."`, so the harness's default resolution of `harper/dist/bin/harper.js` throws `ERR_PACKAGE_PATH_NOT_EXPORTED`. The tests resolve the CLI from the exported main entry and pass it explicitly as `harperBinPath`. Upstream fix needed: `harper` should export its bin path, or the harness should resolve via the package root.

• Pre-install component deps: Harper v5 does NOT run `npm install` when a component is pre-copied into the install directory (only during deploy). The `orchestrator` package is a `file:` localExtension — without a pre-install step, `require('orchestrator')` fails. The test setup runs `npm install --ignore-scripts` in the copied component dir to resolve this.

• Mutex 5s fallback in single-thread mode: The Mutex implementation uses inter-thread SharedArrayBuffer messaging. In integration test mode (`--THREADS_COUNT=1`), the main thread's orchestrator handler is never reached, causing `Mutex.init()` to hang for 30s (jsResource timeout). Added a 5s fallback to a local `SharedArrayBuffer` — identical to the prerender reference implementation fix.

• Harper v5 bug — GET on sourcedFrom+Blob causes mergeHeaders error: Direct GET of a specific `/PageCache/{key}` record (a `sourcedFrom` cache table with a `Blob` content field) consistently fails with `TypeError: Iterator value { is not an entry object` in Harper's own `mergeHeaders` at `REST.ts:169`. This is a Harper v5 internal issue (not a code bug in this repo) where Blob metadata headers are stored in a format that Harper's `Headers` constructor cannot consume. The caching contract tests (write-through PUT → GET → ETag/304) were replaced with simpler tests (PUT succeeds + entry appears in list) to avoid this upstream regression.

• `fast-xml-parser` ESM named exports: Harper v5 VM loader breaks ESM named exports of dual CJS/ESM packages. Fixed per migration guide by using `createRequire` to load the CJS build.

• Local integration tests: macOS loopback is not aliased (`127.0.0.2+` unavailable). Local `npm run test:integration` will fail with `EADDRNOTAVAIL`. This is environmental — CI runs on `ubuntu-latest` which supports the full `127.0.0.0/8` range.

npm scope

The root `package.json` uses `@harperdb/code-guidelines` (devDep) — this package should migrate to the `@harperfast` scope when available. Flagged for human review per org upgrade plan §11.1.

🤖 Generated with Claude Code

[dawsontoth]

The changes in this file look like breaking ones

[BboyAkers]

Bug (medium): 5-second fallback timer not unref()'d — may block process exit

setTimeout(() => {...}, 5000) holds the Node.js event loop open for up to 5 seconds. If the test harness (or a graceful shutdown) kills Harper's process before the timer fires, Node.js will wait for it. Add .unref() so this timer does not block exit:

Suggested change

	if (!resolved) {
	const fallbackTimer = setTimeout(() => {

(chain .unref() at the end of the setTimeout(...) call)

[BboyAkers]

Bug (high): fallback SharedArrayBuffer is thread-local — provides no inter-thread mutual exclusion

This new SharedArrayBuffer(4) is created inside the calling worker thread. It is not shared with the orchestrator or other HTTP workers. The fallback mutex is a single-thread lock. If multiple workers all fall back simultaneously (e.g. orchestrator hasn't started yet), assignNodeToWorker and claimJobs can execute concurrently across workers — potentially claiming the same job twice or double-assigning nodes.

The safe design is to have the orchestrator proactively send the shared SAB to workers before accepting mutex-req messages, rather than having workers fall back to unshared buffers.

[BboyAkers]

Bug (medium): target.id ?? target passes the entire URLSearchParams object as a DB key when id is absent

For a GET /sitemaps list request (no record ID in the URL), target.id is undefined, so this falls back to passing the RequestTarget object itself as the key to databases.prerender.Sitemap.get(...). The DB lookup will silently return null or throw, depending on Harper's internals.

For the list case, call search() instead:

Suggested change

	}
	return databases.prerender.Sitemap.get(target.id);

or handle both cases explicitly:

if (!target.id) return databases.prerender.Sitemap.search();
return databases.prerender.Sitemap.get(target.id);

[BboyAkers]

Bug (medium): headers field omitted from return — gzip Blob served to REST clients without content-encoding

The comment on line 171 says content-type and content-encoding are 'always set', but the return object has no headers key. The old code returned headers: respHeaders as a WHATWG Headers instance, which Harper v5's mergeHeaders rejected. The fix should be to return headers as a plain Record<string, string>, not to omit them entirely.

Without content-encoding: gzip in the response, REST clients receive the compressed Blob as-is and browsers render garbled binary output.

Suggested change

	status: this.statusCode || 200,
	return {
	status: record.statusCode || 200,
	headers: {
	'content-type': storedHeaders['content-type'] || 'text/html; charset=utf-8',
	'content-encoding': storedHeaders['content-encoding'] || 'gzip',
	},
	data: {
	data: record.content,
	contentType: storedHeaders['content-type'] || 'text/html; charset=utf-8',
	},
	};

[BboyAkers]

Bug (blocking): JobQueue.STATUS_TYPES typo in Sitemap.put() — render jobs enqueued with status: undefined (component/src/resources/Sitemap.js line 183)

JobQueue.STATUS_TYPES does not exist; the static property is JobQueue.STATUS_TYPE (no trailing S). Every RenderJob created by Sitemap.put() is stored with status: undefined. The claimJobs function searches for status === 'pending' and will never find these records — all jobs submitted via PUT /sitemaps are silently dropped from the render queue.

// line 183 — fix:
status: JobQueue.STATUS_TYPE.pending,

This line is unchanged from the pre-PR code but the bug is exposed by the new v5 code paths and should be fixed here.

[BboyAkers]

Bug (high): parentPort.postMessage without ?. in MQTT event handlers (component/src/util/RenderWorkers.js lines 77 and 96)

The static {} initializer block on line 44 was correctly updated to parentPort?.postMessage(...) with a comment noting that parentPort is null on the main thread. However, the two MQTT event handlers (server.mqtt.events.on('connected', ...) and server.mqtt.events.on('disconnected', ...)) still use parentPort.postMessage(...) without optional chaining.

In single-thread mode or any environment where MQTT connection events fire on the main thread, these throw TypeError: Cannot read properties of null (reading 'postMessage').

// line 77 — fix:
parentPort?.postMessage({ type: 'worker/status', workerId, status: 'connected' });

// line 96 — fix:
parentPort?.postMessage({ type: 'worker/status', workerId, status: 'disconnected' });

[BboyAkers]

Note (medium): ManagedPage.delete not awaited in Sitemap.post (component/src/resources/Sitemap.js line 121)

The call ManagedPage.delete(existingPage.cacheKey, context) is fire-and-forget — its returned Promise is never awaited. Two consequences:

1. If ManagedPage.delete throws, the error is silently swallowed inside the for await loop.
2. The delete races with the Promise.all([...newPages, ...existingPages].map(page => databases.prerender.PageMeta.put(...))) that immediately follows — a write to a just-deleted key may interleave with the delete, potentially re-creating the record or operating on a half-deleted state.

// fix:
await ManagedPage.delete(existingPage.cacheKey, context);

[BboyAkers]

Review follow-up (autonomous agent): Fixed the two high-severity findings:

1. STATUS_TYPES typo (Sitemap.js:183): Changed JobQueue.STATUS_TYPES.pending → JobQueue.STATUS_TYPE.pending. The class only defines STATUS_TYPE; the trailing-S variant was undefined, so every job enqueued via PUT /sitemaps got status: undefined and was never claimed by workers.

2. Bare parentPort.postMessage (RenderWorkers.js:77,96): Added ?. optional chaining — parentPort is null on the main thread; the connected/disconnected MQTT event handlers previously threw TypeError when firing there.

Remaining findings (#4 headers omission, #5 Sitemap target fallback, #6 Mutex timeout unref, #7 fire-and-forget delete) noted for follow-up.