Add threads.preload config to preload modules (e.g. APM agents) on worker threads

[kriszyp]

Summary

Adds two threads.* config options that preload module(s) into each worker thread at spawn, before the worker's entry script and any other Harper or app module — the earliest point in a worker's lifetime, early enough for an APM agent (e.g. dd-trace, Dynatrace OneAgent) to instrument all subsequent module loads:

• threads.preload — preloads via Node's --import (ESM / loader-hook registration).
• threads.preloadRequire — preloads via Node's --require (CommonJS).

threads:
  preload: dd-trace/register.js     # --import (string, or a list)
  preloadRequire: dd-trace/init     # --require (string, or a list)

Purpose

Operators want to run APM/instrumentation agents that must load before everything else to monkey-patch modules as they're required. Requiring such a module from within threadServer.js is too late — the worker entry file has already required ~20 Harper modules at its top. Injecting the preload into the worker's execArgv runs it before the entry script's body executes at all.

Why both --import and --require: Harper does its real work on worker threads, so instrumenting those threads is the whole point. Per Datadog, recent dd-trace/init (the --require entry) only instruments the main thread; their register entry loaded via --import dd-trace/register.js is what installs the loader hooks that cover worker threads. So threads.preload uses --import as the primary path. threads.preloadRequire remains for CJS agents that document the --require path and don't need ESM loader hooks (e.g. Dynatrace OneAgent) — where --import-ing a CJS agent would be an unvalidated deviation from their docs.

How it works

• New resolver server/threads/resolvePreload.ts reads a config value (string or array) and resolves each specifier to an absolute path. Bare specifiers (e.g. dd-trace/register.js) resolve against installed components' node_modules — each component directory under the components root is tried first, then the components root and the Harper install as fallbacks — so an agent bundled as a dependency of a deployed component is found. Absolute paths are accepted directly. The resolver is mode-agnostic; a configKey label only tailors warning messages.
• server/threads/manageThreads.js startWorker() pushes --import <file-url> for each threads.preload module and --require <path> for each threads.preloadRequire module into execArgv. --import is URL-based, so resolved paths are passed as pathToFileURL(...).href (a bare absolute path can break --import on Windows). Resolution is memoized once per process (config and installed components are fixed for an APM agent's purpose — it must be present at boot to instrument boot). This covers both HTTP workers (threadServer.js) and job workers (jobProcess.js), since both spawn through startWorker.
• Config plumbing: THREADS_PRELOAD / THREADS_PRELOADREQUIRE terms, Joi validation (string | array of strings | null), defaultConfig.yaml, and the JSON config schema.

Where to look

• resolveSpecifier / getResolutionAnchors in resolvePreload.ts — the resolution order and the fallbacks. Relative-path specifiers (./x) are deliberately rejected: they'd resolve against whichever component anchor matched first (non-deterministic). The contract is an absolute path or a package installed in a component.
• --import file URL in manageThreads.js — the ESM preload path is resolved to an absolute filesystem path, then passed to --import as a file:// URL for cross-platform correctness.
• Bun: preload is skipped under Bun, which does not use this execArgv path (it has its own --preload). Called out in a comment; no Bun support in this PR.

Testing

unitTests/server/threads/resolvePreload.test.js — 10 tests covering bare/subpath/array/absolute resolution, unresolvable-warn, relative rejection, configKey label independence, and that a missing/unreadable components root doesn't throw. Full server/threads + config + validation unit suites pass. Build and lint clean.

Review

Cross-model reviewed (Gemini leg + Harper-domain adjudication). Findings folded in before this opened: memoize resolution (avoid per-spawn readdirSync + repeated warn logs), try/catch the components-root readdir so a missing/unreadable root can't crash worker spawning, reject relative specifiers, and a trace log of resolved preloads. The --import/preloadRequire split was added in response to Datadog's guidance that the register/--import entry is required to instrument worker threads.

Generated with assistance from Claude (Opus 4.8); reviewed and committed by Kris.

Docs

Companion documentation PR: HarperFast/documentation#561

[gemini-code-assist]

Code Review

This pull request introduces a new configuration option, threads.preload, which allows preloading specified modules (such as APM agents) on worker threads via --require before other modules load. It includes schema updates, validation logic, resolution logic in a new file resolvePreload.ts, and corresponding unit tests. The review feedback suggests improving the resolution of absolute paths by leveraging Node's built-in createRequire mechanism instead of a simple existsSync check, which would robustly handle file extensions and directory indexes.

[claude]

Reviewed; no blockers found.

[baileydunning]

Synced with Datadog on this — one thing to add: support preloading via --import (ESM), not just --require. Per their guidance, recent dd-trace/init only instruments the main thread, while the register entry (loaded via --import) is what covers worker threads — which is exactly where we run. So --require dd-trace/init alone likely won't produce worker-thread spans; we'd want the --import/register path too.

Otherwise it looks good on their side and we're clear to test once released. Thanks @kriszyp!

[github-actions]

Patch cherry-pick: merged

Cherry-picked onto v5.1.