Barnacle

Point Barnacle at a site, describe the user flow in plain English, and run three recon commands. Barnacle drives a real browser through the flow, captures every API call, replays them with plain HTTP to prove which ones work without a browser, probes rate-limit ceilings, and then generates a complete plugin — Zod schemas inferred from captured JSON, load-bearing headers, rate-limit ceiling, hot-path HTTP client, and Stagehand browser fallback. Register the plugin in one line; Barnacle handles sessions, retries, fallback routing, audit persistence, and response envelope wrapping.

How it works

The mental model

Stagehand drives a real browser through your described user flow. Its only job is to trigger the site's network traffic — not to extract DOM data. While it clicks, a response listener wiretaps every API call to disk. Once that recon run is done, a separate script replays those captures via plain fetch() — no browser, no AI — to prove the endpoints work standalone. The surviving queries and headers become committed constants. In production, the runtime hits those endpoints directly: fast, cheap, deterministic. The browser only re-engages if the direct path breaks.

A nightly smoke test tells you the moment a contract drifts. When it fires, you re-run the same recon command you ran the first time and diff the captures. Human involvement is one recon run up front and a small PR when things change.

The pipeline at a glance

Phase	What runs	What you get
1 — Browser recon	`pnpm run recon:browser`	Every API call the site makes, captured to `<run-dir>/graphql/*.json`
2–3 — HTTP replay + probing	`pnpm run recon:http`	Proof each endpoint works without a browser; rate-limit ceiling; static fixtures
4 — Plugin generation	`pnpm run recon:generate`	A complete plugin: Zod schemas, headers, Bottleneck config, hot-path client, Stagehand fallback
5+ — Runtime	`pnpm start`	Direct HTTP hot path, automatic browser fallback, nightly smoke test, drift detection

Why this approach

You will get asked why not just use the browser for every request, or scrape HTML, or reverse-engineer endpoints by hand. Here is the honest comparison:

Approach	Cost/req	Latency	Fragile to UI	Fragile to API	Effort
Browser on every request	High	5–15 s	Medium	Low	Low
HTML screen scraper	Low	Low	High	Low	Medium
Manual DevTools recon	Low	Low	Low	High (human redo)	High (ongoing)
HAR replay (mitmproxy)	Low	Low	Medium	High	Medium
Recon → codify → direct HTTP + fallback (this)	Low	Low	Low (re-runnable)	Low (fallback covers)	Medium, front-loaded

The browser-on-every-call approach uses Steel minutes and Anthropic tokens on every production call — orders of magnitude more expensive at scale. HTML scrapers break on every UI redesign, and the API response usually carries richer data than what the UI renders anyway. Manual DevTools recon is exactly what this pipeline automates, but committed and re-runnable. Front-loaded recon work buys an integration as cheap as direct HTTP, as robust as a browser fallback, and maintainable in a way none of the hand-rolled options are.

Adding a New Site — The Recon Playbook

Every new site follows the same pipeline (Phases 0–6). The only human-authored input is the flow definition you write once in Phase 0. After that, the scripts run unattended — recon captures, HTTP replay proves endpoints, the generator writes the plugin. When the site changes months later, you re-run the same command and diff the captures. Human time is front-loaded to one recon run and a small PR.

Phase 0 — Define the user flow

Commit the flow steps to a file first — this makes recon re-runnable in one command without retyping it. When the site changes and you need to re-run recon months later, you git pull and run the same command you ran the first time:

# src/sites/my-site/recon-flow.json
["click the Electronics category filter", "open the first product result"]

Phase 1 — Run the browser recon harness

# Preferred: load flow from committed file
pnpm run recon:browser -- \
  --url https://example.com \
  --flow-file src/sites/my-site/recon-flow.json

# Or inline (ephemeral — must be re-typed each recon run):
pnpm run recon:browser -- \
  --url https://example.com \
  --flow '["click the Electronics category filter", "open the first product result"]'

# For sites whose API paths don't match /graph, /api/, /graphql, /v1/, or *.json:
pnpm run recon:browser -- \
  --url https://example.com \
  --flow-file src/sites/my-site/recon-flow.json \
  --capture-all

# Capture page-load XHRs only (no interaction — useful for pure GET-style SPAs):
pnpm run recon:browser -- --url https://example.com

Drives a real Stagehand + Steel browser through your flow. Captures are wired via a single CDP session-level listener (page.getSessionForFrame().on(...)) — Stagehand V3 enables the Network domain internally, so attaching our Network.requestWillBeSent / responseReceived / loadingFinished listeners on the main session catches every response, including the early ones that fire before any page-level handler could be wired.

Captures every network call matching /graph, /api/, /graphql, /v1/, or *.json to <run-dir>/graphql/<NNN>-<phase>-<operationName>.json — one file per call, diffable and greppable. Use --capture-all for sites with non-standard API paths; it captures every response, producing more noise but missing nothing. Omitting both --flow and --flow-file runs zero interaction steps and captures only the network activity that fires during page navigation — useful for pure GET-style SPAs that fetch everything they need on load.

Each step runs through a self-healing cascade (act → observe + act → observe + act with ignoreSelectors → Anthropic-SDK rephrase) verified by network-counter delta or URL change. The script's main() attempts up to two global flow replans before giving up; terminal failures dump a diagnostic bundle to <run-dir>/step-failures/. See docs/playbook.md#1c--self-healing-cascade for the full design.

Total runtime: 20–40 minutes for a typical flow (longer if healing or replans fire), fully unattended.

Every artifact — captures, cookie-jar snapshots, step-failure dumps, DOM dumps — is rooted under one run-scoped directory resolved once at startup: <run-dir> defaults to /tmp/recon/<runId>, where <runId> is a timestamp + random suffix generated per process (e.g. 20260718-120326-a1b2). Set RECON_RUN_ID to pin a deterministic runId (e.g. for tests or replaying a known run) and RECON_OUT_DIR to override the base directory runs are rooted under. This keeps concurrent or repeated runs from intermixing files — the startup log line prints both the resolved runId and out=<run-dir>.

Cookie-jar snapshots

Alongside the network captures, every run snapshots the browser's complete cookie jar (via CDP Network.getAllCookies, which returns the whole-browser jar regardless of the current page's URL — unlike document.cookie or Page.getCookies, it also sees HttpOnly cookies) at each phase boundary: the initial goto, immediately before each flow step (pre-step), immediately after each flow step completes (post-step), and once more at run completion (run-complete).

Snapshots land in <run-dir>/cookies/<NNN>-<label>-<phase>.json — one file per boundary, using the same zero-padded chronological index convention as the network captures. <label> is the boundary kind (goto, pre-step, post-step, run-complete); <phase> is the current step's slugified instruction (e.g. click-the-apply-button), or home before the first step starts.

Each file is a JSON object:

{
  "label": "post-step",
  "phase": "click-the-apply-button",
  "stepIndex": 2,
  "timestamp": "2026-07-18T12:34:56.789Z",
  "cookies": [
    {
      "name": "_acme_attr",
      "value": "abc123",
      "domain": ".acme.example",
      "path": "/",
      "expires": 1234567890,
      "size": 20,
      "httpOnly": true,
      "secure": true,
      "session": false,
      "sameSite": "Lax"
    }
  ]
}

Field reference (mirrors CDP's Network.Cookie type verbatim — no remapping between capture and disk):

Field	Meaning
`name` / `value`	The cookie's name and value.
`domain`	Scope, e.g. `.acme.example` (all subdomains) vs. `apply.acme.example` (exact host) — the detail needed to tell a click-domain cookie from an apply-domain cookie.
`path`	Cookie path scope.
`expires`	Raw CDP epoch-seconds number; `-1` means a session cookie (also reflected in `session: true`). Not reformatted — read it as CDP reports it.
`size`	Cookie size in bytes, as reported by CDP.
`httpOnly` / `secure`	Standard cookie flags.
`session`	`true` for a session cookie (no persistent expiry).
`sameSite`	`"Strict" \| "Lax" \| "None" \| null` — `null` when the cookie doesn't set the attribute.

If the CDP call fails, the file still writes but with an empty cookies array and an error string field carrying the failure message — cookie telemetry is best-effort and never aborts the run.

Diffing what a phase established: to isolate what a specific traversal (e.g. a tracking-click redirect) minted, diff its post-step snapshot against the pre-step snapshot for the next step — cookies present in the later file but absent from the earlier one were established during that step:

diff <(jq -S .cookies <run-dir>/cookies/004-post-step-click-the-apply-button.json) \
     <(jq -S .cookies <run-dir>/cookies/005-pre-step-fill-in-your-name.json)

Cookies actually sent per request: the jar snapshot shows what's available, not what's sent. Each network capture in <run-dir>/graphql/ separately carries the outgoing Cookie header in its requestHeaders (recovered via CDP's Network.requestWillBeSentExtraInfo, since requestWillBeSent omits it by design) — cross-reference that capture's requestHeaders.Cookie against a jar snapshot to see which of the available cookies a given request, e.g. the application submit, actually sent.

Caveat on Set-Cookie: response captures fold responseReceivedExtraInfo headers (which is where Set-Cookie actually appears — responseReceived omits it) into responseHeaders as a flat Record<string, string>. CDP does not guarantee multiple Set-Cookie values on one response stay distinguishable once folded into that shape — if a single response mints more than one cookie, treat the jar snapshot (not the response capture's Set-Cookie header) as the source of truth for what actually landed.

Phase 2–3 — Replay and probe

pnpm run recon:http

Replays every capture via plain fetch() — no browser, no AI — to prove endpoints work standalone. Every replay returning 200 proves the browser is unnecessary for production. Also runs GraphQL introspection, auxiliary fixture detection (static JSON to commit as fixtures), and a rate-limit probe at 1→3→5 rps (run last — if it triggers a ban, all captures are already saved). Results land under the run-scoped root resolved by resolveReconRunDir() — /tmp/recon/<runId>/replays/ by default, rooted elsewhere via --out-dir <path> or RECON_OUT_DIR.

See docs/playbook.md for the full troubleshooting decision matrix when replays fail.

Phase 4 — Generate the plugin

pnpm run recon:generate -- --site-id my-site

Reads every artifact from Phases 1–3 — <run-dir>/graphql/*.json (captures), <run-dir>/replays/*.json (replay results), <run-dir>/replays/rate-limit.json (probe findings), <run-dir>/aux/*.json (static fixtures), and src/sites/my-site/recon-flow.json — and writes a complete plugin to src/sites/my-site/. Pass --run-dir <path> to read a specific run's artifacts instead of the most recently modified run root under /tmp/recon (or RECON_OUT_DIR, if set):

contract.ts — Zod schemas inferred from captured JSON, load-bearing headers, Bottleneck ceiling, and executeHttp / execute implementations
flows/browser-flow.ts — Stagehand fallback wired to your recon-flow.json steps
index.ts — barrel export
fixtures/ — any static JSON found by the auxiliary probe, already copied in

Then review the generated files: trim UI-only fields from the GraphQL query, narrow any z.unknown() entries in the schema you care about, and verify the headers. If you need to regenerate after making changes to the recon flow, pass --force.

Telling the generator which fields carry your caller's data

Barnacle cannot know what your site's forms mean. "Select the departure port from the Country dropdown" and "…select the test candidate's country" are the same sentence shape; only you know that the first is a search facet and the second is your caller's address. So the vocabulary is yours to supply, with --vocabulary:

// src/recon/my-vocabulary.ts
import type { ReconVocabulary } from "@enricai/barnacle/recon/vocabulary";

export const vocabulary: ReconVocabulary = {
  // Only a step naming the SUBJECT may splice off a bare dropdown — a dropdown
  // step carries no quoted constant, so the label alone cannot tell your
  // caller's data from a filter that happens to say "Country".
  subject: /\b(the\s+)?(test\s+)?customer'?s\b/i,
  // Labels that must stay literal even when a table row matches.
  exclusions: [/\bbilling contact\b/i],
  // Ordered label → payload field. First match wins, so specific precedes broad.
  table: [
    [/\bfirst name\b/i, "FirstName"],
    [/\bcity\b/i, "City"],
  ],
};

pnpm run recon:generate -- --site-id my-site --vocabulary ./src/recon/my-vocabulary.ts

The specifier follows the same rule as BARNACLE_PLUGINS: a leading . or / is a filesystem path; anything else resolves from your node_modules. The module may export vocabulary or a default.
--vocabulary none for a site that splices no caller data at all (read-only inventory, search, pricing). Explicit, so it can't happen by accident.
Every regex must be free of the g/y flags and every field name must be a valid JS identifier — the loader rejects both. A stateful regex matches only every other step; a non-identifier emits payload.<name> as a syntax error.
Run the generator with a .ts vocabulary under tsx, or point it at compiled .js — plain node cannot import TypeScript before v22.18.

Deprecated fallback: omit --vocabulary and the generator falls back to a built-in recruiting table (first/last name, email, phone, address), warning as it does. That table is removed in 2.0.0, after which an absent vocabulary on a spliceable flow is an error. Supply one now.

Telling the generator your ATS's form-schema wire keys

Where --vocabulary matches instruction prose, --form-schema names the JSON keys the generator reads out of an ATS's form-definition responses when recovering field ids, option ids, and submitted values. The engine ships no vendor's format; a site whose ATS exposes a form definition declares its keys with --form-schema:

// src/recon/my-form-schema.ts
import type { ReconFormSchema } from "@enricai/barnacle/recon/form-schema";

export const formSchema: ReconFormSchema = {
  fieldIdKey: "fieldId",                    // UUID-valued field identity
  fieldNameKeys: ["code", "label"],         // code, then human label — code preferred
  fieldOptionsKey: "options",
  optionIdKey: "optionId",                  // option id, inside the options array
  optionValueKey: "optionLabel",            // option label, inside the options array
  responseValueKey: "submittedValue",       // submitted free value
  responseOptionIdKey: "submittedOptionId", // submitted option reference
};

pnpm run recon:generate -- --site-id my-site --form-schema ./src/recon/my-form-schema.ts

The specifier follows the same rule as --vocabulary: a leading . or / is a filesystem path; anything else resolves from your node_modules. The module may export formSchema or a default.
--form-schema none for a site with no ATS form definition (a search API, a cruise site) — same as omitting it. "none" is the explicit form.
Wire keys anchor "key":"uuid" markers, so they may be any non-empty string without a quote or backslash — the JS-identifier rule that governs vocabulary field names does not apply here.
fieldNameKeys models two roles: the first key is a machine code (PascalCased directly), the second is a human label (run through the section-heading heuristic). Supply one key for a label-only ATS, or two for one that exposes both. Additional keys are unused.
Omit --form-schema (or pass none) and ATS form-key recovery does not run — the engine hardcodes no vendor's wire format. A site whose ATS exposes a form definition must supply one to recover its option fields. See issue #57.

Mapping the site's screening questions

If the site asks screening questions, tell the generator which payload field answers each one — the same reasoning applies, it cannot know what your site asks:

Env var	Default	Description
`RECON_QUESTION_KEYWORDS`	`{}`	JSON object mapping a payload field name to the keywords that identify its question. A question must match at least 2 keywords to map.
`RECON_TELEMETRY_URL_PATTERNS`	(empty)	Comma-separated extra URL fragments to treat as analytics noise, on top of the built-in list. Put your site's trackers here.

RECON_QUESTION_KEYWORDS='{
  "VisaSponsorship":       ["visa", "sponsor"],
  "RelatedToEmployee":     ["related", "employee"],
  "CanPerformJobFunctions":["perform", "job functions", "duties"]
}' pnpm run recon:generate -- --site-id my-site

Any question that matches no field is logged by prompt rather than skipped — an unmapped required question is how a generated plugin ends up submitting nothing for it. Read those warnings and either add keywords or accept that the question goes unanswered. Malformed JSON logs a warning and is treated as empty, so a typo cannot kill a recon run mid-flight.

Optionally generate the human-readable findings doc alongside:

pnpm run recon:summarize -- --site-id my-site

Writes docs/my-site-recon.md with: endpoints found, replay status, rate-limit ceiling, header frequency table, and hazards (Akamai, Cloudflare). Without --site-id, the default output path is docs/target-recon.md. Accepts --run-dir <path> the same way recon:generate does.

Phase 5 — Register the plugin

See Plugin Authoring Guide → Register the plugin below for plugin registration options.

Phase 6 — Wire up drift detection

See Plugin Authoring Guide → Wire up the nightly smoke test below for the CI step that runs the smoke test nightly.

See docs/playbook.md for the full detection ladder and maintenance loop.

The whole loop, in one picture

The dashed deploys edge is the human-in-the-loop step (the contract PR merges and ships to Runtime). The solid orange edge from smoke-test.ts back into Phase 1 is the self-healing loop: when the contract drifts, recon reruns unattended (~20–40 min) and the next PR is a diff of captures, not a hand-rewrite. See docs/architecture.md for the design rationale behind each lane.

Plugin Authoring Guide

A site plugin is a single TypeScript module that satisfies SitePlugin<TInput, TOutput> from src/site-plugin.ts. Core registers built-in plugins via BUILTIN_SITE_PLUGINS in src/plugins/discover.ts; out-of-tree plugins are loaded at startup via BARNACLE_PLUGINS.

The SitePlugin interface

interface SitePlugin<TPayload, TResult> {
  meta: SitePluginMeta;
  // Optional direct-HTTP hot path — no browser, no LLM tokens, millisecond latency.
  // Core tries this first; falls back to execute() on HttpSchemaError / HttpBotChallengeError / HttpServerError.
  executeHttp?: (
    payload: TPayload,
    context: SitePluginContext
  ) => Promise<SitePluginResult<TResult>>;
  // Browser fallback — Stagehand + Steel session, acquired from the pool by core.
  execute(
    payload: TPayload,
    session: BrowserSession,
    context: SitePluginContext
  ): Promise<SitePluginResult<TResult>>;
  // Async work is supported. Note: NOT called on CaptchaError or EmptyResultsError —
  // p-retry skips onFailedAttempt for AbortError, so those abort paths bypass this hook.
  onRetry?: (error: ScraperError, attempt: number) => void | Promise<void>;
}

SitePluginMeta — required fields

Field	Type	Purpose
`siteId`	`string`	Stable key used for routing (`/v1/<siteId>/run`) and audit rows
`displayName`	`string`	Human-readable label for logs and Swagger docs
`bodySchema`	`ZodTypeAny`	Request body schema — core validates before calling `execute()`
`responseSchema`	`ZodTypeAny`	Success response schema — drives Swagger output shape
`routeOverride?`	`string`	Override the full route path (legacy compatibility only)
`defaultBaseUrl?`	`string`	Fallback base URL when `config.scraper.siteBaseUrls[siteId]` is absent
`taskTimeoutMs?`	`number`	Override the pool's 60-minute per-task hang ceiling for this plugin only — set when the site's normal latency is well below the default and a faster failure is preferable
`apiVersion?`	`string`	Semver range targeting a plugin API version (e.g. `"^1.0.0"`); core disables the plugin on a major-version mismatch. Absent means "accept any version."
`extraRoutes?`	`readonly SitePluginExtraRoute[]`	Extra non-run routes (OTP trigger, resume, etc.) that core registers as authenticated Fastify routes at startup. See `SitePluginExtraRoute` in `src/site-plugin.ts`.
`onShutdown?`	`() => Promise<void>`	Optional cleanup for background work the plugin launched fire-and-forget, awaited during graceful shutdown so in-flight work is not abandoned and sessions are not leaked. Mirrors the engine's own drain functions. Bounded by a per-plugin timeout, so a hanging drain cannot stall shutdown. Module plugins only — config-only `*.plugin.json` manifests are pure JSON and cannot declare a function.

Full plugin skeleton (hot path + browser fallback)

pnpm run recon:generate produces this structure automatically. Use createRateLimitedJsonClient() for REST endpoints that send Chromium client-hint headers (the common case) and createGraphqlClient() for GraphQL endpoints — recon:generate selects the right one based on what it captured. The skeleton below illustrates the REST hot-path pattern; for GraphQL sites, recon-generate uses createGraphqlClient instead and inlines the captured query as a constant.

// src/sites/my-site/contract.ts
import { z } from "zod/v4";
import { createRateLimitedJsonClient } from "@/scraper/rate-limited-json-client";
import type { BrowserSession } from "@/scraper/session";
import type { SitePlugin, SitePluginContext, SitePluginResult } from "@/site-plugin";
import { runMySiteBrowserFlow } from "@/sites/my-site/flows/browser-flow";

// Generated: Zod schemas inferred from captured JSON — tighten z.unknown() fields as needed.
const MySiteResponseSchema = z.object({ data: z.object({ items: z.array(z.object({ id: z.string() })) }) });
const MySitePayloadSchema = z.object({ query: z.string().min(1) });

type MySitePayload = z.infer<typeof MySitePayloadSchema>;
type MySiteResponse = z.infer<typeof MySiteResponseSchema>;

// Generated: rate-limit ceiling (5 rps) + Chromium hints + site-specific headers from recon.
// Use createHttpClient() directly only when you need manual Bottleneck or header control.
const httpClient = createRateLimitedJsonClient({
  minTimeMs: 200,
  userAgent: "Mozilla/5.0 ...",
  secChUa: '"Chromium";v="..."',
  platform: "Linux",
  extraHeaders: {
    "Content-Type": "application/json",
    Accept: "application/json, */*",
  },
  schema: MySiteResponseSchema,
});

export const mySitePlugin: SitePlugin<MySitePayload, MySiteResponse> = {
  meta: {
    siteId: "my-site",
    displayName: "My Site",
    bodySchema: MySitePayloadSchema,
    responseSchema: MySiteResponseSchema,
    defaultBaseUrl: "https://my-site.com",
  },
  // Hot path: direct HTTP — no browser, no LLM tokens.
  async executeHttp(payload: MySitePayload, context: SitePluginContext): Promise<SitePluginResult<MySiteResponse>> {
    const data = await httpClient(`${context.baseUrl}/api/search`, {
      method: "POST",
      body: JSON.stringify({ query: payload.query }),
    });
    return { data };
  },
  // Browser fallback: Stagehand + Steel — invoked automatically when hot path fails.
  async execute(payload: MySitePayload, session: BrowserSession, context: SitePluginContext): Promise<SitePluginResult<MySiteResponse>> {
    const raw = await runMySiteBrowserFlow(session.stagehand, context.baseUrl, payload.query);
    return { data: raw };
  },
};

The auditPayload hook

SitePluginResult accepts an optional auditPayload field alongside data:

return {
  data: responseData,
  auditPayload: { query: payload.query, resultCount: responseData.items.length },
};

When auditPayload is present, core writes it — not data — to the submission-envelope telemetry record. Use this to strip PII or large blobs from the audit trail while keeping the full response in the API reply. When absent, data is written as-is.

Static fixtures

If Phase 3b (auxiliary fixture detection) found static JSON endpoints (markets, currencies, labels), recon:generate copies them to src/sites/<id>/fixtures/. Load them at module init via loadFixture() — zero per-request overhead, fails fast on deploy if the fixture is missing or stale:

import { z } from "zod/v4";
import { loadFixture } from "@/scraper/fixtures";

const MarketsSchema = z.array(z.object({ id: z.string(), name: z.string() }));

// Loaded synchronously at module init. Throws at startup if file is missing
// or shape drifted — surface fixture breakage on deploy, not on the first request.
const markets = loadFixture("my-site", "markets.json", MarketsSchema);

See docs/playbook.md — Phase 3b for how fixtures are detected and when to use them.

Register the plugin

Out-of-tree (recommended for operator-owned plugins): point BARNACLE_PLUGINS at the compiled plugin module — no core edits required:

BARNACLE_PLUGINS=./plugins/my-site/dist/index.js pnpm start

Barnacle validates the export at startup and registers POST /v1/my-site/run automatically. See the Out-of-tree plugins env var table for BARNACLE_PLUGINS_STRICT and BARNACLE_PLUGINS_DIR. A copyable, runnable template lives at examples/plugins/hello-site/.

Config-only (no TypeScript, no compile step): a browser-flow plugin can be a single JSON manifest. Point BARNACLE_PLUGINS at a *.plugin.json file, or drop manifests into a directory named by BARNACLE_PLUGINS_CONFIG_DIR:

BARNACLE_PLUGINS=./plugins/acme-jobs.plugin.json pnpm start
# or, for directory-drop discovery of every *.plugin.json:
BARNACLE_PLUGINS_CONFIG_DIR=./plugins pnpm start

The manifest wears the Kubernetes-style apiVersion / kind / metadata / spec envelope, declares its request/response/extract shapes as JSON Schema, and lists the browser flow as data (the same self-heal step format the recon toolchain authors). Core reads it at startup and registers POST /v1/acme-jobs/run — no per-site code. A site needing the direct-HTTP hot path can reference a compiled executeHttp module via spec.httpModule. A copyable manifest lives at examples/plugins/acme-jobs.plugin.json.

The JSON Schema converter accepts a deliberately small subset — object, string, number, integer, boolean, array (with items), string enum, and required — and rejects anything else (e.g. pattern, minLength, $ref, format constraints) at load time. Flow steps interpolate request values with {{ .request.FieldName }}; a reference to a field the request schema does not declare fails loudly, while an optional declared field the caller omits splices as an empty string.

In-tree (bundled built-ins only): push to BUILTIN_SITE_PLUGINS in src/plugins/discover.ts:

import { mySitePlugin } from "@/sites/my-site";
import { BUILTIN_SITE_PLUGINS } from "@/plugins/discover";

BUILTIN_SITE_PLUGINS.push(mySitePlugin as SitePlugin<unknown, unknown>);

SITE_PLUGINS in src/plugins/loader.ts is an alias for BUILTIN_SITE_PLUGINS kept for backwards compatibility; prefer the discover.ts import for new code.

Core registers POST /v1/my-site/run automatically at startup.

Wire up the nightly smoke test

Add a step to .github/workflows/smoke.yml:

- name: Run smoke test — my-site
  if: steps.check-secrets.outputs.skip == 'false'
  run: |
    pnpm run smoke -- \
      --site my-site \
      --payload '{"query":"test"}' \
      --host "$SMOKE_HOST" \
      --fallback \
      --response-schema src/sites/my-site/contract.ts
  env:
    API_KEY: ${{ secrets.SMOKE_API_KEY }}
    SMOKE_HOST: ${{ secrets.SMOKE_HOST }}
    NODE_ENV: production

--response-schema points to a module whose default export is a Zod schema. The smoke test validates the full response body against it — not just the envelope shape — so any schema drift on the data payload fails the pipeline immediately.

--fallback additionally runs a second request via the Stagehand browser path. This catches Stagehand cache staleness: if the page DOM changed and the cached selector now points at the wrong element, the hot-path test passes but the fallback test fails — alerting you before the fallback is invoked in production.

Maintenance loop

When the smoke test fails: re-run pnpm run recon:browser → diff <run-dir>/graphql/*<operationName>*.json against src/sites/<id>/contract.ts → update query / headers / Zod schema → ship. See docs/playbook.md for the full maintenance loop and change severity table.

Runtime internals

Hot-path fallback triggers

dispatch() (src/plugins/loader.ts) tries executeHttp() first. Which errors trigger the browser fallback and which don't:

Hot-path error	Status	Triggers browser fallback?	Reason
`HttpSchemaError`	Any	Yes	Response shape drifted; browser may still work
`HttpBotChallengeError`	401 / 403	Yes	Residential proxy IP may get through
`HttpServerError`	5xx	Yes	Server-side outage; recovery strategy is the same
`HttpRateLimitError`	429	No	A 429 means the configured rps ceiling is too high. Routing to the browser path would just hit the same ceiling and waste a Steel session. The right response is to lower the Bottleneck `minTime` in `contract.ts` and re-deploy.
`HttpUrlLockedError`	429	No	A plugin's `classifyResponseBody` detected a terminal resource-lock sentinel — the URL is locked at the target's end. Neither a retry nor a browser session can succeed; the caller must back off and surface a "retry later" state.
`UnknownScraperError`	Any	No	Transient network failure or unclassified non-JSON response. `createHttpClient` retries up to 2 times internally; if all attempts fail, the error propagates as `ScrapeFailureError`.

Cache deduplication

getCachedResponse() checks the LRU cache first. On a miss, getOrCreateInFlight() registers a promise in an inFlight map before awaiting it — meaning concurrent identical requests all await the same upstream call rather than fanning out. First caller wins; all others coalesce onto its promise.

Cache key: <endpoint>:<sha256(canonical payload)[:32]> — the endpoint is a literal prefix; the hash covers only the canonical payload. Object key order and primitive array element order are normalized so {a:1,b:2} and {b:2,a:1} hit the same entry. Default TTL: 15 minutes (CACHE_TTL_MS). Max entries: 1000 (CACHE_MAX_ENTRIES). Only successful responses are cached; errors propagate and never poison the cache.

Session pool and timeouts

runWithSession() (src/scraper/pool.ts) queues tasks through a p-queue bounded by SESSION_POOL_SIZE (default: 3). Sessions are created on demand — not pre-warmed — so Steel billing stays proportional to actual traffic.

Per-task hang ceiling: each queued task races against TASK_TIMEOUT_MS (src/scraper/pool.ts, 60 minutes by default). A hung execute() — frozen CDP connection, infinite network wait — converts to SessionTimeoutError, which the retry policy handles by tearing down the broken session and creating a fresh one. The default is sized for long browser flows; shorten per-plugin via SitePluginMeta.taskTimeoutMs. This is a hang-recovery floor, not a p99 latency budget.

Retry policy: withScraperRetry (src/scraper/retry.ts) uses p-retry with factor: 2, minTimeout: 500ms, maxTimeout: 5000ms, randomize: true, and default maxAttempts: 3. EmptyResultsError and abort signals short-circuit retries; SessionTimeoutError triggers a one-time session restart between attempts.

Graceful shutdown: drainPool() is called during graceful shutdown — SIGTERM/SIGINT triggers app.close(), which fires Fastify's onClose hook, which calls drainPool(). It pauses new intake, waits up to 20 seconds for in-flight tasks to close their Steel sessions, then resolves. Without this, process exit leaves live sessions billing until Steel's own timeout.

Viewport rotation

createBrowserSession() (src/scraper/session.ts) picks a random desktop viewport per session from: 1280×720, 1366×768, 1440×900, 1920×1080. A fixed pixel size is an easy bot-detection fingerprint; rotating it makes sessions harder to cluster.

LLM routing: Anthropic vs. AWS Bedrock

By default, Stagehand calls the Anthropic API directly (ANTHROPIC_API_KEY). Set USE_BEDROCK=true to route through AWS Bedrock instead:

USE_BEDROCK=true
AWS_REGION=us-east-1
AWS_ACCESS_KEY_ID=...
AWS_SECRET_ACCESS_KEY=...
BEDROCK_MODEL=us.anthropic.claude-sonnet-4-6[1m]   # default

The [1m] suffix selects the 1-million-token context variant on Bedrock. Both paths run Stagehand with serverCache: true (server-side action cache to skip LLM inference on replay) and selfHeal: false (recon-browser owns its own verify-and-retry cascade; see src/scraper/session.ts for the rationale).

When using Anthropic directly (not Bedrock), the model is controlled by STAGEHAND_MODEL (default: anthropic/claude-sonnet-4-6).

Observability

GET /readyz returns readiness status plus per-site drift-detection metrics exposed by src/scraper/metrics.ts:

{
  "status": "ready",
  "checks": {
    "database": { "ok": true },
    "scraperCredentials": { "ok": true },
    "scraperPool": { "ok": true, "detail": "depth=0" }
  },
  "stats": {
    "scraperPool": { "size": 0, "pending": 0, "concurrency": 3 },
    "cache": { "size": 12, "max": 1000, "inFlight": 0 },
    "metrics": {
      "my-site": {
        "hotPathSuccess": 4821,
        "fallbackActivations": 3,
        "rateLimitRejections": 0,
        "p95LatencyMs": 187
      }
    }
  },
  "telemetry": {
    "currentRunFile": "/path/to/.barnacle/events/run-123.ndjson",
    "currentRunFileSizeBytes": 4096,
    "orphansRecovered": 0
  },
  "heal": {
    "my-site": { "verdict": "SUCCESS", "bestPassRate": 0.95, "reportPath": "heal-out/my-site/healing-my-site.md" }
  }
}

What rising fallbackActivations means: the hot path is failing and the browser fallback is absorbing traffic. Cost and latency rise while error rate stays flat — users don't notice yet, but you will on your bill. This is your signal to re-run recon.

p95LatencyMs is reservoir-sampled (Vitter's Algorithm R, capped at 1000 samples) over actual upstream round-trips. Cache hits are excluded — they're memory reads and must not bias the upstream latency signal.

See docs/playbook.md for the full detection ladder.

NDJSON telemetry files

Barnacle writes two append-only NDJSON files alongside its metrics:

File	Default path	Purpose
LLM call samples	`.barnacle/calls.ndjson`	One line per LLM/Stagehand call; feed to the `judge:llm` and `slm-self-heal` skills
Run event stream	`.barnacle/events/<runId>.ndjson`	Per-run event stream written by the event-stream subsystem; path surfaced in `/readyz` `telemetry.currentRunFile`

LLM call sample schema

Every line in .barnacle/calls.ndjson is a JSON object with these fields (source: src/api/schemas/telemetry.ts):

Field	Type	Description
`callId`	`string`	UUID generated per call
`callType`	`string`	Which LLM call site produced this sample — see table below
`model`	`string`	Model identifier string passed to the SDK
`systemPrompt`	`string \| null`	System-prompt text, or `null` when absent
`userContent`	`string`	Full user-turn content
`responseContent`	`string \| null`	Raw response text, or `null` on SDK error
`parsedOk`	`boolean`	Whether the response was successfully parsed into the expected schema
`inputTokens`	`number \| null`	Input token count from SDK usage metadata
`outputTokens`	`number \| null`	Output token count from SDK usage metadata
`latencyMs`	`number \| null`	Wall-clock latency of the SDK call in milliseconds
`success`	`boolean`	Whether the call site considered the call successful end-to-end
`ts`	`string`	ISO-8601 timestamp at write time

Call types

callType is a stable string constant defined in src/lib/telemetry/call-types.ts:

`callType`	Source	When emitted
`recon-rephrase`	`src/scripts/recon-browser.ts`	Attempt-4 rephrase inside the recon-browser step-healing cascade — Anthropic SDK is asked to reword the failing step
`recon-replan`	`src/scripts/recon-browser.ts`	Global replan after a step terminally fails — Claude rewrites the remaining flow tail
`recon-flow-patch`	`src/scripts/recon-heal.ts`	Patch proposal from the recon-flow-patch-generator during the `recon-heal` self-healing loop
`llm-prompt-patch`	`src/scripts/llm-heal.ts`	Patch proposal from the llm-call-patch-generator during the `llm-heal` self-healing loop

Tailing call samples with jq

# Stream all LLM call samples as they arrive
tail -f .barnacle/calls.ndjson | jq '.'

# Filter to a specific call type
tail -f .barnacle/calls.ndjson | jq 'select(.callType == "recon-rephrase")'

# Show only failures
tail -f .barnacle/calls.ndjson | jq 'select(.success == false) | {callId, callType, latencyMs}'

# Token usage summary by call type
jq -s 'group_by(.callType) | map({callType: .[0].callType, totalInputTokens: map(.inputTokens // 0) | add, totalOutputTokens: map(.outputTokens // 0) | add, n: length})' .barnacle/calls.ndjson

# Tail the current run event stream (path from /readyz telemetry.currentRunFile)
tail -f .barnacle/events/<runId>.ndjson | jq '.'

Environment variables

All variables are read once at process start. Required variables cause the process to exit on missing values; optional ones have safe defaults.

Application

Variable	Default	Required	Purpose
`APP_NAME`	`barnacle`	No	Application name used in logs
`NODE_ENV`	`development`	No	`development` / `production` / `test`
`PORT`	`3000`	No	HTTP listen port
`HOST`	`0.0.0.0`	No	HTTP listen address
`LOG_LEVEL`	`info`	No	Pino log level (`debug`, `info`, `warn`, `error`)

Auth

Variable	Default	Required	Purpose
`API_KEYS_HASHED`	`""`	Yes (prod)	Comma-separated bcrypt hashes of plaintext bearer tokens. See Generating an API key.
`DEV_BYPASS_AUTH`	`false`	No	Skip auth entirely. Local dev only — never set in production.

Browser automation (Steel + Stagehand)

Variable	Default	Required	Purpose
`STEEL_API_KEY`	—	Yes	Steel account API key. Required for all browser automation.
`ANTHROPIC_API_KEY`	—	Yes (if not using Bedrock)	Anthropic API key for Stagehand's LLM calls.
`STAGEHAND_MODEL`	`anthropic/claude-sonnet-4-6`	No	Stagehand model. Use the `anthropic/` prefix — Stagehand 2.x's model map is stale and the prefix routes through AI-SDK's fallback path.
`SCRAPER_PROXY_TYPE`	`residential`	No	`residential` (paid Steel tiers) or `none` (free tier — Steel rejects `useProxy=true` on hobby plans).
`SCRAPER_SOLVE_CAPTCHA`	`true`	No	Enable Steel's built-in CAPTCHA solver. Requires a paid plan; set `false` on the free tier.
`SESSION_POOL_SIZE`	`3`	No	Maximum concurrent Steel browser sessions.
`SCRAPER_MIN_ACTION_DELAY_MS`	`500`	No	Minimum delay between scraper actions (ms). Jitter applied on top.
`SCRAPER_MAX_ACTION_DELAY_MS`	`1500`	No	Maximum delay between scraper actions (ms).
`STAGEHAND_API_TIMEOUT_MS`	`120000`	No	Anthropic SDK request timeout (ms). Raise on slow network paths to `api.anthropic.com`.
`STAGEHAND_CONNECT_TIMEOUT_MS`	`120000`	No	TCP connect timeout for all outbound fetch calls (ms). Raised from the undici default of 10 s to match `STAGEHAND_API_TIMEOUT_MS`.
`STEEL_SESSION_TIMEOUT_MS`	`3600000`	No	Steel session wall-clock timeout (ms). Default is 1 hour; lower on plans that enforce shorter maximum session durations.

AWS Bedrock (alternative LLM provider)

Set USE_BEDROCK=true to route Stagehand's LLM calls through AWS Bedrock instead of the Anthropic API. When enabled, ANTHROPIC_API_KEY is not needed. AWS credentials resolve in standard SDK order: explicit vars → ECS task role → EC2 instance profile → ~/.aws/credentials.

Variable	Default	Required	Purpose
`USE_BEDROCK`	`false`	No	Master switch — routes LLM calls through Bedrock when `true`.
`AWS_REGION`	`us-east-1`	No	AWS region for Bedrock calls.
`AWS_ACCESS_KEY_ID`	—	No	Explicit AWS access key (leave blank for ambient IAM).
`AWS_SECRET_ACCESS_KEY`	—	No	Explicit AWS secret key.
`AWS_SESSION_TOKEN`	—	No	Required only for temporary STS credentials.
`BEDROCK_MODEL`	`us.anthropic.claude-sonnet-4-6[1m]`	No	Bedrock cross-region inference profile ID. The `us.` prefix enables automatic cross-region routing; the `[1m]` suffix selects the 1M-token context variant.

Cache

Variable	Default	Purpose
`CACHE_TTL_MS`	`900000` (15 min)	LRU response cache TTL. Cached responses skip the target API entirely.
`CACHE_MAX_ENTRIES`	`1000`	Maximum entries in the LRU cache.

Rate limiting (inbound)

These limit traffic to Barnacle's own API. See per-plugin Bottleneck config in each contract.ts for outbound rate limits to target sites.

Variable	Default	Purpose
`RATE_LIMIT_MAX`	`120`	Max requests per window per API key (or IP for unauthenticated traffic).
`RATE_LIMIT_WINDOW_MS`	`60000` (1 min)	Rate limit window duration.
`TRUST_PROXY`	`true`	Trust `X-Forwarded-For` when behind a reverse proxy. Set `false` for bare-metal deploys to prevent spoofing.

Readiness / observability

Variable	Default	Purpose
`READINESS_QUEUE_THRESHOLD`	`20`	`/readyz` returns 503 when scraper queue depth exceeds this. Lets orchestrators shed load before the pool is saturated.
`ENABLE_DOCS`	`false`	Serve Swagger UI at `/docs`. Disable in production.

Datadog (opt-in)

APM tracing and DogStatsD metrics are opt-in: dd-trace and hot-shots are optional peer dependencies, so a plain npm i @enricai/barnacle installs neither and Barnacle runs without them. Enable either half independently — install the package and flip its flag. If a flag is on but the package is missing, Barnacle warns and carries on with that feature disabled; it never fails to boot.

# APM tracing
pnpm add dd-trace
DD_TRACE_ENABLED=true node --import dd-trace/initialize dist/server.js

# DogStatsD metrics
pnpm add hot-shots
DD_METRICS_ENABLED=true node dist/server.js

Tracing needs --import dd-trace/initialize for full auto-instrumentation: Datadog requires the tracer to load before any other module so it can patch http/net/dns. Metrics have no such constraint.

Variable	Default	Purpose
`DD_TRACE_ENABLED`	`false`	Enable APM tracing. Requires the `dd-trace` peer dependency.
`DD_METRICS_ENABLED`	`false`	Enable DogStatsD metrics. Requires the `hot-shots` peer dependency. Independent of `DD_TRACE_ENABLED`.
`DD_AGENT_HOST`	`localhost`	Datadog agent hostname (the sidecar, in ECS Fargate).
`DD_DOGSTATSD_PORT`	`8125`	DogStatsD UDP port on the agent host.
`DD_SERVICE`	`barnacle`	Service name tagged on spans and metrics.
`DD_ENV`	`NODE_ENV`	Deployment environment tag.
`DD_VERSION`	`0.1.0`	Application version tag — git SHA or package version.

Telemetry

Variable	Default	Purpose
`TELEMETRY_ENABLED`	`true`	Master switch — set `false` to disable all NDJSON telemetry writes.
`TELEMETRY_EVENTS_DIR`	`.barnacle/events`	Directory for per-run NDJSON event stream files (`<eventsDir>/<runId>.ndjson`).
`CALLS_NDJSON_PATH`	`.barnacle/calls.ndjson`	Append-only NDJSON sink for LLM/Stagehand call samples. One line per call; feed to the judge and self-heal skills.
`SUBMISSIONS_NDJSON_PATH`	`.barnacle/submissions.ndjson`	Append-only NDJSON sink for dispatch submission envelopes. One line per plugin invocation captures siteId, requestId, inbound payload, status, audit payload, and duration — the durable source-of-truth for "what did we submit for jobId X and did it succeed."
`TELEMETRY_MAX_FILE_SIZE_BYTES`	`104857600` (100 MB)	Rotate/drop the calls NDJSON once it exceeds this byte count.
`TELEMETRY_MAX_RETENTION_MS`	`2592000000` (30 days)	Drop event-stream files older than this many milliseconds.
`TELEMETRY_S3_BUCKET`	—	Optional — destination bucket for the buffered S3 telemetry mirror. Sink is entirely inert (no client, no network calls) when unset. Credentials/region resolve the same way as Bedrock (`AWS_REGION`, standard SDK credential order).
`TELEMETRY_S3_PREFIX`	`telemetry`	Key prefix for uploaded NDJSON objects (`<prefix>/<calls\|submissions>/<date>/...`).
`TELEMETRY_S3_FLUSH_INTERVAL_MS`	`60000`	How often buffered lines are flushed to S3.
`TELEMETRY_S3_MAX_BUFFER_LINES`	`500`	Threshold-flush trigger — flush early if either buffer exceeds this many lines, ahead of the next scheduled interval.

LLM judging

Variable	Default	Purpose
`JUDGE_MODEL`	`us.anthropic.claude-sonnet-4-6[1m]`	Anthropic model used by the judge script. Reuses Bedrock creds via the cross-region inference profile.
`JUDGE_TEMPERATURE`	`0.2`	Sampling temperature for judge LLM calls. Keep low (≤ 0.3) for deterministic verdicts.
`JUDGE_BATCH_SIZE`	`10`	Number of call samples sent to the judge in one LLM request.
`JUDGE_TIMEOUT_MS`	`120000` (2 min)	Anthropic SDK request timeout for judge calls.

Self-heal

Variable	Default	Purpose
`SELFHEAL_MAX_ITERATIONS`	`5`	Maximum patch→replay→score iterations before BUDGET_EXHAUSTED.
`SELFHEAL_N_REPLAYS`	`5`	Number of replay runs per iteration arm.
`SELFHEAL_SUCCESS_THRESHOLD`	`0.9`	Minimum pass rate (0–1) to declare SUCCESS and stop iterating.
`SELFHEAL_PLATEAU_WINDOW`	`3`	Consecutive iterations below `SELFHEAL_PLATEAU_DELTA` that triggers PLATEAUED.
`SELFHEAL_PLATEAU_DELTA`	`0.03`	Minimum absolute pass-rate improvement per iteration to count as progress.
`SELFHEAL_TIMEOUT_MS`	`60000` (1 min)	Per-replay LLM request timeout.

Per-site base URL overrides

Set BARNACLE_SITE_<UPPERCASE_SITE_ID>_BASE_URL to override a plugin's defaultBaseUrl without source changes. Underscores in the env key map to hyphens in the siteId:

BARNACLE_SITE_MY_SHOP_BASE_URL="https://staging.my-shop.com"  # overrides plugin `my-shop`

Out-of-tree plugins

Variable	Default	Purpose
`BARNACLE_PLUGINS`	`""`	Comma-separated list of plugin specifiers to load at startup — relative paths (`./plugins/acme`) or package names (`@acme/barnacle-plugin`). Empty by default (built-ins only).
`BARNACLE_PLUGINS_STRICT`	`false`	When `true`, any plugin that fails to load aborts the process instead of producing a disabled record.
`BARNACLE_PLUGINS_DIR`	`process.cwd()`	Base directory used to resolve relative specifiers and locate the operator's `node_modules`. Defaults to wherever the binary is run — not the installed Barnacle package root.
`BARNACLE_PLUGINS_CONFIG_DIR`	(unset)	Directory scanned at startup for `*.plugin.json` config manifests, each loaded as a config-only plugin. Lets operators register sites by dropping a JSON file in a directory instead of editing `BARNACLE_PLUGINS`. An unreadable directory is logged and skipped — it never crashes boot.

Resolution rule: a specifier starting with . or / is treated as a filesystem path resolved relative to BARNACLE_PLUGINS_DIR. Anything else is treated as an npm package name and resolved via require.resolve against the operator's own node_modules inside BARNACLE_PLUGINS_DIR.

Failure policy: by default (non-strict), a plugin that fails to load is logged at warn level and recorded as "disabled" in the load report — the server still boots with the remaining plugins. Set BARNACLE_PLUGINS_STRICT=true to abort startup on any load failure instead.

zod/v4 requirement for plugin authors: import Zod as import { z } from "zod/v4" in your plugin, not as bare "zod". Barnacle uses fastify-type-provider-zod which compiles routes against core's own zod instance; a plugin schema built against a different zod import may pass load-time validation but fail at route registration.

GET /v1/plugins (authenticated) returns the full plugin load report — one record per built-in and out-of-tree specifier — including siteId, displayName, route, specifier, resolvedPath, apiVersion, status ("loaded" or "disabled"), and an optional reason when disabled. Requires a valid Authorization: Bearer <token> header (reveals filesystem paths, so it is separate from the auth-free /healthz//readyz probes).

Usage

Prerequisites

Node.js 22+
pnpm 10.4.1
A Steel account (STEEL_API_KEY) for managed browser sessions
An Anthropic key (ANTHROPIC_API_KEY) for Stagehand's LLM calls, or AWS Bedrock (USE_BEDROCK=true + AWS credentials) — see .env.example for details

Install

pnpm install
cp .env.example .env   # fill in STEEL_API_KEY and either ANTHROPIC_API_KEY or Bedrock creds

Generating an API key

Barnacle validates every request using bcrypt-hashed bearer tokens stored in API_KEYS_HASHED. To create one:

# 1. Generate a random plaintext key — save this, you'll send it as Authorization: Bearer <key>
node -e "console.log(require('crypto').randomBytes(32).toString('hex'))"

# 2. Hash it (bcrypt cost factor 10) — paste the output into API_KEYS_HASHED
node -e "const b=require('bcryptjs');b.hash(process.argv[1],10,(e,h)=>console.log(h))" <your-key>

Comma-separate multiple hashes in API_KEYS_HASHED to support key rotation.

For local development, set DEV_BYPASS_AUTH=true in .env to skip auth entirely — never set this in production.

Dev

pnpm run dev

Build for production

pnpm run build
pnpm start

Try it

Barnacle boots with the built-in plugins registered (see BUILTIN_SITE_PLUGINS in src/plugins/discover.ts). Follow Adding a New Site above to build and register a plugin; core will register POST /v1/<your-siteId>/run automatically at startup.

With the dev server running (pnpm run dev), confirm the server is up:

curl -s http://localhost:3000/health | jq .

Once a plugin is registered, every response follows the same envelope shape. The status block is always present; the plugin's responseSchema fields are spread alongside it at the root:

{
  "status": {
    "httpStatus": "OK",
    "dateTime": "2025-05-16T12:00:00.000Z",
    "details": []
  },
  "items": []
}

The envelope is a flat merge, not nested — status lives at the root and the plugin's response fields are spread alongside it (src/api/helpers/envelope.ts:8-25). Parse as { status, ...pluginData }, not { status, data: pluginData }.

Every response — success or error — uses the same envelope shape so clients share a single parser. Error details appear in status.details[] with numeric codes:

Code	Name	When
1000	`PARTIAL_CONTENT_SUCCESS`	Partial data returned
1001	`DECODING_ERROR`	Request body could not be parsed
1002	`FIELD_VIOLATION`	Schema validation failure on a field
1003	`EMPTY_REQUEST`	Request body was missing or empty
1004	`AUTHORIZATION_ERROR`	Bearer token missing or invalid
1005	`RESOURCE_NOT_FOUND`	Requested resource does not exist
1006	`INDEX_NOT_FOUND`	Internal index lookup failed
1007	`CLIENT_CALL_ERROR`	Downstream client call failed
1008	`GENERIC_ERROR`	Unclassified server error
1009	`EXTRA_DETAIL`	Supplemental detail entry (informational)
1010	`THROTTLED_REQUEST`	Rate limit exceeded (hot path 429)
1011	`TIME_OUT`	Request timed out
2003	`SCRAPE_FAILURE`	Browser automation failed after retries
2004	`CAPTCHA_ENCOUNTERED`	CAPTCHA challenge could not be resolved
2005	`EMPTY_RESULTS`	Scrape succeeded but returned no data
2006	`VERIFICATION_TRIGGER_FAILED`	OTP trigger to the target site failed
2007	`RESUME_INVALID_OTP`	Provided OTP was rejected by the target site
2008	`URL_LOCKED`	Upstream vendor locked the target URL; back off and retry later

Full definitions: src/api/schemas/common.ts.

How scraper exceptions map to API codes (src/plugins/loader.ts:149-153):

CaptchaError → 2004 CAPTCHA_ENCOUNTERED
EmptyResultsError → 2005 EMPTY_RESULTS
HttpRateLimitError → 1010 THROTTLED_REQUEST (no browser fallback)
HttpUrlLockedError → 2008 URL_LOCKED (no browser fallback; distinct from rate-limit for metrics)
Any other ScraperError → 2003 SCRAPE_FAILURE
Task exceeded TASK_TIMEOUT_MS → 1011 TIME_OUT

Endpoints

Each registered plugin exposes a POST route following the default convention: POST /v1/<siteId>/run. When the hot path detects that required applicant answers are absent (e.g. Gender, Degree, EducationLevel, SignatureFullName) or a repeat-applicant OTP challenge, /run returns HTTP 200 with { needsUserInfo: true, missingFields: [{ field, question }], requiresOtp } instead of a submission result, so the caller can collect the gaps and hand back.

Plugins declare their own extra routes via meta.extraRoutes, which core registers uniformly — the engine has no per-site knowledge. Route paths are declared as :siteId templates, so the concrete path is whatever the plugin's siteId is. Two conventional shapes a plugin may add:

POST /v1/<siteId>/resume — body = the full original candidate payload plus collectedData (and otpCode where the site issues an OTP challenge); re-runs the hot path with the collected answers merged in; returns the same { verified } envelope as /run, or 2007 RESUME_INVALID_OTP if the OTP is rejected
POST /v1/<siteId>/trigger-otp — body { offerId, email }; asks the target site to email an OTP to a repeat applicant; returns { success: true } or a 2006 VERIFICATION_TRIGGER_FAILED error envelope

See examples/plugins/acme-jobs.plugin.json for a runnable declaration.

Operational routes:

GET /healthz — liveness probe
GET /readyz — readiness probe (checks scraper credentials, queue depth)
GET /docs — Swagger UI (when ENABLE_DOCS=true)
GET /v1/plugins — authenticated plugin load report (see Out-of-tree plugins)

Commands

Command	What it does
`pnpm run dev`	`tsx watch --env-file=.env src/server.ts` with hot reload
`pnpm run build`	compile to `dist/` (tsc + path alias rewriting + copy `src/sites/` fixtures and `src/testing/fixtures`)
`pnpm start`	`node dist/server.js`
`pnpm run typecheck`	strict TS noEmit
`pnpm run lint` / `lint:fix`	Biome
`pnpm run test`	Vitest unit + integration
`pnpm test src/scraper/fixtures.test.ts`	Run a single test file (NEVER use `--` before the filter)
`pnpm run test:watch`	Vitest in watch mode (re-runs on file changes)
`pnpm run test:coverage`	Vitest with v8 coverage report
`pnpm run format`	Biome format write
`pnpm run recon:browser`	Phase 1 — drive browser + capture API calls
`pnpm run recon:http`	Phases 2–3 — replay, introspect, probe rate limits
`pnpm run recon:generate -- --site-id <id>`	Phase 4 — generate complete plugin from artifacts
`pnpm run recon:summarize -- --site-id <id>`	Phase 4 (optional) — write human-readable findings doc
`pnpm run recon:heal -- --site-id <id> --url <url>`	Self-heal a failing recon flow without modifying the source file
`pnpm run smoke -- --site <id> --payload '...'`	Phase 6 — run nightly drift-detection smoke test
`pnpm run judge:llm -- --calls-ndjson <path> --call-type <type>`	Score captured LLM calls on a three-dimensional rubric; writes a verdict JSON to `judge-out/`
`pnpm run heal:llm -- --verdict-path <path> --call-type <type>`	Self-heal a failing prompt template: iterate patch→replay→score, write `healing-<callType>.md` with the best patch — production prompts are never modified

Architecture

src/
├── server.ts                  # Fastify bootstrap — calls loadAllPlugins(), registerRoutes(), site-agnostic
├── site-plugin.ts             # SitePlugin<TInput,TOutput> interface (engine contract)
├── config.ts                  # frozen env-typed config singleton
├── plugins/
│   ├── loader.ts              # dispatch(), registerRoutes(app, cfg, plugins)
│   └── discover.ts            # BUILTIN_SITE_PLUGINS, loadAllPlugins(), loadPlugins()
├── sites/
│   ├── _shared/               # branch-local cross-plugin guards (coverage-expectations.test.ts)
│   └── <site-id>/             # one directory per registered plugin
├── api/
│   ├── plugins/               # auth, error-handler, request-context
│   ├── routes/                # health
│   ├── schemas/               # common envelope schemas; LLM telemetry + judge-verdict schemas
│   ├── helpers/envelope.ts    # success envelope builder
│   └── errors.ts              # error hierarchy + envelope builder
├── scraper/
│   ├── session.ts             # Steel + Stagehand session factory
│   ├── pool.ts                # p-queue over createBrowserSession
│   ├── throttle.ts            # Bottleneck limiter + jitter
│   ├── retry.ts               # p-retry + failure classification
│   ├── errors.ts              # typed scraper error hierarchy
│   ├── http-client.ts         # typed fetch wrapper (hot path)
│   ├── rate-limited-json-client.ts # factory: Bottleneck + chromiumClientHints + createHttpClient in one call — prefer this over the three-step scaffold for Chromium-hint plugins
│   ├── http-status-classifier.ts # pure status→ScraperError classifier for raw-fetch callers
│   ├── raw-fetch.ts           # site-agnostic undici scaffold: network-error wrap, onResponse hook, optional classifyHttpStatus (skipClassify for callers that classify manually)
│   ├── graphql-client.ts      # GraphQL POST wrapper
│   ├── metrics.ts             # drift-detection counters
│   ├── fixtures.ts            # static JSON fixture loader
│   ├── navigate.ts            # shared awaitActivePage + goto(networkidle) helper
│   ├── behavioral-signals.ts  # CDP synthetic mouse-move + scroll dispatcher for bot-detection warmup
│   ├── session-warmup.ts      # generic pRetry browser-session runner: acquire → callback → close, with caller-supplied exhaustion mapping
│   └── require-response-field.ts # shared helpers for extracting required fields from HTTP response objects (HttpSchemaError on missing/null)
├── cache/
│   ├── response-cache.ts      # lru-cache wrapper for deduplicating concurrent identical scraper requests
│   └── keyed-ttl-cache.ts     # generic per-key TTL + single-flight coalescing cache factory
├── lib/                       # logging, env, bedrock, db client, multipart, option-matcher, chromium-client-hints, telemetry/
├── scripts/                   # recon-browser, recon-http, recon-generate, recon-summarize, recon-heal, recon-shared, smoke-test, judge-llm-batch, llm-heal
├── testing/
│   ├── integration-runner.ts              # site-agnostic scaffold for integration tests (allocate inbox → dispatch → poll)
│   ├── mock-fetch-response.ts             # shared undici-compatible Response stub factory for flow tests that mock fetch
│   ├── replay-integration-suite.ts        # generic describe.skipIf/it.each scaffold; eliminates per-site integration boilerplate
│   ├── contract-parity-suite.ts           # offline schema-parity scaffold; one-call drop-in for accept + rejection-case coverage
│   ├── coverage-guard-suite.ts            # registry-driven structural guard; asserts contract.parity.test.ts exists per registered plugin
│   ├── batch-email-confirmation.ts        # two-phase batch runner: submit jobs → poll inboxes (site-agnostic)
│   └── batch-report.ts                    # markdown table renderer for batch-test verdicts
└── types/

Library choices (battle-tested — no custom reinventions):

API server: fastify + helmet + compress + rate-limit + swagger
Schema: zod via fastify-type-provider-zod
Browser automation: @browserbasehq/stagehand + steel-sdk
Concurrency: p-queue, p-retry, bottleneck
Caching: lru-cache
Logging: pino with CloudWatch 256KB splitting + sensitive-field redaction

Per-site base URL overrides: set BARNACLE_SITE_<UPPERCASE_SITE_ID>_BASE_URL to override a plugin's defaultBaseUrl without source changes. Underscores in the env key map to hyphens in the siteId (e.g. BARNACLE_SITE_MY_SHOP_BASE_URL → plugin my-shop).

Execution header: send x-barnacle-execution: browser on any plugin request to skip the hot path and go directly to the Stagehand browser path. Omit the header (or send any other value) to use the default hot path. Useful for debugging or when you know the hot path is broken. (Fastify lowercases incoming header keys; the dispatcher reads request.headers["x-barnacle-execution"] — supply lowercase to match.)

Deployment

Production checklist

# .env (production)
NODE_ENV=production
ENABLE_DOCS=false         # never expose Swagger in prod
TRUST_PROXY=true          # set false if deploying directly to the internet (no ALB/nginx)
DEV_BYPASS_AUTH=false     # this is the default — confirm it's not set to true
API_KEYS_HASHED="<bcrypt-hash>,<bcrypt-hash>"  # at least one key
STEEL_API_KEY="..."
ANTHROPIC_API_KEY="..."   # or USE_BEDROCK=true + AWS creds

Process management

Barnacle is a plain Node.js process. Use pm2 or systemd to keep it alive and restart it on crash:

# pm2
pm2 start dist/server.js --name barnacle --env production
pm2 save && pm2 startup

# systemd (example unit)
[Service]
ExecStart=/usr/bin/node /srv/barnacle/dist/server.js
WorkingDirectory=/srv/barnacle
EnvironmentFile=/srv/barnacle/.env
Restart=on-failure

Reverse proxy

Route traffic through nginx or an Application Load Balancer (ALB). Set TRUST_PROXY=true so Fastify uses X-Forwarded-For for the client IP (needed for rate limiting on unauthenticated traffic).

location / {
    proxy_pass         http://127.0.0.1:3000;
    proxy_set_header   X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header   X-Forwarded-Proto $scheme;
    proxy_set_header   Host $host;
}

Health probes

Wire /healthz as the liveness probe and /readyz as the readiness probe:

# Kubernetes example
livenessProbe:
  httpGet: { path: /healthz, port: 3000 }
  initialDelaySeconds: 5
readinessProbe:
  httpGet: { path: /readyz, port: 3000 }
  initialDelaySeconds: 10

/readyz returns 503 when the scraper pool queue is saturated (depth > READINESS_QUEUE_THRESHOLD) or when required scraper credentials are missing.

Common issues

Symptom	Cause	Fix
`Error: STEEL_API_KEY is required`	Missing env var	Add `STEEL_API_KEY` to `.env`
`useProxy rejected` / `402` from Steel	Free-tier plan doesn't support residential proxies	Set `SCRAPER_PROXY_TYPE=none` and `SCRAPER_SOLVE_CAPTCHA=false`
`401 Unauthorized` on every request	No API key configured or wrong plaintext key	Verify `API_KEYS_HASHED` is set; double-check the plaintext key. For dev, set `DEV_BYPASS_AUTH=true`
Stagehand throws `model not found`	Wrong model name format	Use the `anthropic/` prefix: `STAGEHAND_MODEL=anthropic/claude-sonnet-4-6`
`/readyz` returns 503 on `scraperCredentials`	`STEEL_API_KEY` or LLM key missing	Set the missing credential
Build succeeds but `dist/sites/` is empty	`tsc` ran but `cp -r src/sites dist/sites` was skipped	Run `pnpm run build` (not `tsc` directly) — the build script copies site sources after compilation

Reference

Coding standards: CLAUDE.md
Architecture & design rationale: docs/architecture.md
Recon playbook (step-by-step): docs/playbook.md
Testing guide: docs/testing.md
Telemetry & LLM judging concept guide: docs/telemetry-and-judging.md
Per-site recon findings: docs/target-recon.md (populated after first pnpm run recon:summarize)

License

MIT © Enricai

Contributions welcome — see CONTRIBUTING.md.

Name		Name	Last commit message	Last commit date
Latest commit History 333 Commits
.aws		.aws
.claude		.claude
.github		.github
.husky		.husky
.leerie		.leerie
docs		docs
examples/plugins		examples/plugins
patches		patches
scripts		scripts
src		src
.dockerignore		.dockerignore
.env.example		.env.example
.gitignore		.gitignore
.nvmrc		.nvmrc
CLAUDE.md		CLAUDE.md
CONTRIBUTING.md		CONTRIBUTING.md
Dockerfile		Dockerfile
LICENSE		LICENSE
README.md		README.md
SECURITY.md		SECURITY.md
biome.json		biome.json
package.json		package.json
pnpm-lock.yaml		pnpm-lock.yaml
tsconfig.build.json		tsconfig.build.json
tsconfig.json		tsconfig.json
vitest.config.ts		vitest.config.ts

Uh oh!

Folders and files

Latest commit

History

Repository files navigation

Barnacle

How it works

The mental model

The pipeline at a glance

Why this approach

Adding a New Site — The Recon Playbook

Phase 0 — Define the user flow

Phase 1 — Run the browser recon harness

Cookie-jar snapshots

Phase 2–3 — Replay and probe

Phase 4 — Generate the plugin

Telling the generator which fields carry your caller's data

Telling the generator your ATS's form-schema wire keys

Mapping the site's screening questions

Phase 5 — Register the plugin

Phase 6 — Wire up drift detection

The whole loop, in one picture

Plugin Authoring Guide

The SitePlugin interface

SitePluginMeta — required fields

Full plugin skeleton (hot path + browser fallback)

The auditPayload hook

Static fixtures

Register the plugin

Wire up the nightly smoke test

Maintenance loop

Runtime internals

Hot-path fallback triggers

Cache deduplication

Session pool and timeouts

Viewport rotation

LLM routing: Anthropic vs. AWS Bedrock

Observability

NDJSON telemetry files

LLM call sample schema

Call types

Tailing call samples with jq

Environment variables

Application

Auth

Browser automation (Steel + Stagehand)

AWS Bedrock (alternative LLM provider)

Cache

Rate limiting (inbound)

Readiness / observability

Datadog (opt-in)

Telemetry

LLM judging

Self-heal

Per-site base URL overrides

Out-of-tree plugins

Usage

Prerequisites

Install

Generating an API key

Dev

Build for production

Try it

Endpoints

Commands

Architecture

Deployment

Production checklist

Process management

Reverse proxy

Health probes

Common issues

Reference

License

About

Resources

License

Contributing

Security policy

Uh oh!

Packages