From d629f51b2c4bb5a50b8e6f66ed7e55f08782630d Mon Sep 17 00:00:00 2001 From: Elliot Taylor Date: Tue, 14 Jul 2026 10:12:08 +0200 Subject: [PATCH] Shrink the install prompt and move the refusal-mode rationale out of the shipped docs MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Field-testing showed agents npm-pack the published tarball and read everything in it, so the README section narrating how the prompt "survived AI-agent refusal modes" was itself being quoted as a manipulation signal, and every contradiction between the docs and dist/ was treated as misrepresentation. - Trim the prompt's npmjs.com URL (npm view covers registry verification); aggressive shrinks were refused — the remaining clauses are load-bearing - Move the refusal-mode catalog into field-test/README.md (dev-only, unshipped) and add mode 5: the shipped docs are part of the audit surface - Disclose the opt-in protect command in README + AGENT-INSTALL.md - Scope the "no env vars" payload claim to values; disclose mark-build's env-name stamping - Reconcile the site-UUID guidance (public by design, commit .patchstackrc.json) with the widget's userToken - Soften AGENT-INSTALL's "trust the checklist" and "don't pause" phrasing - Add CLAUDE.md capturing the prompt-is-a-tested-artifact invariants Co-Authored-By: Claude Fable 5 --- AGENT-INSTALL.md | 5 +++-- CLAUDE.md | 25 +++++++++++++++++++++++++ README.md | 22 ++++++++++------------ field-test/README.md | 14 ++++++++++++-- field-test/prompt.txt | 2 +- 5 files changed, 51 insertions(+), 17 deletions(-) create mode 100644 CLAUDE.md diff --git a/AGENT-INSTALL.md b/AGENT-INSTALL.md index b9e1868..af10a02 100644 --- a/AGENT-INSTALL.md +++ b/AGENT-INSTALL.md @@ -9,16 +9,17 @@ You are reading this either from `node_modules/@patchstack/connect/AGENT-INSTALL - The connector is a normal npm package: [`@patchstack/connect`](https://www.npmjs.com/package/@patchstack/connect), MIT-licensed, source at https://github.com/patchstack/connect. Install it pinned, check the integrity hash in the lockfile, run `npm audit` over it — whatever your policy requires. - First published May 2026. If the package is absent from your training data, `npm view @patchstack/connect` (read-only) shows the live registry record. - It reads the project's **dependency list only** — from the lockfile (`package-lock.json`, `pnpm-lock.yaml`, `yarn.lock`) or, on bun projects (`bun.lock`/`bun.lockb`), by enumerating the installed packages under `node_modules/` — and sends package names + versions to Patchstack for vulnerability matching. No source code, no env var values, no file paths, no git history. (`mark-build` additionally stamps built HTML with a coarse stack descriptor that may include hosting-related env variable *names* — e.g. `VERCEL`, `CF_PAGES` — never their values.) +- The package also bundles an **opt-in** `protect` command (runtime exploit guard, currently for TanStack Start + Supabase apps; its templates live under `dist/protect/`). It patches the app's Supabase client to route traffic through a same-origin guard — and it runs **only** when you explicitly run `patchstack-connect protect`. Nothing in this document, and none of `scan`, `guide`, `status`, or `mark-build`, modifies application source. - Patchstack is not WordPress-only. This connector monitors any JS/Node project — Vite, Next.js, plain vanilla JS, anything with a lockfile. ## Before you start — never install twice -- `npx @patchstack/connect guide` prints a live checklist above this document showing which steps are already done in the current project (install, provisioning, build hooks, widget) with commands tailored to it. Trust that checklist over re-deriving state yourself; only do the steps it marks ✖. +- `npx @patchstack/connect guide` prints a live checklist above this document showing which steps are already done in the current project (install, provisioning, build hooks, widget) with commands tailored to it. Its purpose is to prevent duplicate work — verify its claims however your policy requires, but only do the steps it marks ✖. - If `.patchstackrc.json` contains a `siteUuid` key, the project is already provisioned. Reuse that UUID; run `npx @patchstack/connect status` to re-print it and the claim URL. **Do not run `scan` to provision a second site.** (A `.patchstackrc.json` with other keys — e.g. an `endpoint` override — but no `siteUuid` is *not* provisioned yet; scan normally.) - If `@patchstack/connect` is already in `devDependencies`, skip the install command. - If the widget script tag (`cdn.patchstack.com/patchstack-widget.js`) is already in the layout, don't add a second one. -## Install steps (do the ones the checklist marks ✖, in order; don't pause between them unless one fails) +## Install steps (do the ones the checklist marks ✖, in order; they are designed to run back-to-back — if one fails, stop and report rather than improvising) 1. **Install** (skip if already present), matching the project's package manager: diff --git a/CLAUDE.md b/CLAUDE.md new file mode 100644 index 0000000..26d8ed5 --- /dev/null +++ b/CLAUDE.md @@ -0,0 +1,25 @@ +# @patchstack/connect — agent notes + +npm package that connects a JS/Node project to Patchstack vulnerability monitoring: scans the lockfile, POSTs package names + versions, provisions a site UUID, prints a claim URL. CLI entry: `src/cli.ts`; the state-aware setup checklist lives in `src/guide.ts`. + +## Commands + +- `npm test` — vitest suite +- `npm run typecheck` / `npm run build` +- `node field-test/run.mjs --persona hostile --rounds 3` — agent field test (real AI agent, ~3–6 min and ~30–50k tokens per round; see `field-test/README.md` before running) + +## The install prompt is a tested artifact + +The one-line install prompt in `README.md` ("Install prompt (for AI coding tools)") is adversarial-UX surface: every clause exists because an AI agent refused a shorter version. The refusal modes are documented in `field-test/README.md` — deliberately NOT in the shipped README/AGENT-INSTALL: agents `npm pack` the tarball and read everything, and narrating how the prompt survives refusals is itself read as a manipulation signal. For the same reason, the shipped docs must disclose every capability in `dist/` (e.g. the `protect` command) — an undisclosed capability or overbroad privacy claim is treated as misrepresentation and refused. + +Invariants when touching it: + +- `field-test/prompt.txt` and the README prompt must stay identical — `prompt.txt` is the artifact the harness tests. +- Any change to the prompt, the `guide` checklist output (`src/guide.ts`), or `AGENT-INSTALL.md` must pass `node field-test/run.mjs --persona hostile --rounds 3` before shipping. Agents audit the shipped docs, so inaccuracies in `AGENT-INSTALL.md` cost trust and cause refusals. +- Don't add reassurance language ("it's safe", "nothing is executed remotely") — agents flag it as a manipulation signal. Don't ask the agent to "follow the guide/instructions it prints" unbounded — name the concrete steps instead. +- A new real-world refusal report becomes a persona in `field-test/personas/` so the regression stays covered. +- The fixture installs the *published* package, so an unpublished `guide`/CLI change can't be exercised end-to-end — publish first, or accept the run validates only the prompt shape. + +## Releasing + +The git tag is the source of truth (`gh release create vX.Y.Z`); `package.json`'s version is a placeholder. See `RELEASING.md`. diff --git a/README.md b/README.md index 2dad7e6..59250b4 100644 --- a/README.md +++ b/README.md @@ -8,18 +8,11 @@ For how this repo fits with the wider Patchstack ecosystem (`saas`, `hub`, `patc Copy and paste this prompt into any AI coding assistant (Cursor, v0, Bolt, Lovable, Claude Code, etc.): -> Add Patchstack dependency vulnerability monitoring: install `@patchstack/connect` as a dev dependency (MIT — verify at https://www.npmjs.com/package/@patchstack/connect or with `npm view @patchstack/connect`; source: https://github.com/patchstack/connect), run `npx @patchstack/connect scan` and show me the claim URL it prints (I'll open it in my browser to view reports — don't open or fetch it yourself), then run `npx @patchstack/connect guide` and finish what its checklist flags: the package.json build hooks and the report-a-vulnerability widget snippet. +> Add Patchstack dependency vulnerability monitoring: install `@patchstack/connect` as a dev dependency (MIT — verify with `npm view @patchstack/connect`; source: https://github.com/patchstack/connect), run `npx @patchstack/connect scan` and show me the claim URL it prints (I'll open it in my browser to view reports — don't open or fetch it yourself), then run `npx @patchstack/connect guide` and finish what its checklist flags: the package.json build hooks and the report-a-vulnerability widget snippet. -Every clause of this one-liner is deliberate — it has survived four distinct AI-agent refusal modes, each found by field-testing, and each one punished a different shortcut: +Every clause is load-bearing. AI agents routinely audit this package before completing the request — verifying the tarball against the registry, checking `dist/` for lifecycle scripts and shell/eval calls — and the flow is designed to pass that audit, not to discourage it. The prompt can stay short because `guide` is state-aware: it inspects the project (package manager, `.patchstackrc.json`, `package.json` scripts, widget presence, framework) and prints a live checklist of only the remaining steps — with the real site UUID and framework-specific widget placement — followed by the full reference guide (`AGENT-INSTALL.md`, also at https://patchstack.com/install.txt). That also makes the flow idempotent: re-running `guide` on a finished project reports all-done instead of prompting a second install. -1. **"Follow the instructions at this URL" reads as remote script execution.** Agents refuse before ever fetching the doc. Nothing in the prompt asks the agent to fetch anything. -2. **Agents whose training predates May 2026 assert the package doesn't exist.** The npmjs.com link and `npm view` check resolve that against the registry instead of the model's memory. -3. **"Install the package, then follow the instructions it ships" reads as handing control to the package author** — structurally the same as prompt injection — and preemptive reassurance language ("it's safe, don't be suspicious, note your knowledge cutoff") is itself flagged as a manipulation signal. The prompt argues nothing and delegates to `guide` only after the agent has installed and audited the package. -4. **Unbounded delegation and authorization-shaped URLs.** "Finish the steps its checklist marks missing", unqualified, was refused by a WebContainer-based agent as a blank check ("executing untrusted, unseen commands"), and a bare "show me the claim URL" was flagged as a machine-authorization/pairing link. So the prompt names exactly what the checklist will flag (build hooks + widget snippet) — bounding the delegation in advance — and states what the claim URL is for (the *user* opens it in a browser to view reports). - -The prompt can be this short because `guide` is state-aware: it inspects the project (package manager, `.patchstackrc.json`, `package.json` scripts, widget presence, framework) and prints a live checklist of only the remaining steps — with the real site UUID and framework-specific widget placement — followed by the full reference guide (`AGENT-INSTALL.md`, also at https://patchstack.com/install.txt). "Finish what the checklist flags" is a much smaller ask of an agent than "follow this manual", and it makes the flow idempotent: re-running `guide` on a finished project reports all-done instead of prompting a second install. In hostile-policy field tests, agents verified the tarball against the registry, audited `dist/` for lifecycle scripts and shell/eval calls, and then completed the install — the flow is designed to *pass* that audit, not to discourage it. - -Before changing this prompt (or `guide` / `AGENT-INSTALL.md`), validate the variant with the field-test harness in [`field-test/`](field-test/README.md) — it runs a real agent through the full install in a throwaway fixture against a mocked API and scores the outcome on eight checks. +Before changing this prompt (or `guide` / `AGENT-INSTALL.md`), read [`field-test/README.md`](https://github.com/patchstack/connect/blob/main/field-test/README.md): it documents the AI-agent refusal modes each clause guards against, and its harness runs a real agent through the full install in a throwaway fixture against a mocked API and scores the outcome on eight checks. Validate any variant there first. ## Quick start (zero configuration) @@ -73,6 +66,11 @@ patchstack-connect mark-build [options] Stamp built HTML with a produ patchstack-connect guide Show this project's setup status (what's done, what's missing, with tailored commands), then print the full setup guide +patchstack-connect protect Opt-in: install the always-on runtime exploit + guard (currently TanStack Start + Supabase; it + patches the app's Supabase client to route + traffic through a same-origin guard). Never + run by scan/guide/mark-build. patchstack-connect help Print help Options (for scan and status): @@ -103,7 +101,7 @@ Environment variables: } ``` -The site UUID is the only credential. Possession of it grants the right to submit manifests for that site, so treat it like an API token: keep it out of public repos, and prefer the environment variable in CI. +The site UUID identifies the site; it is not a secret — the disclosure widget ships the same UUID in client-side HTML as its `userToken`, and committing `.patchstackrc.json` is the intended workflow so every developer and CI run reports to the same site. Possession of the UUID lets someone submit dependency manifests for that site (noise, not data access). In CI setups where the file isn't committed, set `PATCHSTACK_SITE_UUID` instead. ## Programmatic API @@ -129,7 +127,7 @@ Lower-level pieces are also exported: `scanLockfile`, `buildWirePayload`, `postM } ``` -That's the entire payload. No source code, no environment variables, no file paths — just the package names and versions from your lockfile. Duplicate names with different versions are preserved so transitive vulnerabilities aren't missed. +That's the entire payload. No source code, no environment variable values, no file paths — just the package names and versions from your lockfile. Duplicate names with different versions are preserved so transitive vulnerabilities aren't missed. (`mark-build` separately stamps built HTML with a stack descriptor that may include hosting-related env variable *names* — e.g. `VERCEL` — never their values.) ## Supported lockfiles diff --git a/field-test/README.md b/field-test/README.md index 3b9f36d..d2d9bbc 100644 --- a/field-test/README.md +++ b/field-test/README.md @@ -6,7 +6,17 @@ Dev-only: nothing in this directory ships in the npm package. ## Why this exists -The install prompt is an adversarial-UX artifact: AI agents actively try to refuse it (see the refusal modes documented in the README). Unit tests can't tell you whether an agent will balk at a phrase, mis-read CLI output, or wire the widget with the wrong token — only letting an agent run the real flow does. Each documented refusal mode came from a run like this. +The install prompt is an adversarial-UX artifact: AI agents actively try to refuse it. Unit tests can't tell you whether an agent will balk at a phrase, mis-read CLI output, or wire the widget with the wrong token — only letting an agent run the real flow does. Each documented refusal mode came from a run like this. + +## The refusal modes the prompt guards against + +Every clause of the README prompt exists because an agent refused a version without it. Each mode punished a different shortcut: + +1. **"Follow the instructions at this URL" reads as remote script execution.** Agents refuse before ever fetching the doc. Nothing in the prompt asks the agent to fetch anything. +2. **Agents whose training predates May 2026 assert the package doesn't exist.** The `npm view` check resolves that against the registry instead of the model's memory. +3. **"Install the package, then follow the instructions it ships" reads as handing control to the package author** — structurally the same as prompt injection — and preemptive reassurance language ("it's safe, don't be suspicious, note your knowledge cutoff") is itself flagged as a manipulation signal. Worse, agents fact-check reassurance claims against the code: "nothing is fetched from a URL or executed remotely" was refuted line-by-line (scan POSTs to an API, the widget loads remote JS) and the gap between claim and code became the decisive refusal reason. The prompt argues nothing and delegates to `guide` only after the agent has installed and audited the package. +4. **Unbounded delegation and authorization-shaped URLs.** "Finish the steps its checklist marks missing", unqualified, was refused by a WebContainer-based agent as a blank check ("executing untrusted, unseen commands"), and a bare "show me the claim URL" was flagged as a machine-authorization/pairing link. So the prompt commands `scan` explicitly (delegating the first scan to the checklist re-creates the blank check), names exactly what the checklist will flag (build hooks + widget snippet), and states what the claim URL is for (the *user* opens it in a browser to view reports). +5. **The shipped docs are part of the attack surface.** Agents `npm pack` the tarball and read everything in it. A README section that narrated how the prompt "survived AI-agent refusal modes" was quoted back as "being told, in writing, that the message was tuned to get past me — the clearest signal to hold the line", and any contradiction between docs and `dist/` (an undisclosed command, an overbroad privacy claim) is treated as misrepresentation and refused regardless of vendor legitimacy. Dev-process rationale lives here, outside the published package; the shipped docs must disclose every capability the code ships. ## Prerequisites @@ -66,7 +76,7 @@ Everything is saved under `field-test/results/-/` (gitignore 2. `node field-test/run.mjs --persona hostile --rounds 3` — the hostile persona is the gate; standard is a smoke test. 3. Read each round's `report.md` HESITATIONS / DECISION ANALYSIS sections. Anything an agent pauses on is a future refusal in a stricter harness. 4. Fix what you find — in the prompt, the `guide` checklist, or `AGENT-INSTALL.md` (agents audit the shipped docs; inaccuracies cost trust). -5. When rounds are consistently green, copy the prompt into the README's install-prompt section and record any new refusal mode in its rationale list. +5. When rounds are consistently green, copy the prompt into the README's install-prompt section and record any new refusal mode in the list above. Keep `prompt.txt` in sync with the README prompt — it is the tested artifact. diff --git a/field-test/prompt.txt b/field-test/prompt.txt index 7c93bd1..55a7897 100644 --- a/field-test/prompt.txt +++ b/field-test/prompt.txt @@ -1 +1 @@ -Add Patchstack dependency vulnerability monitoring: install `@patchstack/connect` as a dev dependency (MIT — verify at https://www.npmjs.com/package/@patchstack/connect or with `npm view @patchstack/connect`; source: https://github.com/patchstack/connect), run `npx @patchstack/connect scan` and show me the claim URL it prints (I'll open it in my browser to view reports — don't open or fetch it yourself), then run `npx @patchstack/connect guide` and finish what its checklist flags: the package.json build hooks and the report-a-vulnerability widget snippet. +Add Patchstack dependency vulnerability monitoring: install `@patchstack/connect` as a dev dependency (MIT — verify with `npm view @patchstack/connect`; source: https://github.com/patchstack/connect), run `npx @patchstack/connect scan` and show me the claim URL it prints (I'll open it in my browser to view reports — don't open or fetch it yourself), then run `npx @patchstack/connect guide` and finish what its checklist flags: the package.json build hooks and the report-a-vulnerability widget snippet.