v1.1.0: LLM/Agentic Threat Expansion

[revsmoke]

Summary

Major coverage expansion (v1.0.2 → v1.1.0) addressing the late-2025/2026 shift toward LLM-native and agentic attacks. Implements 13 vertical-slice passes per PLAN.md, preserving every v1.0 capability.

6 new MCP tools:

• scan_mcp_tool — Tool-poisoning scanner (SHA-256 drift, ignore-previous + imperative + HTML-comment lints)
• check_lethal_trifecta — Willison's private-read + untrusted-fetch + egress classifier
• query_cve — Unified read across NVD + OSV + GHSA-REST + GHSA-GraphQL + KEV + ATLAS
• deploy_canary / verify_canary — UUIDv4 canary tokens (HMAC-signed state, TTL'd) for memory/RAG poisoning detection
• taste_test — User-designed dual-agent sandbox detonator (Taster + Monitor, gated behind TASTE_TESTER_ENABLED)

5 new feed sources: OSV.dev /v1/querybatch (AI-package allowlist), GHSA GraphQL ecosystem-filtered, MITRE ATLAS taxonomy (7-day cache + offline fallback), CISA KEV (24h cache + severity escalator), Hugging Face Hub securityStatus.

6 new detection categories: unicode_smuggling, policy_puppetry, markdown_exfil, mcp_tool_poisoning, many_shot, plus 8 hand-curated prompt_injection IOCs and obfuscation expansion (Cyrillic homoglyphs, Base32, hex, Sneaky Bits encoder).

Architecture: PatternEntry.atlasTechnique? field (additive), evaluatePattern() extracted as shared helper, SkillScanResult gains hasLethalTrifecta / huggingFaceSecurityFlags / atlasTechniques[], VulnFeedResult gains perSource: {nvd, ghsaRest, ghsaGraphql, osv} counts.

Defense in depth, not silver bullet: A 2026 meta-study of 78 defense papers found adaptive attacks still beat ~85% of single defenses. v1.1.0 stacks five complementary layers — disclaimer added to README.

See SPEC.md for full architecture, RESEARCH_THREATS.md and RESEARCH_FEEDS.md for the threat-landscape work that drove this design.

Test plan

• npm run build — TypeScript clean
• npm test — 457/0 across 17 suites (up from 87 in v1.0.2)
• scripts/smoke-v1.1.ts — 11/0 PASS end-to-end, exercises every new tool
• npx tsx scripts/regenerateManifest.ts — manifest integrity PASSED (71 patterns)
• Boot smoke: MCP server starts on stdio in default mode
• Working tree clean before tag — staging side-effect from smoke reverted

Known limitations (carried into SPEC §13)

• CVE-2026-2796 (ClaudeBleed) — Anthropic acknowledged but not yet in NVD at authoring time; runtime exploit out of project scope.
• MemoryGraft arXiv 2512.16962 — narrative citation only; no code path depends on the exact ID.
• ATLAS Feb-2026 IDs (AML.T0070, AML.T0071) — flagged [unverified] in fallback table; live STIX fetch attempts canonical resolution.
• OWASP LLM Top 10 2026 — still draft as of release; v2025 is operative.
• Taste-Tester MAX_TURNS=5 — calibrated only against scripted-mock baseline (20/20). A real-API calibration run (~$0.50–$2 in Anthropic spend) against the 20-sample corpus in src/test/fixtures/taste-tester-corpus.json would surface the real accuracy number; deferred until a future cycle.

🤖 Generated with Claude Code

Summary by CodeRabbit

• New Features

  • New MCP tools: tool scanner, lethal-trifecta analyzer, unified CVE query, canary deploy/verify, taste-tester sandbox checks, and canary echo detection.

• Detection / Patterns

  • Broadened pattern set: unicode smuggling, policy puppetry, markdown exfil, many-shot, MCP tool poisoning, prompt-injection variants, obfuscation, lethal trifecta, AI supply-chain signals.

• Feeds & Taxonomy

  • Added OSV, GHSA GraphQL, CISA KEV, MITRE ATLAS tagging, and Hugging Face security signals; unified CVE cache/view.

• Testing & Docs

  • New smoke/integration tests, test corpora, execution PLAN, research docs, SPEC, and updated CHANGELOG/README.

• Configuration

  • New optional env vars for feeds, HF token, Taste-Tester, canary HMAC/TTL; degrades gracefully.

[coderabbitai]

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

• @coderabbitai resume to resume automatic reviews.
• @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

• ▶️ Resume reviews
• 🔍 Trigger review

Walkthrough

Release v1.1.0 adds agentic/LLM threat coverage: new pattern sets, feed integrations (OSV, GHSA GraphQL, KEV, ATLAS, Hugging Face), many new services (Atlas, OSV, KEV, GHSA, HuggingFace, UnifiedCveCache, TrifectaAnalyzer, McpToolScanner, Canary, TasteTester), MCP tool wiring, extensive tests, and documentation/spec/plan updates.

Changes

Prompt Rejector v1.1.0

Layer / File(s)	Summary
Release docs & plans CHANGELOG.md, PLAN.md, README.md, SPEC.md, RESEARCH_FEEDS.md, RESEARCH_THREATS.md, SKILLS_SECURITY.md, .env.example	Adds v1.1.0 release notes, execution plan, spec, research briefs, security guide updates, and new env config examples.
Patterns, manifest & gitignore patterns/*.json, patterns/manifest.json, patterns/.gitignore	Adds many new pattern rule files (unicode-smuggling, many-shot, policy-puppetry, mcp-tool-poisoning, markdown-exfil, prompt-injection updates, llm-threats placeholder), updates manifest and adds ignore rules for runtime feed/canary state.
package + smoke script package.json, scripts/smoke-v1.1.ts, scripts/calibrate-taste-tester.ts	Bumps version to 1.1.0, adds @anthropic-ai/sdk dependency, expands test script, and adds end-to-end smoke and calibration runners for v1.1.0 checks.
MCP server wiring & schemas src/mcp/mcpServer.ts, src/schemas/PatternSchemas.ts	Registers new MCP tools (scan_mcp_tool, check_lethal_trifecta, query_cve, deploy_canary, verify_canary, taste_test) and extends pattern schema (source enum, optional atlasTechnique).
Feed & core services src/services/{AtlasService,KevFeedService,OsvFeedService,GhsaGraphQLService,HuggingFaceService,aiPackageAllowlist}.ts	New/expanded feed services (ATLAS, KEV, OSV, GHSA GraphQL), Hugging Face lookups, allowlist for GHSA packages, and feed-stage expansion with staging/enrichment and manifest updates.
Detection & analysis services src/services/{StaticCheckService,SkillScanService,TrifectaAnalyzer,McpToolScanner,CanaryService,TasteTesterService,UnifiedCveCache,SecurityService}.ts	Extends static checks (new categories, unicode stripping), SkillScan (HF flags, trifecta, ATLAS mapping), adds TrifectaAnalyzer, McpToolScanner, CanaryService, TasteTester sandbox, UnifiedCveCache, and security report ATLAS aggregation.
Tests & helpers src/test/**, src/test/helpers/*	Adds extensive unit/integration/self-contained test runners and helpers for Atlas/KEV, HuggingFace, Trifecta, TasteTester corpus and behavior, McpToolScanner, Static checks, pattern service, OSV/GHSA/VulnFeed flows, smoke and skeleton tests.

Sequence Diagram(s)

sequenceDiagram
  participant Client
  participant McpServer
  participant McpToolScanner
  participant TrifectaAnalyzer
  participant UnifiedCveCache
  participant CanaryService
  participant TasteTesterService
  Client->>McpServer: CallToolRequest(name, args)
  McpServer->>McpToolScanner: scan_mcp_tool(toolDescriptor)
  McpServer->>TrifectaAnalyzer: check_lethal_trifecta(input)
  McpServer->>UnifiedCveCache: query_cve(filters)
  McpServer->>CanaryService: deploy_canary(opts)
  McpServer->>CanaryService: verify_canary(content, handle?)
  McpServer->>TasteTesterService: taste_test(prompt, mode)

Loading

Estimated code review effort: 🎯 5 (Critical) | ⏱️ ~120 minutes

Possibly related PRs

• revsmoke/promptrejectormcp#3: Overlaps SkillScanService changes and mapping of Gemini-derived signals into scan results.

"A rabbit wrote this release note, hopped through tests,
I sniffed for smuggling, bit by bit in nested nests.
Patterns planted, canaries set, feeds gathered in a chest,
Taste tester nibbles prompts for trouble in the guest.
Hooray — the rabbit shakes a paw: v1.1.0 — now rest."

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch claude/hopeful-brahmagupta-338986

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 31868ea5b8

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 31868ea5b8

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[coderabbitai]

Actionable comments posted: 10

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (1)

src/services/SkillScanService.ts (1)

136-156: ⚠️ Potential issue | 🟠 Major | ⚡ Quick win

Include trifecta in final risk decision.

trifectaResult is computed but excluded from overallSeverity and isDangerous, so a lethal-trifecta hit can still return safe: true.

Suggested patch

         const hfSevIdx = SEVERITIES.indexOf(hfSeverity);
-        const overallSeverity = SEVERITIES[Math.max(geminiSevIdx, staticSevIdx, skillSevIdx, hfSevIdx)];
+        const trifectaSevIdx = trifectaResult.trifectaPresent ? SEVERITIES.indexOf("critical") : SEVERITIES.indexOf("low");
+        const overallSeverity = SEVERITIES[Math.max(geminiSevIdx, staticSevIdx, skillSevIdx, hfSevIdx, trifectaSevIdx)];

         // Decide "safe" status
         const isDangerous =
             overallSeverity === "critical" ||
             overallSeverity === "high" ||
             (geminiResult.isInjection && geminiResult.confidence > 0.6) ||
             skillSpecificResult.hasDangerousToolUsage ||
-            skillSpecificResult.hasNetworkExfiltration;
+            skillSpecificResult.hasNetworkExfiltration ||
+            trifectaResult.trifectaPresent;

Also applies to: 175-176

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/services/SkillScanService.ts` around lines 136 - 156, trifectaResult is
computed but not used in the final risk aggregation; update the aggregation to
include trifectaResult by adding trifectaResult.severity into the SEVERITIES
Math.max calculation that computes overallSeverity, include
...trifectaResult.categories in the categories Set alongside
geminiResult/staticResult/skillSpecificResult, and extend the isDangerous
boolean to consider trifectaResult's relevant danger flags (e.g.,
trifectaResult.isTrifecta / trifectaResult.isLethal /
trifectaResult.hasDangerousBehavior or similar property your code defines) so a
lethal trifecta triggers safe: false; adjust the same logic referenced around
overallSeverity and isDangerous (also applicable to the checks around lines
175-176).

🧹 Nitpick comments (4)

src/services/AtlasService.ts (1)

144-153: ⚡ Quick win

Add a timeout to the ATLAS bundle fetch.

The fetch() call has no timeout, so a hung or slow upstream server could block the refresh() call indefinitely. Consider using AbortController with a reasonable timeout (e.g., 30s).

🛠️ Proposed fix

     async refresh(): Promise<AtlasRefreshResult> {
         // Cache-hit short-circuit: keeps us off the network in CI/loops.
         if (this.cacheIsFresh()) {
             this.tryLoadCache();
             return { count: this.cache.size, fetchedAt: new Date().toISOString() };
         }

         let bundleJson: any;
         try {
+            const controller = new AbortController();
+            const timer = setTimeout(() => controller.abort(), 30_000);
-            const resp = await fetch(this.bundleUrl);
+            const resp = await fetch(this.bundleUrl, { signal: controller.signal });
+            clearTimeout(timer);
             if (!resp.ok) {
                 throw new Error(`ATLAS bundle fetch failed: HTTP ${resp.status}`);
             }
             bundleJson = await resp.json();
         } catch (err: any) {
             throw new Error(`ATLAS refresh failed: ${err?.message || err}`);
         }

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/services/AtlasService.ts` around lines 144 - 153, The fetch to
this.bundleUrl in the refresh() flow has no timeout; modify the refresh() method
to use an AbortController with a ~30s timeout: create an AbortController, start
a setTimeout that calls controller.abort() after 30_000ms, pass
controller.signal into fetch(this.bundleUrl), and clear the timeout on success;
catch the abort case (DOMException/AbortError) and throw a clear Error like
"ATLAS refresh timed out" while preserving other error messages (the current
catch that throws `ATLAS refresh failed: ...` should still wrap non-timeout
errors). Ensure the timeout is cleaned up to avoid leaks.

patterns/mcp-tool-poisoning.json (1)

7-7: Unverified ATLAS technique IDs.

All five pattern descriptions note [unverified ATLAS technique — pending verification]. Per the PR summary, this is a known limitation (ATLAS unverified IDs). Consider tracking verification as a post-merge task.

Also applies to: 27-27, 47-47, 67-67, 87-87

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@patterns/mcp-tool-poisoning.json` at line 7, The pattern descriptions in
patterns/mcp-tool-poisoning.json currently include the literal tag "[unverified
ATLAS technique — pending verification]" in five entries; update each
description to remove that bracketed tag (or replace it with a neutral,
non-ambiguous note like "verification pending") so the description text is
clean, and create a post-merge tracking issue (e.g., ISSUE-XXXX) to verify and,
if verified, annotate or update the patterns later; search for the exact string
"[unverified ATLAS technique — pending verification]" to locate and fix all
occurrences.

src/test/manyShotObfuscationTests.ts (1)

58-61: ⚡ Quick win

Consolidate to a single PatternService instance.

Lines 58-60 create two PatternService instances when one suffices. The first instance svc is used only to call regenerateManifest() and then discarded.

♻️ Simplify to single instance

     const dir = createTestDir();
-    const svc = new PatternService(dir);
-    svc.regenerateManifest();
-    const patSvc = new PatternService(dir);
-    const checker = new StaticCheckService(patSvc);
+    const patSvc = new PatternService(dir);
+    patSvc.regenerateManifest();
+    const checker = new StaticCheckService(patSvc);

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/test/manyShotObfuscationTests.ts` around lines 58 - 61, The test
unnecessarily constructs two PatternService instances (svc and patSvc); call
regenerateManifest() on a single PatternService instance and reuse it when
creating StaticCheckService. Replace the pattern where svc is only used to call
regenerateManifest() and discarded by invoking PatternService dir once (e.g.,
create one PatternService, call its regenerateManifest(), then pass that same
instance into the StaticCheckService constructor).

SPEC.md (1)

42-42: 💤 Low value

Specify language identifier for fenced code block.

The architecture diagram code block lacks a language identifier.

📝 Add language identifier

-```
+```text
 ┌─────────────────────────────────────────────────────────────────────────┐

As per coding guidelines, markdownlint rule MD040 requires language identifiers on fenced code blocks.

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@SPEC.md` at line 42, The fenced code block containing the architecture
diagram in SPEC.md is missing a language identifier; edit that diagram's opening
fence (the architecture diagram code block) to include a language label such as
"text" (e.g., change ``` to ```text) so the block complies with markdownlint
MD040.

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@CHANGELOG.md`:
- Around line 65-66: Update the stale test-count metrics in CHANGELOG.md: locate
the line containing the string "**426 assertions / 0 failures across 16 suites**
(up from 87 in v1.0.2)" and replace the assertion, failure and suite counts and
the "up from" baseline with the final numbers from the release test run; also
verify the adjacent "20-sample labeled Taste-Tester corpus..." line matches the
final validated sample/test pass counts and update if they changed. Ensure the
text formatting and parentheses remain unchanged.

In `@patterns/markdown-exfil.json`:
- Around line 6-8: The detector only matches Markdown image syntax; update the
"pattern" value for the rule named "Markdown image with suspicious query string"
so it also matches standard Markdown links by allowing either leading "!" or not
(i.e., support both "![](...)" and "[](...)"). Modify the regex in the "pattern"
field to accept an optional "!" before the link-start so both image and link
forms with long opaque query values are detected while preserving the existing
query-value length and character constraints.

In `@README.md`:
- Around line 449-454: The OSV.dev `/v1/querybatch` row in the README lists
example package names that don’t match the implemented AI-package allowlist
(e.g., "anthropic-sdk-python", "openai-python", "transformers-js"); update the
README entry so the listed packages exactly match the allowlist identifiers used
by the implementation (replace mismatched names in the bold OSV.dev
`/v1/querybatch` row with the actual package IDs used by the allowlist such as
the canonical registry names the code checks), double-check each token
(langchain, transformers, litellm, mlflow, ollama, llama-index, autogen, crewai,
langgraph, vllm, sglang, anthropic-sdk, openai, etc.) against the allowlist in
the code and make them identical to avoid verification mismatches.

In `@RESEARCH_FEEDS.md`:
- Around line 3-7: Remove the plan-mode/local-path artifact in
RESEARCH_FEEDS.md: locate the section labeled "Plan-mode output" that contains
the environment-specific local absolute path and replace that line with a
repo-relative or generic release-ready description (e.g., reference the doc by
repo-relative path or omit the machine-specific path entirely), ensuring the
file no longer contains any plan-mode instructions or local absolute paths
before merging.

In `@RESEARCH_THREATS.md`:
- Around line 75-86: The Markdown table anchored by the header row "| Item |
Date | Significance |" isn't terminated properly so the following "Sources:"
line is parsed as a table row; fix it by inserting a blank line (one empty
newline) immediately before the "Sources:" line so the table closes cleanly and
"Sources:" becomes a normal paragraph/header rather than part of the table.

In `@scripts/smoke-v1.1.ts`:
- Around line 56-59: The regex used to set looksNetworky (used with
partialOnNetworkErr) has a typo "ecconnreset" so ECONNRESET errors aren't
matched; update the network-error matcher regex (the
/(network|fetch|enotfound|timeout|ecconnreset|ssl|tls|getaddrinfo|429|503)/i
used to compute looksNetworky) to use the correct substring "econnreset" (or
include both "ec" and "econnreset" variants if desired) so ECONNRESET is
properly detected and downgraded to PARTIAL.
- Around line 167-176: The smoke step check_lethal_trifecta unconditionally
records "PASS" even when trifectaAnalyzer.analyze returns trifectaPresent=false
or severity has regressed; update the step to inspect the returned result (the
variable r from trifectaAnalyzer.analyze) and assert expected values (e.g.,
r.trifectaPresent === true and r.severity meets your threshold), then call
record("check_lethal_trifecta", "PASS", ...) only on success and record(...,
"FAIL", ...) (or throw) with diagnostic details when the assertion fails so
failures are surfaced; locate the call site around
tryStep("check_lethal_trifecta", ...) and update handling of r to conditionally
record PASS/FAIL based on r.trifectaPresent and r.severity.

In `@src/mcp/mcpServer.ts`:
- Around line 337-388: Branches for scan_mcp_tool, check_lethal_trifecta,
query_cve, deploy_canary, verify_canary and taste_test call handlers
(mcpToolScanner.scan, trifectaAnalyzer.analyze, unifiedCveCache.query,
canaryService.issueToken, canaryService.checkEcho, tasteTesterService.run) using
blindly-cast args and can throw or start unbounded work on malformed input. Add
explicit runtime validation for required fields and types (e.g., ensure tool is
object for scan_mcp_tool; capabilities/tools/skillContent are correct types for
check_lethal_trifecta; filters conform to QueryCveFilters for query_cve;
context/ttlSeconds types for deploy_canary; content string for verify_canary;
prompt string for taste_test), return a safe structured error response when
validation fails, and only call the corresponding methods after validation.
Ensure the same pattern as check_prompt/scan_skill (guard, validate,
early-return on bad input, then invoke mcpToolScanner.scan /
trifectaAnalyzer.analyze / unifiedCveCache.query / canaryService.issueToken /
canaryService.checkEcho / tasteTesterService.run).

In `@src/services/KevFeedService.ts`:
- Around line 54-59: The loop that climbs directories using
dirname(import.meta.url) can never terminate on non-POSIX roots because it only
checks for "/" — update the traversal in KevFeedService so it also breaks when
the directory stops changing (e.g., dir === dirname(dir)) or when dir equals the
filesystem root (use path.parse(dir).root) as a stop condition; modify the while
condition (and/or add an explicit break) around the existsSync(join(dir,
"package.json")) check so the loop exits on Windows roots, keeping use of
fileURLToPath, dirname, existsSync, join, and maintaining assignment to
this.cacheDir.
- Around line 127-136: The fetch in KevFeedService.refresh() is unbounded and
can hang; wrap the fetch(this.feedUrl) with an AbortController and a timeout
timer (e.g., setTimeout) so the request is aborted after a configurable
interval, pass controller.signal to fetch, clear the timer on success, and catch
AbortError specifically to throw a clear timeout/abort error; ensure body =
await resp.json() only runs when not aborted and that the controller is cleaned
up on all paths.

---

Outside diff comments:
In `@src/services/SkillScanService.ts`:
- Around line 136-156: trifectaResult is computed but not used in the final risk
aggregation; update the aggregation to include trifectaResult by adding
trifectaResult.severity into the SEVERITIES Math.max calculation that computes
overallSeverity, include ...trifectaResult.categories in the categories Set
alongside geminiResult/staticResult/skillSpecificResult, and extend the
isDangerous boolean to consider trifectaResult's relevant danger flags (e.g.,
trifectaResult.isTrifecta / trifectaResult.isLethal /
trifectaResult.hasDangerousBehavior or similar property your code defines) so a
lethal trifecta triggers safe: false; adjust the same logic referenced around
overallSeverity and isDangerous (also applicable to the checks around lines
175-176).

---

Nitpick comments:
In `@patterns/mcp-tool-poisoning.json`:
- Line 7: The pattern descriptions in patterns/mcp-tool-poisoning.json currently
include the literal tag "[unverified ATLAS technique — pending verification]" in
five entries; update each description to remove that bracketed tag (or replace
it with a neutral, non-ambiguous note like "verification pending") so the
description text is clean, and create a post-merge tracking issue (e.g.,
ISSUE-XXXX) to verify and, if verified, annotate or update the patterns later;
search for the exact string "[unverified ATLAS technique — pending
verification]" to locate and fix all occurrences.

In `@SPEC.md`:
- Line 42: The fenced code block containing the architecture diagram in SPEC.md
is missing a language identifier; edit that diagram's opening fence (the
architecture diagram code block) to include a language label such as "text"
(e.g., change ``` to ```text) so the block complies with markdownlint MD040.

In `@src/services/AtlasService.ts`:
- Around line 144-153: The fetch to this.bundleUrl in the refresh() flow has no
timeout; modify the refresh() method to use an AbortController with a ~30s
timeout: create an AbortController, start a setTimeout that calls
controller.abort() after 30_000ms, pass controller.signal into
fetch(this.bundleUrl), and clear the timeout on success; catch the abort case
(DOMException/AbortError) and throw a clear Error like "ATLAS refresh timed out"
while preserving other error messages (the current catch that throws `ATLAS
refresh failed: ...` should still wrap non-timeout errors). Ensure the timeout
is cleaned up to avoid leaks.

In `@src/test/manyShotObfuscationTests.ts`:
- Around line 58-61: The test unnecessarily constructs two PatternService
instances (svc and patSvc); call regenerateManifest() on a single PatternService
instance and reuse it when creating StaticCheckService. Replace the pattern
where svc is only used to call regenerateManifest() and discarded by invoking
PatternService dir once (e.g., create one PatternService, call its
regenerateManifest(), then pass that same instance into the StaticCheckService
constructor).

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: 661a4984-e2c8-4ce8-806a-18c1e13fe9c0

📥 Commits

Reviewing files that changed from the base of the PR and between 292b37d and 31868ea.

⛔ Files ignored due to path filters (1)

• package-lock.json is excluded by !**/package-lock.json

📒 Files selected for processing (54)

• CHANGELOG.md
• PLAN.md
• README.md
• RESEARCH_FEEDS.md
• RESEARCH_THREATS.md
• SKILLS_SECURITY.md
• SPEC.md
• package.json
• patterns/.gitignore
• patterns/llm-threats.json
• patterns/manifest.json
• patterns/many-shot.json
• patterns/markdown-exfil.json
• patterns/mcp-tool-poisoning.json
• patterns/policy-puppetry.json
• patterns/prompt-injection.json
• patterns/unicode-smuggling.json
• scripts/smoke-v1.1.ts
• src/mcp/mcpServer.ts
• src/schemas/PatternSchemas.ts
• src/services/AtlasService.ts
• src/services/CanaryService.ts
• src/services/GeminiService.ts
• src/services/GhsaGraphQLService.ts
• src/services/HuggingFaceService.ts
• src/services/KevFeedService.ts
• src/services/McpToolScanner.ts
• src/services/OsvFeedService.ts
• src/services/SecurityService.ts
• src/services/SkillScanService.ts
• src/services/StaticCheckService.ts
• src/services/TasteTesterService.ts
• src/services/TrifectaAnalyzer.ts
• src/services/UnifiedCveCache.ts
• src/services/VulnFeedService.ts
• src/services/aiPackageAllowlist.ts
• src/test/atlasKevTests.ts
• src/test/canaryTests.ts
• src/test/fixtures/taste-tester-corpus.json
• src/test/helpers/mockAnthropic.ts
• src/test/helpers/mockFetch.ts
• src/test/huggingFaceTests.ts
• src/test/manyShotObfuscationTests.ts
• src/test/markdownExfilTests.ts
• src/test/mcpToolScannerTests.ts
• src/test/patternServiceTests.ts
• src/test/policyPuppetryTests.ts
• src/test/queryCveTests.ts
• src/test/tasteTesterCorpusTests.ts
• src/test/tasteTesterTests.ts
• src/test/trifectaTests.ts
• src/test/unicodeSmugglingTests.ts
• src/test/v11SkeletonTests.ts
• src/test/vulnFeed2Tests.ts

[coderabbitai]

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@src/services/TasteTesterService.ts`:
- Around line 680-686: The branch that tests /\.env|credentials/i using the
variable path currently returns an OpenSSH private key (copy-paste error);
update that branch in TasteTesterService (the if
(/\.env|credentials/i.test(path)) block) to return a realistic .env style string
(e.g., lines of KEY=value pairs separated by newlines such as
"API_KEY=placeholder\nDB_PASSWORD=placeholder\nENV=development") instead of the
SSH key block so .env mocks use KEY=value format.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: 0a26029e-f62a-4138-a9d7-0d993b92ae2a

📥 Commits

Reviewing files that changed from the base of the PR and between c71ea71 and bc541bb.

📒 Files selected for processing (2)

• .env.example
• src/services/TasteTesterService.ts

[coderabbitai]

Actionable comments posted: 5

🧹 Nitpick comments (2)

SPEC.md (1)

251-251: ⚡ Quick win

Clarify conditional requirement wording for ANTHROPIC_API_KEY.

The spec currently says all new env vars are optional, while also marking ANTHROPIC_API_KEY required when Taste-Tester is enabled. A one-line clarification (“optional unless TASTE_TESTER_ENABLED=true”) would prevent config ambiguity.

Also applies to: 378-378

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@SPEC.md` at line 251, Update the SPEC.md entries for ANTHROPIC_API_KEY (and
the similar entry at the other location) to make the conditional requirement
explicit by changing the one-line note to state that the variable is optional
unless TASTE_TESTER_ENABLED=true; reference the exact env var names
ANTHROPIC_API_KEY and TASTE_TESTER_ENABLED in the sentence so readers understand
the dependency and remove ambiguity.

scripts/calibrate-taste-tester.ts (1)

34-37: ⚡ Quick win

Add documentation linking pricing constants to official Anthropic pricing source.

Current values ($5.00 input, $25.00 output, $6.25 cache-write, $0.50 cache-read per 1M tokens) are accurate per official Anthropic pricing, but these constants lack a reference to their source. Since the hard cap check at line 134 depends directly on these rates, adding a comment or documentation URL will help maintainers quickly verify pricing hasn't drifted when making updates in the future.

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@scripts/calibrate-taste-tester.ts` around lines 34 - 37, Add a comment above
the pricing constants (OPUS_4_7_INPUT_PER_MTOK, OPUS_4_7_OUTPUT_PER_MTOK,
OPUS_4_7_CACHE_WRITE_PER_MTOK, OPUS_4_7_CACHE_READ_PER_MTOK) that cites the
official Anthropic pricing URL and the date you verified it, so future
maintainers can quickly validate these rates; ensure the comment also notes that
the hard cap check relies on these values so reviewers know to re-check the
source when updating them.

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@scripts/calibrate-taste-tester.ts`:
- Around line 134-168: The exit logic and summary text are allowing partial-run
success and mislabeling the run as "mocks": update the success gate and
messaging so process.exit only returns 0 when both matches >= 16 and the full
expected corpus was processed (e.g., results.length === EXPECTED_SAMPLE_COUNT or
=== 20) instead of permitting early-abort partial runs (references:
process.exit(...) and results), and change the summary/oneLine string (variable
oneLine) to remove "mocks" (or otherwise reflect that this is a real-API run) so
the output correctly describes the run; keep existing ABORT_ON_FIRST_FAILURE and
COST_HARD_CAP_USD checks but ensure they cause non-zero exit when they produced
a partial run.
- Around line 110-121: The calibration script is missing monitor API token usage
because runMonitor()'s response.usage isn't stored; update the monitor flow to
capture and persist that usage (e.g., add a monitorUsage field on
TasteTesterResult or include it in BehaviorReport), modify
TasteTesterService.runMonitor (call site at TasteTesterService.ts:776) to return
or attach response.usage into the result object, and then alter
scripts/calibrate-taste-tester.ts to aggregate tokens from result.monitorUsage
in addition to result.tasterTranscript (summing input_tokens, output_tokens,
cache_creation_input_tokens, cache_read_input_tokens) so monitor spend is
included in cap accounting.

In `@SPEC.md`:
- Line 42: In SPEC.md update the fenced code block marker (the ``` fence shown)
to include a language tag (e.g., change ``` to ```text) so the block declares
its language and satisfies markdownlint rule MD040; locate the fence in SPEC.md
and add the language identifier to the opening backticks.
- Line 266: Pick a single canonical TTL and make SPEC.md and the config agree:
update the table row that currently reads “MITRE ATLAS (stix bundle) … cache
24h” and the configuration/default for ATLAS_REFRESH_INTERVAL_HOURS (currently
set to 168) so they match the chosen value; ensure any nearby docs/comments that
mention the ATLAS cache duration are updated too to remain consistent.
- Line 136: SPEC.md currently shows an invalid parameter shape for
check_lethal_trifecta; update the signature to match the actual implementation
used in mcpServer.ts and TrifectaAnalyzer.ts by changing the input type to an
object with optional fields (capabilities?: string[]; tools?: string[];
skillContent?: string;) so the SPEC accurately reflects the function parameter
shape for check_lethal_trifecta.

---

Nitpick comments:
In `@scripts/calibrate-taste-tester.ts`:
- Around line 34-37: Add a comment above the pricing constants
(OPUS_4_7_INPUT_PER_MTOK, OPUS_4_7_OUTPUT_PER_MTOK,
OPUS_4_7_CACHE_WRITE_PER_MTOK, OPUS_4_7_CACHE_READ_PER_MTOK) that cites the
official Anthropic pricing URL and the date you verified it, so future
maintainers can quickly validate these rates; ensure the comment also notes that
the hard cap check relies on these values so reviewers know to re-check the
source when updating them.

In `@SPEC.md`:
- Line 251: Update the SPEC.md entries for ANTHROPIC_API_KEY (and the similar
entry at the other location) to make the conditional requirement explicit by
changing the one-line note to state that the variable is optional unless
TASTE_TESTER_ENABLED=true; reference the exact env var names ANTHROPIC_API_KEY
and TASTE_TESTER_ENABLED in the sentence so readers understand the dependency
and remove ambiguity.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: 460e8267-d459-435c-9ecd-a95f374464c3

📥 Commits

Reviewing files that changed from the base of the PR and between bc541bb and 5b8a75a.

📒 Files selected for processing (3)

• CHANGELOG.md
• SPEC.md
• scripts/calibrate-taste-tester.ts

✅ Files skipped from review due to trivial changes (1)

• CHANGELOG.md

[revsmoke]

Review round 1 — addressed in dbc3454

All 18 inline findings from CodeRabbit + Codex have been addressed in a single consolidation commit. Each thread above has a reply linking to the specific fix. Quick summary:

Correctness fixes (11)

• SkillScanService — trifectaResult now folds into overallSeverity + isDangerous. A 3-of-3 lethal trifecta returns safe: false, severity: critical with synthetic lethal_trifecta category and ATLAS T0024 + T0051. Pre-release behavior change, no shipped consumers affected. CHANGELOG entry added.
• StaticCheckService — flagGroup loop scans all matches, takes max severity (Codex finding + Pass 12 known issue).
• KevFeedService — POSIX/Windows-safe root traversal; AbortController-driven 30s fetch timeout.
• mcpServer — runtime argument validation on all 6 new tool handlers via validationError() helper matching existing error envelope.
• markdown-exfil regex — now catches links as well as images; FP-guarded against benign GitHub links.
• TasteTesterService — .env mock returns KEY=value shape (with sk-test-FAKE tripwire preserved); added TasteTesterUsage accumulator so the calibration script can read real token totals.
• smoke-v1.1.ts — ECONNRESET typo + missing trifecta assertion.
• calibrate-taste-tester.ts — fixed broken usage extraction (Path A: service-level counter); exit code gated on full-corpus completion.

Doc consistency (7)

• CHANGELOG test totals, README OSV allowlist, RESEARCH_FEEDS plan-mode artifact, RESEARCH_THREATS table termination, SPEC fence tag / check_lethal_trifecta signature / ATLAS TTL.

Outside-diff finding

The trifecta-in-rollup finding from the third CodeRabbit walkthrough is the same fix as the SkillScanService correctness item above.

CI note for the maintainer

The claude-review workflow ran but logged No buffered inline comments because ANTHROPIC_API_KEY isn't set on the runner. The workflow exits cleanly so it's a green check, but the workflow does no actual review work. To enable it: add ANTHROPIC_API_KEY to repo secrets (or to the claude-review workflow's env). Not a blocker — surfacing it because the check name suggests it's doing real review work, which it currently isn't.

Verification

• npm run build clean
• Per-cluster test suites individually green (atlasKev 24/24, manyShotObfuscation 27/27 with Sneaky Bits now critical, trifecta 49/49, markdownExfil 31/31, tasteTester 89/89, corpus 6/6, skeleton 27/27)
• Manifest regenerated, integrity PASSED, 71 patterns
• Full npm test chain hung on the network-dependent atlasKevTests live-CISA fetch (pre-existing environment quirk); each suite verified individually instead

Pre-existing failure flagged for separate triage

skillScanTests "Social Engineering" case fails because the hardcoded SQLi regex (SELECT|...|UPDATE|...)\s+(FROM|...) matches the literal phrase "security UPDATE FROM Anthropic" in the test prompt. Confirmed unrelated to this PR via git stash baseline. Worth a regex-FP follow-up but out of scope for this round.

[coderabbitai]

Actionable comments posted: 4

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@src/services/StaticCheckService.ts`:
- Around line 97-100: The simple-mode call to regex.test in StaticCheckService
(used by evaluatePattern / ActivePattern via checkWithPatternService) can mutate
RegExp.lastIndex for global regexes and cause flaky results; fix by ensuring the
regex's lastIndex is reset (or use a fresh RegExp instance) before calling
.test(text) in the simple-mode branch so repeated
evaluatePattern/checkWithPatternService calls do not observe a polluted
lastIndex and match behavior remains deterministic.

In `@src/services/TasteTesterService.ts`:
- Around line 547-552: The final severity calculation currently folds only
monitorReport.intents into intentMax and then into monitorReport.severity;
update the logic to also consider raw deterministic tool evidence from
monitorReport.toolCalls (e.g., entries like exec_shell or transfer_funds) so the
final severity is floored by any high-severity toolCalls. Specifically, when
computing the combined severity (using maxSeverity and monitorReport.severity),
include a reduction over monitorReport.toolCalls to compute a toolCallsMax
severity and then take maxSeverity(monitorReport.severity, intentMax,
toolCallsMax); apply the same fix in the other analogous block that calculates
severity (the block currently around lines 765-792).
- Around line 382-395: The withTimeout<T> helper currently only rejects the
outer promise but doesn't cancel the underlying Anthropic request, so wrap the
call to client.messages.create with an AbortController and pass its signal (or
accept a controller into withTimeout) and ensure controller.abort() is invoked
when the timeout fires; specifically, update withTimeout (and the call sites
that invoke messages.create) to create an AbortController, pass
controller.signal into messages.create options, store the controller in the
closure or parameters, and call controller.abort() inside the setTimeout
rejection branch so messages.create is actually cancelled on timeout.

In `@src/test/atlasKevTests.ts`:
- Around line 224-254: The test can hang if svc.refresh() never settles; wrap
the call inside a hard guard using Promise.race so the test always times out
fast: in the withMockedFetch callback, replace the direct await svc.refresh()
usage with awaiting Promise.race([svc.refresh(), new Promise((_, rej) =>
setTimeout(() => rej(new Error("test hard timeout")), <short-ms>))]) and assert
against that rejection; update references around KevFeedService, refresh, and
the local timing/assertion logic (start/elapsed/threw/msg) so the test fails
quickly when the hard guard fires.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: b77dc863-9acc-4be8-be6b-9c6646f91b93

📥 Commits

Reviewing files that changed from the base of the PR and between 5b8a75a and dbc3454.

📒 Files selected for processing (17)

• CHANGELOG.md
• README.md
• RESEARCH_FEEDS.md
• RESEARCH_THREATS.md
• SPEC.md
• patterns/manifest.json
• patterns/markdown-exfil.json
• scripts/calibrate-taste-tester.ts
• scripts/smoke-v1.1.ts
• src/mcp/mcpServer.ts
• src/services/KevFeedService.ts
• src/services/SkillScanService.ts
• src/services/StaticCheckService.ts
• src/services/TasteTesterService.ts
• src/test/atlasKevTests.ts
• src/test/manyShotObfuscationTests.ts
• src/test/markdownExfilTests.ts

✅ Files skipped from review due to trivial changes (5)

• patterns/manifest.json
• SPEC.md
• RESEARCH_FEEDS.md
• RESEARCH_THREATS.md
• CHANGELOG.md

🚧 Files skipped from review as they are similar to previous changes (9)

• patterns/markdown-exfil.json
• src/test/markdownExfilTests.ts
• scripts/smoke-v1.1.ts
• src/mcp/mcpServer.ts
• src/services/SkillScanService.ts
• README.md
• src/services/KevFeedService.ts
• src/test/manyShotObfuscationTests.ts
• scripts/calibrate-taste-tester.ts