feat: semantic scanner + benchmark harness + LangGraph adapter (54 tests) - #15
Open
Pranjal0410 wants to merge 1 commit into
Open
feat: semantic scanner + benchmark harness + LangGraph adapter (54 tests)#15Pranjal0410 wants to merge 1 commit into
Pranjal0410 wants to merge 1 commit into
Conversation
…based on v0.2 scaffolding - Semantic fallback scanner (embedding-based, pluggable backends) — 22 tests - Adversarial benchmark harness (50 payloads, precision/recall/F1) — 18 tests - LangGraph FirewallNode adapter (implements empty stub) — 14 tests - All files under sdk/python/acf/ per v0.2 folder structure - 54 tests passing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Rebased on the latest scaffolding pushed by @tharindupr. Supersedes #10 and #12 (which were against the old repo structure).
Semantic fallback scanner (
sdk/python/acf/scanners/)Embedding-based detection for the scan stage — the "semantic fallback" defined in the v0.2 architecture. Compares normalised inputs against a pre-computed library of 49 attack patterns across 6 categories (instruction override, data exfiltration, role hijack, context manipulation, tool abuse, encoding evasion). Pluggable backends:
SentenceTransformerBackendfor production (~2-4ms),TfidfBackendfor CI (<1ms). Only fires after lexical PROCEED — respects the short-circuit model. Output produces risk_score + semantic_hits for the risk aggregator. 22 tests.Adversarial benchmark harness (
sdk/python/acf/benchmarks/)Scanner-agnostic runner that measures detection quality. 50 curated payloads — 25 malicious across 7 categories + 25 benign including 10 hard negatives containing trigger-like phrases ("ignore previous commits", "system prompt debugging", "override a method in Python"). Reports precision, recall, F1, per-category detection rates, latency percentiles (p50/p95/p99). JSON export for CI. Quality gate exits non-zero if F1 drops. 18 tests.
LangGraph FirewallNode adapter (
sdk/python/acf/adapters/langgraph.py)Implements the empty adapter stub from the scaffolding. Wraps all four v1 hooks as LangGraph-compatible guard nodes:
prompt_guard— evaluates user messages via on_promptcontext_guard— filters RAG chunks via on_context, drops BLOCK, replaces SANITISEtool_guard— evaluates tool calls via on_tool_callmemory_guard— evaluates memory writes via on_memoryBLOCK raises NodeInterrupt. SANITISE replaces content in state. ALLOW passes through. 14 tests (mocked, no langgraph dependency needed).
54 tests passing