Skip to content

test(integration): add benign hard negatives to the adversarial corpus - #72

Open
VibhorGautam wants to merge 1 commit into
c2siorg:mainfrom
VibhorGautam:test/hard-negatives
Open

test(integration): add benign hard negatives to the adversarial corpus#72
VibhorGautam wants to merge 1 commit into
c2siorg:mainfrom
VibhorGautam:test/hard-negatives

Conversation

@VibhorGautam

@VibhorGautam VibhorGautam commented Aug 5, 2026

Copy link
Copy Markdown
Contributor

adds 16 benign hard negatives (ap-109 to ap-124) that Kavishka put together while reproducing the detection numbers. these measure the false positive side that the corpus never covered, we only ever counted catch rate

split into 2 groups, same convention the corpus already uses:

  • ap-109 to ap-117 pass today and pin current behaviour. 4 of them (ap-114 to ap-117) are canaries, each carries exactly 1 of the chars a later shellMetachars change might add (?, bare &, [], *). they pass now and go red the moment that char lands, which is the point, it catches the regression before it ships instead of after
  • ap-118 to ap-124 are real false positives, tracked as tolerant gap probes (expected = current wrong verdict, desired = ALLOW) exactly like the attack-side gaps. e.g. ap-120, the allowlisted calculator getting BLOCKed on (5 > 3) && (2 < 4) because checkToolDangerousParams flattens all params into a single string and substring-scans with no per-tool context

corpus only, no code or policy changes. full integration suite green locally with -count=1, 9 strict cases pass, 7 gaps register as open. authored by Kavishka, landed verbatim

ap-109 to ap-124 from Kavishka's false positive review. 9 pass today and
pin current behaviour, 4 of those are canaries for the exact chars the
ap-097/098 fix wants to add. the other 7 are real false positives tracked
as open gap probes, same pattern as the attack side gaps
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant