feat(skills): add PyRIT AI red-teaming skill - Fixes #11#29
Open
juandresrodca wants to merge 1 commit into
Open
feat(skills): add PyRIT AI red-teaming skill - Fixes #11#29juandresrodca wants to merge 1 commit into
juandresrodca wants to merge 1 commit into
Conversation
Add `pyrit-red-teaming`, an automated LLM red-teaming skill wrapping
Microsoft PyRIT. It drives single-turn (PromptSendingOrchestrator) attacks
against an OpenAI-compatible chat target, scores each response for refusal
(SelfAskRefusalScorer), and writes OWASP-LLM-mapped findings into the
standard engagement OUTPUT_DIR.
Structure (repo skill convention: SKILL.md + reference/ + tools/):
- SKILL.md — methodology, attack coverage, integration with ai-threat-testing.
- reference/setup-and-targets.md — install, OpenAI-compatible target config,
separate judge model, troubleshooting.
- reference/pyrit-orchestrators.md — orchestrator/converter/scorer reference
and how to extend the runner to multi-turn (RedTeaming/Crescendo).
- tools/pyrit_runner.py — CLI. PyRIT imported lazily and confined to one
function; missing/incompatible PyRIT degrades with guidance instead of
crashing. `--dry-run` exercises the whole pipeline with a local mock target
(no PyRIT, no network, no API key).
- tools/finding_writer.py — normalized-result -> OUTPUT_DIR writer
(output-discipline.md tree + formats/data.md finding schema). No PyRIT dep.
- tools/requirements.txt — pyrit (installed separately; 3.10-3.13 only).
Outputs: findings/finding-NNN/{finding.json,description.md,evidence/},
artifacts/pyrit-report.json, reports/pyrit-report.md, logs/attempts.ndjson
(every attempt incl. refusals). CVSS is left null for analyst assignment;
poc_verified=true since PyRIT executed the attack.
Wired into skills/INDEX.md (router) and README.md skill listing.
Tested (Python 3.9, no PyRIT): `finding_writer.py --selftest` passes;
`pyrit_runner.py --dry-run` produces a valid OUTPUT_DIR (6 attempted ->
4 findings, 2 refused), all finding.json parse-valid; both modules
byte-compile. The live PyRIT path targets the documented
PromptSendingOrchestrator API — verify against your installed version.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Add
pyrit-red-teaming, an automated LLM red-teaming skill wrapping Microsoft PyRIT. It drives single-turn (PromptSendingOrchestrator) attacks against an OpenAI-compatible chat target, scores each response for refusal (SelfAskRefusalScorer), and writes OWASP-LLM-mapped findings into the standard engagement OUTPUT_DIR.Structure (repo skill convention: SKILL.md + reference/ + tools/):
--dry-runexercises the whole pipeline with a local mock target (no PyRIT, no network, no API key).Outputs: findings/finding-NNN/{finding.json,description.md,evidence/}, artifacts/pyrit-report.json, reports/pyrit-report.md, logs/attempts.ndjson (every attempt incl. refusals). CVSS is left null for analyst assignment; poc_verified=true since PyRIT executed the attack.
Wired into skills/INDEX.md (router) and README.md skill listing.
Tested (Python 3.9, no PyRIT):
finding_writer.py --selftestpasses;pyrit_runner.py --dry-runproduces a valid OUTPUT_DIR (6 attempted -> 4 findings, 2 refused), all finding.json parse-valid; both modules byte-compile. The live PyRIT path targets the documented PromptSendingOrchestrator API — verify against your installed version.Summary
Related Issue
Closes #
Changes Made
Type of Change
Testing
python -m pytest tests/)Test details:
Checklist
type(scope): descriptionSKILL.mdandreference/directoryCloses #NSkill / agent checklist (if PR touches
skills/)python3 scripts/skill_linter.pyruns clean (or rationale below).skills/hackthebox/(no machine names, no XBEN/Vulnlab IDs, no preservedFLAG{...}literals, no lab IPs).SKILL.md≤ 150 lines;reference/*.md≤ 200 lines;reference/scenarios/*.md≤ 400 lines.DO NOT/MUST NOT/NEVERoutside## Anti-Patterns(unless file is in linter's hard-contract allowlist)./skill-pruneto confirm negative ROI.Screenshots / Evidence