Skip to content

feat(skills): add PyRIT AI red-teaming skill - Fixes #11#29

Open
juandresrodca wants to merge 1 commit into
transilienceai:mainfrom
juandresrodca:feature/pyrit-ai-red-teaming-skill
Open

feat(skills): add PyRIT AI red-teaming skill - Fixes #11#29
juandresrodca wants to merge 1 commit into
transilienceai:mainfrom
juandresrodca:feature/pyrit-ai-red-teaming-skill

Conversation

@juandresrodca

Copy link
Copy Markdown

Add pyrit-red-teaming, an automated LLM red-teaming skill wrapping Microsoft PyRIT. It drives single-turn (PromptSendingOrchestrator) attacks against an OpenAI-compatible chat target, scores each response for refusal (SelfAskRefusalScorer), and writes OWASP-LLM-mapped findings into the standard engagement OUTPUT_DIR.

Structure (repo skill convention: SKILL.md + reference/ + tools/):

  • SKILL.md — methodology, attack coverage, integration with ai-threat-testing.
  • reference/setup-and-targets.md — install, OpenAI-compatible target config, separate judge model, troubleshooting.
  • reference/pyrit-orchestrators.md — orchestrator/converter/scorer reference and how to extend the runner to multi-turn (RedTeaming/Crescendo).
  • tools/pyrit_runner.py — CLI. PyRIT imported lazily and confined to one function; missing/incompatible PyRIT degrades with guidance instead of crashing. --dry-run exercises the whole pipeline with a local mock target (no PyRIT, no network, no API key).
  • tools/finding_writer.py — normalized-result -> OUTPUT_DIR writer (output-discipline.md tree + formats/data.md finding schema). No PyRIT dep.
  • tools/requirements.txt — pyrit (installed separately; 3.10-3.13 only).

Outputs: findings/finding-NNN/{finding.json,description.md,evidence/}, artifacts/pyrit-report.json, reports/pyrit-report.md, logs/attempts.ndjson (every attempt incl. refusals). CVSS is left null for analyst assignment; poc_verified=true since PyRIT executed the attack.

Wired into skills/INDEX.md (router) and README.md skill listing.

Tested (Python 3.9, no PyRIT): finding_writer.py --selftest passes; pyrit_runner.py --dry-run produces a valid OUTPUT_DIR (6 attempted -> 4 findings, 2 refused), all finding.json parse-valid; both modules byte-compile. The live PyRIT path targets the documented PromptSendingOrchestrator API — verify against your installed version.

Summary

Related Issue

Closes #

Changes Made

Type of Change

  • Bug fix (non-breaking change that fixes an issue)
  • New skill or agent
  • Enhancement to existing skill/agent
  • Documentation update
  • CI/CD or infrastructure
  • Refactoring (no functional change)
  • Other:

Testing

  • Tested skill/agent in Claude Code session
  • Tested against vulnerable application (DVWA, WebGoat, Juice Shop, etc.)
  • Verified no false positives
  • Ran existing tests (python -m pytest tests/)
  • Manual review only (documentation/config changes)

Test details:

Checklist

  • My code follows the contribution guidelines
  • Commits use conventional format: type(scope): description
  • I've updated documentation where needed
  • New skills include SKILL.md and reference/ directory
  • No secrets, credentials, or unauthorized target information included
  • This PR links to an issue with Closes #N

Skill / agent checklist (if PR touches skills/)

  • python3 scripts/skill_linter.py runs clean (or rationale below).
  • No new challenge-specific identifiers outside skills/hackthebox/ (no machine names, no XBEN/Vulnlab IDs, no preserved FLAG{...} literals, no lab IPs).
  • SKILL.md ≤ 150 lines; reference/*.md ≤ 200 lines; reference/scenarios/*.md ≤ 400 lines.
  • No new DO NOT / MUST NOT / NEVER outside ## Anti-Patterns (unless file is in linter's hard-contract allowlist).
  • Cross-cutting rules (brute-force, output-discipline, env-reader, skill-update) live in exactly one canonical home; other files reference it.
  • If removing content, ran /skill-prune to confirm negative ROI.

Screenshots / Evidence

Add `pyrit-red-teaming`, an automated LLM red-teaming skill wrapping
Microsoft PyRIT. It drives single-turn (PromptSendingOrchestrator) attacks
against an OpenAI-compatible chat target, scores each response for refusal
(SelfAskRefusalScorer), and writes OWASP-LLM-mapped findings into the
standard engagement OUTPUT_DIR.

Structure (repo skill convention: SKILL.md + reference/ + tools/):
- SKILL.md — methodology, attack coverage, integration with ai-threat-testing.
- reference/setup-and-targets.md — install, OpenAI-compatible target config,
  separate judge model, troubleshooting.
- reference/pyrit-orchestrators.md — orchestrator/converter/scorer reference
  and how to extend the runner to multi-turn (RedTeaming/Crescendo).
- tools/pyrit_runner.py — CLI. PyRIT imported lazily and confined to one
  function; missing/incompatible PyRIT degrades with guidance instead of
  crashing. `--dry-run` exercises the whole pipeline with a local mock target
  (no PyRIT, no network, no API key).
- tools/finding_writer.py — normalized-result -> OUTPUT_DIR writer
  (output-discipline.md tree + formats/data.md finding schema). No PyRIT dep.
- tools/requirements.txt — pyrit (installed separately; 3.10-3.13 only).

Outputs: findings/finding-NNN/{finding.json,description.md,evidence/},
artifacts/pyrit-report.json, reports/pyrit-report.md, logs/attempts.ndjson
(every attempt incl. refusals). CVSS is left null for analyst assignment;
poc_verified=true since PyRIT executed the attack.

Wired into skills/INDEX.md (router) and README.md skill listing.

Tested (Python 3.9, no PyRIT): `finding_writer.py --selftest` passes;
`pyrit_runner.py --dry-run` produces a valid OUTPUT_DIR (6 attempted ->
4 findings, 2 refused), all finding.json parse-valid; both modules
byte-compile. The live PyRIT path targets the documented
PromptSendingOrchestrator API — verify against your installed version.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant