LLM Adversarial Security & Alignment

A comprehensive collection of 12 learning modules and capstone projects for LLM adversarial security, red teaming, and alignment defenses.

---

Welcome to the LLM Adversarial Security & Alignment curriculum. This repository provides a structured, hands-on syllabus for security engineers, AI researchers, and developers seeking to model threats, identify vulnerabilities, and engineer alignment defenses for Large Language Model (LLM) applications.

Through 12 comprehensive modules, you will explore the landscape of AI security — spanning runtime attacks like prompt injection and jailbreaking, training-time hazards like data poisoning, model privacy leakage, supply chain integrity, and automated red teaming protocols. Each module pairs a working attack implementation with a corresponding defense implementation, so you learn both sides of the threat model.

---

Repository Structure

llm-adversarial-security-alignment/
├── 01-introduction-to-ai-security/
├── 02-prompt-injection/
├── 03-jailbreaking-techniques/
├── 04-data-poisoning/
├── 05-model-extraction-and-stealing/
├── 06-adversarial-examples/
├── 07-membership-inference-attacks/
├── 08-model-inversion-attacks/
├── 09-supply-chain-attacks/
├── 10-rag-security/
├── 11-agent-and-agentic-ai-security/
├── 12-defense-mechanisms-and-red-teaming/
├── capstone-projects/
├── ai_security_learning_path_comprehensive_guide.pdf
├── requirements.txt
└── LICENSE

---

Getting Started

Prerequisites

• Python 3.10+
• pip for dependency management

Installation

# Clone the repository
git clone https://github.com/mohd-faizy/llm-adversarial-security-alignment.git
cd llm-adversarial-security-alignment

# Create and activate a virtual environment
python -m venv .venv
source .venv/bin/activate   # On Windows: .venv\Scripts\activate

# Install dependencies
pip install -r requirements.txt

Running a Module Script

Every module ships with standalone, runnable Python scripts. For example, to run the prompt injection demo:

python 02-prompt-injection/direct_injection.py

To run the test suite (where applicable) using pytest:

pytest -v

---

Curriculum & Core Modules

This learning path consists of 12 structured modules. Each module targets a key attack vector or defensive framework, paired with a practical engineering deliverable:

Module	Core Focus	Practical Deliverable
01: Introduction	Threat modeling foundations, OWASP LLM Top 10, and AI attack surface analysis.	AI Asset Inventory: An asset inventory and OWASP risk map for a production-grade AI system.
02: Prompt Injection	Direct, indirect, stored, recursive, and context-window injection vectors.	Prompt Injection Boundary Test: Before-and-after evaluations of structured boundaries and output validation defenses.
03: Jailbreaking Techniques	Safety alignment bypass patterns, automated probing, and guardrail heuristics.	Jailbreak Regression Suite: A versioned test suite assessing allow, refuse, clarify, and escalate outcomes.
04: Data Poisoning	Training dataset backdoors, sleeper agents, label flipping, and statistical detection.	Dataset Poisoning Review: A data card, provenance checklist, and backdoor detection plan.
05: Model Extraction	API-based model cloning, knowledge distillation risks, watermarking, and rate limiting.	Model Extraction Threat Model: API exposure review detailing output filters, rate limiting, and extraction signals.
06: Adversarial Examples	Perturbation mathematics (FGSM, PGD), physical-world evasion, and certified robustness.	Adversarial Robustness Evaluation: Clean-versus-perturbed accuracy matrix under white-box and black-box evaluations.
07: Membership Inference	Data privacy hazards, shadow models, LiRA, and overfitting leakage controls.	Privacy Risk Assessment: Leakage auditing of confidence scores, token probabilities, and memorization markers.
08: Model Inversion	Feature reconstruction, gradient inversion, and federated learning privacy risks.	Model Inversion Exposure Audit: Analysis of exposed embeddings, gradients, and model weight checkpoints.
09: Supply Chain Security	Weight-level vulnerabilities, pickle-based remote code execution, SBOMs, and provenance.	AI Supply Chain Intake: Compliance intake logs documenting model hashes, serialization safetensors, and sandboxing.
10: RAG Security	Vector poisoning, document retrieval hijacking, and database namespace isolation.	Secure RAG Review: Implementation of ingestion, retrieval, and generation guardrails.
11: Agentic AI Security	Tool calling abuse, confused deputy scenarios, LangGraph/MCP safety, and memory poisoning.	Secure Agent & MCP Design: Architecture blueprints, tool permission levels, and MCP capability allowlists.
12: Red Teaming & Defenses	Defense-in-depth engineering, AI firewalls, real-time telemetry, and LLM red-teaming lifecycles.	AI Red Team Report: Comprehensive security assessment detailing system findings, risk ratings, and regression tests.

---

How To Use This Repository

1. Foundations: Start with Modules 01 to 03 to master runtime threats and injection/jailbreaking vectors.
2. Architecture: Progress to Modules 10 and 11 early if you are developing agentic, RAG, or MCP-based AI applications.
3. Deep ML Security: Study Modules 04 to 09 to understand training-time data poisoning, model extraction, membership leakage, and supply chain threats.
4. Operations & Defense: Apply Module 12 to build permanent red-teaming lifecycle routines and deploy telemetry-enabled AI firewalls.
5. Portfolio Capstone: Complete the exercises in capstone-projects to integrate the modules into end-to-end, production-grade security implementations.

---

Code Snippets: Attack & Defense in Action

The examples below are taken directly from the repository scripts to illustrate the attack/defense pairing pattern used throughout the curriculum.

1. Detecting Direct Prompt Injection (Module 02)

02-prompt-injection/direct_injection.py uses heuristic regex rules to flag override attempts and constructs a structurally-delimited prompt to separate trusted instructions from untrusted user input:

INJECTION_RULES = [
    re.compile(r"ignore\s+(all\s+)?(previous|above|prior)\s+instructions?", re.I),
    re.compile(r"(system|admin|developer)\s+(override|note|instruction)", re.I),
    re.compile(r"you\s+are\s+now\s+(an?\s+)?(unrestricted|jailbroken|unfiltered)", re.I),
    re.compile(r"print\s+(your\s+)?(system\s+)?prompt", re.I)
]

def classify_user_intent(message: str) -> IntentResult:
    matches = sum(1 for rule in INJECTION_RULES if rule.search(message))
    risk_score = min(1.0, matches * 0.5)
    is_blocked = risk_score >= 0.5
    return IntentResult(is_blocked, risk_score,
                         "Injection patterns detected" if is_blocked else "Clean request")

2. Validating Tool Calls in Agentic Systems (Module 11)

11-agent-and-agentic-ai-security/tool_call_schema.py demonstrates schema validation, recipient allowlisting, and exfiltration-keyword scanning before an agent is permitted to execute a send_email tool call:

ALLOWED_RECIPIENTS = {"boss@company.com", "manager@company.com", "support@company.com"}
EXFILTRATION_KEYWORDS = {"password", "api key", "secret", "private key"}

def validate_send_email_arguments(args: dict) -> ValidationResult:
    errors = []
    if not isinstance(args.get("to"), str):
        errors.append("Parameter 'to' must be a string")
    if not isinstance(args.get("body"), str):
        errors.append("Parameter 'body' must be a string")
    if errors:
        return ValidationResult(False, errors)

    if args["to"] not in ALLOWED_RECIPIENTS:
        errors.append(f"Recipient '{args['to']}' is not in the authorized recipients allowlist")

    for word in EXFILTRATION_KEYWORDS:
        if word in args.get("body", "").lower():
            errors.append(f"Body contains suspicious keyword: '{word}'")

    return ValidationResult(len(errors) == 0, errors)

3. End-to-End AI Firewall Pipeline (Module 12)

12-defense-mechanisms-and-red-teaming/ai_firewall.py implements a three-stage pipeline — Unicode canonicalization, input classification against injection rules, and output verification for canary-token leaks and PII:

CANARY_TOKEN = "SYS-CANARY-992211"

def canonicalize(text: str) -> str:
    result = unicodedata.normalize("NFKC", text)
    invisible = re.compile(r"[\u200b-\u200f\u2028-\u202f\u00ad\ufeff]")
    return invisible.sub("", result)

def verify_output(text: str) -> dict:
    issues = []
    if CANARY_TOKEN in text:
        issues.append("canary_leaked")
    for pat in PII_PATTERNS:
        if pat.search(text):
            issues.append("pii_found")
    return {"safe": len(issues) == 0, "issues": issues}

def run_firewall_pipeline(user_query: str, simulated_completion: str) -> dict:
    clean_input = canonicalize(user_query)
    input_result = classify_input(clean_input)
    output_result = verify_output(simulated_completion)
    return {"input": input_result, "output": output_result}

4. Safe Model Weight Loading (Module 09)

09-supply-chain-attacks/safe_serialization.py demonstrates loading model weights via safetensors instead of Python's pickle, eliminating arbitrary code execution risk during deserialization:

from safetensors.torch import load_file, save_file

# Vulnerable: torch.load() with pickle can execute arbitrary code on load
# UNSAFE = torch.load("model.pt")

# Safe: safetensors performs no code execution during deserialization
weights = load_file("model.safetensors")
save_file(weights, "verified_model.safetensors")

---

Each Module Includes

• README.md — Concept overview, threat description, and learning objectives.
• Attack scripts — Runnable demonstrations of the vulnerability (e.g. fgsm_attack.py, crescendo_jailbreak.py, corpus_poisoning.py).
• Defense scripts — Mitigations and detection mechanisms (e.g. differential_privacy.py, tenant_isolation.py, dependency_scanner.py).
• Practical deliverable — A reusable artifact (checklist, report template, or test suite) for production hardening.

---

Capstone Projects

The capstone-projects directory contains eight production-style integrations that combine multiple modules:

Capstone	Focus
red_team_platform.py	Distributed red-team test platform with risk scoring and reporting
secure_agent_langgraph.py	Multi-agent LangGraph workflow with state validation and HITL approvals
hardened_mcp_gateway.py	MCP capability allowlisting and tool-call sandboxing
prompt_firewall.py	Standalone inline prompt/response firewall service
rag_vuln_scanner.py	Automated scanner for RAG ingestion and retrieval vulnerabilities
enterprise_security_gateway.py	Unified gateway combining input/output filtering, rate limiting, and logging
agent_telemetry.py	Real-time telemetry and audit logging for agent tool calls
ai_soc_dashboard.py	Security operations dashboard for AI incident monitoring

---

Reference Material

• ai_security_learning_path_comprehensive_guide.pdf — Full written guide accompanying the curriculum.
• OWASP Top 10 for LLM Applications
• MITRE ATLAS

---

Contributing

Contributions are welcome. Please:

1. Fork the repository and create a feature branch.
2. Follow the existing module structure (README.md + attack/defense scripts).
3. Run black, flake8, and mypy before submitting (see requirements.txt).
4. Open a pull request describing the module or fix.

---

License

Distributed under the MIT License. See LICENSE for more information.

---

Connect with Me

_{⭐ If this repository helped you, please star it
Made by mohd-faizy}