🚀Generative AI-based CyberSecurity-focused Prompt Dataset for Benchmarking Large Language Models 🛡️

The largest and most comprehensive Generative AI-based CyberSecurity-focused Dataset for Benchmarking Large Language Models

🌟 Overview

The paper offers:

• 🎯 A cutting-edge dataset of 12662 prompts tailored to cybersecurity challenges.
• 🧠 Novel jailbreaking methods leveraging prompt obfuscation and refinement.
• 📊 Comprehensive performance evaluation of LLMs like ChatGPT, Claude, and Gemini.

Why this?

Existing datasets are too broad and often lack focus on cybersecurity.It fills this gap by providing domain-specific prompts organized into 10 categories, enabling a precise evaluation of LLM security mechanisms.

✨ Features

🗂️ Dataset

• 📁 10 Categories of Prompts:
  • 🌩️ Cloud Attacks
  • ⚙️ Control System Attacks
  • 🔒 Cryptographic Attacks
  • 🕵️ Evasion Techniques
  • 💻 Hardware Attacks
  • 🔐 Intrusion Techniques
  • 📡 IoT Attacks
  • 🦠 Malware Attacks
  • 🌐 Network Attacks
  • 🌍 Web Application Attacks

---

🗂️ Repository Structure

/
├── Code/
│   ├── dataset_generation.py
│   ├── keywords.txt
├── Dataset/
│   ├── Category sets/
│   │   ├── cloud-attacks.csv
│   │   ├── control-system-attacks.csv
│   │   ├── cryptographic-attacks.csv
│   │   ├── evasion-techniques.csv
│   │   ├── hardware-attacks.csv
│   │   ├── intrusion-techniques.csv
│   │   ├── iot-attacks.csv
│   │   ├── malware-attacks.csv
│   │   ├── network-attacks.csv
│   │   ├── web-application-attacks.csv
│   ├── Full dataset/
│   │   ├── data.csv
│   ├── Sample sets/
│       ├── 500.csv
│       ├── 2000.csv
│       ├── 6000.csv

---

🚀 Getting Started

⚙️ Prerequisites

• 🐍 Python 3.8+
• 📦 Required libraries: openai (only for dataset generation)

---

📊 Results

🎯 Evaluation Metrics

• ✅ Success Rate (SR): Percentage of prompts bypassing ethical guidelines.
• 📈 Average Rating (AR): Degree of harmfulness in LLM responses (on a scale of 1-5, where 5 is the most harmful).

⚡ Jailbreaking Performance

LLM	Success Rate (SR)	Average Rating (AR)
🤖 Claude	17.4%	2.00
🤖 ChatGPT	65.4%	4.06
🤖 Gemini	88.4%	4.77

---