SafetyAgents - Intelligent Content Moderation System

Enterprise-grade content moderation using AI orchestration for video chat platforms

---

🚀 Quick Start

# 1. Set API keys
export GROQ_API_KEY="your_groq_api_key"
export ANTHROPIC_API_KEY="your_anthropic_api_key"

# 2. Install dependencies
pip install groq anthropic pyaudio opencv-python ultralytics fastapi uvicorn requests

# 3. Start backend
python backend_server.py

# 4. Start video + audio monitoring with AI
python integrated_video_monitor.py --audio --transcribe

# 5. (Optional) Start frontend dashboard
cd frontend && npm install && npm run dev

That's it! Your intelligent content moderation system is now running.

---

📋 Table of Contents

• Overview
• Architecture
• Features
• Documentation
• Installation
• Usage
• Configuration
• Testing
• API Reference
• Cost Analysis
• Troubleshooting

---

🎯 Overview

SafetyAgents is an intelligent content moderation system that monitors video chat sessions for policy violations using a multi-modal AI approach:

• Video Agent (YOLO v8): Detects nudity, violence, and prohibited objects
• Audio Agent (Whisper + LLaMA): Transcribes speech and detects profanity, harassment, hate speech
• Orchestrator Agent (Claude AI): Makes intelligent decisions when both audio and video violations occur

Why SafetyAgents?

Traditional Approach:

• ❌ Run expensive AI on every frame/audio chunk
• ❌ High false positive rate
• ❌ Cost: ~$11/hour per user

SafetyAgents Approach:

• ✅ Free local processing (YOLO + Whisper)
• ✅ Intelligent orchestration only when needed (~5-10% of cases)
• ✅ Context-aware decisions reduce false positives
• ✅ Cost: ~$0.01/hour per user (1100x cheaper!)

---

🏗️ Architecture

┌─────────────────────────────────────────────────────────┐
│                    USER SESSION                         │
│  ┌──────────────┐              ┌──────────────┐        │
│  │   Camera     │              │  Microphone  │        │
│  └──────┬───────┘              └──────┬───────┘        │
│         │                              │                │
└─────────┼──────────────────────────────┼────────────────┘
          │                              │
          ▼                              ▼
┌─────────────────────┐      ┌─────────────────────┐
│   Video Agent       │      │   Audio Agent       │
│   (YOLO v8)         │      │   (Whisper+LLaMA)   │
│   - Nudity          │      │   - Profanity       │
│   - Violence        │      │   - Harassment      │
│   - Objects         │      │   - Hate Speech     │
└─────────┬───────────┘      └─────────┬───────────┘
          │                            │
          │  Violation Detected        │  Violation Detected
          ▼                            ▼
┌──────────────────────────────────────────────────────┐
│         Backend Server (FastAPI)                     │
│  ┌────────────────────────────────────────────────┐ │
│  │  Session Tracking                              │ │
│  │  - Stores violations by session/agent          │ │
│  │  - Checks time windows (2 minutes)             │ │
│  └────────────────────────────────────────────────┘ │
│                                                       │
│  ┌────────────────────────────────────────────────┐ │
│  │  Orchestrator Trigger Logic                    │ │
│  │  - Both audio + video violations exist?        │ │
│  │  - Within 2-minute window? → TRIGGER           │ │
│  └────────────────────────────────────────────────┘ │
└───────────────────────┬──────────────────────────────┘
                        │
        Both violations detected
                        ▼
         ┌────────────────────────────┐
         │  Orchestrator Agent        │
         │  (Claude Sonnet 4)         │
         │                            │
         │  Analyzes:                 │
         │  - Audio violation context │
         │  - Video violation context │
         │  - User history            │
         │  - Session patterns        │
         │                            │
         │  Decides:                  │
         │  - warn / mute / blur /    │
         │    disconnect              │
         │  - Provides reasoning      │
         │  - Logs for compliance     │
         └────────────┬───────────────┘
                      │
                      ▼
         ┌─────────────────────────────┐
         │  Action Applied             │
         │  + Compliance Logging       │
         │  + Dashboard Update         │
         └─────────────────────────────┘

---

✨ Features

🎥 Video Monitoring (YOLO v8)

• Real-time object detection
• Nudity detection (configurable objects)
• Violence detection (weapons, aggressive objects)
• Automatic blurring of prohibited content
• Policy-based confidence thresholds
• Local processing (FREE)

🎤 Audio Monitoring (Whisper + LLaMA)

• Real-time speech-to-text transcription
• Context-aware profanity detection
• Harassment and threat detection
• Hate speech detection
• Policy-based confidence thresholds
• Conversation transcript logging
• Local processing (FREE via Groq)

🤖 Intelligent Orchestration (Claude AI)

• Multi-modal analysis (audio + video together)
• Context-aware decision making
• Reduces false positives
• Compliance logging
• Cost-effective (only ~5-10% of violations)
• Explainable AI (provides reasoning)

📊 Dashboard & API

• Real-time violation monitoring
• Live metrics and statistics
• Violation history and details
• Orchestrator decision logs
• RESTful API for integration
• WebSocket support for live updates

---

📚 Documentation

• QUICKSTART.md - Get started in 5 minutes
• WHISPER_AUDIO_SETUP.md - Detailed audio system guide
• ORCHESTRATOR_SETUP.md - Orchestrator configuration
• WHISPER_IMPLEMENTATION_SUMMARY.md - Implementation details
• INTEGRATION_GUIDE.md - Architecture overview

---

🛠️ Installation

Prerequisites

• Python 3.8+
• Node.js 16+ (for frontend)
• Webcam (for video monitoring)
• Microphone (for audio monitoring)

1. Clone Repository

git clone <repository-url>
cd adobe-hec

2. Install Python Dependencies

pip install groq anthropic pyaudio opencv-python ultralytics fastapi uvicorn requests pydantic

3. Install Frontend Dependencies (Optional)

cd frontend
npm install
cd ..

4. Set API Keys

# Groq API (FREE) - for Whisper + LLaMA
export GROQ_API_KEY="your_groq_api_key"

# Anthropic API - for Claude Orchestrator
export ANTHROPIC_API_KEY="your_anthropic_api_key"

Get API keys:

• Groq (FREE): https://console.groq.com
• Anthropic: https://console.anthropic.com

---

🎮 Usage

Full System (Video + Audio + Orchestrator)

# Terminal 1: Start backend
python backend_server.py

# Terminal 2: Start monitoring
python integrated_video_monitor.py --audio --transcribe

# Terminal 3 (Optional): Start frontend
cd frontend && npm run dev

Video Only

python integrated_video_monitor.py

Audio Only

python integrated_audio_monitor.py

Common Commands

# List audio devices
python integrated_video_monitor.py --list-audio-devices

# Custom transcription interval (5 seconds)
python integrated_video_monitor.py --audio --transcribe --transcribe-interval 5

# Specify audio devices
python integrated_video_monitor.py --audio --transcribe --audio-input 0 --audio-output 1

# Disable blur
python integrated_video_monitor.py --no-blur

# Custom YOLO weights
python integrated_video_monitor.py --weights yolov8n.pt

---

⚙️ Configuration

Audio Policy (config/audio_policy.json)

{
  "enabled_violations": {
    "profanity": {
      "enabled": true,
      "strictness": 70,
      "confidence_threshold": 0.815
    },
    "hate_speech": {
      "enabled": true,
      "strictness": 10,
      "confidence_threshold": 0.545
    },
    "harassment": {
      "enabled": false,
      "strictness": 50,
      "confidence_threshold": 0.725
    }
  },
  "escalation_rules": [
    {
      "violation_number": 1,
      "action": "warn",
      "message": "First warning",
      "time_window_minutes": 30
    },
    {
      "violation_number": 2,
      "action": "disconnect",
      "message": "Disconnected",
      "time_window_minutes": 30
    }
  ]
}

Video Policy (config/video_policy.json)

{
  "enabled_violations": {
    "nudity": {
      "enabled": true,
      "strictness": 20,
      "confidence_threshold": 0.7,
      "objects": ["banana"]
    },
    "violence": {
      "enabled": true,
      "strictness": 40,
      "confidence_threshold": 0.68,
      "objects": ["cup", "fork", "knife"]
    }
  }
}

Adjusting Thresholds:

• Lower threshold (e.g., 0.5) → More sensitive, more violations
• Higher threshold (e.g., 0.9) → Less sensitive, fewer violations

---

🧪 Testing

Test 1: Audio Violation

# Start system
python integrated_video_monitor.py --audio --transcribe

# Say profanity
"What the fuck is this?"

# Expected output:
📝 Transcribed: What the fuck is this?
[12:34:56] 🚨 AUDIO VIOLATION: PROFANITY → Backend: violation_abc123

Test 2: Video Violation

# Show prohibited object (cup) to camera

# Expected output:
[12:34:56] 📡 VIOLENCE: cup (0.85) → Backend: violation_xyz789

Test 3: Orchestrator Trigger

# 1. Say profanity: "Fuck this"
# 2. Within 2 minutes, show cup to camera

# Expected backend output:
🤖 ORCHESTRATOR: DISCONNECT - User displayed prohibited object while using profanity...

# Check orchestrator logs:
curl http://localhost:8000/api/v1/orchestrator/logs | jq

---

📡 API Reference

Base URL

http://localhost:8000

Endpoints

GET / - Health Check

curl http://localhost:8000/

Response:

{
  "status": "active",
  "service": "SafetyAgents API",
  "total_violations": 42
}

POST /api/v1/violations - Create Violation

curl -X POST http://localhost:8000/api/v1/violations \
  -H "Content-Type: application/json" \
  -d '{
    "sessionId": "session_123",
    "userId": "user_456",
    "type": "profanity",
    "severity": 8.0,
    "confidence": 0.9,
    "agent": "speech",
    "transcription": "What the fuck?",
    "decision": {
      "action": "warn",
      "message": "Please watch your language"
    }
  }'

GET /api/v1/violations - Get Violations

curl http://localhost:8000/api/v1/violations?limit=50 | jq

GET /api/v1/orchestrator/logs - Get Orchestrator Logs

curl http://localhost:8000/api/v1/orchestrator/logs | jq

GET /api/v1/orchestrator/stats - Get Orchestrator Stats

curl http://localhost:8000/api/v1/orchestrator/stats | jq

DELETE /api/v1/violations/clear - Clear Violations (Testing)

curl -X DELETE http://localhost:8000/api/v1/violations/clear

Full API Docs: http://localhost:8000/docs

---

💰 Cost Analysis

SafetyAgents (Intelligent Orchestration)

Component	Technology	Cost	Frequency
Video Processing	YOLO v8 (local)	FREE	Every 0.5s
Audio Transcription	Whisper (Groq)	FREE	Every 10s
Audio Moderation	LLaMA 3.3 (Groq)	FREE	Per transcription
Orchestrator	Claude Sonnet 4	$0.003	~5-10% of violations

Total: ~$0.01/hour per user

Traditional Cloud-Only Approach

Component	Technology	Cost	Frequency
Video Processing	Cloud Vision API	$1.50/1000 images	Every frame
Audio Transcription	Cloud Speech-to-Text	$1.44/hour	Continuous
Moderation	Cloud AI Moderation	$5/hour	Continuous

Total: ~$11/hour per user

Savings: 1100x cheaper with SafetyAgents! 💰

---

🐛 Troubleshooting

No Audio Violations

Problem: Audio is transcribed but no violations are detected.

Solutions:

1. Check --transcribe flag is enabled
2. Verify GROQ_API_KEY is set: echo $GROQ_API_KEY
3. Check confidence thresholds in config/audio_policy.json
4. Ensure you're speaking loud enough
5. Try lowering confidence threshold (e.g., 0.5)

No Video Violations

Problem: Objects are detected but not sent to backend.

Solutions:

1. Verify object is in config/video_policy.json
2. Check YOLO confidence meets threshold
3. Ensure object is clearly visible in camera
4. Try lowering confidence threshold
5. Check backend is running: curl http://localhost:8000/

Orchestrator Not Triggering

Problem: Both audio and video violations occur but orchestrator doesn't trigger.

Solutions:

1. Verify ANTHROPIC_API_KEY is set: echo $ANTHROPIC_API_KEY
2. Check violations occurred within 2 minutes of each other
3. Verify backend server is running
4. Check backend logs for orchestrator messages
5. Test orchestrator directly: curl http://localhost:8000/api/v1/orchestrator/stats

PyAudio Installation Issues

Linux:

sudo apt-get install portaudio19-dev
pip install pyaudio

macOS:

brew install portaudio
pip install pyaudio

Windows:

pip install pipwin
pipwin install pyaudio

---

📄 License

[Your License Here]

---

🤝 Contributing

Contributions are welcome! Please read our contributing guidelines.

---

📞 Support

For issues and questions:

• GitHub Issues: [Link to issues]
• Email: [Your email]
• Documentation: See docs folder

---

🎉 Acknowledgments

• YOLO v8 by Ultralytics
• Whisper AI by OpenAI (via Groq)
• LLaMA 3.3 by Meta (via Groq)
• Claude AI by Anthropic
• FastAPI by Sebastián Ramírez

---

Built with ❤️ for safer online communities