Mapping Multilingual Blind Spots in AI Safety Evaluation Infrastructure
International AI Safety Institutes cannot coordinate on safety evaluation if they lack shared visibility into what evaluation infrastructure actually exists.
Our analysis reveals:
- 75% of EU languages have no jailbreak resistance benchmarks
- 96% of EU languages have no bias detection benchmarks
- 96% of EU languages have no adversarial robustness benchmarks
A model could pass every available safety evaluation in Greek, Hungarian, or Polish while having completely untested vulnerabilities in those languages.
SafetyGap is an open-source tool that:
- Maps coverage — Which benchmarks exist for which languages
- Identifies gaps — Which language × category combinations have zero coverage
- Prioritizes action — Which gaps affect the most people
- Enables coordination — Shareable format for international collaboration
# View the interactive dashboard
# Open SafetyGapDashboard.jsx in any React environment
# Or run the Python analysis
python analyze_coverage.py| Risk Category | EU Language Coverage | Status |
|---|---|---|
| Truthfulness | 22/24 (92%) | ✅ Adequate |
| Over-refusal | 24/24 (100%) | ✅ Adequate |
| Toxicity | 14/24 (58%) | |
| Jailbreak | 2/24 (8%) | 🚨 Critical |
| Bias | 1/24 (4%) | 🚨 Critical |
| Adversarial | 1/24 (4%) | 🚨 Critical |
| Stakeholder | Use Case |
|---|---|
| 🏛️ AI Safety Institutes | Coordinate benchmark development, avoid duplication |
| ⚖️ Regulators | Understand which compliance claims are verifiable |
| 🔬 Benchmark Developers | Identify high-impact targets for new work |
SafetyGapDashboard.jsx— Interactive React visualizationbenchmarks.json— Structured benchmark databasesubmission_report.md— Full hackathon report
This tool is designed to be community-maintained. To add a benchmark:
- Fork this repo
- Add entry to
benchmarks.json - Submit PR
- Deng et al. (2024). Multilingual jailbreak challenges in LLMs. ICLR 2024.
- Wang et al. (2024). All languages matter: Multilingual safety of LLMs. ACL 2024.
- Mazeika et al. (2024). HarmBench. arXiv:2402.04249.
MIT
Track 4: International Verification & Coordination — Technical AI Governance Hackathon 2026