An open-source Retrieval-Augmented Generation (RAG) system for analyzing publicly released documents from the Jeffrey Epstein investigation. Built to enable researchers, journalists, and investigators to search, analyze, and find connections across millions of pages of official DOJ, House Committee, and FBI releases.
This project provides:
- Hybrid Search: Combined BM25 keyword + vector semantic search for accurate retrieval
- Entity Graph: Extracted people, organizations, and locations with relationship mapping
- Evidence-Grade Citations: Every result includes source, page number, and Bates numbers
- MCP Server: Integrate with Claude, GPT, and other LLM agents via Model Context Protocol
- Multi-Backend Embeddings: Support for Google Gemini API and OpenRouter
This system is designed to work with official public releases:
| Source | Description |
|---|---|
| DOJ Epstein Library | ~3.5M pages, 180K images, 2K+ videos |
| House Oversight Committee | Additional document releases |
| FBI FOIA Vault | FOIA releases |
Note: This repository contains processing code and pre-computed embeddings. Raw documents should be downloaded from official sources.
- Python 3.10+
- PostgreSQL (optional, SQLite works for development)
- Meilisearch (for keyword search)
# Clone the repository
git clone https://github.com/yourusername/epsteindataset.git
cd epsteindataset
# Create virtual environment
python -m venv venv
source venv/bin/activate # or `venv\Scripts\activate` on Windows
# Install dependencies
pip install -r requirements.txt
# Download spaCy model for entity extraction
python -m spacy download en_core_web_sm
# Copy environment template
cp .env.example .envEdit .env with your API key (choose one):
# Option 1: Google Gemini API (Direct)
GEMINI_API_KEY=your_gemini_api_key
# Option 2: OpenRouter API
OPENROUTER_API_KEY=your_openrouter_api_keyThe repository includes pre-processed data:
corpus.sqlite- SQLite database with extracted text and metadatadata_archive.zip- Additional processed data
import sqlite3
# Connect to the corpus
conn = sqlite3.connect('corpus.sqlite')
cursor = conn.cursor()
# Query example
cursor.execute("SELECT * FROM documents LIMIT 10")
for row in cursor.fetchall():
print(row)# Start the MCP server for LLM integration
python -m src.mcp_server.server┌─────────────────────────────────────────────────────────────┐
│ LLM Agent / User │
└─────────────────────────────────────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────────────┐
│ MCP Server │
│ Tools: search, fetch_page, entity_lookup, evidence_pack │
└─────────────────────────────────────────────────────────────┘
│
┌───────────────┼───────────────┐
▼ ▼ ▼
┌─────────────────┐ ┌─────────────────┐ ┌─────────────────┐
│ Hybrid Search │ │ Entity Graph │ │ Citation Builder│
│ (BM25+Vector) │ │ (Co-occurrence)│ │ (Provenance) │
└─────────────────┘ └─────────────────┘ └─────────────────┘
│ │ │
└───────────────┼───────────────┘
▼
┌─────────────────────────────────────────────────────────────┐
│ Storage Layer │
│ PostgreSQL/SQLite │ Meilisearch │ LanceDB/Qdrant │
└─────────────────────────────────────────────────────────────┘
The MCP server exposes these tools for LLM agents:
| Tool | Description |
|---|---|
search |
Hybrid keyword + vector search with filters |
fetch_page |
Get full page content by document ID and page number |
entity_lookup |
Find entities by name |
entity_neighbors |
Discover entity connections |
timeline |
Chronological events for an entity |
evidence_pack |
Curated evidence with strict citations |
User: Who visited Little Saint James in 2005?
Agent: [calls search tool]
Query: "Little Saint James visitors 2005"
Filters: {date_start: "2005-01-01", date_end: "2005-12-31"}
Results: [
{
"text": "Flight log entry showing...",
"citation": {
"source": "DOJ Dataset 3",
"page_number": 142,
"bates_number": "DOJ-00045821"
}
}
]
epsteindataset/
├── src/
│ ├── embeddings/ # Gemini/OpenRouter embedding providers
│ ├── processing/ # PDF extraction, OCR, chunking
│ ├── search/ # Hybrid search, vector stores
│ ├── database/ # SQLAlchemy models, schema
│ └── mcp_server/ # MCP server implementation
├── scripts/ # Data processing scripts
├── config/ # Configuration files
├── tests/ # Test suite
├── corpus.sqlite # Pre-processed document database
├── data_archive.zip # Additional processed data
├── requirements.txt # Python dependencies
├── .env.example # Environment template
└── README.md
Using Gemini gemini-embedding-001 at $0.15/1M tokens:
| Scenario | Pages | Est. Tokens | Standard | Batch API |
|---|---|---|---|---|
| Small corpus | 100K | 35M | ~$5 | ~$2.50 |
| Medium | 500K | 175M | ~$26 | ~$13 |
| Full DOJ | 3.5M | 1.2B | ~$180 | ~$90 |
This project handles sensitive investigation materials. Please:
- Do not redistribute raw media - Link to official sources only
- Preserve redactions - Never attempt to reveal redacted content
- Respect victim privacy - Do not identify or speculate about victims
- Report PII leaks - If you find inadvertent PII exposure, report to DOJ
The DOJ explicitly warns that releases may contain inadvertent non-public PII. This system mirrors that posture.
Contributions are welcome! Please see CONTRIBUTING.md for guidelines.
Areas where help is needed:
- Improving OCR quality for handwritten documents
- Entity resolution and deduplication
- Building visualization interfaces
- Performance optimization
This project is licensed under the MIT License - see LICENSE for details.
This is a research tool for analyzing publicly available documents. The presence of a name in these documents does not imply wrongdoing. Always:
- Frame results as "mentions in documents," not conclusions
- Provide full citations
- Consider the context of surrounding pages
- DOJ Epstein Library for public document releases
- Epstein Files Archive for OCR processing work
- PDF Association for document structure analysis
Note: This project is not affiliated with any government agency or official investigation.