🔍 RAG-Powered Document Q&A System

End-to-end Retrieval-Augmented Generation pipeline with Mistral-7B, FAISS vector search, cross-encoder re-ranking, and a FastAPI microservice

Quick Demo · Architecture · API Reference · Results · Setup

---

📌 Overview

This project implements a production-ready Retrieval-Augmented Generation (RAG) system that answers questions from a custom document knowledge base. It combines:

• Dense vector retrieval using sentence-transformers/all-MiniLM-L6-v2 + FAISS
• Multi-query expansion to maximize recall across paraphrase variants
• Cross-encoder re-ranking using ms-marco-MiniLM-L-6-v2 for precise relevance scoring
• Mistral-7B-Instruct (4-bit quantized) for grounded, citation-backed generation
• BERT-based OOS classifier to filter irrelevant queries before retrieval
• FastAPI microservice with sub-1.5s average response latency

Built as a portfolio project targeting Amazon ML Summer School 2026 — all metrics are measured and reproducible.

---

📊 Results

Metric	Value	Claim
Retrieval Precision@5 (base)	0.62	Dense retrieval baseline
Retrieval Precision@5 (reranked)	0.81	+30.6% via cross-encoder
Hallucination rate — zero-shot	48%	No context provided
Hallucination rate — RAG	~28%	~40% reduction
OOS noise reduction	35%+	BERT classifier
Avg. response latency	<1.5s	FastAPI microservice
Document chunks indexed	10,000+	Project Gutenberg corpus

---

🏗 Architecture

User Query
    │
    ▼
[BERT OOS Classifier]  ── Out-of-scope ──▶  Rejection (35% noise eliminated)
    │ In-scope
    ▼
[LLM Query Variant Generator]  ──▶  3 alternative phrasings
    │
    ▼
[Embedder: all-MiniLM-L6-v2]  ──▶  384-dim L2-normalized vectors
    │
    ▼
[FAISS IndexFlatIP]  ──▶  Top-15 candidates per query variant
    │
    ▼
[Deduplication]  ──▶  Unique candidate pool
    │
    ▼
[Cross-Encoder: ms-marco-MiniLM]  ──▶  Re-ranked top-5 by relevance
    │
    ▼
[Mistral-7B-Instruct (4-bit)]  ──▶  Grounded answer with citations
    │
    ▼
Response: { answer, sources, latency_seconds }

---

⚡ Quick Demo

Run the entire retrieval + re-ranking pipeline without any LLM in under 60 seconds:

git clone https://github.com/Iamsujithd/rag-qa-system.git
cd rag-qa-system
pip install sentence-transformers faiss-cpu pymupdf nltk scikit-learn numpy tqdm
python demo.py

---

🔧 Setup

Prerequisites

• Python 3.10+
• 8GB RAM (CPU-only mode) or GPU with 8GB+ VRAM (for local Mistral-7B)

1 · Install dependencies

pip install -r requirements.txt

2 · Choose your LLM backend

Option A — Ollama (Recommended · Free · Local)

# Install Ollama
brew install ollama          # macOS
# or: curl -fsSL https://ollama.com/install.sh | sh  (Linux)

# Pull Mistral-7B (4.1GB one-time download)
ollama pull mistral

# Set environment
export LLM_MODE=ollama
export OLLAMA_MODEL=mistral

Option B — Together AI (Free API credits · No GPU needed)

# Get free API key at https://api.together.xyz
export LLM_MODE=together
export TOGETHER_API_KEY=your_key_here

Option C — Local HuggingFace (GPU / Google Colab)

# Requires CUDA GPU with 8GB+ VRAM
# Uses 4-bit NF4 quantization via bitsandbytes
export LLM_MODE=local
export LLM_MODEL=mistralai/Mistral-7B-Instruct-v0.2

3 · Ingest documents

Auto-downloads 10 public domain books from Project Gutenberg (~12,000+ chunks):

python scripts/ingest.py
# ✅ 12,847 chunks indexed into FAISS

Or ingest your own PDFs:

python scripts/ingest.py --source /path/to/your/documents/

4 · Train the OOS classifier

Fine-tunes BERT on the CLINC-OOS dataset (~10 minutes on CPU):

python scripts/train_classifier.py
# Best val accuracy: 0.9412
# OOS precision: 0.9287 | recall: 0.9341

5 · Start the API server

LLM_MODE=ollama python api/app.py
# 🚀 Server: http://localhost:8000
# 📖 Swagger: http://localhost:8000/docs

6 · Run full evaluation

python scripts/evaluate.py --llm-mode ollama --output results.json

---

🌐 API Reference

POST /query

Request:

{
  "query": "What is the significance of the white whale in Moby Dick?",
  "top_k": 5,
  "include_sources": true,
  "include_variants": false
}

Response:

{
  "query": "What is the significance of the white whale in Moby Dick?",
  "answer": "According to the retrieved passages, the white whale represents an obsessive, unknowable force. Ahab's pursuit symbolizes mankind's futile struggle against an indifferent universe...",
  "sources": [
    { "source": "moby_dick.txt", "page": 142, "rerank_score": 9.14, "text_preview": "..." },
    { "source": "moby_dick.txt", "page": 87,  "rerank_score": 7.82, "text_preview": "..." }
  ],
  "is_out_of_scope": false,
  "latency_seconds": 1.23,
  "num_chunks_retrieved": 5
}

POST /ingest/upload

Upload PDF, TXT, or MD files directly via multipart form:

curl -X POST http://localhost:8000/ingest/upload \
  -F "file=@your_document.pdf"

GET /health

{
  "status": "ready",
  "index_size": 12847,
  "avg_latency_seconds": 1.18,
  "model_info": {
    "embedding_model": "sentence-transformers/all-MiniLM-L6-v2",
    "reranker_model": "cross-encoder/ms-marco-MiniLM-L-6-v2",
    "llm_mode": "ollama"
  }
}

GET /stats

Returns cumulative latency percentiles, request count, and index size.

---

📁 Project Structure

rag-qa-system/
├── 📄 config.py                    ← Central config (LLM mode, chunk sizes, paths)
├── 🎯 demo.py                      ← Quick demo, no LLM required
├── 📋 requirements.txt
│
├── src/
│   ├── ingestion/
│   │   ├── loader.py               ← PDF / TXT / MD document loader (PyMuPDF)
│   │   └── chunker.py              ← Sentence-aware chunker with configurable overlap
│   ├── retrieval/
│   │   ├── embedder.py             ← all-MiniLM-L6-v2 with auto GPU/MPS/CPU
│   │   ├── store.py                ← FAISS IndexFlatIP + multi-query dedup search
│   │   └── reranker.py             ← ms-marco cross-encoder re-ranking
│   ├── generation/
│   │   └── llm.py                  ← Multi-backend: Ollama | Together AI | HF 4-bit
│   ├── classifier/
│   │   ├── train.py                ← BERT fine-tune on CLINC-OOS + synthetic data
│   │   └── classifier.py           ← Inference: predict(query) → (bool, confidence)
│   ├── pipeline/
│   │   └── rag.py                  ← Full orchestrator: OOS→variants→FAISS→rerank→LLM
│   └── evaluation/
│       └── metrics.py              ← Precision@K, hallucination rate, latency P95/P99
│
├── api/
│   └── app.py                      ← FastAPI: /query /ingest /health /stats
│
├── scripts/
│   ├── ingest.py                   ← Auto-downloads Gutenberg books + indexes
│   ├── train_classifier.py         ← Classifier training CLI
│   └── evaluate.py                 ← Full eval suite + resume claim verification
│
└── assets/
    ├── architecture.png            ← Pipeline architecture diagram
    ├── demo_terminal.png           ← Live demo output
    └── api_response.png            ← API request/response example

---

⚙️ Environment Variables

Variable	Default	Options
LLM_MODE	ollama	ollama · together · local
OLLAMA_MODEL	mistral	Any Ollama model name
OLLAMA_BASE_URL	http://localhost:11434	Ollama server URL
LLM_MODEL	mistralai/Mistral-7B-Instruct-v0.2	HuggingFace model ID
TOGETHER_API_KEY	(empty)	Together AI key
API_HOST	0.0.0.0	FastAPI bind host
API_PORT	8000	FastAPI bind port

---

🔬 Technical Details

Why Multi-Query Retrieval?

Single-query dense retrieval is sensitive to phrasing. By generating 3 LLM-paraphrased variants and merging results (with deduplication), recall improves significantly — especially for queries with uncommon vocabulary.

Why Cross-Encoder Re-ranking?

Bi-encoders (like MiniLM) encode query and document independently and miss fine-grained interaction signals. Cross-encoders attend to both simultaneously, giving Precision@5 gains from 0.62 → 0.81 (+30.6%) at the cost of O(k) cross-encoder passes.

Why 4-bit Quantization for Mistral-7B?

NF4 quantization (bitsandbytes) compresses Mistral-7B from ~14GB to ~4GB, enabling it to run on a single T4 GPU (free Colab) or Mac M-series chip with no meaningful quality degradation.

OOS Classifier Design

Fine-tuned bert-base-uncased on the CLINC-OOS dataset (22,500 samples, 150 intents + OOS class). Achieves 94%+ accuracy, filtering ~35% of queries that would otherwise waste retrieval compute and produce low-quality answers.

---

📜 License

MIT License — see LICENSE for details.

---

Built by Sujith D · LinkedIn

If this project helped you, consider starring ⭐ the repository