🧠 DocuMind — Multi-PDF Intelligent Research & RAG Assistant

DocuMind is a premium, high-performance RAG (Retrieval-Augmented Generation) application designed to assist researchers, students, and professionals in analyzing, summarizing, and querying multiple PDF documents simultaneously.

By combining locally executed semantic embeddings, BM25 keyword retrieval, and high-speed cloud LLMs via Groq, DocuMind matches semantic context to user queries in real-time. It features a custom-engineered, theme-adaptive glassmorphic interface that delivers a visually stunning experience in both dark and light modes.

---

📖 Table of Contents

1. 🌟 Key Features
2. ⚙️ Internal Architecture (System Design)
   • Data Ingestion & Indexing Pipeline
   • Hybrid Retrieval & Generation Pipeline
3. 💻 External Working (User Flow)
4. 🎨 Premium Theme & UI Override System
   • Aesthetic Foundation
   • Specific Widget Tweaks
5. 📂 Project Directory Structure
6. 🔍 File-by-File Code Walkthrough
   • Backend Modules (utils/)
   • Frontend App (app.py)
   • Test Pipeline (scratch/test_pipeline.py)
7. 🛠️ Setup & Installation
8. 🧪 Verification & Testing

---

🌟 Key Features

• BM25 + FAISS Hybrid Retrieval: Combines semantic search with exact-keyword matching using Reciprocal Rank Fusion (RRF). This improves retrieval of specific register names, mathematical formulas, tabular details, and abbreviations.
• Abbreviation Expansion: Automatically expands common computer architecture abbreviations (e.g., PC -> Program Counter, IR -> Instruction Register, AR -> Address Register, DR -> Data Register, RTL -> Register Transfer Language) before generating embeddings.
• Strict Grounding & Answerability Check: Uses a user-configurable similarity threshold (via a sidebar slider). If no chunks exceed this threshold, the system skips Groq API calls entirely and outputs: "Information not found in uploaded documents." preventing hallucinations and saving API tokens.
• Document-Level Filtering: Sidebar selector allowing queries to be restricted to all files or a specific uploaded PDF document.
• Isolated Document-Wide Mode: Groups and isolates document-wide actions (summaries, cheat sheets, MCQs) to the selected document only, preventing cross-document confusion when multiple files are indexed.
• Developer/Debug Panel: Toggle checkbox displaying a tabular view of all retrieved chunks, RRF and similarity scores, source documents, pages, and their keep/discard status.
• Strict Prompting with Inline Citations: Forces the LLM to write inline citations (e.g. [Filename, Page X]) directly within sentences rather than only grouping them at the end.
• Theme-Adaptive Glassmorphic UI: High-contrast, custom-styled interface supporting manual toggle and system defaults (Adaptive, Premium Light, Premium Dark) with no unreadable text.

---

⚙️ Internal Architecture (System Design)

DocuMind divides its operations into two main asynchronous stages: the Data Ingestion Pipeline and the Retrieval & Generation Pipeline.

1. Data Ingestion & Indexing Pipeline

This pipeline processes raw files uploaded by the user, chunks their content, converts it to numerical representations, and stores it locally for fast query lookups.

graph TD
    A[Raw PDF Files] -->|Upload bytes/path| B(utils/pdf_loader.py: PyMuPDF Extraction)
    B -->|Page-level Text & Metadata| C(utils/chunking.py: Recursive Character Splitter)
    C -->|1000 Char Chunks with 200 Overlap| D(utils/embeddings.py: SentenceTransformer all-MiniLM-L6-v2)
    D -->|384-Dimensional Normalized Vector Embeddings| E(utils/retrieval.py: FAISS Index FlatIP & BM25Okapi fitting)
    E -->|Write files| F[data/faiss.index & data/chunks.json]

Loading

• Extraction: PyMuPDF parses the layout and extracts clean page-level text blocks. The metadata object captures the original source filename and page numbers (1-indexed for readability).
• Chunking: The page text is segmented by Langchain's RecursiveCharacterTextSplitter into manageable pieces of 1,000 characters with an overlap of 200 characters to preserve structural and narrative context.
• Embeddings & BM25 Initialization: A singleton instance of EmbeddingsManager generates 384-dimensional vector representations using all-MiniLM-L6-v2. In parallel, the RetrievalManager tokenizes the chunks and initializes a BM25 model (BM25Okapi) for keyword-matching.
• Vector Indexing: The dense embeddings are stored in a FAISS inner product index (IndexFlatIP) while the raw texts and metadata objects are persisted as JSON in chunks.json. Both are loaded automatically when the app boots up.

---

2. Hybrid Retrieval & Generation Pipeline

When a query is entered, this pipeline expands abbreviations, classifies user intent, queries both the FAISS index and the BM25 model, merges and reranks candidates, applies similarity threshold filters, and queries the LLM.

sequenceDiagram
    autonumber
    actor User as Researcher
    participant App as app.py
    participant Retrieval as utils/retrieval.py
    participant LLM as utils/llm.py
    participant Groq as Groq API

    User->>App: Submits Query ("Explain RTL logic in PC")
    Note over App: Step 1: Expand query abbreviations<br/>(PC -> Program Counter, RTL -> Register Transfer Language)
    App->>Retrieval: detect_question_type(expanded_query)
    Retrieval-->>App: Returns Category ("formula")
    Note over App: Map Category to context limit (k = 20 chunks)
    App->>Retrieval: search(query_embedding, query=expanded_query, k=20, filter_document)
    Note over Retrieval: Search top-100 FAISS & top-100 BM25<br/>Filter by document metadata if selected<br/>Compute RRF scores and exact cosine similarity
    Retrieval-->>App: Returns ranked chunks & raw similarity scores
    Note over App: Step 2: Answerability check<br/>Filter chunks by Similarity Threshold<br/>If no chunks pass, return failure response & bypass Groq
    App->>LLM: query_llm(api_key, query, filtered_chunks, model_name)
    Note over LLM: Filter overlapping duplicate chunks
    Note over LLM: Inject System Rules, Fallback Constraints & Citations
    LLM->>Groq: Sends formatted instructions & context
    Groq-->>LLM: Generates response with inline citations
    LLM-->>App: Returns response text
    App->>User: Displays chat message, relevance metrics, & developer logs

Loading

• Intent Classification: The query is mapped to one of five categories based on heuristic keywords:
  • document_wide: Summaries, notes, MCQs, or cheat-sheet creation. Bypasses FAISS vector retrieval. Instead, extracts the sequential document chunks for the selected document (capped at 10 to fit context windows).
  • overview ($k=25$ chunks): General concepts, introductions, abstracts, and table-of-contents summaries.
  • formula ($k=20$ chunks): Mathematical expressions, derivations, theorems, or algorithmic calculation queries.
  • definition ($k=8$ chunks): Target conceptual terms, acronym breakdowns, and short references.
  • default ($k=10$ chunks): General standard search configuration.
• RRF Reranking: Merges the top 100 semantic search candidate indices with the top 100 BM25 search candidate indices using Reciprocal Rank Fusion ($k_{rrf} = 60$). For candidates retrieved via BM25 that were not in the top FAISS results, their vectors are reconstructed from the flat FAISS index to calculate their exact cosine similarity score.
• Grounding Validation: Discards candidates below the user-specified threshold. If no chunks pass the threshold, the LLM API call is bypassed, returning "Information not found in uploaded documents." to prevent hallucination.
• Inference with Groq: The structured context block and user query are sent to Groq's high-speed endpoint using the selected LLM. Strictly instructs the LLM to restrict answers to the provided context and cite sources inline (e.g. [Filename, Page X]).

---

💻 External Working (User Flow)

The application provides a smooth, step-by-step layout for interacting with academic papers:

1. Sidebar Configuration:
   • App Theme: Toggle between Adaptive (System), Premium Light, and Premium Dark.
   • Groq API Key: Input your Groq API key.
   • LLM Model: Dropdown menu to swap the active LLM.
   • Filter by Document: Restrict search queries to "All Documents" or a specific PDF.
   • Retrieval Similarity Threshold: Slider (0.0 to 1.0) to define the minimum relevance score.
   • Developer/Debug Mode: Toggle to reveal detailed search logs.
   • Clear DB: Wipes out the FAISS data, BM25 model, and chat histories to start fresh.
2. Uploading Documents:
   • Drop multiple PDFs into the upload zone and click ✨ Process PDFs to index them.
3. Conversational Dashboard:
   • Metrics at the top display total documents, chunks, and database status.
   • Enter queries in the chat input. The system will expand abbreviations and retrieve relevant chunks.
4. Validating Citations & Debugging:
   • Click 🔍 View Grounding Sources & Relevance Scores to inspect specific file snippets and their actual float scores.
   • If Developer/Debug Mode is enabled, expand 🛠️ Developer Retrieval Debugger to see query metadata, question classifications, and a tabular log of all candidates (retrieved vs. discarded) in real-time.

---

🎨 Premium Theme & UI Override System

Streamlit applications can look default and sometimes run into contrast issues. DocuMind bypasses this with specific CSS injections in app.py.

Aesthetic Foundation

• Font Integration: Loads Google Fonts' Outfit family, applying it across all elements.
• Adaptive Glassmorphic Backdrop: Dynamic HSL color mixing overlays subtle accents onto the selected theme's background.
• Theme Variables Override: Intercepts system defaults dynamically. When selecting a custom theme, it changes core system selectors:

  :root, .stApp, [data-testid="stSidebar"] {
      --background-color: #f8fafc !important; /* Light slate */
      --secondary-background-color: #f1f5f9 !important;
      --text-color: #0f172a !important; /* Deep dark slate */
      --primary-color: #8B5CF6 !important; /* Violet */
  }

Specific Widget Tweaks

1. Dropdown selectboxes & lists: Custom rules target dropdowns and individual <li> hover styles to map back to the selected theme's text color.
2. File Uploader Zone: Hides standard Streamlit upload text and injects styled CSS pseudo-elements (::before and ::after) to display a custom "📄 Drop PDFs Here" design.
3. Password Eye-Toggle Button: Removes dark block backgrounds and inherits the text color for the eye icon in light mode.
4. Chat Bubbles: Uses borders, color-mix highlights, and drop shadows to separate user bubbles from the assistant's grounded responses.

---

📂 Project Directory Structure

DocuMind/
│
├── .env.template            # Template for environment variables (copy to .env)
├── app.py                   # Main Streamlit app, custom CSS styles, and UI layout
├── requirements.txt         # Project dependencies (Streamlit, PyMuPDF, FAISS, rank-bm25)
│
├── data/                    # Vector store and metadata storage (created dynamically)
│   ├── chunks.json          # Extracted text chunks with original page metadata
│   └── faiss.index          # Local FAISS index binary file
│
├── utils/                   # Modular backend RAG logic
│   ├── chunking.py          # Document chunking logic (Recursive Character Splitter)
│   ├── embeddings.py        # HuggingFace Embeddings Manager (Singleton)
│   ├── llm.py               # Groq LLM orchestration and prompt template engine
│   ├── pdf_loader.py        # Document text parser (PyMuPDF wrapper)
│   └── retrieval.py         # FAISS + BM25 Hybrid manager and query intent classifier
│
└── scratch/                 # Temporary directories and tests
    └── test_pipeline.py     # End-to-end integration test script

---

🔍 File-by-File Code Walkthrough

Backend Modules (utils/)

1. pdf_loader.py

Contains extract_text_from_pdf(file_source: Union[str, bytes], filename: str) -> List[Dict[str, Any]].

• Purpose: Parses page-by-page text blocks.
• Internal Mechanism: Opens PDF streams or file paths via PyMuPDF, extracts clean unicode text, and returns structured dictionaries with text and metadata containing source and page.

2. chunking.py

Contains split_documents(documents: List[Dict[str, Any]], chunk_size: int = 500, chunk_overlap: int = 100).

• Purpose: Breaks documents into smaller chunks for vector storage.
• Internal Mechanism: Iterates over page documents, uses RecursiveCharacterTextSplitter to split text at natural boundaries. (In production, the app runs with chunk_size = 1000 and chunk_overlap = 200).

3. embeddings.py

Contains the EmbeddingsManager class.

• Purpose: Generates vector representations of text.
• Internal Mechanism: Built as a Python Singleton. Uses SentenceTransformer("all-MiniLM-L6-v2") to generate 384-dimensional float32 arrays, normalized for cosine similarity calculations.

4. retrieval.py

Contains the RetrievalManager class, abbreviation dictionary ABBREVIATIONS, helper expand_query(query: str) -> str, and helper detect_question_type(question: str) -> str.

• Purpose: Manages index state (load, save, add, search, clear), tokenizes and fits the BM25Okapi keyword index, runs RRF hybrid queries, and detects query intent.
• Internal Mechanism:
  • expand_query(): Uses case-insensitive word boundaries to replace architecture acronyms with full terms.
  • search(): Executes semantic searches in FAISS and keyword searches in BM25. Intersects results, calculates RRF scores, filters candidates by selected document early, and reconstructs FAISS vectors for out-of-index candidates to compute raw cosine similarity.

5. llm.py

Contains query_llm(...).

• Purpose: Orchestrates prompt formatting and queries the Groq API.
• Internal Mechanism:
  • Filters out duplicate chunks.
  • Hardens the system instruction templates to strictly refuse external knowledge, return "Information not found in uploaded documents." on gaps, and generate inline citations (e.g. [Filename, Page X]).

---

Frontend App (app.py)

• State Management: Uses st.session_state to hold models, database manager instances, and chat histories.
• Dynamic CSS Injection: Generates styling based on the selected theme, inserting overridden class rules into the HTML page header.
• Layout & Interaction:
  • Sets up the custom sidebar settings panel (theme selection, document filter dropdown, threshold slider, dev mode checkbox).
  • Implements the early check for the similarity threshold. If no chunks pass, it writes the fallback response and skips Groq API calls entirely.
  • Renders grounding sources with exact float scores.
  • Renders a pandas-driven st.dataframe showing real-time retrieved chunk status logs under the Developer Retrieval Debugger box.

---

Test Pipeline (scratch/test_pipeline.py)

• Purpose: Verifies the backend components.
• Internal Mechanism:
  1. Generates a mock two-page PDF file (test_paper.pdf) using PyMuPDF.
  2. Extracts and displays text page-by-page.
  3. Verifies chunking behavior.
  4. Generates embeddings and asserts that the vector shape is correct (n, 384).
  5. Verifies query abbreviation expansion.
  6. Performs a hybrid FAISS + BM25 search query, displaying Relevance and RRF scores.
  7. Clears the test index and database files.

---

🛠️ Setup & Installation

Prerequisites

• Python 3.9 or higher
• A Groq API Key (Free tier key available at console.groq.com)

1. Navigate to Project Directory

Open your terminal or PowerShell and change directories to the project root:

cd DocuMind

2. Set Up a Virtual Environment

Create and activate a virtual environment to manage project dependencies:

# Create environment
python -m venv .venv

# Activate on Windows (PowerShell):
.venv\Scripts\Activate.ps1

# Activate on macOS/Linux:
source .venv/bin/activate

3. Install Dependencies

Install the required packages listed in requirements.txt:

pip install -r requirements.txt

4. Configure Environment Variables

Copy .env.template to a new file named .env:

# Windows
copy .env.template .env

# macOS/Linux
cp .env.template .env

Open .env in a text editor and add your Groq API key:

GROQ_API_KEY=gsk_your_actual_key_here

5. Launch the Application

Run the Streamlit application:

streamlit run app.py

The application will open in your default browser at http://localhost:8501/.

---

🧪 Verification & Testing

To test the backend ingestion pipeline locally, run the integration test script:

python scratch/test_pipeline.py

Upon successful execution, the console output will show:

Created dummy PDF: test_paper.pdf

--- Testing PDF Text Extraction ---
Extracted 2 pages.
Page 1: Attention mechanisms have revolutionized natural language processing.
Transforme...
Page 2: In medical imaging, artificial intelligence has improved tumor detection rates.
...

--- Testing Chunking ---
Generated 4 chunks.
Chunk #1 [Page 1]: Attention mechanisms have revolutionized natural language processing.
Transformers utilize self-attention to model long-range dependencies in text.
Chunk #2 [Page 1]: This makes them superior to recurrent neural networks for many tasks.
Chunk #3 [Page 2]: In medical imaging, artificial intelligence has improved tumor detection rates.
Chunk #4 [Page 2]: Deep learning algorithms analyze MRI scans to find early signs of cancer.
Healthcare systems worldwide are adopting these machine learning tools.

--- Testing Embeddings Generation ---
Generated embeddings shape: (4, 384)

--- Testing Abbreviation Expansion ---
Original: 'Explain AR, DR, PC, IR, and RTL operations'
Expanded: 'Explain Address Register, Data Register, Program Counter, Instruction Register, and Register Transfer Language operations'
[PASS] Abbreviation expansion verified successfully.

--- Testing Hybrid Indexing & Search ---
Total chunks indexed: 4
Querying: 'How do transformers work?'

Search Results:
Result #1 (Relevance Score: 0.2864, RRF Score: 0.032787, Page: 1):
  Attention mechanisms have revolutionized natural language processing.
Transformers utilize self-attention to model long-range dependencies in text.
Result #2 (Relevance Score: 0.1438, RRF Score: 0.016129, Page: 1):
  This makes them superior to recurrent neural networks for many tasks.

[SUCCESS] End-to-end backend pipeline test PASSED successfully!

This confirms that text extraction, recursive splitting, embedding generation, abbreviation expansion, hybrid search execution, and RRF calculations are working as expected.