Question Organizer 🎓

A comprehensive, AI-powered system for managing a personal bank of math and science questions. Features a powerful CLI, a dynamic web dashboard, and an automated PDF ingestion pipeline driven by Gemini Vision OCR.

✨ Features

• Store & Tag: Organize questions by Subject/Module → School → Assignment Number.
• Difficulty & Modules: Track course modules, filter questions quickly.
• Log Attempts: Record every attempt at a question, track progress, and view performance over time with success/fail notes.
• AI Ingestion: Drag-and-drop or use CLI to automatically extract questions and solutions using Gemini Vision OCR from PDF homework sheets.
• Auto-tagging & Bulk Ingestion: Ingest entire folders of PDFs. Folders named MODULE_SCHOOL and files named ASSIGNMENT.pdf are automatically parsed to tag all extracted questions!
• Search & Filter: Find questions by module, school, assignment, or original source file.
• LaTeX Support: Full support for KaTeX in the web dashboard for beautiful mathematical rendering.
• Bulk Operations: Select multiple questions to delete or restore in the web UI.

---

🏗️ Architecture

graph TD
    subgraph Frontend [Frontend (Vanilla JS/HTML/CSS)]
        WebUI[Web Dashboard]
    end

    subgraph Backend [FastAPI Backend]
        API[api/main.py]
        CLI[cli.py]
    end

    subgraph Data [Data Layer (SQLite & SQLAlchemy)]
        DB[(question_organizer.db)]
        ORM[db/models.py]
        FolderParser[folder_parser.py]
    end

    subgraph Ingestion [Ingestion Pipeline]
        Ingest[ingest.py]
        Pdf2Img[pdf_to_images.py]
        OCR[llm_tagger.py]
    end

    WebUI -->|HTTP Requests| API
    API --> ORM
    CLI --> ORM
    ORM --> DB
    
    CLI --> Ingest
    API --> Ingest
    Ingest --> Pdf2Img
    Ingest --> OCR
    Ingest --> FolderParser
    OCR -->|API Key| Gemini((Gemini Vision API))

Loading

---

🔍 Detailed Component Documentation

💻 Frontend (frontend/)

The user interface is built as a Single Page Application (SPA) using purely Vanilla Javascript, HTML, and CSS.

• app.js: Handles all the dynamic interactions, fetching data from the backend API endpoints. It manages state for filtering, rendering question cards, tracking selections for bulk delete, and handling the drag-and-drop file upload interface.
• index.html & styles.css: Defines a clean, responsive layout. It includes KaTeX scripts to automatically parse and beautifully render LaTeX strings returned from the backend.
• API Interaction: Communicates asynchronously via fetch requests to the FastAPI backend on endpoints like /api/questions and /api/ingest.

⚙️ Backend API (api/main.py)

Built with FastAPI, this server acts primarily as the core HTTP backend connecting the database with the Web UI.

• Static files serving: Automatically mounts the frontend/ directory to serve the SPA on the root (/) path.
• CRUD Operations: Exposes REST endpoints to fetch (GET /api/questions), update tags (PUT /api/questions/{q_id}/tag), log attempts (POST /api/attempts), and handle complex queries (e.g. GET /api/filter-options).
• File Uploads: Handles single file (/api/ingest) and bulk folder (/api/ingest-folder) UploadFile requests. Inbound PDFs are temporarily saved, processed by the ingestion pipeline, and then cleaned up.

⚡ Command Line Interface (cli.py)

The CLI is built with Python's native argparse and serves as an alternative to the Web UI for quick operations and database manipulation.

• Database Initialization: python cli.py init creates tables and seeds default data.
• Interactive Prompts: Commands like add and tag walk the user through terminal inputs.
• List & View: Highly filterable CLI tables to read questions and solutions entirely from the terminal (python cli.py list --module MA2001).
• Ingestion Hooks: Capable of invoking the ingest_pdf and ingest_folder pipelines directly from local paths. Also supports a --dry-run flag to preview OCR extraction without writing to the database.

🧠 Ingestion Pipeline

The AI backbone of the project, converting static PDFs into structured, taggable database rows.

• ingest.py (Orchestrator): Coordinates the entire flow. It accepts a PDF path, defers to pdf_to_images for rendering, sends the images to llm_tagger for OCR, parses optional folder tags via folder_parser, and writes the final rows to the SQLite database.
• pdf_to_images.py: Uses the pdf2image library (backed by poppler) to convert PDF pages into high-resolution PIL Image objects suitable for machine vision.
• llm_tagger.py (OCR Engine): Interfaces directly with the google-genai SDK using the gemini-3.1-flash-lite-preview model. It sends the PIL Images along with a rigorously designed system prompt.
  • Note on output format: The prompt forces the LLM to return pure XML (<questions><question><content>...</content><solution>...</solution></question></questions>). This intentionally bypasses JSON parsers, which frequently break on heavily backslash-escaped LaTeX mathematics.
• folder_parser.py: Extracts auto-tagging metadata from file structures. If a directory is named MA2001_NTU and contains TUT_1.pdf, the system automatically tags extracted questions with module="MA2001", school="NTU", and assignment_number="TUT_1".

🗄️ Database (db/)

Powered by SQLite and SQLAlchemy for lightweight, robust persistence.

• database.py: Sets up the SQLite engine and provides the get_session() context manager for safe transactional operations.
• models.py: Defines the DeclarativeBase ORM.
  • Question: Stores content, solutions, taxonomy tags (module, school, assignment), and raw source file tracking.
  • Attempt: A relational table linked via question_id. Tracks timestamp, boolean was_correct, and optional textual notes, providing a historical progression of learning.

---

🛠️ Setup

1. Prerequisites

• Python 3.10+
• A Gemini API Key (for PDF ingestion)
• Poppler: Required for PDF image rendering.
  • Linux/Ubuntu: sudo apt install poppler-utils
  • macOS: brew install poppler
  • Windows: Download Poppler and add the bin folder to your PATH.

2. Installation

# Clone the repository
git clone https://github.com/low1408/Question-Organizer.git
cd Question-Organizer

# Create and activate a virtual environment
python -m venv virtualenv
source virtualenv/bin/activate  # Windows: virtualenv\Scripts\activate

# Install dependencies
pip install -r requirements.txt

3. Environment Configuration

Create a .env file in the project root:

GEMINI_API_KEY=your_actual_api_key_here
GEMINI_MODEL_NAME=gemini-3.1-flash-lite-preview

4. Database Initialization

python cli.py init

This creates the SQLite database and seeds it with default subjects, topics, and difficulties.

---

🚀 Usage

🌐 Web Dashboard (Recommended)

Launch the FastAPI development server:

uvicorn api.main:app --reload

Open http://localhost:8000 in your browser.

• Dashboard: Filter questions, log attempts (Reveal Solution), and manage your question bank.
• Ingest: Drag and drop PDF files or entire folders to extract questions directly into your database.
• Manage: Create and organize subjects, topics, and modules.

💻 Command Line Interface

The CLI is powerful for quick lookups and local ingestion.

Command	Usage	Description
list	python cli.py list	List all questions with snippets. filter with --module, --school, --assignment.
view	python cli.py view <id>	View full question, solution, and attempt history.
add	python cli.py add	Interactively add a new question.
attempt	python cli.py attempt <id> --correct	Log a success/fail attempt.
ingest	python cli.py ingest hw1.pdf	Extract questions from a local PDF. Use --dry-run to test first.
ingest-folder	python cli.py ingest-folder ./MA2001_NTU_DIR/	Extract all PDFs in a folder, auto-tagging them.

---

🧪 Testing

Run the automated test suite using pytest:

pytest

---

📝 License

Distributed under the MIT License. See LICENSE for more information (if applicable).