DeepDecode – Multimodal RAG & Video Intelligence Platform

Live Demo: https://deepdecode.onrender.com

DeepDecode is a multimodal Retrieval-Augmented Generation (RAG) platform that enables users to upload videos, automatically generate transcripts, and interact with video content through conversational AI. Instead of manually searching through lengthy videos, users can ask natural language questions and receive accurate, timestamped, citation-backed answers linked to the relevant video segments.

Features

• 🎥 Upload and process video files
• 🎙 Automatic transcription using Whisper V3
• 🤖 Conversational video search powered by Gemini 2.5 Pro
• 🔍 Hybrid retrieval combining vector and keyword search
• ⏱ Timestamped, citation-backed responses
• ⚡ Asynchronous processing with Celery
• 📈 Scalable pipeline for long-duration videos

Tech Stack

Frontend: React, Tailwind CSS

Backend: FastAPI, Celery, PostgreSQL, Qdrant

AI/ML: Whisper V3, Gemini 2.5 Pro

Architecture

                +------------------+
                |   React Client   |
                +--------+---------+
                         |
                         v
                  FastAPI Backend
                         |
      +------------------+------------------+
      |                                     |
      v                                     v
 Video Upload                        Chat Query API
      |                                     |
      v                                     |
 Celery Background Worker                   |
      |                                     |
      v                                     |
 Whisper V3 Transcription                   |
      |                                     |
      v                                     |
Chunking + Metadata Extraction              |
      |                                     |
      +------------------+------------------+
                         |
                         v
                Embedding Generation
                         |
          +--------------+--------------+
          |                             |
          v                             v
      PostgreSQL                   Qdrant
 (Metadata/Transcript)       (Vector Embeddings)
          |                             |
          +--------------+--------------+
                         |
                  Hybrid Retrieval
                         |
                    Gemini 2.5 Pro
                         |
                  Citation Generation
                         |
                         v
                Timestamped Response

How It Works

1. Users upload a video through the React frontend.
2. FastAPI queues the processing task using Celery.
3. Whisper V3 transcribes the video's audio.
4. The transcript is divided into semantic chunks and converted into embeddings.
5. Transcript metadata is stored in PostgreSQL, while vector embeddings are stored in Qdrant.
6. When a user asks a question, hybrid retrieval combines semantic vector search and keyword matching to fetch the most relevant transcript chunks.
7. Gemini 2.5 Pro generates a context-aware response with timestamps and citations pointing to the relevant parts of the video.

Project Structure

deepdecode/
├── frontend/
├── backend/
│   ├── api/
│   ├── workers/
│   ├── services/
│   ├── rag/
│   ├── embeddings/
│   └── database/
├── docs/
└── README.md

Performance

• Processes 10-minute videos in under 2 minutes
• Delivers sub-second hybrid search latency
• Reduces manual video search time by approximately 40%
• Supports scalable asynchronous background processing

Future Improvements

• Multi-video search
• Speaker diarization
• OCR for presentation slides
• Live meeting transcription
• Streaming AI responses
• User authentication
• Distributed Celery workers
• Cross-encoder reranking

Installation

git clone https://github.com/yourusername/deepdecode.git
cd deepdecode
pip install -r requirements.txt

cd frontend
npm install

Run the backend:

uvicorn app.main:app --reload

Run Celery:

celery -A app.worker worker --loglevel=info

Run the frontend:

npm run dev

License

MIT License