RAG Vector Database

A hands-on implementation of Retrieval-Augmented Generation (RAG) concepts, including document chunking, embeddings, vector search, BM25 retrieval, and hybrid search.

Features

• Document chunking
• Embedding generation
• Vector database storage
• Semantic similarity search
• BM25 keyword search
• Hybrid retrieval (BM25 + semantic search)
• Context retrieval for LLMs
• Claude API integration

Repository Contents

File	Description
001_chunking.ipynb	Demonstrates document chunking strategies for RAG pipelines.
002_embeddings.ipynb	Generates vector embeddings for document chunks.
003_vector_index.ipynb	Implements semantic search using embeddings and a custom vector index.
004_bm25.ipynb	Implements BM25 keyword search for document retrieval.
005_hybrid.ipynb	Combines BM25 and semantic search into a hybrid retrieval pipeline.

Learning Path

1. Document chunking
2. Embedding generation
3. Vector-based semantic search
4. BM25 keyword retrieval
5. Hybrid retrieval (BM25 + semantic search)

Technologies

• Python
• Anthropic Claude API
• NumPy
• Pandas

Project Goal

This repository explores the fundamental building blocks of Retrieval-Augmented Generation (RAG) through practical notebook implementations, covering document preprocessing, embedding generation, semantic retrieval, keyword retrieval, and hybrid search techniques.

Disclaimer

This repository is shared for educational purposes and personal learning.

Acknowledgments

This repository is based on and adapted from Anthropic's educational materials on Retrieval-Augmented Generation (RAG). It is shared for personal learning and experimentation.