Skip to content

kushin25/chatvector-ai

BranchesTags
 
 

Repository files navigation

ChatVector

Open-Source Backend-First RAG Engine for Document Intelligence

ChatVector is an open-source Retrieval-Augmented Generation (RAG) engine for ingesting, indexing, and querying unstructured documents such as PDFs and text files.

Think of it as an engine developers can use to build document-aware applications — such as research assistants, contract analysis tools, or internal knowledge systems — without having to reinvent the RAG pipeline.

Status PRs Welcome AI RAG CI

Python Version FastAPI Version

⭐ Star the repo to follow progress and support the project!

GitHub stars

🔗 Quick Links

Good First Issues Roadmap Quick Setup Project Board Dev Notes Architecture License: MIT Contributing Docs Contributing Video Discussions

📌 Table of Contents

🔎 What is ChatVector?

ChatVector provides a clean, extensible backend foundation for RAG-based document intelligence. It handles the full lifecycle of document Q&A:

  • Document ingestion (PDF, text) with configurable chunking strategies
  • Text extraction, cleaning, and semantic chunking
  • Vector embedding and storage via pgvector
  • Semantic retrieval with optional query transformations
  • LLM-powered answer generation with cited responses
  • Background processing queue with rate limiting and retry logic

The goal is to offer a developer-focused RAG engine you can deploy as a service and integrate via HTTP — not a polished end-user SaaS, and not a framework you have to assemble yourself.

ChatVector vs Frameworks

ChatVector is designed as a production-ready backend service, not a general-purpose framework. Here's how it compares:

Aspect ChatVector (This Project) General AI Framework (e.g., LangChain)
Primary Goal Deliver a deployable backend service for document intelligence. Provide modular components to build a wide variety of AI applications.
Out-of-the-Box Experience A fully functional FastAPI service with logging, testing, rate limiting, and a clean API. A collection of tools and abstractions you must wire together and productionize.
Architecture Batteries-included, opinionated engine. Get a working system for one use case. Modular building blocks. Assemble and customize components for many use cases.
Best For Developers, startups, or teams who need a document Q&A API now and want to focus on their application layer. Developers and researchers building novel, complex AI agents or exploring multiple LLM patterns from the ground up.
Path to Production Short. Configure, deploy, and integrate via API. Built-in observability, rate limiting, and scaling patterns. Long. Requires significant additional work on API layers, monitoring, deployment, and performance tuning.

👥 Who is this for?

ChatVector is designed for:

  • Developers building document intelligence tools or internal knowledge systems
  • Backend engineers who want a solid RAG foundation without heavy abstractions
  • AI/ML practitioners experimenting with chunking, retrieval, and prompt strategies
  • Open-source contributors interested in retrieval systems, embeddings, and LLM orchestration

🚀 Current Status

Phases 1–2.5 Complete | Phase 3 In Progress

Phases 1, 2, and 2.5 are complete. Phase 3 is actively underway — most backend quality and provider work has shipped, but production API-key enforcement is still in progress. See ROADMAP.md for the full breakdown.

What's working today:

Backend

  • ✅ PDF and text document ingestion
  • ✅ Configurable chunking strategies (fixed, paragraph, semantic)
  • ✅ Vector embeddings + semantic search via pgvector
  • ✅ Hybrid retrieval (PostgreSQL full-text + vector, RRF fusion)
  • ✅ Baseline retrieval reranking (similarity + lexical overlap)
  • ✅ Session-scoped and tenant-wide retrieval modes
  • ✅ LLM-powered answers with source citations and relevance scores
  • ✅ Query transformations (rewrite, expand, stepback)
  • ✅ Configurable response personas and system prompt
  • ✅ Session-based chat with persisted conversation history
  • ✅ SSE streaming chat (/chat/stream)
  • ✅ Background ingestion queue with rate limiting, retry, and DLQ
  • ✅ Redis-backed ingestion queue (production default; in-memory fallback for local dev)
  • ✅ Structured logging with request ID tracing
  • ✅ Health checks with TTL caching on /status
  • ✅ Per-tenant rate limiting on all authenticated API routes
  • ✅ Security headers, CORS hardening, input validation
  • ✅ Production Compose config + GitHub Actions CI
  • ✅ Pluggable LLM providers (Gemini, OpenAI, Ollama, Anthropic Claude)
  • ✅ Pluggable embedding providers (Gemini, OpenAI, Ollama, Voyage AI)
  • ✅ Mixed-provider configurations (e.g. Claude + Voyage)
  • ✅ Response metadata: latency_ms and model on chat responses
  • ✅ Python client SDK (core synchronous workflows)

Frontend Demo

  • ✅ Document upload with live pipeline stage display and SSE progress
  • ✅ Real-time ingestion status polling
  • ✅ Full RAG chat with source citations and session sidebar
  • ✅ SSE streaming chat, batch query demo, and live system status page
  • ✅ Responsive design with dark developer aesthetic

Active Phase 3 work:

  • 🚧 Production API-key authentication and strict tenant enforcement (plumbing scaffolded; validation not yet active)
  • 🚧 Redis-backed distributed rate limiting
  • 🚧 Python SDK parity (sessions, streaming, retrieval scopes)
  • 🚧 Node.js/TypeScript SDK (planned)
  • 🚧 Query transformation debug metadata and retrieval inspection tooling

🧠 Architecture Overview

Backend Layer (Core)

  • FastAPI — modern Python API framework
  • Uvicorn — high-performance ASGI server
  • slowapi — per-tenant rate limiting
  • Design goals: clarity, extensibility, resilience, and security by default

AI & Retrieval Layer

  • Pluggable providers — Gemini, OpenAI, Ollama, Anthropic Claude (LLM), and Voyage AI (embeddings); mix and match independently
  • Hybrid retrieval — vector similarity + PostgreSQL full-text search with RRF fusion
  • Baseline reranking — deterministic similarity + lexical overlap reranker
  • Retrieval scopes — session-scoped (default) or tenant-wide search
  • Configurable chunking — fixed, paragraph, or semantic strategies
  • Query transformations — rewrite, expand, or stepback before retrieval
  • Response personas — default, concise, conversational, academic, technical

Data Layer

  • PostgreSQL + pgvector — vector similarity search
  • SQLAlchemy (development) / Supabase (production)
  • Strategy pattern — swap backends without touching business logic

Reference Frontend (Non-Core)

  • Next.js + TypeScript
  • Full end-to-end demo — upload, ingest, chat, citations
  • Not production-ready — exists to demonstrate and test backend capabilities

See ARCHITECTURE.md for full system design details.

🎯 Quick Start

After installing Docker and Node.js, start the complete ChatVector development environment with one command.

Prerequisites

  • Docker with Docker Compose (docker compose)
  • Node.js and npm
  • Either:
    • an API key for a supported hosted provider (Gemini or OpenAI), or
    • a local Ollama installation

Gemini is the recommended default and the simplest guided setup.

One-command setup

make quickstart

The command creates the env file, pauses while you add provider credentials, then continues after you press Enter. It installs frontend dependencies, builds the backend Docker image, starts backend services and the non-containerized frontend demo, waits for both to become ready, and opens the frontend and API docs when supported.

Setup is safe to rerun — existing backend/.env and frontend-demo/.env.local files are never overwritten.

If provider configuration is already complete, make quickstart continues immediately without pausing.

Provider configuration

Edit backend/.env to choose providers and set credentials. Gemini is the recommended default:

LLM_PROVIDER=gemini
EMBEDDING_PROVIDER=gemini
GEN_AI_KEY=your_google_ai_studio_api_key

Supported combinations include Gemini, OpenAI, Ollama, Anthropic Claude (generation), and Voyage AI (embeddings), including mixed setups (for example Claude + Voyage). See backend/.env.example for all variables.

URLs

Alternative non-interactive flow

make setup
# edit backend/.env
make
  • make setup — create env files, install dependencies, and build Docker images (prints editing instructions if configuration is incomplete; does not wait)
  • make — start backend and frontend, then open browser tabs

Returning contributors

Returning contributors normally use:

make

Start without browser tabs

make dev

Useful for SSH sessions, CI, or when you prefer to open URLs yourself.

Useful commands

Command Purpose
make quickstart Create env, pause for credentials, then start everything
make setup Create env files, install dependencies, and build Docker images
make Start backend + frontend, open browser tabs (default)
make dev Start backend + frontend without opening tabs
make backend Start only the backend Docker stack
make frontend Start only the frontend demo
make open Open the frontend and API docs URLs
make stop Stop this repo's frontend process and Docker services
make help Show all Make commands

Notes:

  • Setup is safe to rerun and preserves existing env files.
  • Provider credentials are edited in backend/.env — the setup scripts do not prompt for or read API keys in the terminal.
  • Press Ctrl+C while make, make dev, or make quickstart is running to stop the frontend; backend containers keep running until you run make stop.
  • The frontend demo is a non-core, non-containerized reference UI for testing the backend.

Try the API

  1. POST /upload — upload a PDF, get a document_id and status_endpoint
  2. GET /documents/{document_id}/status — poll ingestion stage and progress
  3. POST /chat — ask questions using the document_id

Extra Docker Commands

Command Purpose
docker compose up Start containers without rebuilding
docker compose down Stop containers, preserve data
docker compose down -v Stop containers and delete all DB data
docker compose up --build Rebuild containers after code changes
docker compose logs -f api Follow API logs in real time
docker compose exec db psql -U postgres Connect to Postgres directly

Or use the Makefile shortcuts above — run make help for the full list.

Manual setup (alternative)

If you prefer not to use Make, copy backend/.env.example to backend/.env and configure your providers. Gemini is the simplest default:

LLM_PROVIDER=gemini
EMBEDDING_PROVIDER=gemini
GEN_AI_KEY=your_key_here

See backend/.env.example for OpenAI, Ollama, Anthropic Claude, Voyage AI, and mixed-provider configurations. Then:

docker compose up --build

cd frontend-demo
npm ci
echo "NEXT_PUBLIC_API_URL=http://localhost:8000" > .env.local
npm run dev

Frontend runs at http://localhost:3000

Python SDK

A synchronous Python client is available today. A Node.js/TypeScript SDK is planned for Phase 3.

pip install ./sdk/python
from chatvector import ChatVectorClient

with ChatVectorClient("http://localhost:8000") as client:
    doc = client.upload_document("report.pdf")
    client.wait_for_ready(doc.document_id, timeout=90)
    answer = client.chat("What are the key findings?", doc.document_id)
    print(answer.answer, answer.latency_ms, answer.model)
    for source in answer.sources:
        print(source.file_name, source.page_number, source.score)

Session management, streaming chat, and retrieval scope options are not yet exposed in the SDK. See sdk/python/README.md for details.

🤝 Contributing

High-impact contribution areas:

  • Ingestion & indexing pipelines
  • Retrieval quality & evaluation
  • Chunking and query transformation strategies
  • API design & refactoring
  • Performance & scaling
  • Security hardening
  • SDK development
  • Documentation & examples

Frontend contributions are welcome but considered non-core.

See CONTRIBUTING.md for details and Good First Issues to get started.

📄 License

License: MIT

About

Open-source RAG engine for ingesting, indexing, and querying unstructured documents

Resources

Readme
License

Contributing

Contributing
Activity

Stars

0 stars

Watchers

0 watching

Forks

0 forks
Report repository

Releases

Packages

Contributors

Languages