🤖 Clerk-RAG: Authenticated RAG-based Chatbot with Streamlit + Next.js + Clerk

This project implements a fully functional Retrieval-Augmented Generation (RAG) system with:

• 🔐 Clerk authentication (Next.js frontend)
• 🧠 LangChain RAG pipeline (Streamlit backend)
• 🗂️ ChromaDB for persistent vector storage
• 🌐 Document & Website ingestion
• 💬 Secure chat interface powered by Groq LLMs

---

🏗️ Project Structure

Clerk-RAG/
├── backend/
│ ├── app.py # Streamlit UI with Clerk-protected access
│ ├── requirements.txt
│ └── utils/
│ ├── auth.py # Clerk JWT/session validation
│ ├── data/
│ │ ├── scraper.py
│ │ └── doc_loader.py
│ ├── llm/
│ │ └── llm_generator.py
│ ├── rag/
│ │ ├── retriever.py
│ │ └── query_constructor.py
│ └── vectorstore/
│ ├── indexer.py
│ └── chroma_handler.py
│
├── frontend /
│ ├── app/
│ │ ├── layout.tsx # ClerkProvider + redirect logic
│ │ ├── page.tsx # Home landing
│ │ ├── sign-in/page.tsx # Sign-in page
│ │ ├── sign-up/page.tsx # Sign-up page
│ │ └── redirect/page.tsx # Auto-redirects to Streamlit
│ ├── middleware.ts # Clerk auth for protected routes
│ ├── .env.local # Clerk API keys
│ ├── package.json
│ ├── tsconfig.json
│ └── tailwind.config.js # Optional (styling) \

yaml Copy Edit

---

🚀 How to Run

1. 🔧 Backend Setup (Python)

cd backend
python3 -m venv venv
source venv/bin/activate
pip install -r requirements.txt

To run the backend:

streamlit run app.py

1. ⚙️ Frontend Setup (Next.js + Clerk)

cd frontend
npm install

Create .env.local inside frontend/:

CLERK_PUBLISHABLE_KEY=<your_clerk_publishable_key>
CLERK_SECRET_KEY=<your_clerk_secret_key>
NEXT_PUBLIC_CLERK_FRONTEND_API=<your-clerk-frontend-api>
NEXT_PUBLIC_CLERK_PUBLISHABLE_KEY=<your_clerk_publishable_key>

To run the frontend:

npm run dev

✅ Features

Clerk auth via hosted UI or embedded login

Website scraping via crawl4ai

PDF/TXT file ingestion using unstructured & PyMuPDF

Vectorization using Ollama/Gemini Embeddings

RAG query pipeline using LangChain

Persistent ChromaDB vector store

Simple chatbot UI via Streamlit

🔐 Auth Flow

User lands on /sign-in in Next.js frontend.

Upon login, Clerk redirects to /redirect.

Clerk session or JWT is passed to Streamlit app via query param.

Streamlit verifies token and grants access.

📌 Notes

JWT/session expiry handled using st.session_state to avoid constant revalidation.

RAG works with both website and file content.

Supports Groq (via LangChain).

🙏 Acknowledgements

Clerk – Authentication made easy

LangChain – Powerful RAG orchestration

ChromaDB – Fast local vector DB

Streamlit – Chat UI framework