NoteKeeperAI

Local-first, AI-powered meeting summarization platform.
Built for privacy, speed, and modularity using FastAPI, Next.js, MLX (Whisper), vLLM, PostgreSQL, and MongoDB.

🏗 Architecture

The system follows a Polyglot Persistence architecture, decoupling metadata management from high-volume text data.

🔄 End-to-End Workflow

1. Meeting Capture

   • Frontend: A Next.js dashboard connects to the Backend.
   • Bot Service: Uses MeetingBaas API to spawn a bot that joins Google Meet/Zoom/Teams calls.
   • Audio Stream: The bot streams raw audio (16kHz PCM) via WebSocket to the FastAPI Gateway.

2. Audio Processing Pipeline

   • Gateway: Buffers audio into 30-second chunks.
   • Transcription Worker:
     • Consumes audio chunks.
     • Runs Whisper Large v3 Turbo locally via MLX.
     • Pushes raw text to MongoDB (transcripts collection) and notifies the frontend via SSE.

3. Intelligence Layer

   • Intelligence Worker:
     • Monitors new transcripts for active sessions.
     • Aggregates context (sliding window).
     • Queries a local Qwen 2.5 (1.5B) LLM via vLLM for summarization and action item extraction.
     • Saves insights to MongoDB (summaries collection) and notifies the frontend via SSE.

4. Data Persistence Layer

   • PostgreSQL: Stores relational metadata (Sessions, Bot IDs, Meeting Status).
   • MongoDB: Stores unstructured, high-volume data (Transcripts, Append-only Summaries).

5. Real-Time UI

   • The Dashboard listens to Server-Sent Events (SSE).
   • Updates are filtered by session_id to show only relevant live data.

---

🚀 Tech Stack

• Frontend: Next.js 14, TailwindCSS, Framer Motion, SSE.
• Backend API: FastAPI, WebSockets (/stream), SSE (/events).
• AI/ML:
  • ASR: mlx-whisper (Optimized for Apple Silicon).
  • LLM: vLLM serving Qwen/Qwen2.5-1.5B-Instruct.
• Database:
  • PostgreSQL: Session Management.
  • MongoDB: Content Storage.

🛠 Local Setup

Prerequisites

• Python 3.11+
• Node.js 18+
• Docker/Podman (for DBs) or local installs of Postgres & Mongo.

1. Databases

Ensure PostgreSQL and MongoDB are running.

# Example Docker command
docker run -d -p 5432:5432 -e POSTGRES_PASSWORD=password postgres
docker run -d -p 27017:27017 mongo

2. Backend

cd app
python -m venv venv
source venv/bin/activate
pip install -r requirements.txt

# Start vLLM Server (Separate Terminal)
python -m vllm.entrypoints.openai.api_server --model Qwen/Qwen2.5-1.5B-Instruct --port 8000

# Start Gateway
uvicorn main:app --reload --port 8008

3. Frontend

cd frontend
npm install
npm run dev

Visit http://localhost:3000 to start recording.

---

🧩 Key Design Decisions

• Polyglot Persistence: chosen to optimize for relational integrity (sessions) vs. rapid write throughput (transcriptions).
• Event-Driven: Workers communicate via internal queues, decoupled from the API loop to prevent blocking.
• Local-First: All inference (ASR & LLM) runs on-device for privacy and zero latency.