LLM API Gateway with Semantic Caching

A high-performance FastAPI-based LLM API gateway designed to reduce application latency and API consumption costs using intelligent Semantic Caching powered by vector similarity search.

---

🚀 Features

• FastAPI Routing: High-throughput asynchronous routing.
• Semantic Caching:
  • Encodes user queries to vector embeddings using API-based embedding services. Currently supports Voyage AI (default) and Cohere API.
  • The hosted demo link uses Voyage AI's free tier for embeddings, which is rate-limited to 3 requests per minute.
  • Option for local encoding using sentence-transformers (all-MiniLM-L6-v2, 384 dimensions) is present in the codebase but not active by default.
  • Queries a PostgreSQL database using the pgvector extension for cosine distance (<=> operator).
  • Automatically returns cached responses for semantically similar queries (default threshold $\ge$ 90% similarity).
• Multi-Model & Provider Support:
  • Built-in integration with Groq for default low-latency inference.
  • Support for Anthropic (Claude) and Google Gemini models.
• Bring Your Own Key (BYOK):
  • Allows clients to supply their own API keys (e.g., Anthropic, Gemini) dynamically via the X-API-Key request header.
• Sliding-Window Rate Limiting:
  • Custom memory-based sliding window rate limiter (default: max 5 requests per 60 seconds per IP, with automatic blocks) to prevent API abuse.
• Asynchronous Request Logging: Records metadata including:
  • Raw query and generated response
  • Active model used
  • Cache status (Hit / Miss)
  • Execution/network latency in milliseconds

---

📋 Prerequisites & Infrastructure

1. Python 3.8+
2. PostgreSQL with the pgvector extension installed.
   • The database tables and extension configuration are defined in schema.sql.
   • You can automatically run the schema migration script init_db.py (see details in the Setup section).

---

🛠️ Getting Started

1. Setup Environment

Clone or navigate to the project directory, then create and activate a virtual environment:

# Create Virtual Environment
python -m venv venv

# Activate Virtual Environment (Windows)
.\venv\Scripts\activate

# Activate Virtual Environment (Unix/macOS)
source venv/bin/activate

2. Install Dependencies

pip install -r requirements.txt

3. Configure Environment Variables

Copy the template configuration file:

cp .env.example .env

Open .env and fill in your actual credentials:

• GROQ_API_KEY: Your Groq platform API token.
• DATABASE_URL: Your PostgreSQL connection string (e.g. Neon PostgreSQL, AWS RDS, or local PostgreSQL instance).
• VOYAGE_API_KEY: Your Voyage AI API key (used for default embedding service).
• COHERE_API_KEY: Your Cohere API key (used if using Cohere embedding service).

If you are using the hosted demo link, keep in mind that embeddings are served through Voyage AI's free tier and are limited to 3 requests per minute.

4. Initialize the Database

Run the schema migration script to enable pgvector and automatically create the required database tables:

python app/db/init_db.py

5. Running the Server

Start the development server using uvicorn:

uvicorn app.main:app --reload

The server will start on http://127.0.0.1:8000.

---

🔌 API Endpoints

Health Check

• GET /
• GET /api/health
• Response: {"status": "ok"}

Chat Completion (with Semantic Cache)

• POST /api/chat
• Headers:
  • X-API-Key (Optional): Client-supplied API key for Bring Your Own Key (BYOK) providers (e.g., Anthropic, Gemini).
• Note: When using the hosted demo link, embedding requests go through Voyage AI's free tier, so requests are rate-limited at 3 requests per minute.
• Request Body:

  {
    "query": "What is the capital of France?",
    "model": "claude-3-5-sonnet-20241022"
  }

  Note: The model field is optional. If not provided, it defaults to "openai/gpt-oss-120b" via Groq. Models starting with claude or gemini will use the key provided in the X-API-Key header.
• Response:

  {
    "response": "The capital of France is Paris.",
    "cached": false,
    "latency_ms": 420
  }

  (Subsequent calls with identical or semantically similar queries will return "cached": true and low latencies like <10ms)

---

🗺️ Roadmap & Future Implementation

• Fallback & Routing Policies: Automatically routing to a secondary model/provider if the primary provider hits rate limits or encounters downtime.
• DeepSeek and Native OpenAI Integration: Add native client integrations for DeepSeek and OpenAI (beyond Groq routing).
• Billing & Tenant Segmentation: Dynamic rate limiting and usage metrics based on authenticated tenants or user tokens.
• Security Auditing: Validation checks on user-supplied BYOK credentials to prevent security issues.