CodeCompass

Grounded answers over a code repository, with citations you can verify.

What it does

Dropping into an unfamiliar codebase means reconstructing three kinds of knowledge that live in three different places: how the code actually works (source files), how it's meant to be used (documentation, which is often stale or incomplete), and why it is the way it is (issue and PR discussion — the institutional memory). Searching by keyword misses anything phrased differently than the code; asking a general-purpose AI assistant gets you a fluent answer it cannot prove and cannot point you to inside the actual repository.

CodeCompass ingests a single public GitHub repository's code, documentation, and issue history into a local, searchable index, and answers natural-language questions with responses grounded in — and cited back to — the specific files, line ranges, and issues that support them. Every citation resolves to a real, clickable link into the repository at the exact commit that was indexed, so an answer is something you check, not something you trust.

It runs entirely on free, local infrastructure by default — a local embedding model, a local vector store, and a local language model via Ollama. No API keys, no paid services, and no cloud dependency are required to ask a question.

A second, deliberate focus of the project is edge-constraint benchmarking: measuring how the retrieval pipeline's quality and latency degrade when run under simulated memory, CPU, and network limits, with the portable simulator validated against real OS-enforced limits on Linux.

Demo

streaming-demo.mp4

The recording above shows a full round trip against the real, indexed kubeedge/ianvs repository: asking a question, the answer streaming in token by token, clicking a citation to confirm it opens the real source file on GitHub, and the light/dark theme toggle.

The mem-256 row above is an expected, validated result, not a bug — see Benchmarks.

How it works

1. Point it at a repository. codecompass ingest --repo <url> clones the code and docs and pulls issue/PR history through the GitHub API, recording the commit it indexed.
2. Ask a question. Retrieval combines dense vector search and keyword search, reranks the result with a local cross-encoder, and always includes a pinned summary of the repository so broad questions ("what is this about?") are answerable, not just narrow lookups.
3. Get a grounded answer. A local language model is given only the retrieved context and instructed to cite every claim or say it doesn't know — never to fill gaps from its own training data. Citations are resolved mechanically against the real index, not trusted from the model, so a citation that doesn't check out is a caught error, not a silent one.

Features

• Grounded, checkable answers. Citations resolve to a real file, line range, or issue/PR, with a clickable link to the exact indexed commit on GitHub — never a bare claim.
• Hybrid retrieval. Dense vector search and BM25 keyword search are fused with Reciprocal Rank Fusion, then re-ranked with a local cross-encoder, recovering both semantic matches and exact keyword/identifier lookups that pure vector search tends to miss.
• Repository-aware on broad questions. A pinned overview chunk, set at ingestion time, is always available so "what is this repository about?" gets a real answer instead of an abstention.
• Fully local by default. Embeddings, vector search, and generation all run on a single machine via Ollama — no API keys, no per-query cost, no network dependency at query time.
• Streaming responses over Server-Sent Events, so answers appear as they're generated rather than after a long silent wait.
• Two answer styles. A detailed, jargon-explaining "guided" mode aimed at newcomers, and a terse "concise" mode used for benchmarking, so quality results stay comparable across changes.
• Edge-constraint benchmarking. A two-layer constraint simulator — a portable, application-level model that runs anywhere, and real OS-enforced limits on Linux — measures how retrieval quality and latency hold up under memory, CPU, and network pressure.
• A CLI, an HTTP API, and a web UI, all built on the same retrieval/generation pipeline, with light and dark themes and citation references rendered in a monospace type built for scanning file paths and line numbers.

Architecture

Indexing:  GitHub repo --> Ingestion --> Chunking --> Embedding --> Vector store
           (clone + REST)  (Document)    (Chunk x3)   (bge-small)   (Milvus Lite)

Querying:  question --> Retrieval --> Prompt builder --> LLM provider --> grounded answer
                        (hybrid +      (cite-or-abstain)  (Ollama/Gemini)  + resolved citations
                         rerank +
                         overview)

Code, documentation, and issue/PR history are split into retrieval-sized chunks with a strategy specific to each content type (structure-aware for code, heading-aware for docs, thread-aware for issues), embedded with bge-small-en-v1.5, and stored in a local Milvus Lite index using an exact (FLAT) index — a deliberate choice so that quality differences in benchmarking are attributable to the pipeline, not to approximate-search noise. Generation runs locally via Ollama (qwen2.5:3b-instruct by default), behind a provider interface that also supports a hosted Gemini backend.

Full design rationale, including the reasoning behind every major technical decision, is in docs/ARCHITECTURE.md.

Benchmarks

Retrieval and answer quality are measured against a hand-built gold question set over the real target repository, kubeedge/ianvs. Adding hybrid search, reranking, and the pinned overview chunk over a pure-vector baseline improves retrieval substantially:

metric	vector only	hybrid + rerank + overview
hit@k	0.533	0.667
recall@k	0.500	0.633
MRR	0.208	0.486
nDCG@k	0.262	0.504

The constraint simulator's results are equally direct: under a memory budget of 256 MB, this specific model/embedding stack cannot even finish loading — confirmed two independent ways, first by a portable application-level simulator and then by real OS-enforced memory limits (RLIMIT_AS) on Linux. The two layers agree at every tested budget except one: at a looser budget where the workload's actual memory use fits comfortably but the virtual address space needed to memory-map PyTorch's native library does not, the portable simulator is optimistic and only the real OS-enforced check catches the failure — a genuine, documented limit of modeling memory pressure by resident-set size alone.

Full methodology, the complete edge-constraint sweep, and the two-layer validation are in docs/BENCHMARKS.md, with raw report files under docs/results/.

Getting started

Prerequisites

• Python 3.12+
• Ollama installed, with a model pulled: ollama pull qwen2.5:3b-instruct
• Node.js 18+ (only needed for the web UI)
• A GitHub personal access token (for ingesting repositories without hitting the unauthenticated rate limit)

Install

git clone https://github.com/Fahmid-Arman/CodeCompass.git
cd CodeCompass
python3 -m venv .venv
source .venv/bin/activate
pip install -e ".[dev,api,embeddings]"

Create a .env file in the project root:

GITHUB_TOKEN=your_github_token_here

Make sure Ollama is running (ollama serve, or installed as a background service).

Index a repository and ask questions

codecompass ingest --repo https://github.com/kubeedge/ianvs
codecompass ask "What are the core components of the Ianvs architecture?"
codecompass ask "How do I install Ianvs on my machine?" --style concise

Run the benchmark harness

codecompass bench --gold-set data/benchmarks/ianvs_qa.yaml --profile all --repeats 3

Run the web UI

# terminal 1 - API
uvicorn codecompass.api.app:app --port 8000

# terminal 2 - frontend
cd frontend
npm install
npm run dev

Open http://localhost:5173.

CLI reference

Command	Purpose
codecompass ingest --repo <url>	Clone a repository, fetch its issues/PRs, and build a local index.
codecompass ask "<question>" [--style guided|concise]	Ask a question against the most recently built index.
codecompass bench --gold-set <path> [--profile all|<name>] [--repeats N]	Run the benchmark harness, optionally under a named constraint profile.

Tech stack

Language / API	Python 3.12, FastAPI, Typer
Retrieval	Milvus Lite, bge-small-en-v1.5, rank-bm25, a local cross-encoder reranker
Generation	Ollama (qwen2.5:3b-instruct), optional Gemini provider
Frontend	React, TypeScript, Vite, Tailwind CSS
Quality	pytest, mypy --strict, ruff, GitHub Actions CI

Repository structure

src/codecompass/
  ingestion/      clone a repo, fetch issues/PRs
  chunking/       per-content-type chunking (code, prose, issues)
  embedding/      pluggable embedding models
  store/          Milvus Lite vector store
  retrieval/      hybrid search, reranking, the pinned overview
  generation/     prompt construction, LLM providers, streaming
  constraints/    the two-layer edge-constraint simulator
  benchmark/      the gold-set harness and metrics
  api/            the FastAPI HTTP layer
  cli.py          the command-line interface

frontend/         the React web UI
docs/             architecture, benchmarks, raw results, screenshots
data/benchmarks/  the gold question sets
configs/          model, chunking, and constraint-profile configuration
tests/            unit and integration tests

Testing and CI

ruff check .
mypy --strict src
pytest

GitHub Actions runs linting, strict type checking, and the full test suite on every push to main. A separate, manually-triggered workflow validates the constraint simulator against real OS-enforced memory limits on a Linux runner.

Project status

Actively developed as a personal project exploring retrieval-augmented generation, hybrid search, and resource-constrained ("edge") system benchmarking, using kubeedge/ianvs — a CNCF distributed AI benchmarking framework — as the real-world target repository throughout.

License

MIT