Evaluate your RAG retrieval without labeled datasets — a regression-gated quality harness for retrieval ranking, built around a small hybrid retriever. Measure whether a change helps or hurts when you have no labels, no users to A/B against, and no budget for manual annotation.
Building RAG systems is straightforward. Knowing whether a change made retrieval better or worse is hard — especially with a small corpus and no production telemetry. You can't A/B test with one user. You can't afford hand-labeled golden sets. LLM judges are expensive and opaque.
hitgate solves one specific problem: detect retrieval regressions label-free. Like a test suite for ranking quality.
✓ Proves whether retrieval improved or regressed between two runs (regression detection)
✓ Works with zero labeled data — no golden sets required to start
✓ Gates on measurable deltas — reports Hit@K, MRR, per-intent breakdown
✓ Plugs into your retriever — point --retriever at any callable (query, top, scope) → [results]
✗ Does not measure absolute quality — Hit@5=1.0 on a self-indexed corpus is optimistic by design
✗ Not a framework or hosted service — a harness you fork and customize
✗ Not a maintained product — personal tooling shared for the methodology
Comparison to RAGAS / DeepEval / Braintrust: Those tools gate answer quality (faithfulness, relevancy) via labeled golden sets or LLM judges. hitgate gates retrieval ranking label-free. Different layer, complementary — and the only one that requires zero labels to start. Full comparison: docs/COMPARISONS.md.
pip install -e ".[hybrid]" # harness + bundled hybrid retriever
# Index a repo and run the eval gate
RAG_SOURCE_ROOTS="$PWD" python -m ragcore.build
RAG_RERANK_AUTO=off python -m hitgate.run --label demo
# ...or point the gate at YOUR retriever
python -m hitgate.run --retriever myretriever:retrieve --label mineWhat you get: Hit@5, Hit@1, MRR + per-case breakdown showing which queries failed and why.
After pip install hitgate, the following commands are available on $PATH. Each also works as python -m <module> (e.g. python -m hitgate.run --label demo).
rag-build — Build a searchable index from source code and docs.
RAG_SOURCE_ROOTS="/path/to/corpus" rag-build--incremental FILES— Reindex only specific files (e.g.--incremental src/core.py docs/api.md)--no-code— Skip source code; index docs only
rag-query QUERY — Search the indexed corpus.
rag-query "how does pagination work" --top 5 --format json--top K(default 5) — Return top K results--scope TYPE— Filter by source type (e.g.--scope codeor--scope docs)--rerank {on|off|auto}(defaultauto) — Cross-encoder reranking--format {text|json}— Output format
hitgate-run — Run the gate on a golden set and measure Hit@K, MRR.
hitgate-run --dataset eval/golden.jsonl --label baseline--top K(default 5) — Measure up to top K--label NAME(defaultbaseline) — Label for output file--dataset PATH— Golden set (JSONL)--retriever MODULE:FUNC— Custom retriever (omit to use bundled hybrid)--rerank— Force cross-encoder reranking (bundled retriever only)--auto-rerank— Auto-trigger reranking on weak/ambiguous queries--verbose— Show per-case details
hitgate-init [ROOT] — Scaffold a label-free golden set from a corpus.
hitgate-init /path/to/corpus --output eval/golden.jsonl--output PATH(defaulteval/golden.jsonl) — Where to write the golden set--min-confidence {high|medium|low}(defaultmedium) — Heuristic confidence threshold--limit N(default 0, unlimited) — Cap number of candidate cases
hitgate-demo [ROOT] — One-shot demo: build index, mine cases, run gate, print verdict.
hitgate-demo /path/to/corpus --limit 12--limit N(default 12) — Number of golden cases to mine
hitgate-generate — Mine candidate golden cases from a corpus (auto-generated, needs curation).
hitgate-generate --output candidates.jsonl --min-confidence medium--output PATH(defaulthitgate/candidates.jsonl) — Where to write candidates--existing JSONL— Skip files already in an existing golden set--min-confidence {high|medium|low}(defaultmedium) — Confidence threshold--llm— Use an LLM to paraphrase queries (requiresOPENAI_API_KEY)--limit N— Cap total output cases
hitgate-diff BASELINE.json HEAD.json — Diff two eval result JSON files case-by-case.
hitgate-diff eval/baseline.json eval/ci.json--quiet— Summary line only, no per-case detail
hitgate-audit-contamination — Check golden set for data leakage (eval queries appear in corpus).
hitgate-audit-contamination --dataset eval/golden.jsonl--dataset PATH(defaulteval/golden.jsonl) — Golden set to audit
- Dense embeddings (
intfloat/multilingual-e5-small) + BM25 lexical search - Reciprocal Rank Fusion to combine both signals
- Code-aware tokenizer — splits
camelCase/snake_caseso "get user profile" matchesgetUserProfile - Language-aware chunking — Python via AST, TS/JS/Shell via regex, fallback to word count
- Optional reranking — cross-encoder scoped to code queries only, graceful fallback if unavailable
- Run your retriever against a golden set of queries (you write 20–99 JSON cases; this demo scores 99)
- Measure Hit@1/3/5, MRR — does the expected file rank in top K?
- Freeze a baseline when you like the results
- Gate future changes — CI exits 1 if any metric regresses by >5pp
Config via env var (RAG_*); see ragcore/config.py.
| Metric | Meaning |
|---|---|
| Hit@K | % of queries where the expected file appears in top K results |
| Hit@5 | Regression-gated headline metric (allows room for ranking imprecision) |
| Hit@1 | Perfect-ranking metric (strict but useful for identifying misses) |
| MRR | Mean Reciprocal Rank — average of 1/rank across queries |
Self-indexed results are optimistic by construction — point the gate at your own retriever for numbers that mean something.
A retriever is any callable:
retrieve(query: str, top: int, scope: str | None) -> Sequence[Mapping]
# Results ranked best-first; each a Mapping with at least "path" (optionally "start_line")RAG_SOURCE_ROOTS="/path/to/corpus" python -m hitgate.generate \
--output hitgate/candidates.jsonl \
--min-confidence mediumReview and curate to remove vague queries or wrong expected paths.
{"query": "how does pagination work", "expect_path_contains": "api/pagination.py", "expect_scope": "code"}
{"query": "rate limit config", "expect_path_contains": "config/limits.yaml", "expect_scope": "code"}Aim for 20–50 cases mixing identifier lookups and paraphrased queries. Save as any .jsonl.
python -m hitgate.run \
--retriever mypkg.myretriever:retrieve \
--dataset my_golden.jsonl \
--label baseline-v1Outputs hitgate/baseline-v1.json with Hit@1/3/5/MRR + per-case breakdown.
cp hitgate/baseline-v1.json hitgate/baseline.myproject.jsonEdit _note to record conditions: corpus, model, date.
BASELINE_FILE=hitgate/baseline.myproject.json \
python -m hitgate.run --retriever mypkg.myretriever:retrieve --dataset my_golden.jsonl --label ci
bash hitgate/check.sh hitgate/ci.json hitgate/baseline.myproject.json
# Exits 1 if any metric regresses >5ppDrop-in CI: examples/retrieval-gate.yml
python -m hitgate.diff hitgate/baseline-v1.json hitgate/ci.json| Metric | Value |
|---|---|
| Hit@5 (code scope, pure hybrid) | 0.99 |
| Hit@1 | 0.636 |
| MRR | 0.784 |
| Corpus | this repo, self-indexed · 99 cases |
63 of 99 hit at rank 1; misses kept on purpose (no quiet dropping). Why Hit@5 is the gate: as the corpus grows, borderline cases drift rank. Hit@5 tolerates that noise; Hit@1 is for strict ranking. See CONTEXT.md (search "Drift over self-indexed corpus") for the fuller drift/self-indexed discussion.
| Corpus | Language | n | Hit@5 | Hit@1 | MRR |
|---|---|---|---|---|---|
| FastAPI v0.115 | Python | 25 | 1.0 | 0.64 | 0.79 |
| forge-space / mcp-gateway | TypeScript | 20 | 1.0 | 0.70 | 0.821 |
| portfolio / src | React/TS | 15 | 1.0 | 0.60 | 0.778 |
| ai-dev-toolkit / packages/core | Python + TS | 20 | 1.0 | 0.85 | 0.925 |
| homelab / homelab_manager | Python | 20 | 0.950 | 0.85 | 0.900 |
| Lucky / packages/backend | TypeScript | 21 | 0.905 | 0.71 | 0.810 |
| Criativaria / web-app | Next.js/TS | 27 | 0.741 | 0.59 | 0.660 |
Finding: corpus module clarity predicts Hit@1 better than language or size. Clean functional boundaries → 0.85. Same-layer UI components → 0.59. Same tokenizer, zero corpus-specific tuning.
Selection bias: FastAPI is third-party; six of seven are the author's own repos. See docs/SWEEP.md for a broader 63-repo sample.
Full methodology and miss taxonomy: docs/METHODOLOGY.md
Gate retrieval quality in RAG-backed chatbots or assistants. Detect when corpus updates or embedding model swaps regress search. Deployed version of this exact use case: docs/two-channel-fastapi.md (auto-mined Hit@5=1.0 vs hand-labeled=0.92).
Keep search quality stable as your docs grow. No user click data? Gate on retrievability. Ecosystem wrappers for LangChain / LlamaIndex: adapters/README.md.
Find code references and definitions. The hybrid retriever knows camelCase — useful for codebases with identifier-heavy queries. Same gate works for private codebases.
Core indexes code + docs + commits, nothing else. Tool-specific sources (assistant transcripts, code-graphs, other memory stores) plug in as optional adapters — adapters/README.md.
Adopt: The harness (hitgate/) — label-free regression gating and the --retriever interface. Plug in your own retriever.
Skip:
- Not a framework or hosted service (no plugin marketplace, no SaaS)
- Not state-of-the-art retrieval research (pragmatic, single-user, knows its ceiling)
- Not a maintained product (solo operator's personal tool; best-effort, no SLA)
Deliberately omitted: CONTRIBUTING, issue templates, badge walls. DECISIONS.md records what's left out on purpose and when it might reopen.
- Quick Start — run in 10 seconds
- METHODOLOGY.md — how the measurements work and their limits
- COMPARISONS.md — vs RAGAS, DeepEval, Braintrust, and hand-labeling
- ROADMAP.md — candidate experiments (no commitments)
- DECISIONS.md — what's out of scope and why
- CONTEXT.md — full vocabulary and decision history
MIT — see LICENSE.