Add BEIR benchmark evaluation script

[vignesh2027]

Task

Add a benchmark evaluation script that runs VORTEXRAG against the standard BEIR benchmark suite.

What's needed:

• Script at benchmarks/eval_beir.py
• Load BEIR datasets via the beir library
• Run VORTEXRAG on each dataset
• Output a results table: dataset → NDCG@10, Recall@100, MAP

Relevant BEIR datasets: MSMARCO, NQ, HotpotQA, FiQA, SCIDOCS, FEVER, SciFact

Skills needed: Python, familiarity with BEIR

Reference: https://arxiv.org/abs/2104.08663

Comment below if you'd like this assigned to you!

[vignesh2027]

Reference implementation is now merged! 🎉

What's in commit 8f5ddff:

benchmarks/eval_beir.py

Run it:

# Install BEIR first
pip install beir

# Evaluate on default 9 datasets (downloads automatically)
python benchmarks/eval_beir.py

# Specific datasets + save CSV
python benchmarks/eval_beir.py --datasets nq scifact bioasq --output results/beir.csv

Output format:

Dataset                Domain         NDCG@10    R@100      MAP    ms/q
---------------------------------------------------------------------------
nq                     general         0.xxxx    0.xxxx    0.xxxx    xx.x
scifact                biomedical      0.xxxx    0.xxxx    0.xxxx    xx.x

If you want to extend this:

• Add more BEIR datasets (Touché, DBPedia, TREC-COVID)
• Add comparison against BM25 baseline
• Add latency profiling per layer

If you run it on any dataset and get results, please share them here — would love to add benchmark numbers to the README!

[vignesh2027]

This is still open for anyone who wants to contribute! Running VORTEXRAG on BEIR would be a meaningful benchmark — the script is already there at benchmarks/eval_beir.py.

What would make this even better:

• Async evaluation (run multiple datasets concurrently — inspired by the async work in Add LangChain integration — VortexRAGRetriever #4)
• Comparison table against BM25 baseline in the output
• Results posted back here so we can add real numbers to the README

If you'd like to take this on, comment and I'll assign it.