AIBox Benchmarking

Benchmarks that decide how to build the document Q&A pipeline for AIBox on real-estate PDFs (Swedish + English: annual reports, energy declarations, inspection protocols, bylaws, garden proposals). It answers three practical questions with data:

Which embedding model should we use for retrieval? → bge-m3 (local).
Which PDF parser turns the PDFs into the cleanest text? → MinerU (best), Docling (cheapest).
RAG or full-context (retrieve chunks vs. stuff the whole document into the LLM)? → both work for these small docs; RAG scales and is far cheaper.

The repo contains two independent benchmarks that share the same documents and tooling:

Benchmark	Question	Code	Gold set	Headline result
1. Embedding models	which embedder retrieves best?	`bench/`, `run_retrieval.ipynb`	`eval/gold.jsonl` (236 queries)	`bge-m3` — best recall, free, ~20–200× faster
2. Parsers + RAG vs long-context	which parser, and RAG or full-context?	`scripts/`	`eval/parser_gold.jsonl` (66 queries)	`MinerU` best quality; `Docling` best value

Full parser analysis with plots: results/results_bench.md.

TL;DR recommendations

Embedder: bge-m3 — ties for top quality, highest recall (0.93), runs locally for free, and is dramatically faster than the API models. Use a GPU-hosted qwen3-embed-4b only if you need its slightly higher raw quality.
Parser: MinerU for best quality (best retrieval and best full-context answers, strongest on tables and Swedish). Docling if cost matters (fewest tokens, lowest cost, best hit@1). Marker is the weakest trade-off (highest token cost, middling quality).
RAG vs full-context: for a handful of small docs per property, full-context is simplest and affordable (~80% correct). As the corpus grows, RAG + a good parser is competitive and much cheaper — and there the parser matters more, since retrieval can only surface what the parser put into the markdown.

Benchmark 1 — Embedding models

Setup: 132 chunks across 6 properties · 236 evaluation queries (125 SV / 111 EN) · property-scoped retrieval (each query only searches its own property's chunks, mirroring production) · TOP_K = 10.

Each model only produces embeddings (text → vectors). Retrieval is a single identical procedure for every model — cosine similarity over the vectors, then top-K (bench/retrieval.py); no reranker. So the metric differences isolate embedding quality measured via a fixed retrieval task (the same idea as MTEB), not the model's own ranking.

Composite = 0.6·nDCG@10 + 0.25·Recall@10 + 0.15·MRR@10. Higher = better quality; lower = better latency/cost/storage.

Model	Composite	nDCG@10	Recall@10	MRR@10	Hit@5	p50 (ms)	p95 (ms)	Storage	Index cost
bge-m3 (local)	0.707	0.650	0.932	0.560	0.805	17	34	0.54 MB	$0.00
qwen3-embed-4b	0.709	0.662	0.894	0.587	0.809	729	6811	1.35 MB	$0.0009
gemini-embed-001	0.631	0.581	0.831	0.501	0.742	420	960	1.62 MB	$0.0071
openai-3-large	0.621	0.566	0.835	0.481	0.703	666	898	1.62 MB	$0.0061

(Index cost = one-time cost to embed the 132-chunk corpus via OpenRouter; local models are free. Local latencies are pure compute, API latencies include the network round-trip.)

Takeaways

bge-m3 is the best practical choice — ties for top composite, highest recall (0.93), runs locally for free, ~20–200× faster (p95 34 ms vs 0.9–6.8 s), smallest storage.
qwen3-embed-4b has the highest raw quality but a ~6.8 s p95 through OpenRouter — only viable if self-hosted on a GPU.
openai-3-large / gemini-embed-001 underperform the free local model on this Swedish-heavy corpus while costing more.
All models score lower on English queries (English questions against Swedish docs); the cross-lingual gap is smallest for qwen3-embed-4b.

Run it: open run_retrieval.ipynb (indexes chunks, benchmarks every enabled model, writes results/retrieval.csv). Enable/disable models and tune weights in bench/config.py.

Benchmark 2 — Parsers + RAG vs long-context

Setup: 66 parser-agnostic gold questions across 6 properties — generated from the original PDFs (not from any parser's output, so no parser is favored) and hand-cleaned so each gold answer is a literal string in the source. Embedder = bge-m3, property-scoped, chunk ≈ 1200 chars / 200 overlap. Full-context answer + LLM judge = google/gemini-2.5-flash.

We compare three parsers — Docling, MinerU, Marker — in three scenarios: retrieval recall (S1), RAG end-to-end answer quality (S1b), and full-context answer quality (S2). S1b and S2 share the same metric and judge, so RAG vs full-context is a true head-to-head.

flowchart TD
  PDFs["data/docs/*.pdf"] --> Parsers["Parsers: docling / mineru / marker"]
  Parsers --> MD["results/parsed/&lt;parser&gt;/*.md"]
  MD --> Chunk["chunk + bge-m3 embed"]
  Chunk --> S1["S1: retrieval recall@5"]
  Chunk --> RAGQA["S1b: RAG end-to-end QA (top-k -> LLM -> judge)"]
  MD --> S2["S2: full-context QA (whole doc -> LLM -> judge)"]
  S1 --> Stats["bench/stats.py: bootstrap CIs + McNemar"]
  RAGQA --> Stats
  S2 --> Stats
  S2 --> JV["validate_judge.py: judge vs human kappa"]
  Stats --> Rep["results/*_report.md (with CIs + significance)"]
  JV --> Rep

On the numbers below: with only 66 questions the point estimates carry wide 95% confidence intervals and the parser gaps are not statistically significant (paired McNemar p > 0.25). Treat the ranking as a weak preference, not a definitive result — the per-run reports (results/parser_*_report.md) print the CIs and significance tests.

Scenario 1 — RAG retrieval quality (`recall@5` headline)

Parser	hit@1	hit@3	recall@5	MRR
MinerU	0.652	0.833	0.894	0.751
Docling	0.682	0.788	0.848	0.748
Marker	0.530	0.758	0.818	0.652

Scenario 1 measures only the retrieval ceiling (did a chunk containing the answer reach the top-k); it does not ask the LLM to answer.

Scenario 1b — RAG end-to-end QA (closes the loop)

scripts/run_parser_rag_qa_eval.py feeds only the retrieved top-k chunks to the same answer model and judge as Scenario 2, producing a directly comparable correctness / faithfulness at a fraction of the input tokens.

Parser	correctness	faithfulness	avg input tokens	cost/query
MinerU	0.833	0.924	2,500	$0.00075
Docling	0.803	0.939	1,803	$0.00054
Marker	0.727	0.909	1,415	$0.00042

Headline: end-to-end RAG correctness (0.73–0.83) matches or beats full-context (0.74–0.80, below) at ~4x fewer tokens / ~4x lower cost and with higher faithfulness — so RAG + a good parser is the better default, not just a cheaper fallback. Full write-up in results/parser_rag_qa_report.md.

Scenario 2 — Full-context QA (whole markdown → LLM)

Parser	correctness	faithfulness	avg input tokens	cost/query
MinerU	0.803	0.833	9,601	$0.00288
Marker	0.758	0.758	10,470	$0.00314
Docling	0.742	0.758	8,587	$0.00258

Takeaways

RAG matches full-context at ~4x lower cost (headline). End-to-end RAG correctness (0.73–0.83) equals/beats full-context (0.74–0.80) using ~1.4–2.5k vs ~8.6–10.5k tokens, with higher faithfulness. RAG + a good parser is the better default.
MinerU is the (weak) overall leader, strongest on tables and Swedish; Docling is the value pick (cheapest, best hit@1); Marker trails for its token cost.
Mind the statistics: with n = 66 the parser gaps are not statistically significant (paired McNemar p > 0.14); the recall@5 ranking is stable across chunk sizes (sweep).
Judge sanity check: the LLM judge agrees with an independent reviewer on correctness 87.5% of the time (Cohen's kappa 0.40, "fair") — usable but treat absolutes with caution.
Cleaning the gold (correcting unit/format mismatches) lifted recall@5 from ~0.65–0.71 to 0.82–0.89 — the earlier "tables are hard" story was mostly gold noise, not parser failure.

See results/results_bench.md for the full write-up and plots.

Run it — one command (recommended):

# Reuses already-parsed markdown; runs S1 -> S1b -> S2 -> judge template -> plots.
uv run python -m scripts.run_all --resume          # full evals, resuming API work
uv run python -m scripts.run_all --smoke           # 2 queries/eval, fast sanity check
uv run python -m scripts.run_all --rebuild         # also rebuild gold + re-parse first

Or step by step (one parser at a time keeps memory flat):

# 1) Build the parser-agnostic gold set from the original PDFs (LLM-generated)
uv run python -m scripts.build_parser_gold

# 2) Parse the PDFs with each parser -> results/parsed/<parser>_bench/...
uv run python -m scripts.run_docling                 # Docling
uv run python -m scripts.run_marker                  # Marker
#   MinerU is produced with the MinerU CLI -> results/parsed/mineru_bench/<sc>/<stem>/auto/<stem>.md

# 3) Scenario 1: RAG retrieval eval (offline; HF_HUB_OFFLINE=1 if the model is cached)
uv run python -m scripts.run_parser_rag_eval --parsers docling
uv run python -m scripts.run_parser_rag_eval --parsers mineru
uv run python -m scripts.run_parser_rag_eval --parsers marker

# 3b) Scenario 1b: RAG end-to-end QA (OpenRouter API; comparable to Scenario 2)
uv run python -m scripts.run_parser_rag_qa_eval --resume

# 4) Scenario 2: full-context QA eval (uses the OpenRouter API; --resume to skip done work)
uv run python -m scripts.run_parser_longctx_eval --resume

# 5) Audit any gold answer no parser could retrieve (offline)
uv run python -m scripts.analyze_unretrievable

# 5b) Sensitivity sweep: confirm the ranking holds across chunk size / top-k (offline)
uv run python -m scripts.run_parser_sweep

# 6) Validate the LLM judge against human labels (sample -> fill -> score)
uv run python -m scripts.validate_judge --sample 40        # writes eval/judge_human_labels.csv
uv run python -m scripts.validate_judge --score           # after filling human_* columns

# 7) Regenerate all plots
uv run python -m scripts.make_plots

All Benchmark-2 knobs (chunk size, top-k, models, price, seed) live in one place — bench/config.py — and are stamped into every report header for traceability. Run the unit tests with uv run pytest.

Full parser analysis with plots (click to expand)

Scenario 1 — RAG retrieval quality

Relevance = a retrieved chunk actually contains the gold answer (number-aware match). recall@5 is the headline.

Parser	hit@1	hit@3	recall@5	MRR
MinerU	0.652	0.833	0.894	0.751
Docling	0.682	0.788	0.848	0.748
Marker	0.530	0.758	0.818	0.652

Read: MinerU leads recall@5 and hit@3; Docling is a close 2nd and actually best at hit@1 (when it retrieves the answer it ranks it first most often). Marker trails mainly on hit@1/MRR (it retrieves, but ranks worse). All three clear ~0.82+ recall@5 — retrieval is reliable for these documents.

recall@5 by answer location & language

Parser	prose (text)	table/number	EN	SV
MinerU	0.900	0.885	0.870	0.95
Docling	0.875	0.808	0.848	0.85
Marker	0.825	0.808	0.804	0.85

Read: tables are nearly as retrievable as prose (0.81–0.88). MinerU is best on both axes and clearly strongest on Swedish; Docling and Marker tie on tables, with Docling ahead on prose.

Scenario 1b — RAG end-to-end QA (top-5 chunks → LLM)

Retrieved top-5 chunks are sent to the LLM and graded with the same metric as Scenario 2 — comparable correctness at a fraction of the tokens.

Parser	correctness	faithfulness	avg input tokens	p50 latency	avg cost / query
MinerU	0.833	0.924	2,500	0.69 s	$0.00075
Docling	0.803	0.939	1,803	0.67 s	$0.00054
Marker	0.727	0.909	1,415	0.66 s	$0.00042

Read: RAG correctness (0.73–0.83) equals or beats full-context (0.74–0.80, below) at ~4x fewer tokens and higher faithfulness (0.91–0.94 vs 0.76–0.83).

Scenario 2 — Full-context QA (whole markdown → LLM)

Parser	correctness	faithfulness	avg input tokens	p50 latency	avg cost / query
MinerU	0.803	0.833	9,601	0.79 s	$0.00288
Marker	0.758	0.758	10,470	0.73 s	$0.00314
Docling	0.742	0.758	8,587	0.76 s	$0.00258

correctness by answer location

Parser	prose (text)	table/number
MinerU	0.850	0.731
Marker	0.800	0.692
Docling	0.750	0.731

Read: MinerU is the most correct and faithful overall and best on prose. Docling and MinerU tie on tables (0.73) — Docling does so while using the fewest tokens / lowest cost, its main advantage; but on prose and overall correctness it trails both others.

RAG vs Full-context — head to head

Both scenarios measure final answer correctness with the same metric, so this is a direct comparison.

Aspect	RAG end-to-end (Scenario 1b)	Full-context (Scenario 2)
What it measures	final answer correct (judge)	final answer correct (judge)
Best correctness	MinerU 0.833	MinerU 0.803
Correctness range	0.73–0.83	0.74–0.80
Faithfulness range	0.91–0.94	0.76–0.83
Tokens per question	1.4k–2.5k (top-5 chunks)	8.6k–10.5k (whole doc)
Cost per question	~$0.0004–0.0008	~$0.0026–0.0031
Scales to large corpora	yes	no (context limit + cost grows with doc size)

Verdict: the assumption that full-context is needed for correctness does not hold here — end-to-end RAG matches or beats full-context correctness at ~4x lower cost and with higher faithfulness, because the reader sees only relevant chunks instead of a whole noisy document. RAG + a good parser is the better default for AIBox, with full-context as a simple fallback for a single small document. In RAG the parser matters more, because retrieval can only surface what the parser put into the markdown. (All parser-vs-parser gaps remain within noise — see the limitations note.)

The trade-off is clearest as correctness vs. token cost — up-and-to-the-left is better (more correct, fewer tokens). MinerU sits highest; Docling is left-most (cheapest); Marker is pushed right (most tokens) without a quality payoff:

Which parser is better?

Rank	Parser	Why
1	MinerU	Best retrieval (recall@5 0.89) and best full-context quality (0.80 correct / 0.83 faithful). Strongest on tables and Swedish. Mid token cost.
2	Docling	Close 2nd in retrieval (recall@5 0.85, best hit@1), cheapest (fewest tokens, lowest cost), and ties MinerU on tables (0.73) in full-context. Trails on prose / overall full-context correctness (0.74). Best value pick.
3	Marker	Competitive recall (0.82) but lowest hit@1/MRR and the highest token cost (10.5k) for middling full-context quality.

Bottom line: default to MinerU for quality on both retrieval and full-context. If cost is the priority, Docling is the cheaper alternative (fewest tokens, best hit@1, matches MinerU on tables) at a small hit to overall full-context correctness. Marker isn't justified given its token cost.

Operational plots

Full-context operational cost (tokens) and latency:

Threats to validity / limitations

Read the Benchmark-2 parser ranking as a weak, directional preference, not a settled result. Known limitations:

Small sample (n = 66). Per-parser gaps fall inside overlapping 95% confidence intervals and are not statistically significant (paired McNemar p > 0.25 for every pair on recall@5). The per-run reports print these CIs and tests.
Single domain. Six Swedish real-estate properties. Results may not transfer to other document types or languages.
Judge == answer model. Both default to google/gemini-2.5-flash, which risks self-grading bias. scripts/validate_judge.py quantifies judge-vs-human agreement (Cohen's kappa); treat the correctness/faithfulness numbers as provisional until that kappa is computed on a human-labeled sample.
Single retrieval config. One embedder (bge-m3), fixed chunking, no reranker by default. The chunk-size sweep (scripts/run_parser_sweep.py, results/parser_sweep_report.md) shows the recall@5 winner (MinerU) is stable across chunk sizes 800/1200/1600, but embedder and reranker choices are not yet swept.
String-containment relevance (S1). A chunk "contains the answer" via number-aware substring match; paraphrased or reformatted answers can read as misses. Scenario 1b (end-to-end QA) mitigates this by actually asking the LLM to answer.

Setup

This project uses uv for package and environment management.

uv sync                       # create .venv and install dependencies
echo "OPENROUTER_API_KEY=sk-..." > .env   # needed for gold generation + long-context eval

The embedding benchmark and Scenario-1 retrieval run fully locally (bge-m3 is cached on first use). Gold generation, Scenario 1b (RAG QA), and the Scenario-2 long-context eval call the OpenRouter API.

Repository structure

bench/                         # Benchmark 1 — embedding-model library
  prepare_docs.py              #   parse + clean + chunk PDFs -> data/chunks.json
  pdf_parser.py / chunker.py   #   PyMuPDF extraction; page-aware paragraph chunking
  doc_registry.py              #   property short codes + document-type classifier
  embedders.py / rerankers.py  #   local (BGE) + API (OpenRouter) embedders/rerankers
  retrieval.py                 #   L2-normalized cosine retrieval + MD5-hashed .npz cache
  metrics_ir.py / opsmeter.py  #   ranx IR metrics; latency/storage/cost meters
  stats.py                     #   bootstrap CIs, McNemar test, Cohen's kappa (Benchmark 2)
  build_dataset.py / clean_gold.py  # LLM gold-set generation + cleaning
  config.py                    #   paths, model specs, metric weights, Benchmark-2 eval knobs
run_retrieval.ipynb            # Benchmark 1 — runner notebook

scripts/                       # Benchmark 2 — parser + RAG-vs-long-context
  build_parser_gold.py         #   parser-agnostic gold from original PDFs
  run_docling.py / run_marker.py    # parse PDFs with each parser
  run_parser_rag_eval.py       #   Scenario 1: RAG retrieval recall eval
  run_parser_rag_qa_eval.py    #   Scenario 1b: RAG end-to-end QA (comparable to S2)
  run_parser_longctx_eval.py   #   Scenario 2: full-context QA eval (LLM-as-judge)
  validate_judge.py            #   sample + score judge-vs-human agreement (Cohen's kappa)
  analyze_unretrievable.py     #   diagnose/clean gold answers no parser retrieves
  make_plots.py                #   generate comparison plots
  run_all.py                   #   one-command pipeline (--smoke / --resume / --rebuild)
  run_docling_bench.py / run_page_eval.py / run_llm_eval.py  # earlier eval iterations (kept)

tests/                         # pytest unit tests (chunking, normalize, stats) — no API
  test_chunking.py

data/
  docs/<Address>/*.pdf         # raw property PDFs (input)
  chunks.json                  # chunks for Benchmark 1
  parser_bench/                # per-parser chunk sets for early parser iterations
eval/
  gold.jsonl                   # Benchmark 1 gold (236 queries)
  parser_gold.jsonl            # Benchmark 2 gold (66 clean, parser-agnostic queries)
  judge_human_labels.csv       # judge-validation template (human fills 0/1 labels)
results/
  retrieval.csv                # Benchmark 1 leaderboard
  parser_rag_*.{csv,md}        # Scenario 1 outputs (recall) + Scenario 1b (RAG QA)
  parser_longctx_*.{csv,md}    # Scenario 2 outputs
  parser_sweep.{csv,md}        # chunk-size/top-k sensitivity sweep
  judge_validation_report.md   # judge-vs-human agreement + kappa
  parsed/<parser>_bench/...    # parsed markdown per parser
  plots/                       # all charts
  results_bench.md             # Benchmark 2 full analysis

Metrics glossary

Retrieval quality (Information Retrieval metrics, bench/metrics_ir.py via ranx)

Hit Rate@K — fraction of queries with at least one relevant doc in the top-K. Good proxy for "did the search bar surface something useful on page one".
MRR@K (Mean Reciprocal Rank) — average of 1/rank of the first relevant result; rewards putting the best answer at the very top.
nDCG@K — overall ranking quality with a logarithmic position discount; the industry standard when the ordering of multiple relevant docs matters.
Recall@K — fraction of all relevant docs retrieved in the top-K; matters when missing any relevant doc is costly.

For the parser benchmark, relevance = a retrieved chunk literally contains the gold answer (number-aware match), so recall@5 measures "did the parser+retriever surface the answer".

Answer quality (Scenario 2, LLM-as-judge)

correctness — share of answers matching the gold fact (formatting/unit/language allowed).
faithfulness — share of answers grounded in the document (no hallucination).

Operational (bench/opsmeter.py)

Latency p50 / p95 — median and tail time to embed a query (warmup queries excluded).
Storage — dimensions × 4 bytes × chunks for fp32 vectors.
Index / query cost — USD from provider per-token pricing (long-context uses the real gemini-2.5-flash input rate, $0.30 / 1M).

Name		Name	Last commit message	Last commit date
Latest commit History 15 Commits
bench		bench
data		data
eval		eval
prompts		prompts
results		results
scripts		scripts
tests		tests
.gitignore		.gitignore
.python-version		.python-version
DocString_openDTlab.ipynb		DocString_openDTlab.ipynb
README.md		README.md
doclingMiner.md		doclingMiner.md
example_chunking.md		example_chunking.md
pyproject.toml		pyproject.toml
run_retrieval.ipynb		run_retrieval.ipynb
uv.lock		uv.lock

Navigation Menu

Repository files navigation

AIBox Benchmarking

TL;DR recommendations

Benchmark 1 — Embedding models

Benchmark 2 — Parsers + RAG vs long-context

Scenario 1 — RAG retrieval quality (`recall@5` headline)

Scenario 1b — RAG end-to-end QA (closes the loop)

Scenario 2 — Full-context QA (whole markdown → LLM)

Scenario 1 — RAG retrieval quality

recall@5 by answer location & language

Scenario 1b — RAG end-to-end QA (top-5 chunks → LLM)

Scenario 2 — Full-context QA (whole markdown → LLM)

correctness by answer location

RAG vs Full-context — head to head

Which parser is better?

Operational plots

Threats to validity / limitations

Setup

Repository structure

Metrics glossary

About

Releases

Packages

Contributors

Languages

Folders and files

Latest commit

History

Repository files navigation

AIBox Benchmarking

TL;DR recommendations

Benchmark 1 — Embedding models

Benchmark 2 — Parsers + RAG vs long-context

Scenario 1 — RAG retrieval quality (recall@5 headline)

Scenario 1b — RAG end-to-end QA (closes the loop)

Scenario 2 — Full-context QA (whole markdown → LLM)

Scenario 1 — RAG retrieval quality

recall@5 by answer location & language

Scenario 1b — RAG end-to-end QA (top-5 chunks → LLM)

Scenario 2 — Full-context QA (whole markdown → LLM)

correctness by answer location

RAG vs Full-context — head to head

Which parser is better?

Operational plots

Threats to validity / limitations

Setup

Repository structure

Metrics glossary

About

Resources

Stars

Watchers

Forks

Releases

Packages

Contributors

Languages

Scenario 1 — RAG retrieval quality (`recall@5` headline)