feat(search): hybrid retrieval fusing full-corpus BM25 with dense ranking

[scarletkc]

Motivation

Roadmap P0: pure semantic search is weak on exact identifiers, and the existing --rerank bm25 only reorders the dense top candidates (clamp(top*2, 20, 150)) — lexical-only matches that dense retrieval misses can never surface. True hybrid scores the full chunk set on both paths before fusing.

Design

• Persisted term statistics (cache schema v6 → v7): bm25_doc (chunk_id → token_count) + bm25_posting (index_id, term, chunk_id → tf, WITHOUT ROWID inverted index). Query cost is O(query terms × matching postings), not O(corpus). Incremental updates ride the existing FK cascades (changed files rewrite postings in-transaction; removed files cascade; touched files are provable no-ops since tokens derive from the unchanged label). Corpus stats (N, avgdl) are computed per query by aggregate — no stats table to drift.
• Tokenization: BERT sub-tokens PLUS whole identifier tokens (load_index_vectors is kept intact alongside load/index/vectors), so exact-symbol queries get a high-IDF precise match while natural-language queries match through sub-tokens.
• Fusion: weighted reciprocal rank fusion over the full chunk set — 0.7·(k+1)/(k+dense_rank) + 0.3·(k+1)/(k+bm25_rank), k=60, scores normalized to [0,1] (1.0 = ranked first on both paths). The 0.7/0.3 split mirrors the legacy reranker''s fusion weights; equal-weight RRF measurably let full-corpus BM25 noise displace correct dense top hits on common-word queries.
• Fallbacks (deliberate, user-visible): dense-only ranking with reranker: null when the query yields no tokens or the index has no BM25 data. --no-cache/in-memory searches score lexically from transient chunk term data.
• Surface: vexor config --rerank hybrid (fifth SUPPORTED_RERANKERS value), init wizard option 5, doctor check (warns when tokenizers is missing → regex fallback degrades CJK), MCP reranker response field can now be "hybrid" (no tool-schema change; injectable via VEXOR_CONFIG_JSON), bundled skill tip for exact-identifier lookups.

DEFAULT_RERANK stays "off" — see evaluation below; the default flip remains gated per the roadmap.

Evaluation (scripts/eval_hybrid.py, 30 QA pairs over this repo, top=10)

Local provider (intfloat/multilingual-e5-small):

Rerank	MRR@10	Hit@1	Hit@5
off	0.581	0.467	0.767
bm25	0.575	0.400	0.867
hybrid	0.607	0.533	0.733

Custom provider (BAAI/bge-m3, OpenAI-compatible):

Rerank	MRR@10	Hit@1	Hit@5
off	0.712	0.600	0.900
bm25	0.639	0.500	0.900
hybrid	0.671	0.533	0.900

Honest read: hybrid beats dense-only with the weaker local embedding model and structurally guarantees lexical-only matches are retrievable (unit-tested: a chunk dense-ranked last but lexically exact surfaces in hybrid top-k while bm25 rerank cannot see it), but a strong dense model still wins on aggregate MRR for this query mix. Two fusion iterations are already included (whole-token tokenization: hybrid MRR 0.531→0.671 on bge-m3; dense-weighted RRF). Further tuning belongs to the roadmap''s evaluation work; hybrid ships opt-in.

Compatibility (reviewers please check)

• Cache schema v7: older index caches are invalidated and rebuilt automatically on next use; the shared embedding cache is reused, so migration costs extraction only (no re-embedding). Called out in docs/configuration.md.
• Score semantics under hybrid: normalized rank-fusion score in [0,1], not cosine (documented in docs/mcp.md; no score thresholds exist in the pipeline, and existing rerankers already overwrite scores).
• Config schema: rerank accepts new value "hybrid". No Python API signature changes. Supervisor fix during review: cache._relative_path now returns posix form to match the rel_paths index_service stores — without this, compare_snapshot would mark nested paths permanently stale on Windows (regression-tested).
• Roadmap: P0 hybrid item marked shipped-behind-flag; added a P2 item for whitelisted project-level config (behavior fields only, credentials/endpoints rejected).
• AGENTS.md: docs-sync guideline now explicitly lists the bundled skill and docs/mcp.md.

Tests

• python -m pytest tests -q (.venv, offline): 515 passed (19 new: bm25 math incl. hand-computed BM25/weighted-RRF values, cache postings persistence/incremental/cascade/fallback, hybrid ranking incl. the lexical-only-outside-clamp case and exact-identifier-vs-subtoken-flood case, config/init/doctor/MCP/CLI surface, nested-path staleness regression)
• Real e2e (AGENTS.md): VEXOR_CONFIG_JSON={"rerank":"hybrid"} + vexor search "_apply_bm25_rerank" --path . with the configured custom provider (BAAI/bge-m3) — correct file at Add Claude Skills template for Vexor repository #1, Reranker: hybrid label shown. Both eval tables above produced with real embeddings (local + custom).

Implementation by Codex CLI (gpt-5.6-sol, high reasoning) under Claude supervision; two supervisor-directed fix iterations (tokenization, fusion weights) driven by benchmark results.

🤖 Generated with Claude Code