feat(search): hybrid retrieval fusing full-corpus BM25 with dense ranking

AGENTS.md

@@ -31,7 +31,7 @@ Let errors surface instead of swallowing them: degraded paths (a missing optiona
 Tests live in `test_<subject>.py` under `tests/unit/` or `tests/integration/`; integration suites drive the Typer CLI end to end, including porcelain output. Use fixtures/stubs instead of real APIs and keep provider and network interactions mocked so tests stay offline. Assert behavior or structured output rather than brittle console formatting. Pair each behavior change with happy-path and failure coverage, especially for invalid modes, empty queries, malformed config, stale or missing indexes, optional extras such as `flashrank`, and platform-specific shell behavior. The offline suite cannot catch real provider regressions: before opening a PR that touches providers, indexing/search, or the MCP server, run a real end-to-end check that exercises your change (for example, `python -m vexor index --path . --mode code` plus a search) using your configured API credentials, or the local provider if you have none, and note the result in the PR.
 
 ## Commit & Pull Request Guidelines
-Commit subjects and PR titles follow [Conventional Commits](https://www.conventionalcommits.org): `type(scope): description`, e.g. `feat(mcp): add stdio server` or `fix(cache): handle locked index database`. Allowed types are `feat`, `fix`, `docs`, `style`, `refactor`, `perf`, `test`, `build`, `ci`, `chore`, and `revert`; the scope is optional but encouraged and should name the touched area (`mcp`, `cache`, `cli`, `gui`, `providers`, ...). Use the imperative mood, keep subjects under ~72 characters, and mark breaking changes with `!` (for example, `feat(config)!: drop legacy keys`). CI enforces the format on PR titles (`.github/workflows/conventional-commits.yml`); squash merges inherit the PR title as the commit subject, so a compliant title keeps `main` history clean. Name branches `type/short-slug` aligned with the commit type (e.g. `feat/hybrid-search`, `docs/roadmap-recover`); do all work on a branch and land it through a PR rather than committing directly to `main`. PRs should explain motivation, list the commands/tests exercised, link issues, and include screenshots for GUI changes or pasted terminal output for CLI output changes. Call out config-schema, cache-schema, Python API, bundled skill, or provider/reranker changes so reviewers can check compatibility. When behavior or contributor workflow changes, update whichever of `README.md`, docs, plugin metadata, and this guide the change affects.
+Commit subjects and PR titles follow [Conventional Commits](https://www.conventionalcommits.org): `type(scope): description`, e.g. `feat(mcp): add stdio server` or `fix(cache): handle locked index database`. Allowed types are `feat`, `fix`, `docs`, `style`, `refactor`, `perf`, `test`, `build`, `ci`, `chore`, and `revert`; the scope is optional but encouraged and should name the touched area (`mcp`, `cache`, `cli`, `gui`, `providers`, ...). Use the imperative mood, keep subjects under ~72 characters, and mark breaking changes with `!` (for example, `feat(config)!: drop legacy keys`). CI enforces the format on PR titles (`.github/workflows/conventional-commits.yml`); squash merges inherit the PR title as the commit subject, so a compliant title keeps `main` history clean. Name branches `type/short-slug` aligned with the commit type (e.g. `feat/hybrid-search`, `docs/roadmap-recover`); do all work on a branch and land it through a PR rather than committing directly to `main`. PRs should explain motivation, list the commands/tests exercised, link issues, and include screenshots for GUI changes or pasted terminal output for CLI output changes. Call out config-schema, cache-schema, Python API, bundled skill, or provider/reranker changes so reviewers can check compatibility. When behavior or contributor workflow changes, update whichever of `README.md`, docs, plugin metadata, and this guide the change affects — the bundled skill (`plugins/vexor/skills/vexor-cli/SKILL.md`) is the easiest to forget.
 
 ## Security, Configuration & Maintenance
 Never commit API keys, private proxy endpoints, or local credentials; use `vexor config --set-api-key`, provider env vars, or ignored `.env` files. Persistent data lives under `~/.vexor`: config/cache data in the main directory, FlashRank assets under `~/.vexor/flashrank`, and local embedding models under `~/.vexor/models`. Sanitize filesystem paths, extension filters, and exclude patterns before using them. Treat extracted document text, embedding payloads, and remote rerank/provider responses as untrusted input before writing to disk or surfacing them elsewhere. Keep docs, bundled skills, and release metadata in sync when commands, packaging, or setup flows change.

README.md

@@ -35,6 +35,8 @@ When you remember what a file *does* but forget its name or location, Vexor find
 
 Designed for both humans and AI coding assistants, enabling semantic file discovery in autonomous agent workflows.
 
+Optional hybrid retrieval (`vexor config --rerank hybrid`) fuses full-corpus BM25 with dense search, so exact identifiers can surface even when dense retrieval misses them.
+
 Per-project `.vexorignore` files give you full gitignore-style control over what gets indexed, and you can opt in to per-project indexes by creating a `.vexor/` directory or running `vexor index --local`.
 
 ## Install

docs/api/python.md

@@ -140,7 +140,7 @@ The `config` payload (dict/JSON) supports:
 - `embedding_dimensions`: integer or null
 - `auto_index`: boolean
 - `local_cuda`: boolean (local provider only)
-- `rerank`: `off`, `bm25`, `flashrank`, `remote`
+- `rerank`: `off`, `bm25`, `flashrank`, `remote`, `hybrid`
 - `flashrank_model`: string or null
 - `remote_rerank`: object with `base_url`, `api_key`, `model`
 

docs/cli.md

@@ -39,6 +39,10 @@
 | `--no-cache` | In-memory only; do not read/write index cache |
 | `--local` | With `index`, create and use `<path>/.vexor/index.db` |
 
+Rerank values are `off`, `bm25`, `flashrank`, `remote`, and `hybrid`.
+`hybrid` fuses full-corpus BM25 and dense rankings, allowing exact lexical
+matches outside the dense candidate window to be retrieved.
+
 Porcelain output fields: `rank`, `similarity`, `path`, `chunk_index`,
 `start_line`, `end_line`, `preview` (line fields are `-` when unavailable).
 

docs/configuration.md

@@ -30,6 +30,7 @@ vexor config --set-update-check false       # disable the daily update notice (d
 vexor config --rerank bm25                  # optional BM25 rerank for top-k results
 vexor config --rerank flashrank             # FlashRank rerank (requires optional extra)
 vexor config --rerank remote                # remote rerank via HTTP endpoint
+vexor config --rerank hybrid                # full-corpus BM25 + dense rank fusion
 vexor config --set-flashrank-model ms-marco-MultiBERT-L-12  # multilingual model
 vexor config --set-flashrank-model          # reset FlashRank model to default
 vexor config --clear-flashrank              # remove cached FlashRank models
@@ -65,10 +66,17 @@ CI), merged over `~/.vexor/config.json`. Credential fields inside
 Rerank reorders the semantic results with a secondary ranker. Candidate
 sizing uses `clamp(int(--top * 2), 20, 150)`.
 
+`hybrid` is different from the candidate rerankers: it scores BM25 over every
+indexed chunk and fuses that full lexical ranking with the full dense ranking
+using reciprocal rank fusion. Hybrid result scores are normalized rank-fusion
+scores in `[0, 1]`, not cosine similarities. A score of `1.0` means a chunk was
+ranked first by both retrieval paths.
+
 Recommended defaults:
 
 - Keep `off` unless you want extra precision.
 - Use `bm25` for lightweight lexical boosts; it is fast and lightweight.
+- Use `hybrid` when exact identifiers and lexical-only matches matter.
 - BM25 uses a multilingual tokenizer (Bert pre-tokenizer), so it can handle CJK better.
 - Use `flashrank` for stronger reranking (requires `pip install "vexor[flashrank]"` and
   downloads a model to `~/.vexor/flashrank`).
@@ -77,6 +85,10 @@ Recommended defaults:
 - For Chinese or multi-language content, set `--set-flashrank-model ms-marco-MultiBERT-L-12`.
 - If unset, FlashRank defaults to `ms-marco-TinyBERT-L-2-v2`.
 
+This release raises the index cache schema version. Older index caches are
+invalidated and rebuilt automatically; cached embeddings are shared and reused,
+so the migration does not require re-embedding unchanged content.
+
 ## Providers: Remote vs Local
 
 Vexor supports both remote API providers (`openai`, `gemini`, `voyageai`,

docs/mcp.md

@@ -45,6 +45,10 @@ codex mcp add vexor -- vexor mcp
 }
 ```
 
+The `reranker` response field can be `"hybrid"`. In that case, each result's
+`score` is a normalized reciprocal-rank-fusion score in `[0, 1]`, rather than a
+cosine similarity.
+
 If `vexor` is not on the client's PATH, use an absolute command path, or
 `python` with `"args": ["-m", "vexor", "mcp"]`.
 

docs/roadmap.md

@@ -10,7 +10,8 @@ never leaving the machine.
 
 ## P0 — Agent-first distribution
 
-- Hybrid retrieval as a first-class path: fuse BM25 and dense scores
+- **Shipped behind `--rerank hybrid`:** hybrid retrieval as a first-class path
+  fuses BM25 and dense scores
   (e.g. reciprocal rank fusion) during search instead of offering BM25
   only as an opt-in reranker. Dependencies (`rank-bm25`, `tokenizers`)
   are already present. Pure semantic search is weak on exact identifiers;
@@ -22,7 +23,7 @@ never leaving the machine.
   - Full-corpus BM25 needs term statistics persisted alongside the index
     cache — rebuilding them per query is O(corpus). Design this together
     with the `vectors.npy`/memmap work in P1.
-  - Flip the ranking default only after the evaluation benchmark (next
+  - The default flip remains pending. Flip the ranking default only after the evaluation benchmark (next
     item) confirms hybrid beats dense-only, and call the change out in
     release notes since result ordering shifts for existing users.
 - Publish an evaluation: token cost + answer quality of agent+Vexor vs
@@ -58,6 +59,15 @@ never leaving the machine.
 - Add AST-aware `code` mode chunking for Go and Rust (tree-sitter support).
 - Support `.vexorignore` for per-project ignore rules.
 - Project-level local cache (per-folder cache root override).
+- Project-level config (`<project>/.vexor/config.json`) for behavior-only
+  settings such as mode preferences, rerank, extensions, and exclude
+  patterns. Security constraint: repo files are attacker-controllable
+  input, so the loader must whitelist behavior fields and reject
+  credentials and endpoints (`api_key`, `base_url`, `remote_rerank`) with
+  an explicit error — stricter than the `VEXOR_CONFIG_JSON` env override,
+  which permits `base_url`. Reuse the guarded `config_from_json(base=...)`
+  merge; `vexor config --show` and `vexor doctor` should display each
+  field's origin (global vs project).
 - Additional embedding providers (Azure).
 - Evaluate an optional LLM reranker that reads a bounded set of retrieved
   candidates and judges their relevance to the query. Keep dense/BM25

plugins/vexor/skills/vexor-cli/SKILL.md

@@ -47,6 +47,7 @@ vexor "<QUERY>" [--path <ROOT>] [--mode <MODE>] [--ext .py,.md] [--exclude-patte
 
 ## Troubleshooting
 
+- Searching for an exact identifier (function/class/constant name) with weak results: suggest `vexor config --rerank hybrid` once — it fuses exact lexical matching with semantic search.
 - Need ignored or hidden files: add `--include-hidden` and/or `--no-respect-gitignore`.
 - Scriptable output: use `--format porcelain` (TSV) or `--format porcelain-z` (NUL-delimited).
 - Get detailed help: `vexor search --help`.

scripts/eval_hybrid.py

@@ -0,0 +1,154 @@
+"""Evaluate dense, legacy BM25 reranking, and full-corpus hybrid retrieval."""
+
+from __future__ import annotations
+
+import argparse
+import json
+from pathlib import Path
+from typing import Sequence
+
+from vexor import api
+from vexor.config import DEFAULT_LOCAL_MODEL, load_config
+
+ARMS = ("off", "bm25", "hybrid")
+
+
+def _parse_args() -> argparse.Namespace:
+    parser = argparse.ArgumentParser(description=__doc__)
+    parser.add_argument("--path", type=Path, default=Path("."), help="Repository root")
+    parser.add_argument("--mode", default="auto")
+    parser.add_argument("--top", type=int, default=10)
+    parser.add_argument("--provider")
+    parser.add_argument("--model")
+    parser.add_argument(
+        "--queries",
+        type=Path,
+        default=Path(__file__).with_name("eval_queries.jsonl"),
+    )
+    parser.add_argument("--json", action="store_true", dest="as_json")
+    parser.add_argument("--verbose", action="store_true")
+    return parser.parse_args()
+
+
+def _load_queries(path: Path) -> list[dict[str, str]]:
+    queries: list[dict[str, str]] = []
+    with path.open("r", encoding="utf-8") as handle:
+        for line_number, line in enumerate(handle, start=1):
+            if not line.strip():
+                continue
+            item = json.loads(line)
+            if not isinstance(item.get("query"), str) or not isinstance(
+                item.get("expected"), str
+            ):
+                raise ValueError(f"Invalid query record on line {line_number}")
+            queries.append(item)
+    return queries
+
+
+def _relative_result_paths(response, root: Path) -> list[str]:
+    paths: list[str] = []
+    for result in response.results:
+        try:
+            relative = result.path.resolve().relative_to(root)
+        except ValueError:
+            relative = result.path
+        paths.append(relative.as_posix())
+    return paths
+
+
+def _metrics(details: Sequence[dict[str, object]]) -> dict[str, float]:
+    count = len(details)
+    if not count:
+        return {"mrr_at_10": 0.0, "hit_at_1": 0.0, "hit_at_5": 0.0}
+    reciprocal_sum = 0.0
+    hit_one = 0
+    hit_five = 0
+    for detail in details:
+        rank = detail["rank"]
+        if isinstance(rank, int) and rank <= 10:
+            reciprocal_sum += 1.0 / rank
+        hit_one += int(rank == 1)
+        hit_five += int(isinstance(rank, int) and rank <= 5)
+    return {
+        "mrr_at_10": reciprocal_sum / count,
+        "hit_at_1": hit_one / count,
+        "hit_at_5": hit_five / count,
+    }
+
+
+def main() -> int:
+    args = _parse_args()
+    root = args.path.expanduser().resolve()
+    config = load_config()
+    provider = args.provider or config.provider
+    model = args.model or config.model
+    if provider == "local" and args.model is None:
+        model = DEFAULT_LOCAL_MODEL
+    common = {
+        "path": root,
+        "mode": args.mode,
+        "provider": provider,
+        "model": model,
+        "use_config": True,
+    }
+    api.index(**common)
+
+    queries = _load_queries(args.queries)
+    by_arm: dict[str, list[dict[str, object]]] = {arm: [] for arm in ARMS}
+    for item in queries:
+        for arm in ARMS:
+            response = api.search(
+                item["query"],
+                top=args.top,
+                auto_index=False,
+                config={"rerank": arm},
+                **common,
+            )
+            returned = _relative_result_paths(response, root)
+            try:
+                rank: int | None = returned.index(item["expected"]) + 1
+            except ValueError:
+                rank = None
+            by_arm[arm].append(
+                {
+                    "query": item["query"],
+                    "expected": item["expected"],
+                    "rank": rank,
+                    "results": returned,
+                }
+            )
+
+    output = {
+        "query_count": len(queries),
+        "top": args.top,
+        "arms": {
+            arm: {"metrics": _metrics(details), "queries": details}
+            for arm, details in by_arm.items()
+        },
+    }
+    if args.as_json:
+        print(json.dumps(output, indent=2))
+        return 0
+
+    print("| Rerank | MRR@10 | Hit@1 | Hit@5 |")
+    print("|---|---:|---:|---:|")
+    for arm in ARMS:
+        metrics = output["arms"][arm]["metrics"]
+        print(
+            f"| {arm} | {metrics['mrr_at_10']:.3f} | "
+            f"{metrics['hit_at_1']:.3f} | {metrics['hit_at_5']:.3f} |"
+        )
+    if args.verbose:
+        for item in queries:
+            ranks = []
+            for arm in ARMS:
+                detail = next(
+                    row for row in by_arm[arm] if row["query"] == item["query"]
+                )
+                ranks.append(f"{arm}={detail['rank'] or '-'}")
+            print(f"- {item['query']} -> {item['expected']} ({', '.join(ranks)})")
+    return 0
+
+
+if __name__ == "__main__":
+    raise SystemExit(main())

scripts/eval_queries.jsonl

@@ -0,0 +1,30 @@
+{"query": "_apply_bm25_rerank", "expected": "vexor/services/search_service.py"}
+{"query": "load_index_vectors", "expected": "vexor/cache.py"}
+{"query": "EMBED_CACHE_TTL_DAYS", "expected": "vexor/cache.py"}
+{"query": "compare_snapshot", "expected": "vexor/cache.py"}
+{"query": "BertPreTokenizer", "expected": "vexor/bm25.py"}
+{"query": "_resolve_rerank_candidates", "expected": "vexor/services/search_service.py"}
+{"query": "VEXOR_CONFIG_JSON", "expected": "vexor/config.py"}
+{"query": "backfill_chunk_lines", "expected": "vexor/cache.py"}
+{"query": "build_exclude_spec", "expected": "vexor/utils.py"}
+{"query": "DEFAULT_FLASHRANK_MODEL", "expected": "vexor/config.py"}
+{"query": "where is the sqlite schema created", "expected": "vexor/cache.py"}
+{"query": "how does incremental indexing decide what to re-embed", "expected": "vexor/services/index_service.py"}
+{"query": "render search results as a table", "expected": "vexor/cli.py"}
+{"query": "load cached query embedding", "expected": "vexor/cache.py"}
+{"query": "extract keywords from document content", "expected": "vexor/services/keyword_service.py"}
+{"query": "validate embedding dimensions for a model", "expected": "vexor/config.py"}
+{"query": "filter indexed results by extension", "expected": "vexor/services/search_service.py"}
+{"query": "detect stale index cache", "expected": "vexor/services/cache_service.py"}
+{"query": "configure an OpenAI compatible provider", "expected": "vexor/providers/openai.py"}
+{"query": "start the MCP stdio server", "expected": "vexor/services/mcp_service.py"}
+{"query": "mcp search tool schema", "expected": "vexor/services/mcp_service.py"}
+{"query": "flashrank model cache directory", "expected": "vexor/config.py"}
+{"query": "plugins/vexor skill installer", "expected": "vexor/services/skill_service.py"}
+{"query": "docs configuration rerank strategies", "expected": "docs/configuration.md"}
+{"query": "tests CLI doctor command", "expected": "tests/integration/test_cli.py"}
+{"query": "local embedding provider ONNX", "expected": "vexor/providers/local.py"}
+{"query": "JavaScript TypeScript parser chunks", "expected": "vexor/services/js_parser.py"}
+{"query": "query cache sqlite table", "expected": "vexor/cache.py"}
+{"query": "porcelain search output fields", "expected": "docs/cli.md"}
+{"query": "roadmap hybrid retrieval benchmark", "expected": "docs/roadmap.md"}