diff --git a/README.md b/README.md index 2cd357c..b31329b 100644 --- a/README.md +++ b/README.md @@ -35,10 +35,6 @@ When you remember what a file *does* but forget its name or location, Vexor find Designed for both humans and AI coding assistants, enabling semantic file discovery in autonomous agent workflows. -Optional hybrid retrieval (`vexor config --rerank hybrid`) fuses full-corpus BM25 with dense search, so exact identifiers can surface even when dense retrieval misses them. - -Per-project `.vexorignore` files give you full gitignore-style control over what gets indexed, and you can opt in to per-project indexes by creating a `.vexor/` directory or running `vexor index --local`. - ## Install Download standalone binary from [releases](https://github.com/scarletkc/vexor/releases) (no Python required), or: @@ -161,17 +157,17 @@ The server exposes two tools: `vexor_search` (semantic file search) and `vexor_i vexor init # guided setup (recommended) vexor config --set-api-key "YOUR_KEY" # or env: VEXOR_API_KEY / OPENAI_API_KEY / ... vexor config --set-provider openai # default; also gemini/voyageai/custom/local -vexor config --rerank bm25 # recommended: improves search accuracy +vexor config --rerank hybrid # optional: fuse exact keyword + semantic ranking vexor config --show # view current settings ``` Config lives in `~/.vexor/config.json`. Any field can also be injected via the `VEXOR_CONFIG_JSON` environment variable (useful for MCP client configs and CI), and fully offline use is supported through local embedding models. -See [`docs/configuration.md`](https://github.com/scarletkc/vexor/blob/main/docs/configuration.md) for the complete reference: all config commands, API keys and environment variables, rerank strategies (BM25 / FlashRank / remote), remote vs local providers, embedding dimensions, and offline local model setup. +See [`docs/configuration.md`](https://github.com/scarletkc/vexor/blob/main/docs/configuration.md) for the complete reference: all config commands, API keys and environment variables, rerank strategies (hybrid / BM25 / FlashRank / remote), remote vs local providers, embedding dimensions, and offline local model setup. ## CLI Reference -Everyday usage fits in `vexor "query"`, `vexor search`, and `vexor index` (see Quick Start). The full command table, common flags, index modes (`--mode auto/name/head/brief/full/code/outline`), cache behavior, and porcelain output format are documented in [`docs/cli.md`](https://github.com/scarletkc/vexor/blob/main/docs/cli.md). +Everyday usage fits in `vexor "query"`, `vexor search`, and `vexor index` (see Quick Start). The full command table, common flags, index modes (`--mode auto/name/head/brief/full/code/outline`), `.vexorignore` files, project-local indexes (`vexor index --local`), cache behavior, and porcelain output format are documented in [`docs/cli.md`](https://github.com/scarletkc/vexor/blob/main/docs/cli.md). ## Documentation diff --git a/docs/cli.md b/docs/cli.md index 8047546..a7f24c4 100644 --- a/docs/cli.md +++ b/docs/cli.md @@ -39,9 +39,9 @@ | `--no-cache` | In-memory only; do not read/write index cache | | `--local` | With `index`, create and use `/.vexor/index.db` | -Rerank values are `off`, `bm25`, `flashrank`, `remote`, and `hybrid`. -`hybrid` fuses full-corpus BM25 and dense rankings, allowing exact lexical -matches outside the dense candidate window to be retrieved. +Reranking is a config setting rather than a search flag — see +[Configuration → Rerank](configuration.md#rerank) for the available +strategies (`off`, `bm25`, `flashrank`, `remote`, `hybrid`). Porcelain output fields: `rank`, `similarity`, `path`, `chunk_index`, `start_line`, `end_line`, `preview` (line fields are `-` when unavailable). diff --git a/docs/configuration.md b/docs/configuration.md index f4f4de6..c1132be 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -13,6 +13,10 @@ Run `vexor index --local` to create the project directory and its ignore file. Only the index database, including embedding and query cache tables, is project-local. +When an upgrade changes the index cache schema, existing index caches are +invalidated and rebuilt automatically on next use; cached embeddings are +reused, so the migration does not re-embed unchanged content. + ## Commands ```bash @@ -75,7 +79,7 @@ ranked first by both retrieval paths. Recommended defaults: - Keep `off` unless you want extra precision. -- Use `bm25` for lightweight lexical boosts; it is fast and lightweight. +- Use `bm25` for a lightweight lexical boost with almost no extra latency. - Use `hybrid` when exact identifiers and lexical-only matches matter. - BM25 uses a multilingual tokenizer (Bert pre-tokenizer), so it can handle CJK better. - Use `flashrank` for stronger reranking (requires `pip install "vexor[flashrank]"` and @@ -85,10 +89,6 @@ Recommended defaults: - For Chinese or multi-language content, set `--set-flashrank-model ms-marco-MultiBERT-L-12`. - If unset, FlashRank defaults to `ms-marco-TinyBERT-L-2-v2`. -This release raises the index cache schema version. Older index caches are -invalidated and rebuilt automatically; cached embeddings are shared and reused, -so the migration does not require re-embedding unchanged content. - ## Providers: Remote vs Local Vexor supports both remote API providers (`openai`, `gemini`, `voyageai`, diff --git a/docs/development.md b/docs/development.md index 51dec8d..834eded 100644 --- a/docs/development.md +++ b/docs/development.md @@ -7,4 +7,4 @@ pytest ``` Tests rely on fake embedding backends, so no network access is required. -Cache files and configuration live in `~/.vexor`. Adjust `_label_for_path` or `VexorSearcher._prepare_text` if you need to encode additional context (e.g., relative paths). \ No newline at end of file +Cache files and configuration live in `~/.vexor` (a project with a `.vexor/` directory keeps its index there instead). The text embedded for each chunk is built by the mode strategies in `vexor/modes.py`; adjust the strategy `label` construction there if you need to encode additional context. \ No newline at end of file diff --git a/docs/mcp.md b/docs/mcp.md index 484f26d..43a4293 100644 --- a/docs/mcp.md +++ b/docs/mcp.md @@ -45,10 +45,6 @@ codex mcp add vexor -- vexor mcp } ``` -The `reranker` response field can be `"hybrid"`. In that case, each result's -`score` is a normalized reciprocal-rank-fusion score in `[0, 1]`, rather than a -cosine similarity. - If `vexor` is not on the client's PATH, use an absolute command path, or `python` with `"args": ["-m", "vexor", "mcp"]`. @@ -164,6 +160,10 @@ Returns JSON text: } ``` +When the configured rerank strategy is `hybrid`, the `reranker` field is +`"hybrid"` and each result's `score` is a normalized reciprocal-rank-fusion +score in `[0, 1]` rather than a cosine similarity. + ### `vexor_index` Build or refresh the index for a directory. Use it to warm the cache or when diff --git a/docs/roadmap.md b/docs/roadmap.md index 62a0800..a6a2eb9 100644 --- a/docs/roadmap.md +++ b/docs/roadmap.md @@ -10,25 +10,18 @@ never leaving the machine. ## P0 — Agent-first distribution -- **Shipped behind `--rerank hybrid`:** hybrid retrieval as a first-class path - fuses BM25 and dense scores - (e.g. reciprocal rank fusion) during search instead of offering BM25 - only as an opt-in reranker. Dependencies (`rank-bm25`, `tokenizers`) - are already present. Pure semantic search is weak on exact identifiers; - hybrid is the ecosystem default now. - - Why the existing `--rerank bm25` is not hybrid: it only reorders the - dense top candidates (`clamp(top*2, 20, 150)`), so lexical-only - matches that dense retrieval misses never reach the reranker. True - hybrid scores the full chunk set on both paths before fusing. - - Full-corpus BM25 needs term statistics persisted alongside the index - cache — rebuilding them per query is O(corpus). Design this together - with the `vectors.npy`/memmap work in P1. - - The default flip remains pending. Flip the ranking default only after the evaluation benchmark (next - item) confirms hybrid beats dense-only, and call the change out in - release notes since result ordering shifts for existing users. +- Flip the default ranking to hybrid retrieval (shipped opt-in behind + `--rerank hybrid` in 0.25.0) once the benchmark confirms it beats + dense-only across embedding models. Current `scripts/eval_hybrid.py` + status on this repo: hybrid wins with the small local model but still + trails a strong remote model (bge-m3) on MRR@10. Tune the fusion + (RRF k, dense/BM25 weights, doc-length normalization) against a larger + query set and more corpora first, and call the flip out in release + notes since result ordering shifts for existing users. - Publish an evaluation: token cost + answer quality of agent+Vexor vs grep-only workflows (30–50 QA tasks), feature the chart in the README. Benchmarks are what make these tools travel (see mgrep's launch). + `scripts/eval_hybrid.py` and `scripts/eval_queries.jsonl` are the seed. ## P1 — Performance & experience @@ -57,17 +50,23 @@ never leaving the machine. ## P2 — Coverage & polish - Add AST-aware `code` mode chunking for Go and Rust (tree-sitter support). -- Support `.vexorignore` for per-project ignore rules. -- Project-level local cache (per-folder cache root override). -- Project-level config (`/.vexor/config.json`) for behavior-only - settings such as mode preferences, rerank, extensions, and exclude - patterns. Security constraint: repo files are attacker-controllable - input, so the loader must whitelist behavior fields and reject - credentials and endpoints (`api_key`, `base_url`, `remote_rerank`) with - an explicit error — stricter than the `VEXOR_CONFIG_JSON` env override, - which permits `base_url`. Reuse the guarded `config_from_json(base=...)` - merge; `vexor config --show` and `vexor doctor` should display each - field's origin (global vs project). +- Project-level config (`/.vexor/config.json`). + - v1: overlay limited to behavior fields that already exist in the + Config schema (`rerank`, `auto_index`, `model`, + `embedding_dimensions`, batch/concurrency). Security constraint: repo + files are attacker-controllable input, so the loader must whitelist + those fields and reject credentials and endpoints (`api_key`, + `base_url`, `remote_rerank`) with an explicit error — stricter than + the `VEXOR_CONFIG_JSON` env override, which permits `base_url`. + Precedence: global config < project config < env overrides < explicit + arguments. Reuse the guarded `config_from_json(base=...)` merge; the + structural change is making config resolution directory-aware + (`load_config()` takes no path today). `vexor config --show` and + `vexor doctor` should display each field's origin (global vs project). + - v2 (only if v1 sees real use): per-project scan defaults (`mode`, + `extensions`, `exclude_patterns`) — these are per-invocation CLI + arguments today, not config fields, so supporting them means new + config surface and CLI-default plumbing. - Additional embedding providers (Azure). - Evaluate an optional LLM reranker that reads a bounded set of retrieved candidates and judges their relevance to the query. Keep dense/BM25 @@ -106,8 +105,9 @@ never leaving the machine. - Package for homebrew, scoop, and winget (standalone binaries already exist in releases). -- Write a launch post when MCP + hybrid search ship (Show HN, Chinese - dev community follow-up — Vexor appeared in Ruan Yifeng's Weekly #379). +- Write the launch post — MCP and hybrid search have both shipped as of + 0.25.0 (Show HN, Chinese dev community follow-up — Vexor appeared in + Ruan Yifeng's Weekly #379). - README: add a comparison table vs mgrep / claude-context highlighting local-first, no account, provider-agnostic, reranking options. @@ -122,13 +122,6 @@ never leaving the machine. remains on its own version. This is workable, but release notes and asset naming should make the split explicit, or the GUI should get an independent documented release track. -- Split provider-specific config validation out of `config.py` if provider - support keeps growing. - - `config.py` now owns default model resolution, provider environment - variables, base URLs, embedding dimension validation, and config - persistence. Keep it stable for now, but consider moving provider - capability metadata into a dedicated module before adding more - provider-specific rules. - Make user-facing error handling more systematic. - Most messages are centralized in `text.py`, but several runtime validation paths still build detailed errors inline. Consider adding diff --git a/plugins/vexor/.claude-plugin/plugin.json b/plugins/vexor/.claude-plugin/plugin.json index 08fbd89..c0d1184 100644 --- a/plugins/vexor/.claude-plugin/plugin.json +++ b/plugins/vexor/.claude-plugin/plugin.json @@ -1,6 +1,6 @@ { "name": "vexor", - "version": "0.24.3", + "version": "0.25.0", "description": "A semantic search engine for files and code (Vexor skill bundle).", "author": { "name": "scarletkc" diff --git a/server.json b/server.json index aec5fd4..70c14f7 100644 --- a/server.json +++ b/server.json @@ -7,13 +7,13 @@ "url": "https://github.com/scarletkc/vexor", "source": "github" }, - "version": "0.24.3", + "version": "0.25.0", "packages": [ { "registryType": "pypi", "registryBaseUrl": "https://pypi.org", "identifier": "vexor", - "version": "0.24.3", + "version": "0.25.0", "runtimeHint": "uvx", "transport": { "type": "stdio" diff --git a/vexor/__init__.py b/vexor/__init__.py index 4a3ef1e..cc3c6ed 100644 --- a/vexor/__init__.py +++ b/vexor/__init__.py @@ -22,7 +22,7 @@ "set_data_dir", ] -__version__ = "0.24.3" +__version__ = "0.25.0" _API_EXPORTS = frozenset( {