Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 3 additions & 7 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -35,10 +35,6 @@ When you remember what a file *does* but forget its name or location, Vexor find

Designed for both humans and AI coding assistants, enabling semantic file discovery in autonomous agent workflows.

Optional hybrid retrieval (`vexor config --rerank hybrid`) fuses full-corpus BM25 with dense search, so exact identifiers can surface even when dense retrieval misses them.

Per-project `.vexorignore` files give you full gitignore-style control over what gets indexed, and you can opt in to per-project indexes by creating a `.vexor/` directory or running `vexor index --local`.

## Install

Download standalone binary from [releases](https://github.com/scarletkc/vexor/releases) (no Python required), or:
Expand Down Expand Up @@ -161,17 +157,17 @@ The server exposes two tools: `vexor_search` (semantic file search) and `vexor_i
vexor init # guided setup (recommended)
vexor config --set-api-key "YOUR_KEY" # or env: VEXOR_API_KEY / OPENAI_API_KEY / ...
vexor config --set-provider openai # default; also gemini/voyageai/custom/local
vexor config --rerank bm25 # recommended: improves search accuracy
vexor config --rerank hybrid # optional: fuse exact keyword + semantic ranking
vexor config --show # view current settings
```

Config lives in `~/.vexor/config.json`. Any field can also be injected via the `VEXOR_CONFIG_JSON` environment variable (useful for MCP client configs and CI), and fully offline use is supported through local embedding models.

See [`docs/configuration.md`](https://github.com/scarletkc/vexor/blob/main/docs/configuration.md) for the complete reference: all config commands, API keys and environment variables, rerank strategies (BM25 / FlashRank / remote), remote vs local providers, embedding dimensions, and offline local model setup.
See [`docs/configuration.md`](https://github.com/scarletkc/vexor/blob/main/docs/configuration.md) for the complete reference: all config commands, API keys and environment variables, rerank strategies (hybrid / BM25 / FlashRank / remote), remote vs local providers, embedding dimensions, and offline local model setup.

## CLI Reference

Everyday usage fits in `vexor "query"`, `vexor search`, and `vexor index` (see Quick Start). The full command table, common flags, index modes (`--mode auto/name/head/brief/full/code/outline`), cache behavior, and porcelain output format are documented in [`docs/cli.md`](https://github.com/scarletkc/vexor/blob/main/docs/cli.md).
Everyday usage fits in `vexor "query"`, `vexor search`, and `vexor index` (see Quick Start). The full command table, common flags, index modes (`--mode auto/name/head/brief/full/code/outline`), `.vexorignore` files, project-local indexes (`vexor index --local`), cache behavior, and porcelain output format are documented in [`docs/cli.md`](https://github.com/scarletkc/vexor/blob/main/docs/cli.md).

## Documentation

Expand Down
6 changes: 3 additions & 3 deletions docs/cli.md
Original file line number Diff line number Diff line change
Expand Up @@ -39,9 +39,9 @@
| `--no-cache` | In-memory only; do not read/write index cache |
| `--local` | With `index`, create and use `<path>/.vexor/index.db` |

Rerank values are `off`, `bm25`, `flashrank`, `remote`, and `hybrid`.
`hybrid` fuses full-corpus BM25 and dense rankings, allowing exact lexical
matches outside the dense candidate window to be retrieved.
Reranking is a config setting rather than a search flag — see
[Configuration → Rerank](configuration.md#rerank) for the available
strategies (`off`, `bm25`, `flashrank`, `remote`, `hybrid`).

Porcelain output fields: `rank`, `similarity`, `path`, `chunk_index`,
`start_line`, `end_line`, `preview` (line fields are `-` when unavailable).
Expand Down
10 changes: 5 additions & 5 deletions docs/configuration.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,10 @@ Run `vexor index --local` to create the project directory and its ignore file.
Only the index database, including embedding and query cache tables, is
project-local.

When an upgrade changes the index cache schema, existing index caches are
invalidated and rebuilt automatically on next use; cached embeddings are
reused, so the migration does not re-embed unchanged content.

## Commands

```bash
Expand Down Expand Up @@ -75,7 +79,7 @@ ranked first by both retrieval paths.
Recommended defaults:

- Keep `off` unless you want extra precision.
- Use `bm25` for lightweight lexical boosts; it is fast and lightweight.
- Use `bm25` for a lightweight lexical boost with almost no extra latency.
- Use `hybrid` when exact identifiers and lexical-only matches matter.
- BM25 uses a multilingual tokenizer (Bert pre-tokenizer), so it can handle CJK better.
- Use `flashrank` for stronger reranking (requires `pip install "vexor[flashrank]"` and
Expand All @@ -85,10 +89,6 @@ Recommended defaults:
- For Chinese or multi-language content, set `--set-flashrank-model ms-marco-MultiBERT-L-12`.
- If unset, FlashRank defaults to `ms-marco-TinyBERT-L-2-v2`.

This release raises the index cache schema version. Older index caches are
invalidated and rebuilt automatically; cached embeddings are shared and reused,
so the migration does not require re-embedding unchanged content.

## Providers: Remote vs Local

Vexor supports both remote API providers (`openai`, `gemini`, `voyageai`,
Expand Down
2 changes: 1 addition & 1 deletion docs/development.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,4 +7,4 @@ pytest
```
Tests rely on fake embedding backends, so no network access is required.

Cache files and configuration live in `~/.vexor`. Adjust `_label_for_path` or `VexorSearcher._prepare_text` if you need to encode additional context (e.g., relative paths).
Cache files and configuration live in `~/.vexor` (a project with a `.vexor/` directory keeps its index there instead). The text embedded for each chunk is built by the mode strategies in `vexor/modes.py`; adjust the strategy `label` construction there if you need to encode additional context.
8 changes: 4 additions & 4 deletions docs/mcp.md
Original file line number Diff line number Diff line change
Expand Up @@ -45,10 +45,6 @@ codex mcp add vexor -- vexor mcp
}
```

The `reranker` response field can be `"hybrid"`. In that case, each result's
`score` is a normalized reciprocal-rank-fusion score in `[0, 1]`, rather than a
cosine similarity.

If `vexor` is not on the client's PATH, use an absolute command path, or
`python` with `"args": ["-m", "vexor", "mcp"]`.

Expand Down Expand Up @@ -164,6 +160,10 @@ Returns JSON text:
}
```

When the configured rerank strategy is `hybrid`, the `reranker` field is
`"hybrid"` and each result's `score` is a normalized reciprocal-rank-fusion
score in `[0, 1]` rather than a cosine similarity.

### `vexor_index`

Build or refresh the index for a directory. Use it to warm the cache or when
Expand Down
65 changes: 29 additions & 36 deletions docs/roadmap.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,25 +10,18 @@ never leaving the machine.

## P0 — Agent-first distribution

- **Shipped behind `--rerank hybrid`:** hybrid retrieval as a first-class path
fuses BM25 and dense scores
(e.g. reciprocal rank fusion) during search instead of offering BM25
only as an opt-in reranker. Dependencies (`rank-bm25`, `tokenizers`)
are already present. Pure semantic search is weak on exact identifiers;
hybrid is the ecosystem default now.
- Why the existing `--rerank bm25` is not hybrid: it only reorders the
dense top candidates (`clamp(top*2, 20, 150)`), so lexical-only
matches that dense retrieval misses never reach the reranker. True
hybrid scores the full chunk set on both paths before fusing.
- Full-corpus BM25 needs term statistics persisted alongside the index
cache — rebuilding them per query is O(corpus). Design this together
with the `vectors.npy`/memmap work in P1.
- The default flip remains pending. Flip the ranking default only after the evaluation benchmark (next
item) confirms hybrid beats dense-only, and call the change out in
release notes since result ordering shifts for existing users.
- Flip the default ranking to hybrid retrieval (shipped opt-in behind
`--rerank hybrid` in 0.25.0) once the benchmark confirms it beats
dense-only across embedding models. Current `scripts/eval_hybrid.py`
status on this repo: hybrid wins with the small local model but still
trails a strong remote model (bge-m3) on MRR@10. Tune the fusion
(RRF k, dense/BM25 weights, doc-length normalization) against a larger
query set and more corpora first, and call the flip out in release
notes since result ordering shifts for existing users.
- Publish an evaluation: token cost + answer quality of agent+Vexor vs
grep-only workflows (30–50 QA tasks), feature the chart in the README.
Benchmarks are what make these tools travel (see mgrep's launch).
`scripts/eval_hybrid.py` and `scripts/eval_queries.jsonl` are the seed.

## P1 — Performance & experience

Expand Down Expand Up @@ -57,17 +50,23 @@ never leaving the machine.
## P2 — Coverage & polish

- Add AST-aware `code` mode chunking for Go and Rust (tree-sitter support).
- Support `.vexorignore` for per-project ignore rules.
- Project-level local cache (per-folder cache root override).
- Project-level config (`<project>/.vexor/config.json`) for behavior-only
settings such as mode preferences, rerank, extensions, and exclude
patterns. Security constraint: repo files are attacker-controllable
input, so the loader must whitelist behavior fields and reject
credentials and endpoints (`api_key`, `base_url`, `remote_rerank`) with
an explicit error — stricter than the `VEXOR_CONFIG_JSON` env override,
which permits `base_url`. Reuse the guarded `config_from_json(base=...)`
merge; `vexor config --show` and `vexor doctor` should display each
field's origin (global vs project).
- Project-level config (`<project>/.vexor/config.json`).
- v1: overlay limited to behavior fields that already exist in the
Config schema (`rerank`, `auto_index`, `model`,
`embedding_dimensions`, batch/concurrency). Security constraint: repo
files are attacker-controllable input, so the loader must whitelist
those fields and reject credentials and endpoints (`api_key`,
`base_url`, `remote_rerank`) with an explicit error — stricter than
the `VEXOR_CONFIG_JSON` env override, which permits `base_url`.
Precedence: global config < project config < env overrides < explicit
arguments. Reuse the guarded `config_from_json(base=...)` merge; the
structural change is making config resolution directory-aware
(`load_config()` takes no path today). `vexor config --show` and
`vexor doctor` should display each field's origin (global vs project).
- v2 (only if v1 sees real use): per-project scan defaults (`mode`,
`extensions`, `exclude_patterns`) — these are per-invocation CLI
arguments today, not config fields, so supporting them means new
config surface and CLI-default plumbing.
- Additional embedding providers (Azure).
- Evaluate an optional LLM reranker that reads a bounded set of retrieved
candidates and judges their relevance to the query. Keep dense/BM25
Expand Down Expand Up @@ -106,8 +105,9 @@ never leaving the machine.

- Package for homebrew, scoop, and winget (standalone binaries already
exist in releases).
- Write a launch post when MCP + hybrid search ship (Show HN, Chinese
dev community follow-up — Vexor appeared in Ruan Yifeng's Weekly #379).
- Write the launch post — MCP and hybrid search have both shipped as of
0.25.0 (Show HN, Chinese dev community follow-up — Vexor appeared in
Ruan Yifeng's Weekly #379).
- README: add a comparison table vs mgrep / claude-context highlighting
local-first, no account, provider-agnostic, reranking options.

Expand All @@ -122,13 +122,6 @@ never leaving the machine.
remains on its own version. This is workable, but release notes and
asset naming should make the split explicit, or the GUI should get an
independent documented release track.
- Split provider-specific config validation out of `config.py` if provider
support keeps growing.
- `config.py` now owns default model resolution, provider environment
variables, base URLs, embedding dimension validation, and config
persistence. Keep it stable for now, but consider moving provider
capability metadata into a dedicated module before adding more
provider-specific rules.
- Make user-facing error handling more systematic.
- Most messages are centralized in `text.py`, but several runtime
validation paths still build detailed errors inline. Consider adding
Expand Down
2 changes: 1 addition & 1 deletion plugins/vexor/.claude-plugin/plugin.json
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
{
"name": "vexor",
"version": "0.24.3",
"version": "0.25.0",
"description": "A semantic search engine for files and code (Vexor skill bundle).",
"author": {
"name": "scarletkc"
Expand Down
4 changes: 2 additions & 2 deletions server.json
Original file line number Diff line number Diff line change
Expand Up @@ -7,13 +7,13 @@
"url": "https://github.com/scarletkc/vexor",
"source": "github"
},
"version": "0.24.3",
"version": "0.25.0",
"packages": [
{
"registryType": "pypi",
"registryBaseUrl": "https://pypi.org",
"identifier": "vexor",
"version": "0.24.3",
"version": "0.25.0",
"runtimeHint": "uvx",
"transport": {
"type": "stdio"
Expand Down
2 changes: 1 addition & 1 deletion vexor/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@
"set_data_dir",
]

__version__ = "0.24.3"
__version__ = "0.25.0"

_API_EXPORTS = frozenset(
{
Expand Down
Loading