A cold grans search invocation costs ~5s of wall time: ~3.7s of ONNX runtime and embedding-model init (measured via search --fast, which pays everything except the reranker), ~1.2s more for the reranker (session load plus ~0.33s inference), and ~130ms of actual retrieval. The stages run strictly in sequence today: init embedder, embed query, FTS plus fusion, init reranker, rerank.
The reranker's init depends on nothing upstream. Start it concurrently when the search command begins (a thread or task that yields the ready session), let embedding and retrieval run as they do now, and join on the reranker only when the top-50 fused pool is ready. Its load then hides almost entirely behind the ~3.8s the pipeline already spends, and a full search should land within ~0.3-0.4s of what --fast costs today.
Measurement notes: cold-process invocations on main @ a55bec8, release build with --features directml, RTX 3080, against a fully embedded 872-doc database, 4 runs per mode. Timings observed: grep ~0.09s, search --fast ~3.7-4.0s, full search ~4.7-5.9s. Warm-process per-query times from the benchmark harness for contrast: fts ~7ms, hybrid ~130ms, rerank ~334ms.
Out of scope: the ~3.7s embedder init itself, which stays on the critical path (nothing can run before the query is embedded). That floor is a separate investigation.
A cold
grans searchinvocation costs ~5s of wall time: ~3.7s of ONNX runtime and embedding-model init (measured viasearch --fast, which pays everything except the reranker), ~1.2s more for the reranker (session load plus ~0.33s inference), and ~130ms of actual retrieval. The stages run strictly in sequence today: init embedder, embed query, FTS plus fusion, init reranker, rerank.The reranker's init depends on nothing upstream. Start it concurrently when the search command begins (a thread or task that yields the ready session), let embedding and retrieval run as they do now, and join on the reranker only when the top-50 fused pool is ready. Its load then hides almost entirely behind the ~3.8s the pipeline already spends, and a full search should land within ~0.3-0.4s of what
--fastcosts today.Measurement notes: cold-process invocations on main @ a55bec8, release build with
--features directml, RTX 3080, against a fully embedded 872-doc database, 4 runs per mode. Timings observed:grep~0.09s,search --fast~3.7-4.0s, fullsearch~4.7-5.9s. Warm-process per-query times from the benchmark harness for contrast: fts ~7ms, hybrid ~130ms, rerank ~334ms.Out of scope: the ~3.7s embedder init itself, which stays on the critical path (nothing can run before the query is embedded). That floor is a separate investigation.