Every grans search invocation pays ~3.7s to initialize the ONNX runtime and embedding model before the query runs (cold-invocation measurements in #82). #82 can hide the reranker's share behind retrieval, but the embedder init is on the critical path: nothing runs until the query is embedded, so per-invocation wall time has a hard floor near 4s. The only way past that floor is to stop paying init per invocation.
Options to investigate:
- A background daemon holding warm ORT sessions, with the CLI as a thin client over a named pipe or local socket; started on demand by the first search, idling out after N minutes.
- A
grans serve subcommand that does the same thing opt-in, with no lifecycle magic.
- Prior art for daemon lifecycle on Windows: watchman, gopls, the bazel client/server split.
Questions the investigation should answer: lifecycle and staleness (the db changes under the daemon on every sync), Windows process semantics (named pipes, supervision, install-less operation), memory cost of resident models, and failure behavior (a dead or wedged daemon must degrade to today's in-process path rather than error). grep stays daemon-free; it loads no models and already runs in ~90ms.
Outcome: a written recommendation with rough sizing, not an implementation.
Every
grans searchinvocation pays ~3.7s to initialize the ONNX runtime and embedding model before the query runs (cold-invocation measurements in #82). #82 can hide the reranker's share behind retrieval, but the embedder init is on the critical path: nothing runs until the query is embedded, so per-invocation wall time has a hard floor near 4s. The only way past that floor is to stop paying init per invocation.Options to investigate:
grans servesubcommand that does the same thing opt-in, with no lifecycle magic.Questions the investigation should answer: lifecycle and staleness (the db changes under the daemon on every sync), Windows process semantics (named pipes, supervision, install-less operation), memory cost of resident models, and failure behavior (a dead or wedged daemon must degrade to today's in-process path rather than error). grep stays daemon-free; it loads no models and already runs in ~90ms.
Outcome: a written recommendation with rough sizing, not an implementation.