Skip to content

Investigate a persistent process to amortize model init across invocations #83

Description

@dmwyatt

Every grans search invocation pays ~3.7s to initialize the ONNX runtime and embedding model before the query runs (cold-invocation measurements in #82). #82 can hide the reranker's share behind retrieval, but the embedder init is on the critical path: nothing runs until the query is embedded, so per-invocation wall time has a hard floor near 4s. The only way past that floor is to stop paying init per invocation.

Options to investigate:

  • A background daemon holding warm ORT sessions, with the CLI as a thin client over a named pipe or local socket; started on demand by the first search, idling out after N minutes.
  • A grans serve subcommand that does the same thing opt-in, with no lifecycle magic.
  • Prior art for daemon lifecycle on Windows: watchman, gopls, the bazel client/server split.

Questions the investigation should answer: lifecycle and staleness (the db changes under the daemon on every sync), Windows process semantics (named pipes, supervision, install-less operation), memory cost of resident models, and failure behavior (a dead or wedged daemon must degrade to today's in-process path rather than error). grep stays daemon-free; it loads no models and already runs in ~90ms.

Outcome: a written recommendation with rough sizing, not an implementation.

Metadata

Metadata

Assignees

No one assigned

    Labels

    prio: p2Medium priority, normal queuesize: mMedium, a few hours to a daystatus: readyReady to be worked ontype: choreMaintenance, refactoring, dependencies

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions