Skip to content

fix: lazy load node-llama-cpp for no-op embed#604

Open
aceclaw826 wants to merge 1 commit into
tobi:mainfrom
aceclaw826:ace/lazy-node-llama-noop-embed
Open

fix: lazy load node-llama-cpp for no-op embed#604
aceclaw826 wants to merge 1 commit into
tobi:mainfrom
aceclaw826:ace/lazy-node-llama-noop-embed

Conversation

@aceclaw826
Copy link
Copy Markdown

Summary

Avoid importing node-llama-cpp until QMD actually needs a local LLM/model operation.

This keeps no-op embed/status-style paths lightweight. In particular, store.embed() already checks for pending documents before calling into the LLM, but the previous static import meant the native node-llama-cpp module was still loaded as soon as llm.ts was imported.

This matters for OpenClaw multi-agent gateways where boot/update cycles may call qmd embed frequently and most runs often have no missing embeddings.

Changes

  • Convert node-llama-cpp value imports to type-only imports.
  • Add a memoized dynamic loader for node-llama-cpp.
  • Load getLlama, resolveModelFile, LlamaLogLevel, and LlamaChatSession only from code paths that actually need them.
  • Add a regression test proving a no-op embed does not import node-llama-cpp.

Related

This addresses part of openclaw/openclaw#72144: avoiding heavy native/LLM initialization when all content hashes already have embeddings.

Tests

Passed:

corepack pnpm exec vitest run test/llm-lazy-load.test.ts test/sdk.test.ts -t "lazy node-llama-cpp loading|store.embed rejects invalid batch limits|store.embed forwards batch limit options" --reporter=verbose

Also checked:

corepack pnpm exec tsc -p tsconfig.build.json --noEmit

That currently fails on main with an unrelated existing error:

src/store.ts(2142,22): error TS2339: Property 'transaction' does not exist on type 'Database'.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant