Skip to content

RFC: LLM backend and orchestration (ADR 0001)#335

Open
slobentanzer wants to merge 1 commit into
stagingfrom
rfc/backend-orchestration
Open

RFC: LLM backend and orchestration (ADR 0001)#335
slobentanzer wants to merge 1 commit into
stagingfrom
rfc/backend-orchestration

Conversation

@slobentanzer

Copy link
Copy Markdown
Contributor

Adds docs/decisions/0001-llm-backend-orchestration.md: a verified research synthesis to support a decision, not code.

Question: should LangChain stay the load-bearing spine, or move to a thin OpenAI-compatible spine with LangChain at the edges?

Finding: all major local backends (vLLM, SGLang, llama.cpp, Ollama) expose OpenAI /v1; LiteLLM (already a dependency) normalizes hosted and local providers; vLLM gives grammar-based structured outputs that are more reliable for local models than native tool calling. LangChain v1 is breaking work (langchain-classic split, agent API rename), which #334 is currently paying.

Recommendation: route model calls through a thin spine (LiteLLM), use grammar-based structured outputs for local models, keep LangChain at the edges. Even if #334 lands as-is, wrap its LangChain usage behind a biochatter-owned interface so it can be demoted later. For #297: prefer the spine's normalized response over a bespoke return type.

Relates to #334 and #297. See the document for findings with confidence levels, open questions, and refuted claims.

Records a verified research synthesis on whether LangChain should remain
the load-bearing spine or be demoted to the edges in favour of a thin
OpenAI-compatible spine (LiteLLM). For discussion; relates to #334 and #297.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@VladSam42

Copy link
Copy Markdown
Collaborator

The decision should be made based on the intended use cases. LangChain and LangGraph are useful if we want more complex agent behavior. But even if complex looping and tool use won't be required, completely removing lang[things] will be a lot of work. Will see how much problem lang[things] cause. A better approach would be a gradual migration, starting by implementing new capabilities in a simplified way.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants