feat: per-user LLM model resolution — BYOK goes live (BYOK 3/4)#408
Open
edspencer wants to merge 3 commits into
Open
feat: per-user LLM model resolution — BYOK goes live (BYOK 3/4)#408edspencer wants to merge 3 commits into
edspencer wants to merge 3 commits into
Conversation
This is the breaking PR of the BYOK series: the module-level model
singletons in apps/web/lib/ai/index.ts (routerModel, chatModel,
documentWritingModel, extractAchievementsModel, getLLM, customModel and
friends) are deleted and replaced by resolveModelForUser(user, task) in
lib/ai/resolve-model.ts. Demo users keep running on the platform
OPENAI_API_KEY with the historical task->model mapping (extraction:
gpt-4o-mini, generation: gpt-4.1-mini, chat: gpt-4o); everyone else gets
their default user_llm_config row decrypted and instantiated via
@bragdoc/ai's createLLMFromConfig, with no env-key fallback path left
anywhere. Embeddings are the one deliberate exception and stay on the
platform key (OpenAI-only 1536-d vectors power Workstreams; documented in
lib/ai/embeddings.ts).
Every AI call site now threads a per-user model: chat routes (documents
and performance review) resolve chat + generation models before creating
the stream, generate routes resolve before streaming, the artifact
handler interface and document tools accept a model, standup document
creation takes a model resolved by its routes, workstream naming resolves
from the already-threaded user, and chat title generation gains a user
param. Routes catch the typed NoLLMConfigError and return HTTP 409
{ error: 'no_llm_configured' } before any stream starts.
Client surfaces detect the 409 and show a "Connect your AI provider" CTA
pointing at /account instead of generic failure toasts: an inline
NoLLMConfigAlert in the generate-document dialog and the performance
review zero state, and an actionable toast for document chat,
performance-review chat, standup regeneration and workstream generation.
The getting-started checklist gains a "Connect your AI provider" item
that completes when the user has a stored config (demo users are exempt)
and the banner no longer auto-hides until it is done.
Dead code is removed along the way: the legacy /api/cli/commits route and
its tests (the shipping CLI extracts locally and never calls it),
lib/llm-utils.ts (raw-fetch OpenAI helpers with no importers),
lib/ai/llm-router.ts plus its eval, scorer and prompt, and the unused
lib/ai/models.ts and custom-middleware.ts. Evals for the remaining
prompts now construct their own platform models explicitly. New unit
tests cover the resolver (demo mapping, decrypt-and-build, keyless
providers, missing-config and missing-key throws) and the workstreams
generate route's 409 pre-check.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Contributor
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
Contributor
Braintrust eval reportNo experiments to report |
This was referenced Jul 7, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part 3 of 4 of the BYOK stack (#405). Base:
byok/2-key-storage-settings(#407) — merge #406 and #407 first.This is the breaking change: the platform
OPENAI_API_KEYno longer serves AI features for regular users. Every AI call site now resolves a model from the user's stored BYOK config; users without one get a clear 409 + "add your API key" CTA. Demo mode and embeddings stay on the platform key (explicit v1 decision, documented in code).What's here
lib/ai/resolve-model.ts—resolveModelForUser(user, task): demo users → platform key with the exact pre-BYOK task→model mapping (extraction gpt-4o-mini, generation gpt-4.1-mini, chat gpt-4o); everyone else → defaultuser_llm_configrow, decrypted, built via@bragdoc/ai. No config → typedNoLLMConfigError→ HTTP 409{ error: 'no_llm_configured' }. The old model singletons (chatModel,documentWritingModel, …) are deleted outright — grep confirms no non-demo path can reach the platform chat key.modelparam — threaded through chat (document + performance review), document generate, performance-review generate, standup regeneration, workstream naming, artifact handlers, and the document tools. Chat title generation gained a user param (three callers, one more than the spec knew about). Streaming routes resolve the model before opening the stream so clients get a clean 409, never a broken SSE.NoLLMConfigAlertinline in the generate-document dialog and performance-review zero state; CTA toasts with an "Add API key" action for chat, standups, and workstreams. Getting-started banner gains a "Connect your AI provider" item (auto-satisfied for demo users)./api/cli/commitsroute (the shipping CLI extracts locally),lib/llm-utils.ts(raw-fetch OpenAI helpers, zero importers),lib/ai/llm-router.ts+ its eval/prompt/scorer,lib/ai/models.ts,custom-middleware.ts.embeddings.tsrecords the decision.wrangler secret put BYOK_ENCRYPTION_KEY(staging + production) before this deploys — and the equivalent env var on Vercel while that's still the host. Generate withopenssl rand -base64 32.Verification
pnpm build6/6 and rootpnpm testgreentasks/405-byok/TEST_PLAN.md(synced to Bring Your Own Key (BYOK): per-user LLM API keys #405 workflow)🤖 Generated with Claude Code