Skip to content

feat: warn when configured model belongs to a different provider#289

Open
Greg Land (bikeusaland) wants to merge 3 commits into
langchain-ai:mainfrom
bikeusaland:fix/provider-model-mismatch-warning
Open

feat: warn when configured model belongs to a different provider#289
Greg Land (bikeusaland) wants to merge 3 commits into
langchain-ai:mainfrom
bikeusaland:fix/provider-model-mismatch-warning

Conversation

@bikeusaland

@bikeusaland Greg Land (bikeusaland) commented Jul 12, 2026

Copy link
Copy Markdown
Contributor

Problem

isValidModelId only validates the shape of the model string:

export function isValidModelId(value: string): boolean {
  const modelId = normalizeModelId(value);
  return (
    modelId.length > 0 &&
    modelId.length <= 120 &&
    /^[A-Za-z0-9][A-Za-z0-9._:/+-]*$/u.test(modelId) &&
    !modelId.includes("://")
  );
}

So a saved OPENWIKI_MODEL_ID left over from a previous provider — e.g. claude-opus-4-8 while the provider is now openai — passes validation and gets sent to the wrong backend, failing at request time with an opaque provider-side 400/404. Provider/model mismatch is the most common misconfiguration, and today it produces a cryptic error one turn later instead of an actionable local message.

Fix

Add two pure helpers to constants.ts:

  • getProvidersForKnownModelId(modelId, excludeProvider) — providers (other than the one given) whose known modelOptions include the model by exact match.
  • isModelIdForOtherProvider(modelId, provider) — true when the model is a known model of some other provider and not known for the configured provider.

resolveModelId now emits a non-fatal, actionable warning on mismatch (via the existing onEvent text channel) and still proceeds — a custom endpoint or gateway may legitimately serve the model.

Why exact matching (not substring)

Provider model lists overlap heavily via namespacing:

  • Anthropic lists claude-opus-4-8; OpenRouter lists anthropic/claude-opus-4-8.
  • nvidia lists openai/gpt-oss-120b.

Exact-match on the full normalized ID avoids false positives from these overlaps, and returns empty for custom/gateway IDs so openai-compatible model names are never flagged. Because openai and openai-chatgpt share OPENAI_MODEL_OPTIONS, switching between those two never warns.

Tests

Adds unit tests for both helpers covering: exact match, the excludeProvider behavior, namespaced-overlap non-matching, shared openai/openai-chatgpt models, custom/unknown IDs, and whitespace trimming.

Full gate passes: typecheck, lint:check, format:check, test suite (204 tests, +8 new).

🤖 Generated with Claude Code

Related PRs

Part of a batch from a single codebase-review pass. Independent, main-based branches that can merge in any order.

Security

Correctness

Feature

(This PR is #289.)

isValidModelId only checks the shape of the model string, so a saved
OPENWIKI_MODEL_ID left over from a previous provider (e.g. an Anthropic
model while the provider is now OpenAI) passed validation and failed
later with an opaque provider-side 400/404.

Add isModelIdForOtherProvider / getProvidersForKnownModelId to
constants.ts, which use exact matching against each provider's known
model options (so namespaced overlaps like OpenRouter's
"anthropic/claude-..." don't false-match, and custom/gateway model IDs
are never flagged). resolveModelId now emits a non-fatal, actionable
warning on mismatch; the run still proceeds since a custom endpoint may
legitimately serve the model.

Add unit tests for both helpers, including exact-match, shared
openai/openai-chatgpt models, and custom-id cases.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
audichuang added a commit to audichuang/openwiki that referenced this pull request Jul 15, 2026
…tream langchain-ai#289)

Adds getProvidersForKnownModelId / isModelIdForOtherProvider (constants.ts,
exact-match, union-safe over PROVIDER_CONFIGS) and a non-fatal
warnOnProviderModelMismatch emitted from resolveModelId: if OPENWIKI_MODEL_ID
is a known model of another provider, warn instead of failing later with an
opaque provider-side error. Run still proceeds (gateways may serve it).

Fork guard: agent-cli aliases (opus/default/grok-4.5) stay known to their own
provider and never trip the warning — covered by a regression test.

Verified: typecheck, lint, format, pnpm test (344), build.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
audichuang added a commit to audichuang/openwiki that referenced this pull request Jul 16, 2026
…tream langchain-ai#289)

Adds getProvidersForKnownModelId / isModelIdForOtherProvider (constants.ts,
exact-match, union-safe over PROVIDER_CONFIGS) and a non-fatal
warnOnProviderModelMismatch emitted from resolveModelId: if OPENWIKI_MODEL_ID
is a known model of another provider, warn instead of failing later with an
opaque provider-side error. Run still proceeds (gateways may serve it).

Fork guard: agent-cli aliases (opus/default/grok-4.5) stay known to their own
provider and never trip the warning — covered by a regression test.

Verified: typecheck, lint, format, pnpm test (344), build.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…smatch-warning

# Conflicts:
#	src/agent/index.ts
#	test/constants.test.ts
@bikeusaland

Copy link
Copy Markdown
Contributor Author

Merged main — resolved a semantic conflict beyond the textual ones

Rebased/merged latest main into this branch. The three textual conflicts (import lists in src/agent/index.ts + test/constants.test.ts, and the warnOnProviderModelMismatch function landing next to main's export function createModel change) were straightforward unions.

There was also a semantic conflict that the merge tool couldn't flag, worth calling out:

main introduced a new gemini-enterprise provider whose modelOptions legitimately include claude-opus-4-8 — it's a gateway that also serves Claude models. That means claude-opus-4-8 is now a known model of two providers (anthropic and gemini-enterprise), which is exactly the case getProvidersForKnownModelId is designed to surface.

Two of this PR's tests were written before gemini-enterprise existed and asserted the model belonged to anthropic only:

  • finds the provider(s) whose known models include the id — now expects ["anthropic", "gemini-enterprise"]
  • excludes the provider passed in — excluding anthropic now leaves ["gemini-enterprise"] instead of []

I updated both expectations (with a comment explaining why) to reflect the new reality rather than mask it. The feature's behavior is correct here: a claude-opus-4-8 configured under, say, openai still warns and correctly names both providers it belongs to.

No production logic changed in the merge — exact-match semantics are unchanged; the helper simply now finds a second legitimate owner.

Full gate is green after the merge: typecheck, lint:check, format:check, and the full test suite (341 passing).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant