Add model tiers with cross-model start-time failover - #1048
Draft
ferrislucas wants to merge 31 commits into
Draft
Conversation
Implements the plan in model-tiers-plan.md: users can define named,
ordered lists of models across providers ("tiers") and bind sessions,
templates, kanban lanes, project defaults, and the summary model to a
tier instead of a concrete model. When a session bound to a tier
starts, it tries the first healthy member and transparently fails
over to the next member on a service/token-limit error, crossing
providers if needed — before the first assistant message only.
- shared: tier ref helpers (isTierRef/parseTierRef/buildTierRef), Zod
contracts, WS_MESSAGE_TYPES.TIER_FAILOVER, tier cooldown constants.
- server: model_tiers/model_tier_members tables + ModelTierRepository,
REST API (/api/tiers), tierResolutionService (cooldown Map + member
resolution), and the failover loop wired into the session start path
(sessionTierFailover.js) via a tier-aware shouldRescheduleOnError.
Extracted continueSessionCore into sessionContinuation.js to keep
sessionExecution.js under the line-count lint limit.
- web: tiers Pinia store + API client, ModelSelector tier support
(all 11 consumers inherit it), and a new Settings → Model Tiers tab
for CRUD + reordering.
- sessions gain resolved_model/resolved_provider_id snapshot columns
so they remember which concrete model they actually ran on.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Update E2E test to expect 5 tabs after Model Tiers tab was added to SettingsView. Add missing modelTiers contract test file for the shared package. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…ckout-feature-description-b
Fixes eight high-priority bugs from the model tiers remediation plan: Fix 1 — sessionContinuation: resolve tier ref to concrete model before building agent env so continue turns never pass a raw `tier::<id>` sentinel to the agent SDK. Uses resolvedModel snapshot; falls back to live tier lookup if snapshot is absent. Fix 2 — WebSocket: add `onTierFailover` event handler to useSessionSubscription composable and display a toast in SessionChatContent when a failover occurs. Fix 3 — UI: ModelSelector shows a "Tier: <name>" chip badge when the session is bound to a tier; ConversationTab's activeModelDisplayName resolves tier refs to "Resolved Model (Tier: Name)". Fix 4 — agentCallLogger: add _logFailoverEvent() to write tier failover events to the agent call log (visible in Settings → Logs). Fix 5 — sessionTierFailover: after runSessionWithTierFailover, check `session.status === 'scheduled'` before writing the resolvedModel snapshot so auto-rescheduled sessions don't snapshot the failed member. Fix 7 — sessionTierFailover: import DEFAULT_MAX_FAILOVER_ATTEMPTS from @circuschief/shared and cap the failover loop at Math.min(members.length, DEFAULT_MAX_FAILOVER_ATTEMPTS). Fix 8 — ModelTiersView: prevent adding duplicate (providerId, modelId) pairs to a tier in the add-member dialog. Fix 9 — summaryModelResolver: gracefully fall back to the default summary model instead of throwing when a tier-bound summary model has all members in cooldown. Tests: new sessionContinuation.test.js (4 tests, Fix 1); extended sessionTierFailover.test.js (Fixes 5, 7); extended summaryModelResolver.test.js (Fix 9); extended useWebSocket.test.js (Fix 2); updated test mocks in SessionChatContent/Overlay tests. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Extract focused helpers from larger functions to clarify responsibilities: - Split resolveConcreteContinueModel out of buildContinueModelAndEnv - Extract emitTierFailoverEvent from handleTierMemberFailure - Drop unused timestamp param/locals from failover logging paths - Consolidate duplicate imports in summaryModelResolver test and ConversationTab Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Co-authored-by: z.ai <noreply@z.ai>
ferrislucas
marked this pull request as draft
July 4, 2026 18:18
Fix 1 (blocker): Make buildModelAndProvider tier-aware so continueSessionWithExistingMessage never forwards a raw tier:: sentinel to the agent. Resolves to stored resolvedModel snapshot or falls back to live resolveActiveModel lookup on the reschedule-retry path. Fix 2: Make cooldown successor-aware — markUnhealthy is only called when hasNextHealthyMember is true, preventing a dead-end where the terminal member is cooled down and blocks both new starts and reschedule-retries. Fix 3: Preserve resolvedModel when a successful run is proactively rescheduled — snapshots the member that ran (has assistant messages) regardless of subsequent scheduled status. Fix 4: Introduce matchesStartFailoverEligibleError — a tighter matcher for start-time tier failover that avoids spurious cross-provider failover on non-quota errors (e.g. "Unexpected token in JSON"). Both handleTierMemberFailure and isTierFailoverEligibleError now use it. matchesTokenLimitError retains its intentional breadth for reschedule decisions. Fix 5: Confirm and test that tierFailover log entries are isolated in their own call_type bucket — they do not inflate runSession call_count or token sums in getSessionStats/getGlobalStats. Fix 6: Add doc comment to cooldown map documenting per-process scope and restart-clears-cooldown behavior with a note for future multi-process support. All new fixes ship with regression tests. Server suite: 4804 passed, 0 failed. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Multer stops reading the request stream when it rejects an upload, so responding immediately can reset the socket mid-upload instead of closing cleanly. Drain the remaining body before sending the error response to make the rejection path deterministic. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
ferrislucas
marked this pull request as ready for review
July 6, 2026 01:59
matchesStartFailoverEligibleError treated prompt-too-long conditions (context length / context window / max_tokens) as failover-eligible. These aren't outages or quota exhaustion, so failing over to another provider won't fix an oversized prompt and can mask a real prompt-size bug by silently bouncing across providers. Removed those patterns from the tighter start-failover matcher; matchesTokenLimitError keeps its intentional breadth for the separate auto-reschedule decision. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
_logFailoverEvent always wrote agentType: 'claude-code' and success: false. Failing over *from* a Codex/Gemini member logged the wrong agent type, and success:false mapped the row to status 'error' (AgentCallLogRepository.complete), putting a benign system event in the error bucket. - agentCallLogger._logFailoverEvent now accepts an explicit agentType (falling back to 'claude-code') and completes with success: true — a failover that successfully advances is not a call failure. Stats are grouped by call_type, so tierFailover stays isolated from runSession cost/failure rollups either way. - sessionTierFailover.emitTierFailoverEvent derives the source member's agentType via resolveAgentTypeFromModel(member.modelId) and passes it through. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Adds tests/e2e/model-tiers.spec.ts covering the deterministic surfaces per the amended DoD (option A): Model Tiers settings tab render, tier CRUD (create/rename/add-member/delete), member reorder persistence, and ModelSelector's tier::<id> sentinel (mutually exclusive with a concrete model). The failover *notice* toast is exercised by injecting a TIER_FAILOVER message via page.routeWebSocket rather than provoking a real provider outage — the cassette harness can't express a start-time provider error deterministically. Actual failover *behavior* stays covered by the server integration suite (sessionTierFailover.test.js). While wiring the CRUD test, found ModelTiersView.vue calling uiStore.showSuccess()/showError(), which don't exist on the ui store (it exposes success()/error()/warning()/info()) — every create/update/ delete threw after the API call succeeded, leaving the modal stuck open with a bogus "g.showSuccess is not a function" error. Fixed to use the real store methods. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Resolve conflicts to make PR #1048 ready for review: - migrations/index.js: keep both appended migration blocks (model tiers from this branch + normalize-stale-claude-model-ids from main) - ModelSelector.test.js: keep both describe blocks (tier support + orphaned/unknown model id); re-close the tier describe block ModelSelector.vue auto-merged cleanly (unknown-model badge from main is independent of the tier optgroup from this branch). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Co-authored-by: z.ai <noreply@z.ai>
…embers Built-in providers use fixed non-UUID ids (e.g. anthropic-default), so the uuid() constraint on TierMember.providerId was incorrectly rejecting valid requests. Relaxed to min(1) throughout the contract and updated tests accordingly. Also added Issue 3 fix to getTierMembersResolved: filter out members whose model was deleted from an otherwise-present provider (model_tier_members has no FK on model_id, so orphaned members must be filtered at query time). Updated unit and E2E tests to register model ids so they are not treated as orphans by this new filter. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Closes the remaining gaps between the model-tiers FRD and the implementation after PR #1048: - Fix 1: sessionProvider's resolve*FromModel helpers now accept an explicit providerId hint to disambiguate a modelId shared by two providers. - Fix 2: unify tier-ref resolution for continuation paths behind a new tierResolutionService.resolveTierRefForContinue helper, shared by sessionContinuation.buildContinueModelAndEnv and sessionManager.buildModelAndProvider. Switching from one bound tier to a different one now resolves the new tier live instead of reusing the old tier's snapshot, and an explicit concrete-model override always clears the stored resolvedModel/resolvedProviderId snapshot. - Fix 3: refresh/backfill the active concrete snapshot whenever a session's tier binding changes or a legacy row is missing one. - Fix 4: thread the exact tier member's providerId through the start-time failover path (attemptRunWithModel, reconcileAgentTypeForRun, resolveInitialSessionModelEnv, and the failover log's agentType) so cross-provider members sharing a modelId resolve correctly. - Fix 5: add tierResolutionService.findNextHealthyTierMember — a single ordered, attempt-cap-aware scan shared by the cooldown decision and the failover notice/log so they can't disagree about which member is next. - Fix 6: degrade safely when a new/scheduled session starts on a tier ref that no longer resolves (deleted or emptied) — fall back to the session's last concrete snapshot, or the server default, with a visible notice and log entry, instead of failing the session outright. - Fix 7: allow summary settings to store a tier ref (summaryProviderId must be null) in the settings API validation, matching what summaryModelResolver and the web form already supported. - Fix 8: keep the ModelSelector tier chip visible (and mark it stale) when the bound tier has been deleted or emptied, instead of silently falling back to a concrete default with no visible trace of the original binding. Covered by new unit/integration tests across sessionProvider, sessionAgentGuard, tierResolutionService, sessionTierFailover, sessionContinuation, sessionManager, settings, and ModelSelector, plus the existing model-tiers/settings E2E specs. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…model tiers Adds write-time validation of `tier::<id>` sentinel values across all session and kanban write paths, ensuring broken tier bindings are never persisted. Extends sessionAgentGuard to resolve the active tier member before performing cross-kind drift checks, and clears stale providerId when a tier ref is set. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…ckout-feature-description-b Co-authored-by: z.ai <noreply@z.ai> # Conflicts: # packages/web/src/components/ModelSelector.vue
…E reliability Separates the archive and unarchive flows into distinct emitted events (archive vs unarchive) so each handler is unambiguous regardless of session state. Also updates model references from gpt-5.5 to gpt-5.6-sol and hardens E2E tests with explicit element waits and expect.poll instead of fixed timeouts. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Closes the correctness gaps found in review against the Model Tiers PRD: - Reject/degrade a summary tier binding that resolves to an unsupported provider kind (e.g. Google) at both write-time (settings API) and resolve-time (summaryModelResolver), so it can never silently mis-route to the Anthropic client. - Unify the failover suppression check and the failover advance check on the same cap-aware resolver so they can't disagree for tiers with more than DEFAULT_MAX_FAILOVER_ATTEMPTS members, guaranteeing an exhausted tier always reaches terminal error handling instead of a silent hang. - Tighten the 'billing' start-failover trigger to specific phrases so it no longer fires on unrelated text that merely mentions billing. - Rename the create-time-only resolveAgentTypeFromModel in db/session-helpers.js to resolveInitialAgentTypeFromModel to disambiguate it from the provider-aware run-time resolver in sessionProvider.js.
Add a README features bullet with settings/edit-tier screenshots, a technical Model Tiers section in docs/development.md, and update CLAUDE.md's architecture notes to reflect the new tables, services, and views introduced by the feature. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Adds a scripted spawn-outcome seam (e2eSpawnOutcomes.js) and captured agent event helpers (e2eSpawnEvents.js) so E2E tests can simulate provider quota/rate-limit/outage/auth failures and assert cross-model failover routing, plus a new model-tiers-failover.spec.ts suite that exercises this against tierResolutionService and sessionExecution. Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
…into circus-chief/c17f-checkout-feature-description-b Co-authored-by: Codex <noreply@openai.com> # Conflicts: # packages/web/src/components/ModelSelector.test.js # packages/web/src/components/ModelSelector.vue
…into circus-chief/c17f-checkout-feature-description-b
…into circus-chief/c17f-checkout-feature-description-b
…into circus-chief/c17f-checkout-feature-description-b
…into circus-chief/c17f-checkout-feature-description-b
Resolves conflicts between the structured lane-run/workflow-session work on main and the model-tiers work on this branch: - sessionExecution.js: kept this branch's extraction of continueSessionCore into sessionContinuation.js and ported main's `interactive` flag onto the extracted copy. Kept main's workflow imports (workflowSessionService, drainLaneEntryTrigger) alongside the re-export. - kanbanTriggers.js: kept this branch's tier-aware deriveAgentTypeForModelOrTier and resolveCommitAttributionOverrideForModel, combined with main's attachRootSession and its move of parentSessionId into sessions.create() (required by the new parent immutability trigger). - kanban.js: kept both branches' imports. - schemaBaseline.test.js: combined both column sets in physical order — main's workflow columns from schema.sql, then this branch's migration-added resolved_model / resolved_provider_id. Merge-induced lint fixes: - Extracted finishWorkflowTurn() and shouldRethrowForTierFailover() out of _executeSession, which exceeded max-statements/complexity once both branches' additions landed in it. - Extracted resolveInheritedLaneRunId() into session-helpers.js, which put SessionRepository.js back under max-lines. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Minor fixes across 5 E2E spec files (archive-worktree-cleanup, file-attachments, kanban-lane-run-structured, model-tiers, session-navigation) to align with current app behavior. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…ckout-feature-description-b Co-authored-by: Codex <noreply@openai.com> # Conflicts: # packages/server/src/db/SessionRepository.js
ferrislucas
marked this pull request as draft
August 4, 2026 02:28
Co-authored-by: Codex <noreply@openai.com>
…ckout-feature-description-b Co-authored-by: Codex <noreply@openai.com> # Conflicts: # packages/server/src/api/kanban.test.js # packages/server/src/db/SessionRepository.js # packages/server/src/services/queryParamBuilder.js # packages/server/src/services/sessionExecution.js # packages/server/src/services/sessionProvider.test.js # packages/shared/src/index.js # packages/web/src/components/SessionChatContent.test.js # packages/web/src/components/SessionChatContent.vue # packages/web/src/components/SessionChatOverlay.test.js # packages/web/src/composables/useSessionSubscription.js
The origin/main merge (da92d4b) resolved several conflicted files by keeping only the branch side, silently dropping code main had added. That surfaced as 21 failing tests and 4 lint errors. - SessionRepository: restore claimScheduled() (and its claimScheduledRow import) so scheduled-session claiming works again — this fixes the run-scheduled-now API and repository claim tests. - queryParamBuilder: restore the conversationId param, interaction callbacks, and askUserQuestion toolConfig on the Claude path, and thread conversationId from the standard start and tier-failover start paths so the callbacks bind to the right conversation. - sessionExecution/sessionContinuation/sessionTierFailover: restore the structured start-result contract ({started, sessionId, reason}). Both start paths now propagate _executeSession's rejection result and otherwise return startedSessionExecution(). A rejected dispatch inside the tier loop is surfaced verbatim rather than snapshotted as a healthy member or failed over from. - sessionExecution: extract beginSessionStart() and _runTierBoundSession() out of runSessionCore, which was over the complexity and max-statements budgets after the merge. - kanban.test.js: restore main's Idempotency-Key and cross-project ownership suites alongside the branch's tier-ref lane tests. - kanbanService.test.js: mock resolveCommitAttributionOverrideForModel, which the model-tiers git-setup path now calls before dispatch. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Test plan
🤖 Generated with Claude Code