feat(deepseek): flip operator default to v4-flash (post-eval)#40
Merged
Conversation
…val) This lands the default flip held back by PR #39 ("default flip happens after the eval produces a data-driven recommendation"). The recommendation is now in: `daemon/docs/v1-deepseek/PRO_VS_FLASH_EVAL_RESULTS.md`. ## Empirical summary (full data in EVAL_RESULTS.md) Across 130 consults / 13 distinct task types over four eval waves on promptfoo / DeepSeek `https://api.deepseek.com/v1`: | Wave | Sample | Pro pass | Flash pass | Notes | |---|---|---|---|---| | v1 | 6 × 1 | 6/6* | 6/6* | *after fixing assertion bugs | | v3 | 9 × 5 | 44/45 | 44/45 | both misses were assertion false-positives | | v4 | 4 × 5 | **19/20** | **19/20** | 4 tasks specifically designed for Pro to win | v4 covers the four categories where published DeepSeek benchmarks claim a material Pro advantage: - S10 long-context cross-file synthesis (Pro 97% needle / Flash drops in middle) - S11 complex 15-field nested JSON (Flash reportedly fails on 10+ field schemas) - S12 algorithm design with space-complexity constraint (multi-step reasoning) - S13 SimpleQA-Verified-style factual recall (Pro 57.9 / Flash 34.1) **Quality: tied (19/20 vs 19/20).** The one Pro failure (S11 trial 1) was Pro emitting two valid JSON objects concatenated with leaked meta-reasoning prose between them — i.e., Pro broke the explicit "Output ONLY the JSON object" instruction. The one Flash failure (S12) was an assertion-regex mismatch on correct output. The only quality signal that emerged across all 130 consults cut against Pro, not for it. **Cost: 3.4× more for Pro** at v4 promo pricing (Pro $0.053 vs Flash $0.016 for the 20-call wave). Post-promo on 2026-06-01 the gap widens. **Token economics: Pro uses 8.6% more total tokens, 12.4% more reasoning tokens** at indistinguishable answer quality. ## What this changes - `deepseek_config.rs` — default model `"deepseek-v4-pro"` → `"deepseek-v4-flash"`. - `from_env_returns_documented_defaults_for_all_15_knobs` — assert updated. - `shared-types/src/lib.rs` — doc-comment on `AskAgentRequest.deepseek_model` updated to reflect Flash as default. - `daemon/docs/deepseek-operator-runbook.md` — env-var + per-call tables updated. - `daemon/docs/v1-deepseek/PRO_VS_FLASH_EVAL_RESULTS.md` (new) — full data, methodology, and the explicit boundaries of what this eval does NOT cover (no multi-turn agentic flows; context ≤ 3K tokens; binary assertions; n=5 per task). For the daemon's current single-shot workload, the empirical case is firm. If multi-turn or 100K-token flows are added later, re-evaluate. - `daemon/docs/v1-deepseek/PRO_VS_FLASH_DRIVER_SPEC.md` (new) — co-located driver spec referenced by EVAL_RESULTS. ## Operator escape hatches preserved - `TRIUMVIRATE_DEEPSEEK_MODEL=deepseek-v4-pro` — flip back operator-wide. - `AskAgentRequest.deepseek_model = "deepseek-v4-pro"` — per-call opt-in. Workspace: 292/292 unit tests pass with the new default. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Lands the default-model flip that PR #39 held back pending empirical eval. Data is in
daemon/docs/v1-deepseek/PRO_VS_FLASH_EVAL_RESULTS.md.cfg.modelchanges fromdeepseek-v4-pro→deepseek-v4-flash.AskAgentRequest.deepseek_modeloverride andTRIUMVIRATE_DEEPSEEK_MODEL=deepseek-v4-prooperator escape hatch both preserved.Empirical case (130 consults / 13 task types over 4 eval waves)
v4 specifically probed the four categories where published DeepSeek benchmarks claim a material Pro advantage (long-context cross-file synthesis, complex 15-field nested JSON, hard algorithm with space optimisation, SimpleQA-Verified-style factual recall). Quality tied. The single Pro failure was Pro emitting two valid JSONs concatenated with leaked meta-reasoning prose between them — i.e., breaking the explicit "Output ONLY the JSON object" instruction. The single Flash failure was an assertion-regex mismatch on a correct algorithm output. The only quality signal that emerged across all 130 consults cut against Pro, not for it.
Token economics: Pro used 8.6% more total tokens, 12.4% more reasoning tokens, at indistinguishable answer quality.
Cost (v4 promo pricing): Pro $0.053 vs Flash $0.016 for 20 calls each — ~3.4× more for Pro. Post-promo on 2026-06-01 the gap widens.
Honest test boundaries (called out explicitly in EVAL_RESULTS.md)
This eval is strong evidence for the daemon's current single-shot, bounded-context workload. It does NOT cover:
thinking: enabled,effort: high)If/when the daemon adds multi-turn flows or routes 100K-token prompts, re-evaluate. Both escape hatches above remain available immediately.
Files changed
daemon/crates/mcp-bridge/src/deepseek_config.rs— default value + doc-comment +from_env_returns_documented_defaults_for_all_15_knobstestdaemon/crates/shared-types/src/lib.rs— doc-comment onAskAgentRequest.deepseek_modeldaemon/docs/deepseek-operator-runbook.md— per-call + env-var tablesdaemon/docs/v1-deepseek/PRO_VS_FLASH_EVAL_RESULTS.md(new) — full resultsdaemon/docs/v1-deepseek/PRO_VS_FLASH_DRIVER_SPEC.md(new) — co-located driver spec referenced by EVAL_RESULTSTest plan
cargo test -p mcp-bridge --lib— 106/106 passcargo test --workspace --lib— 292/292 passmcp__triumvirate__ask_agent { agent: "deepseek", message: "..." }post-merge should route to Flash by defaultdeepseek_model: "deepseek-v4-pro"per-call should still route to Pro🤖 Generated with Claude Code