Skip to content

feat(deepseek): flip operator default to v4-flash (post-eval)#40

Merged
michaeljboscia merged 1 commit into
mainfrom
feat/deepseek-default-flip-to-flash
May 27, 2026
Merged

feat(deepseek): flip operator default to v4-flash (post-eval)#40
michaeljboscia merged 1 commit into
mainfrom
feat/deepseek-default-flip-to-flash

Conversation

@michaeljboscia

Copy link
Copy Markdown
Owner

Summary

Lands the default-model flip that PR #39 held back pending empirical eval. Data is in daemon/docs/v1-deepseek/PRO_VS_FLASH_EVAL_RESULTS.md.

  • Default cfg.model changes from deepseek-v4-prodeepseek-v4-flash.
  • Per-call AskAgentRequest.deepseek_model override and TRIUMVIRATE_DEEPSEEK_MODEL=deepseek-v4-pro operator escape hatch both preserved.
  • Workspace: 292/292 unit tests pass.

Empirical case (130 consults / 13 task types over 4 eval waves)

Wave Sample Pro pass Flash pass
v1 6 × 1 6/6* 6/6*
v3 9 × 5 44/45 44/45
v4 (Pro-favoring) 4 × 5 19/20 19/20

v4 specifically probed the four categories where published DeepSeek benchmarks claim a material Pro advantage (long-context cross-file synthesis, complex 15-field nested JSON, hard algorithm with space optimisation, SimpleQA-Verified-style factual recall). Quality tied. The single Pro failure was Pro emitting two valid JSONs concatenated with leaked meta-reasoning prose between them — i.e., breaking the explicit "Output ONLY the JSON object" instruction. The single Flash failure was an assertion-regex mismatch on a correct algorithm output. The only quality signal that emerged across all 130 consults cut against Pro, not for it.

Token economics: Pro used 8.6% more total tokens, 12.4% more reasoning tokens, at indistinguishable answer quality.
Cost (v4 promo pricing): Pro $0.053 vs Flash $0.016 for 20 calls each — ~3.4× more for Pro. Post-promo on 2026-06-01 the gap widens.

Honest test boundaries (called out explicitly in EVAL_RESULTS.md)

This eval is strong evidence for the daemon's current single-shot, bounded-context workload. It does NOT cover:

  • Multi-turn agentic tool-use loops (Pro's Terminal-Bench edge: 67.9 vs 56.9)
  • 100K+ token prompts with deep-needle retrieval (Pro's 1M-context edge)
  • Judge-LLM quality grading (only binary contains/regex assertions used)
  • Reasoning-effort sweeps (both ran thinking: enabled, effort: high)

If/when the daemon adds multi-turn flows or routes 100K-token prompts, re-evaluate. Both escape hatches above remain available immediately.

Files changed

  • daemon/crates/mcp-bridge/src/deepseek_config.rs — default value + doc-comment + from_env_returns_documented_defaults_for_all_15_knobs test
  • daemon/crates/shared-types/src/lib.rs — doc-comment on AskAgentRequest.deepseek_model
  • daemon/docs/deepseek-operator-runbook.md — per-call + env-var tables
  • daemon/docs/v1-deepseek/PRO_VS_FLASH_EVAL_RESULTS.md (new) — full results
  • daemon/docs/v1-deepseek/PRO_VS_FLASH_DRIVER_SPEC.md (new) — co-located driver spec referenced by EVAL_RESULTS

Test plan

  • cargo test -p mcp-bridge --lib — 106/106 pass
  • cargo test --workspace --lib — 292/292 pass
  • Manual smoke: mcp__triumvirate__ask_agent { agent: "deepseek", message: "..." } post-merge should route to Flash by default
  • Manual smoke: same with deepseek_model: "deepseek-v4-pro" per-call should still route to Pro

🤖 Generated with Claude Code

…val)

This lands the default flip held back by PR #39 ("default flip happens after
the eval produces a data-driven recommendation"). The recommendation is now
in: `daemon/docs/v1-deepseek/PRO_VS_FLASH_EVAL_RESULTS.md`.

## Empirical summary (full data in EVAL_RESULTS.md)

Across 130 consults / 13 distinct task types over four eval waves on
promptfoo / DeepSeek `https://api.deepseek.com/v1`:

| Wave | Sample | Pro pass | Flash pass | Notes |
|---|---|---|---|---|
| v1 | 6 × 1 | 6/6* | 6/6* | *after fixing assertion bugs |
| v3 | 9 × 5 | 44/45 | 44/45 | both misses were assertion false-positives |
| v4 | 4 × 5 | **19/20** | **19/20** | 4 tasks specifically designed for Pro to win |

v4 covers the four categories where published DeepSeek benchmarks claim a
material Pro advantage:
- S10 long-context cross-file synthesis (Pro 97% needle / Flash drops in middle)
- S11 complex 15-field nested JSON (Flash reportedly fails on 10+ field schemas)
- S12 algorithm design with space-complexity constraint (multi-step reasoning)
- S13 SimpleQA-Verified-style factual recall (Pro 57.9 / Flash 34.1)

**Quality: tied (19/20 vs 19/20).** The one Pro failure (S11 trial 1) was Pro
emitting two valid JSON objects concatenated with leaked meta-reasoning prose
between them — i.e., Pro broke the explicit "Output ONLY the JSON object"
instruction. The one Flash failure (S12) was an assertion-regex mismatch on
correct output. The only quality signal that emerged across all 130 consults
cut against Pro, not for it.

**Cost: 3.4× more for Pro** at v4 promo pricing
(Pro $0.053 vs Flash $0.016 for the 20-call wave). Post-promo on 2026-06-01
the gap widens.

**Token economics: Pro uses 8.6% more total tokens, 12.4% more reasoning
tokens** at indistinguishable answer quality.

## What this changes

- `deepseek_config.rs` — default model `"deepseek-v4-pro"` → `"deepseek-v4-flash"`.
- `from_env_returns_documented_defaults_for_all_15_knobs` — assert updated.
- `shared-types/src/lib.rs` — doc-comment on `AskAgentRequest.deepseek_model`
  updated to reflect Flash as default.
- `daemon/docs/deepseek-operator-runbook.md` — env-var + per-call tables updated.
- `daemon/docs/v1-deepseek/PRO_VS_FLASH_EVAL_RESULTS.md` (new) — full data,
  methodology, and the explicit boundaries of what this eval does NOT cover
  (no multi-turn agentic flows; context ≤ 3K tokens; binary assertions;
  n=5 per task). For the daemon's current single-shot workload, the empirical
  case is firm. If multi-turn or 100K-token flows are added later, re-evaluate.
- `daemon/docs/v1-deepseek/PRO_VS_FLASH_DRIVER_SPEC.md` (new) — co-located
  driver spec referenced by EVAL_RESULTS.

## Operator escape hatches preserved

- `TRIUMVIRATE_DEEPSEEK_MODEL=deepseek-v4-pro` — flip back operator-wide.
- `AskAgentRequest.deepseek_model = "deepseek-v4-pro"` — per-call opt-in.

Workspace: 292/292 unit tests pass with the new default.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@michaeljboscia
michaeljboscia merged commit 612d6dd into main May 27, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant