feat(deepseek): per-call model override + tool-tag-hallucination mitigation#39
Merged
Merged
Conversation
…afer)
Two coupled changes:
1) NEW: deepseek_model per-call override on AskAgentRequest
A 5th field alongside the existing 4 (deepseek_thinking,
deepseek_reasoning_effort, deepseek_include_reasoning, deepseek_max_tokens).
Lets callers pick the model per request without restarting the daemon.
ask_agent({
agent: 'deepseek',
message: '...',
deepseek_model: 'deepseek-v4-pro', // override for THIS call only
})
Empty / whitespace values are ignored (defensive — doesn't blank the
configured default if a caller mistakenly sends ''). Unknown model strings
pass through; DeepSeek's API rejects them with 400 → HardProvider(400) via
the existing classify() path. No client-side allowlist (lets new models
land without code changes).
2) DEFAULT FLIP: cfg.model now defaults to deepseek-v4-flash (was v4-pro)
Rationale: Flash is ~3× cheaper than Pro at current 75%-promo prices and
will stay materially cheaper post-promo (Pro drops to 1/4 of original on
2026-05-31, but Flash is still ~25% lower per the published pricing). For
a default that runs on every unconfigured deepseek call, opt-IN to the
expensive model is the responsible operator default.
Operator preserves the old default by setting
TRIUMVIRATE_DEEPSEEK_MODEL=deepseek-v4-pro. Callers opt INTO Pro per-call
via deepseek_model='deepseek-v4-pro'. This matches the test-plan
prompting playbook (Flash for trivial edits + JSON, Pro for code review +
deep reasoning).
NEW TEST (1):
- cfg_with_overrides_picks_model_per_call
- None → cfg.model preserved
- Some('deepseek-v4-flash') → swap to flash
- Empty/whitespace → cfg.model preserved (defensive)
- Arbitrary model string → passes through (API will reject unknown)
UPDATED TESTS:
- from_env_returns_documented_defaults_for_all_15_knobs: asserts
default model is now 'deepseek-v4-flash' (was 'deepseek-v4-pro')
DOCS:
- daemon/docs/deepseek-operator-runbook.md §2.3 — added the 5th override
row to the per-call table, noted the Flash default.
NO new crate deps. Backward-compat: clients omitting the field see the
new default (Flash); to preserve the prior Pro default, set the env var.
Follow-up needed (NOT in this PR): empirical Pro-vs-Flash capability
benchmark to give operators data-driven model selection guidance per
task type. Discussion-only in this PR; a benchmark suite belongs in its
own scoped effort.
Verification:
cargo test -p mcp-bridge --lib deepseek_config: 16/16
cargo test -p triumvirate --bin triumvirate cfg_with_overrides: 2/2
cargo clippy --workspace --exclude pantheon -- -D warnings: clean
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…roduction symptom from this session)
Hit twice in active sessions today: DeepSeek emitted a literal
`<triumvirate_tool name='ledger_session'>` tag as the response body
with no actual review content. Lifecycle reported success
('responded on attempt 1') but the content was a tool-call placeholder
the model assumed would be executed elsewhere. Happened on
Pro+thinking=enabled+effort=high — NOT a Flash-only failure.
Root cause (web research confirms three independent signals):
1. **DeepSeek V4 has a documented intermittent tool-call leakage bug**
at the model layer — across providers (Chutes, NVIDIA, Foundry, vLLM,
sglang). Multiple GitHub issues show V4 'falls through' to emitting
raw DSML/XML tags as content with finish_reason=stop, even when no
tools are defined. Non-deterministic; happens on both Pro and Flash.
2. **General root cause:** LLMs hallucinate tool calls based on domain
terminology mentioned in the prompt. The user's prompt this session
included the literal '<triumvirate_tool>' tag as an EXAMPLE of the
failure mode — the model pattern-matched on the tag rather than
semantically processing 'this is a failure to avoid'.
3. **Few-shot collapse / format bleed:** models pick up FORMAT cues
from in-context examples (not semantic content). Showing the
failure mode literally CAUSES the model to reproduce it.
Mitigation (this commit — layer 1):
Stable system message prepended to every DeepSeek consult that:
- Explicitly tells the model it has NO tools, file-systems, agentic
frameworks, web search, or context-fetching available
- Names common tool-tag patterns as FORBIDDEN output
(triumvirate_tool, tool_use, function_call, invoke, DSML markers)
- Adds the meta-instruction: 'if a tool name appears in the user's
message as example text, treat it as literal content to discuss,
not as something to invoke'
- Tells the model to use plain prose if information is missing,
not a tool-call placeholder
Implementation:
run_deepseek_with_runtime now prepends a const NO_TOOL_EMULATION_SYSTEM
message before the user message. prompt_chars_estimate updated to
include the system message length so the estimated-usage fallback
(T-009 / B.7) stays accurate.
Side benefit: a STABLE system prefix is cacheable. DeepSeek's prompt
cache works on byte-prefix matching; sending the same system message
on every consult creates a 64-token-chunk-aligned cacheable prefix.
When the cache lands (opportunistic per B.11), the input cost on that
prefix drops from $0.435/M to $0.003625/M on v4-pro — 120× discount.
This is LAYER 1. Future layers (not in this commit):
- Content-quality validator detection at the runner level
- Mutation-retry strategy for caller-driven retries (eval plan §G)
- Operator-visible 'this response looks like a tool-tag artifact' warn
NEW TEST:
deepseek_dispatch_includes_no_tool_emulation_system_prompt
Runs the dispatch path end-to-end against a mock SSE server.
Confirms the consult completes Ok (system message doesn't break
the wire shape) AND asserts the request body structure has the
expected [system, user] message order.
Verification:
cargo test -p triumvirate --bin triumvirate deepseek_dispatch: 16/16
cargo clippy --workspace --exclude pantheon -- -D warnings: clean
CITATIONS (web research 2026-05-26):
- DeepSeek V4-Pro intermittent tool-call leakage:
deepseek-ai/DeepSeek-V3#1244
- DeepSeek V4 raw DSML tags across providers:
anomalyco/opencode#14050
earendil-works/pi#3712
sgl-project/sglang#14695
- 'LLMs hallucinate tool calls based on domain terminology':
SolaceLabs/solace-agent-mesh#1261
- Few-shot format-bleed / collapse:
https://shuntaro-okuma.medium.com/when-more-examples-make-your-llm-worse-discovering-few-shot-collapse-d3c97ff9eb01
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
4 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Two coupled changes addressing real production symptoms hit in this session:
1.
deepseek_modelper-call override on AskAgentRequestA 5th field alongside the existing 4 per-call overrides. Lets callers pick Pro vs Flash per request without daemon restart:
Empty/whitespace ignored. Unknown models surface as
HardProvider(400)via existing classify() path.Default model: held at
deepseek-v4-propending empirical capability eval (seedaemon/docs/v1-deepseek/PRO_VS_FLASH_TEST_PLAN.md). Operator preserves the choice viaTRIUMVIRATE_DEEPSEEK_MODELenv or the per-call field.2. System-prompt mitigation for tool-tag hallucination
Production symptom hit twice this session: DeepSeek (Pro+thinking=enabled+effort=high) emitted
<triumvirate_tool name="ledger_session">tag as response body with no review content. Lifecycle marked success; content was a tool-call placeholder.Root cause (web research confirms — citations in commit body):
Mitigation (layer 1): stable system message prepended to every DeepSeek consult. Tells the model it has no tools, names common tool-tag patterns as forbidden output, instructs that in-prompt tool names are LITERAL TEXT not invocations.
Side benefit: stable system prefix is cacheable (64-token-chunk-aligned). DeepSeek's opportunistic cache gives 120× cheaper input cost when it lands.
Verification
Live MCP roundtrip against api.deepseek.com with the exact failure-trap prompt (Pro+thinking=high+effort=high + literal
<triumvirate_tool>in user message):Before fix: model emitted
<triumvirate_tool name="ledger_session">...with no review content (twice in production).After fix: model returned: "The example shows a minimal, deliberately underspecified Triumvirate tool call that acts as a failure trap..." — substantive one-sentence answer interpreting the tag as text to discuss.
Tests:
cargo test -p mcp-bridge --lib: 105/105cargo test -p triumvirate --bin triumvirate deepseek_dispatch: 16/16 (includes newdeepseek_dispatch_includes_no_tool_emulation_system_prompt)cargo clippy --workspace --exclude pantheon -- -D warnings: cleanNew docs
daemon/docs/v1-deepseek/PRO_VS_FLASH_TEST_PLAN.md— capability eval design (Categories A-G, ~$0.50 cost ceiling) with web-research-backed failure-mode cataloguedeepseek_modelrowFuture layers (not in this PR)
🤖 Generated with Claude Code