feat(advisor): executor+advisor strategies on any Chat wire (tool_call default, review_gate)#49
Open
eric-liu-nvidia wants to merge 2 commits into
Open
feat(advisor): executor+advisor strategies on any Chat wire (tool_call default, review_gate)#49eric-liu-nvidia wants to merge 2 commits into
eric-liu-nvidia wants to merge 2 commits into
Conversation
…_gate strategies Signed-off-by: zengyuanl <zengyuanl@nvidia.com>
WalkthroughAdds configurable executor-plus-advisor backends with ChangesAdvisor-backed execution
Estimated code review effort: 5 (Critical) | ~120 minutes Poem
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 3
🧹 Nitpick comments (2)
switchyard/lib/profiles/advisor_config.py (1)
55-58: 🎯 Functional Correctness | 🔵 Trivial | ⚡ Quick win
advisor_tool_nameoverride isn't reflected in the steering/tool prompts.The docstring states
advisor_tool_name"Must match what the steering prompt references ("advisor")," but nothing enforces this. If a caller overridesadvisor_tool_namewithout also overridingexecutor_steering/advisor_tool_description, the executor will be steered to call a tool namedadvisorthat doesn't exist (the synthetic tool is registered under the overridden name inadvisor_tool_def()), silently breaking the tool_call strategy.Consider validating that overriding
advisor_tool_namealso requires overriding the prompts referencing it, or better, template the tool name intoexecutor_steering/advisor_tool_descriptioninstead of hardcoding "advisor" verbatim.Also applies to: 102-108
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@switchyard/lib/profiles/advisor_config.py` around lines 55 - 58, The advisor_tool_name override is not synchronized with prompts that reference the hardcoded "advisor" name. Update the advisor configuration and related symbols such as advisor_tool_def(), executor_steering, and advisor_tool_description so the configured tool name is consistently templated into generated prompts, or validate that custom names require corresponding prompt overrides; ensure default behavior remains unchanged.tests/test_route_bundle.py (1)
1450-1464: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winAdd the positive-path stats assertion alongside the negative one.
This test only checks that
enable_stats: Falseremoves the stats processors. Given the advisor backends use a non-standard_statscompatibility hook instead ofStatsLlmBackendwrapping (peradvisor.py's docstring), it'd be worth also asserting the default (enable_statsunset/True) case actually includesStatsRequestProcessor/StatsResponseProcessor, to catch a regression in that wiring.✅ Proposed additional test
+ def test_enable_stats_default_true_enables_stats_processors(self): + from switchyard.lib.processors.stats_request_processor import ( + StatsRequestProcessor, + ) + from switchyard.lib.processors.stats_response_processor_accumulator import ( + StatsResponseProcessor, + ) + + table = build_route_bundle_table(self._bundle()) + components = list(table.iter_components()) + assert any(isinstance(c, StatsRequestProcessor) for c in components) + assert any(isinstance(c, StatsResponseProcessor) for c in components) + def test_enable_stats_false_disables_stats_processors(self):Based on learnings, "Define verifiable success criteria, write regression tests for bugs and validation changes, and verify each implementation step."
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@tests/test_route_bundle.py` around lines 1450 - 1464, Extend test_enable_stats_false_disables_stats_processors to first build the default bundle with enable_stats unset or true, then assert its components include both StatsRequestProcessor and StatsResponseProcessor; retain the existing false-case assertions confirming both are absent.Source: Learnings
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@switchyard/lib/backends/advisor_loop_backend.py`:
- Around line 176-191: Record the initial executor turn’s token usage before it
is discarded on the REDO path. In _run_executor, capture the usage returned by
_review or the executor response, then add it to the routing statistics before
returning _passthrough(ctx, redo_request); avoid double-counting when _finish is
used for non-REDO verdicts.
In `@switchyard/lib/backends/advisor_tool_call_backend.py`:
- Around line 219-226: At the _HARD_ITERATION_CAP fallback in
AdvisorToolCallBackend, do not pass the last executor turn with advisor tool_use
blocks directly to _finish(), since it exposes and duplicates the internal
planner turn. Before returning, remove all advisor tool-use blocks from turn or
construct a client-safe fallback response, then pass that sanitized result to
_finish(); preserve the existing no-turn guard and warning.
In `@switchyard/lib/profiles/advisor.py`:
- Around line 44-69: Update the advisor profile’s build method to pass the
configured fallback target into ComponentChainProfile via
fallback_target_on_evict, matching the behavior of sibling profiles. Use the
existing advisor configuration field and preserve the current request processor
and backend setup.
---
Nitpick comments:
In `@switchyard/lib/profiles/advisor_config.py`:
- Around line 55-58: The advisor_tool_name override is not synchronized with
prompts that reference the hardcoded "advisor" name. Update the advisor
configuration and related symbols such as advisor_tool_def(), executor_steering,
and advisor_tool_description so the configured tool name is consistently
templated into generated prompts, or validate that custom names require
corresponding prompt overrides; ensure default behavior remains unchanged.
In `@tests/test_route_bundle.py`:
- Around line 1450-1464: Extend
test_enable_stats_false_disables_stats_processors to first build the default
bundle with enable_stats unset or true, then assert its components include both
StatsRequestProcessor and StatsResponseProcessor; retain the existing false-case
assertions confirming both are absent.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: e3c4e573-3ad4-408e-ba3d-26336d0580af
📒 Files selected for processing (18)
.agents/skills/switchyard-lib-core/SKILL.mdAGENTS.mdcrates/switchyard-components/src/backends/anthropic.rscrates/switchyard-components/tests/adversarial_native_backends.rsswitchyard/__init__.pyswitchyard/cli/route_bundle.pyswitchyard/lib/backends/advisor_loop_backend.pyswitchyard/lib/backends/advisor_tool_call_backend.pyswitchyard/lib/cost_estimator.pyswitchyard/lib/profiles/__init__.pyswitchyard/lib/profiles/advisor.pyswitchyard/lib/profiles/advisor_config.pyswitchyard/lib/profiles/advisor_presets.pyswitchyard/lib/profiles/advisor_prompts.pytests/test_advisor_loop_backend.pytests/test_advisor_profile.pytests/test_advisor_tool_call_backend.pytests/test_route_bundle.py
Comment on lines
+176
to
+191
| self._reviewed.add(session) | ||
| verdict, plan = await self._review(messages, turn.content) | ||
| if verdict != "REDO": | ||
| return await self._finish(ctx, turn) | ||
|
|
||
| # REDO: feed the optimized plan back and re-invoke so the executor keeps | ||
| # working instead of stopping. The session is now reviewed, so the redo | ||
| # turn (and everything after it) is plain passthrough. | ||
| redo_messages = [ | ||
| *messages, | ||
| {"role": "assistant", "content": turn.content or ""}, | ||
| {"role": "user", "content": REDO_FEEDBACK_PREFIX + plan}, | ||
| ] | ||
| redo_body = {**body, "messages": redo_messages} | ||
| redo_request = request_with_type("anthropic", redo_body) | ||
| return await self._passthrough(ctx, redo_request) |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟠 Major | 🏗️ Heavy lift
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
# Map the target file and nearby related backend files.
git ls-files 'switchyard/lib/backends/*' 'switchyard/lib/**/*.py' | sed -n '1,200p'
echo
echo '--- outline: advisor_loop_backend.py ---'
ast-grep outline switchyard/lib/backends/advisor_loop_backend.py --view expanded
echo
echo '--- search for _record_internal_turn and related accounting hooks ---'
rg -n "_record_internal_turn|StatsResponseProcessor|usage|token|cost|reviewed|REDO_FEEDBACK_PREFIX|_run_executor|_passthrough" switchyard/lib -g '*.py'Repository: NVIDIA-NeMo/Switchyard
Length of output: 50378
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo '--- advisor_loop_backend.py (relevant slices) ---'
sed -n '1,240p' switchyard/lib/backends/advisor_loop_backend.py | cat -n | sed -n '1,240p'
echo
sed -n '240,340p' switchyard/lib/backends/advisor_loop_backend.py | cat -n | sed -n '1,140p'
echo
echo '--- advisor_tool_call_backend.py (relevant slices) ---'
sed -n '160,340p' switchyard/lib/backends/advisor_tool_call_backend.py | cat -n | sed -n '1,220p'Repository: NVIDIA-NeMo/Switchyard
Length of output: 27242
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo '--- stats_response_processor_accumulator.py ---'
sed -n '1,260p' switchyard/lib/processors/stats_response_processor_accumulator.py | cat -n | sed -n '1,260p'
echo
echo '--- stats_accumulator.py ---'
sed -n '1,260p' switchyard/lib/stats_accumulator.py | cat -n | sed -n '1,260p'
echo
echo '--- stats_response_processor_live_collector.py ---'
sed -n '1,260p' switchyard/lib/processors/stats_response_processor_live_collector.py | cat -n | sed -n '1,260p'Repository: NVIDIA-NeMo/Switchyard
Length of output: 11924
Record the REDO executor turn in stats
On REDO, the first executor turn is dropped before _finish(), so its token usage never reaches stats. That undercounts routing_stats_final.json by one full executor call; capture that usage in _run_executor and record it on the REDO path before returning the passthrough response.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@switchyard/lib/backends/advisor_loop_backend.py` around lines 176 - 191,
Record the initial executor turn’s token usage before it is discarded on the
REDO path. In _run_executor, capture the usage returned by _review or the
executor response, then add it to the routing statistics before returning
_passthrough(ctx, redo_request); avoid double-counting when _finish is used for
non-REDO verdicts.
Comment on lines
+219
to
+226
| log.warning( | ||
| "AdvisorToolCallBackend: hit hard iteration cap (%d) without a terminal " | ||
| "executor turn; returning the last result.", | ||
| self._HARD_ITERATION_CAP, | ||
| ) | ||
| if turn is None: # unreachable: _HARD_ITERATION_CAP >= 1 | ||
| raise RuntimeError("AdvisorToolCallBackend produced no executor turn") | ||
| return await self._finish(ctx, turn) |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
# Map the file before reading it.
ast-grep outline switchyard/lib/backends/advisor_tool_call_backend.py --view expanded
# Inspect the relevant section around the hard-cap fallback and nearby helpers.
sed -n '1,320p' switchyard/lib/backends/advisor_tool_call_backend.py | cat -n
# Find the test that mentions the hard-cap behavior.
rg -n "test_hard_cap_returns_last_round|HARD_ITERATION_CAP|advisor" switchyard -S
# Read the specific test file(s) if found.
fd -a "test*advisor*|*advisor*test*" switchyardRepository: NVIDIA-NeMo/Switchyard
Length of output: 44941
🏁 Script executed:
#!/bin/bash
set -euo pipefail
# Inspect the stats accumulator and response processor behavior around terminal responses.
ast-grep outline switchyard/lib/stats_accumulator.py --view expanded || true
ast-grep outline switchyard/lib/response_processors --view expanded || true
# Find where terminal responses are recorded in the chain.
rg -n "record_success|record_planner_usage|record_response|StatsResponseProcessor|selected_model|backend_call_latency_ms" switchyard -S
# Inspect the hard-cap test(s) in the advisor backend test file.
fd -a "test*advisor*backend*.py|*advisor*backend*test*.py|*advisor*tool*test*.py" switchyardRepository: NVIDIA-NeMo/Switchyard
Length of output: 6661
🏁 Script executed:
#!/bin/bash
set -euo pipefail
# Inspect the response processor that runs after backend.call().
ast-grep outline switchyard/lib/processors/stats_response_processor_live_collector.py --view expanded
sed -n '1,220p' switchyard/lib/processors/stats_response_processor_live_collector.py | cat -n
# Inspect the stats accumulator API to see what backend.record_success does vs response processing.
ast-grep outline switchyard/lib/stats_accumulator.py --view expanded
sed -n '1,260p' switchyard/lib/stats_accumulator.py | cat -n
# Find advisor backend tests.
fd -a --glob '*advisor*backend*test*.py' --glob '*advisor*tool*test*.py' --glob '*advisor*test*.py' switchyardRepository: NVIDIA-NeMo/Switchyard
Length of output: 10775
🏁 Script executed:
#!/bin/bash
set -euo pipefail
# Find the advisor backend test file and the hard-cap test.
fd -a --glob '*advisor*backend*' --glob '*advisor*tool*call*' switchyard
rg -n "hard cap|HARD_ITERATION_CAP|stop_reason|tool_use|advisor unavailable|max_uses exceeded" switchyard -g '*test*.py' -S
# Read the likely test file(s) around the hard-cap case.
for f in $(fd -a --glob '*test*.py' switchyard); do
if rg -q "HARD_ITERATION_CAP|max_uses exceeded|stop_reason.*tool_use|advisor" "$f"; then
echo "=== $f ==="
sed -n '1,260p' "$f" | cat -n
fi
doneRepository: NVIDIA-NeMo/Switchyard
Length of output: 365
🏁 Script executed:
#!/bin/bash
set -euo pipefail
# Inspect the accumulator-backed response processor.
sed -n '1,220p' switchyard/lib/processors/stats_response_processor_accumulator.py | cat -n
# Inspect chain wiring to see whether backend stats and response stats both run.
sed -n '1,260p' switchyard/lib/profiles/chain.py | cat -n
# Search for the hard-cap test by its behavior rather than file name.
rg -n "hit hard iteration cap|produced no executor turn|stop_reason|advisor unavailable|max_uses exceeded|tool_use" switchyard -g '*test*.py' -SRepository: NVIDIA-NeMo/Switchyard
Length of output: 12919
🏁 Script executed:
#!/bin/bash
set -euo pipefail
# Find all uses of record_success to infer its accounting behavior.
rg -n "record_success\(" switchyard -S
# Inspect neighboring backend code that uses the same stats API.
sed -n '500,570p' switchyard/lib/backends/latency_service_llm_backend.py | cat -n
sed -n '230,310p' switchyard/lib/backends/advisor_loop_backend.py | cat -nRepository: NVIDIA-NeMo/Switchyard
Length of output: 9466
Strip the advisor turn at the hard cap
switchyard/lib/backends/advisor_tool_call_backend.py:219-226 — when _HARD_ITERATION_CAP is reached, turn is still the last executor reply that contains advisor tool_use blocks. Returning it through _finish() sends those blocks to the client and counts the same turn twice: once as an internal planner turn, then again as the terminal response. Strip the advisor blocks or synthesize a client-safe fallback before returning.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@switchyard/lib/backends/advisor_tool_call_backend.py` around lines 219 - 226,
At the _HARD_ITERATION_CAP fallback in AdvisorToolCallBackend, do not pass the
last executor turn with advisor tool_use blocks directly to _finish(), since it
exposes and duplicates the internal planner turn. Before returning, remove all
advisor tool-use blocks from turn or construct a client-safe fallback response,
then pass that sanitized result to _finish(); preserve the existing no-turn
guard and warning.
Comment on lines
+44
to
+69
| def build(self) -> ComponentChainProfile: | ||
| """Build the advisor profile runtime for the configured strategy.""" | ||
| from switchyard.lib.processors.reasoning_effort_normalizer import ( | ||
| ReasoningEffortNormalizer, | ||
| ) | ||
|
|
||
| config = self.config | ||
| backend: Any | ||
| if config.strategy == "review_gate": | ||
| from switchyard.lib.backends.advisor_loop_backend import AdvisorLoopBackend | ||
|
|
||
| backend = AdvisorLoopBackend(config) | ||
| else: | ||
| from switchyard.lib.backends.advisor_tool_call_backend import ( | ||
| AdvisorToolCallBackend, | ||
| ) | ||
|
|
||
| backend = AdvisorToolCallBackend(config) | ||
|
|
||
| # Normalize unsupported ``reasoning_effort`` values (Claude Code's | ||
| # ``/effort xhigh`` in particular) before they reach the executor. | ||
| # Same placement as the plan-execute profile. | ||
| return ComponentChainProfile( | ||
| request_processors=[ReasoningEffortNormalizer()], | ||
| backend=backend, | ||
| ) |
There was a problem hiding this comment.
🩺 Stability & Availability | 🟠 Major | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
rg -n "fallback_target_on_evict" switchyard/lib/profiles/chain.py switchyard/lib/profiles/advisor_config.py switchyard/lib/profiles/plan_execute.py
rg -n -B3 -A8 "ContextWindowExceeded" switchyard/lib/profiles/chain.pyRepository: NVIDIA-NeMo/Switchyard
Length of output: 3052
🏁 Script executed:
#!/bin/bash
set -euo pipefail
sed -n '1,260p' switchyard/lib/profiles/advisor.py
printf '\n--- advisor_config ---\n'
sed -n '1,240p' switchyard/lib/profiles/advisor_config.py
printf '\n--- chain ---\n'
sed -n '1,360p' switchyard/lib/profiles/chain.py
printf '\n--- advisor_loop_backend ---\n'
sed -n '1,260p' switchyard/lib/backends/advisor_loop_backend.py
printf '\n--- advisor_tool_call_backend ---\n'
sed -n '1,260p' switchyard/lib/backends/advisor_tool_call_backend.pyRepository: NVIDIA-NeMo/Switchyard
Length of output: 46715
🏁 Script executed:
#!/bin/bash
set -euo pipefail
rg -n "AdvisorProfileConfig|AdvisorConfig|fallback_target_on_evict|ContextWindowExceeded" switchyard/lib switchyard/testsRepository: NVIDIA-NeMo/Switchyard
Length of output: 8436
Pass fallback_target_on_evict through the advisor profile. ComponentChainProfile only evicts/retries ContextWindowExceeded when a fallback target is configured, and this build path never supplies one. Long advisor sessions will therefore fail on overflow instead of falling back like the sibling profiles.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@switchyard/lib/profiles/advisor.py` around lines 44 - 69, Update the advisor
profile’s build method to pass the configured fallback target into
ComponentChainProfile via fallback_target_on_evict, matching the behavior of
sibling profiles. Use the existing advisor configuration field and preserve the
current request processor and backend setup.
…re formats Signed-off-by: zengyuanl <zengyuanl@nvidia.com>
eric-liu-nvidia
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Ports the advisor strategy from the internal dev repo (
advisor-strategybranch, including its final in-flight state) into this repo's profiles architecture, and generalizes the shippingtool_callstrategy into a wire-format-agnostic performance gear — it works with any Chat-shaped executor/advisor pair, not just Anthropic:type: advisorroute /AdvisorProfileConfig— pairs a faster executor with a stronger advisor model. Each tier'sformatselects its wire independently and tiers mix freely:anthropic— served native Anthropic Messages (/v1/messages); the executor call is delegated verbatim toAnthropicNativeBackend, so the client'scache_controlbreakpoints survive and prompt caching is honored.openai— served OpenAI Chat Completions (/chat/completions) viaOpenAiNativeBackend, covering OSS models (Qwen, DeepSeek on vLLM/NIM) and OpenAI. The loop runs natively on OpenAI wire: function-shaped advisor tool,tool_callsinterception,role:"tool"feedback (1:1 with tool_call ids), delta reassembly + final-usage chunk, verbatim stream replay.responsestargets are rejected at config validation (the loop is Chat-shaped);format: autoresolves at build time.AdvisorConfig.strategyselects the design:tool_call(shipping default, any wire) —AdvisorToolCallBackend: a proxy-side re-creation of Anthropic'sadvisor_20260301server tool for gateways/models that cannot run it server-side. A real, parameterlessadvisortool is injected; each advisor call is intercepted before it reaches the client, the advisor model is consulted on the transcript, and the advice loops back as the tool result (up tomax_uses, hard cap 8 executor turns). Wire specifics live in two small private dialect objects (_AnthropicDialect/_OpenAiDialect); the loop itself is format-independent.review_gate(Anthropic executors only, validated) —AdvisorLoopBackend: no tool injected; at the executor's first no-tool-call turn (a plan, or a claim of "done") the advisor reviews once per session → APPROVE (return as-is) or REDO (feed the optimized plan back and re-invoke). Preserved for A/B comparison.advisor.format: Anthropic Messages (httpx, Bearer) or OpenAI Chat Completions (OpenAILLMClient,max_retries=0so failures fall through to the loop's ownfail_open); targetextra_headers/extra_bodyare forwarded so gateway auth and vLLM chat-template hints work.AnthropicNativeBackend::outbound_body) — a native Anthropic request is now forwarded byte-faithful (only the model id is rewritten + operatorextra_bodymerged). The strip/normalize/sanitize steps now apply only to the translated (OpenAI/Responses→Anthropic) path. This preservescontext_management, signed thinking blocks, mid-conversation system turns, client tool ids — and, critically,cache_controlbreakpoints.cost_estimator.py(3 id variants), public exports (AdvisorConfig,AdvisorPresets,AdvisorProfileConfig), an advisor row inswitchyard-lib-core's Quick Reference, and AGENTS.md structure sync.Port mapping (dev repo → here):
lib/factories/advisor/{config,factory,advisor_presets,prompts}.py+SwitchyardRecipes.advisor_recipe→lib/profiles/advisor{_config,,_presets,_prompts}.py, per this repo's profile-owned-construction contract (no factories/recipes/registries). The internal benchmark runbook skill intentionally stays in the internal repo.Why
This migrates the complete advisor line of work — code and the design rationale behind it — so it ships from this repo, and makes it usable across every endpoint family Switchyard fronts. Condensed design history:
v1 — executor-triggered tool loop: a mostly spec-faithful re-creation of Anthropic's advisor tool. Trace analysis showed front-loaded advice suppressed the executor's own test-and-iterate loop — it trusted the plan, one-shot it, and declared done prematurely, losing tasks the unadvised baseline solved by iterating. Retired in favor of:
v2 — once-per-session review gate: the executor works with its own tools; its first no-tool-call turn is reviewed once → APPROVE/REDO. A near-superset of solo behavior (identical until "done", plus one quality gate), so it is downside-protected while catching premature convergence.
v3 — native Anthropic everywhere: the cost lever. Serving the executor through OpenAI-Chat translation strips
cache_control(the OpenAI schema has no such field), so upstream prompt caching never engages and all input bills at the full rate — on agentic workloads (~95% input tokens) that was roughly a 7× cost difference vs the cached path. Both tiers therefore run native/v1/messages, and the executor call is delegated verbatim toAnthropicNativeBackend(this PR's Rust change) so the client's breakpoints reach the upstream. For gateways that authenticate/v1/messageswithAuthorization: Bearerrather than Anthropic'sx-api-key, the preset setsapi_key: ""(suppresses x-api-key) and carries the key inextra_headers.Benchmark outcome (internal coding-agent benchmark, review-gate arm vs executor-solo, both native-Anthropic on a healthy cluster): the gate edged the solo baseline directionally but not statistically significantly — it rescued more tasks than it regressed (premature-convergence losses recovered via REDO), at a small cost and latency premium. Earlier runs suggesting the advisor lost were traced to an infrastructure incident and discarded.
v4 — advisor-as-tool-call restored as the shipping default: ship the strategy that matches the public product surface (Anthropic's
advisor_20260301; OpenRouter's analogousopenrouter:advisor), keeping the gate selectable for A/B. The rebuiltAdvisorToolCallBackendfixes the v1 deviations found by auditing against Anthropic's advisor-tool doc:max_usesfollows the spec's semantics: over-cap calls receive amax_uses exceededtool result and the executor continues (v1 withdrew the tool, which the Anthropic wire format rejects once history containstool_useblocks).One v1 behavior kept deliberately: a turn mixing advisor + client tool calls is regenerated with the sibling client calls dropped (they are re-issued advice-informed on the next iteration). The native API pauses with the client calls pending; a proxy cannot — it can neither execute the client's tools nor return a turn containing a tool the client was never offered.
v5 — generic wire formats (this PR's final shape): Switchyard users front OSS models (Qwen, DeepSeek) on OpenAI-compatible endpoints alongside commercial Claude/OpenAI, and want the advisor as a generic model-performance gear rather than an Anthropic-only feature. This is also where the feature most plausibly pays off: Anthropic's published guidance puts tool-call lifts on weaker executors (strong executors tend to over-call), which is exactly the OSS-executor + strong-advisor pairing. The loop was refactored behind a private wire-dialect seam — the Anthropic dialect delegates to the pre-existing helpers (the original Anthropic test suite passes unmodified, which is the byte-for-byte regression proof), and the OpenAI dialect adds Chat-wire handling with OSS-endpoint hardening: advisor detection by tool-call presence (never
finish_reason, which some servers mislabel), synthesized ids for servers that omit them in stream deltas,"{}"for empty arguments, vendor fields (reasoning_content) whitelisted out of rebuilt turns so strict endpoints accept the replayed history, and missing-usage tolerance.Cost accounting: advisor consults and the intercepted executor turns (which the client never sees, so the outer stats processor cannot count them) are priced into the stats planner bucket (OpenAI cached tokens read from
prompt_tokens_details); each consult emits anadvisor_call={…}audit line (advisor_review={…}on the gate arm).Open follow-ups carried from the dev discussion: (a) verify end-to-end that planner-bucket advisor cost lands in the deployed cost output; (b) request-declared advisor (parsing/stripping a client-sent advisor tools entry, with an opt-in route mode) was scoped but not built — today the advisor is route-configured, always-on, and invisible to the API contract.
How tested
uv run ruff check .cleanuv run mypy switchyardclean (strict)uv run pytest tests/— 2070 passed (100 advisor tests: the Anthropic tool-call suite — unmodified across the format generalization — plus its OpenAI mirror, review-gate contract, both advisor callers via respx, profile wiring + format validation, route-bundletype: advisorincl. OSS-tier and rejected-format cases, presets, exports)cargo test -p switchyard-components(246 tests, incl. the rewritten native-passthrough adversarial suite),cargo clippyclean,cargo fmt --checkcleanChecklist
snake_caseof the primary class.switchyard/__init__.py.__all__(AdvisorConfig,AdvisorPresets,AdvisorProfileConfig).switchyard-lib-coreadvisor row); AGENTS.md backends list updated.Notes for reviewers
reasoning_effort,context_management, unsigned thinking blocks, message-level system turns, unsanitized tool ids) now forward verbatim, and the upstream API decides what to accept. The adversarial tests were rewritten to pin the new contract; translated-path (OpenAI/Responses→Anthropic) stripping is unchanged.strategy:explicitly — the config default istool_call, so an unpinnedtype: advisorroute means the tool-call arm. Notereview_gatenow validates the executor as Anthropic-format (previously an OpenAI executor under the gate would silently mis-wire).AdvisorToolCallBackend.supported_request_typesfollows the executor's wire; the chain's TranslationEngine normalizes any inbound client format (OpenAI Chat, Anthropic Messages, Responses) to it once, so e.g. Claude Code can front a Qwen executor and an OpenAI SDK client can front a Claude executor.LatencyServiceLLMBackend): do not wrap them inStatsLlmBackend;with_runtime_componentsattaches the shared accumulator via the_statshook (covered by a test).🤖 Generated with Claude Code