feat(plugin)!: Add Switchyard Decision API routing plugin

[bbednarski9]

Warning

BREAKING CHANGE: [Rust source compatibility] This PR adds FlowError::Upstream and new fields to the public AtofEndpointConfig, AtofEndpointSectionConfig, and EmitMarkEventParams types. Downstream exhaustive FlowError matches must handle the new variant, and direct struct literals must initialize the new fields or migrate to the provided constructor/builder APIs. Builder-based mark emission and default Relay configurations remain compatible.

Overview

This adds Relay's first-party integration with the Switchyard Decision API. The plugin routes buffered and streaming LLM requests through an external Switchyard service, executes the selected Relay-owned backend, and records causal model-routing optimization evidence.

Switchyard owns provider-protocol translation through its switchyard-translation Rust library. Relay consumes that library only when the native Switchyard plugin is enabled; default Relay builds do not include either the plugin or the translation dependency.

• I confirm this contribution is my own work, or I have the right to submit it under this project's license.
• I searched existing issues and open pull requests, and this does not duplicate existing work.

Details

Plugin and execution behavior

• Add the standalone nemo-relay-switchyard first-party plugin.
• Normalize stable Relay identity and build switchyard.routing_request.v1 requests.
• Validate selected routes and optional counterfactual baselines against exact Relay-owned backend bindings.
• Support enforce and observe-only modes, six request-materialization modes, bounded retry-aware dispatch, protocol-specific trusted fallbacks, and no retry after the first upstream stream item.
• Record one model-routing optimization contribution only after the selected route commits. Failed decisions, failed provider attempts, and fallback dispatches do not claim routing savings.
• Preserve Switchyard routing metadata, including snapshot freshness, decision source, event count, and turn depth.
• Continue using Relay's existing ATOF exporter and lifecycle; the plugin does not install local routers, feature accumulators, or a duplicate event subscriber.

Translation ownership and optional dependency

Request, buffered-response, and SSE translation across OpenAI Chat Completions, OpenAI Responses, and Anthropic Messages use Switchyard's public switchyard-translation library instead of a Relay-local translation engine.

The dependency is pinned to the immutable topic-branch commit 8f9db9a6a47f848cdff1d262276ba25a8ae9cbc8, which includes the Decision API series, StageRouter baseline/freshness behavior, and Switchyard #46.

The Relay CLI exposes this through --features switchyard:

• Without that feature, nemo-relay-cli contains neither nemo-relay-switchyard nor switchyard-translation.
• With that feature, the plugin and its pinned Switchyard translation library are compiled and registered.
• Relay continues to own target bindings, credentials, dispatch, retries, fallbacks, lifecycle, and observability.
• Relay retains narrow fail-open policy checks for provider extensions that cannot be translated without data loss.

Runtime requirements

The current experimental deployment requires a separately running Switchyard service from the pinned topic commit. The plugin does not start or supervise that service.

Switchyard availability is not a Relay startup requirement. It is a request-time dependency: connection failures, timeouts, malformed decisions, selected-route drift, unsupported extensions, and missing required identity fail open to the configured same-protocol default. ATOF-backed profiles additionally require Relay's ATOF HTTP exporter to target Switchyard's /v1/atof/events endpoint with preserved field names and environment-referenced authentication.

The public examples document this service boundary, verify the exact Switchyard revision before startup, and intentionally remain manual compatibility workflows rather than production orchestration.

Optimization accounting and observability

This PR builds on the plugin-neutral accounting contract merged in NeMo Relay #389. Applied routing contributions carry the selected and baseline model transition, decision identity, backend and tier, attempt, rollout mode, reason, and router metadata. Observe-only decisions remain visible as non-applied evidence.

Relay core combines routing and other plugin contributions at LLM close while retaining explicit token evidence for downstream repricing. Existing requested, decision, retry, fallback, and error routing marks remain available alongside the canonical optimization contribution.

The generic ATIF change that attributes step.model_name to the effective response model is intentionally separated into NeMo Relay #399. This PR still ensures translated provider responses carry the actual selected model; #399 independently controls how ATIF presents it.

Testing summary

• cargo fmt --all -- --check
• cargo clippy -p nemo-relay-switchyard --all-targets -- -D warnings
• cargo clippy -p nemo-relay-cli --features switchyard --all-targets -- -D warnings
• cargo check -p nemo-relay-cli — proves the default CLI builds without Switchyard.
• cargo check -p nemo-relay-cli --features switchyard — proves the opt-in dependency path.
• cargo test -p nemo-relay-switchyard — 31 passed.
• cargo test -p nemo-relay-cli gateway::tests — 33 passed without the feature.
• cargo test -p nemo-relay-cli --features switchyard gateway::tests — 33 passed with the feature.
• cargo test -p nemo-relay-cli --features switchyard --test switchyard_process_e2e — buffered, streaming, retry, and fail-open process boundary passed after the split.
• examples/switchyard/run-real-e2e.sh against the exact pinned Switchyard service — passed with the expected StageRouter route sequence provider/weak, provider/strong, provider/strong.
• The pinned Switchyard translation crate separately passes 61 request, response, streaming, extension, and round-trip tests under Rust 1.93.

Previous feedback addressed

• Split the implementation at repository ownership boundaries.
• Replaced Relay's custom translation engine with the Switchyard-owned reusable library.
• Kept the dependency compile-time optional for users who do not enable the Relay-native Switchyard plugin.
• Removed in-process Random, LLM, and Cascade routers, Relay-side feature accumulation, duplicate Switchyard subscriptions, and the global fallback target.
• Replaced literal credentials with environment-referenced headers and activation-time validation.
• Hardened internal dispatch headers and structured upstream-failure metadata.
• Standardized routing terminology on StageRouter.
• Standardized model-routing evidence on Relay's shared optimization contract instead of adding a second post-call savings mark.
• Moved generic ATIF effective-model attribution into focused PR fix(observability)!: attribute ATIF steps to effective response model #399.
• Moved Fern documentation into focused experimental docs PR docs: document experimental Switchyard integration #400 so this functional PR remains code-focused.
• Kept public examples free of internal-only providers, credentials, Visor source references, and private infrastructure assumptions.

Where should the reviewer start?

1. crates/switchyard/src/component.rs — Decision API lifecycle, exact target and baseline validation, retries, fallback, and terminal route commitment.
2. crates/switchyard/src/translation.rs and stream_translation.rs — thin policy adapters over the Switchyard-owned translation library.
3. crates/cli/Cargo.toml and the cfg(feature = "switchyard") registration paths — compile-time optionality.
4. crates/cli/src/gateway.rs — reserved dispatch headers and structured retry-aware upstream failures.
5. crates/cli/tests/switchyard_process_e2e.rs — process-boundary buffered, streaming, and fail-open coverage.
6. examples/switchyard/README.md and run-real-e2e.sh — pinned experimental service workflow.

Related Issues: (use one of the action keywords Closes / Fixes / Resolves / Relates to)

• Relates to NVIDIA/NeMo-Relay #389.
• Relates to NVIDIA/NeMo-Relay #399.
• Relates to NVIDIA/NeMo-Relay #400.
• Relates to NVIDIA-NeMo/Switchyard #46.
• Paired with the Switchyard Decision API topic series:
  • NVIDIA-NeMo/Switchyard #8
  • NVIDIA-NeMo/Switchyard #9
  • NVIDIA-NeMo/Switchyard #11
  • NVIDIA-NeMo/Switchyard #35

[coderabbitai]

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

• @coderabbitai resume to resume automatic reviews.
• @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

• ▶️ Resume reviews
• 🔍 Trigger review

Walkthrough

Adds Switchyard routing, optimization contribution and summary plumbing, retry-aware gateway handling, environment-based ATOF headers, Hermes/session wiring, and example configs and scripts with expanded coverage.

Changes

Optimization contracts and bindings

Layer / File(s)	Summary
Optimization data contracts and API surface crates/types/src/api/llm.rs, crates/types/src/codec/optimization.rs, crates/types/src/codec/response.rs, crates/core/src/api/mod.rs, crates/core/src/codec/mod.rs, crates/core/src/codec/streaming.rs, crates/core/src/codec/traits.rs, crates/core/src/api/scope.rs, crates/core/src/api/shared.rs, crates/core/src/api/runtime/state.rs, crates/core/src/codec/anthropic.rs, crates/core/src/codec/openai_chat.rs, crates/core/src/codec/openai_responses.rs, crates/ffi/src/api/mod.rs, crates/node/src/api/mod.rs, crates/node/src/callable.rs, crates/plugin/src/lib.rs, crates/python/src/py_types/*.rs, crates/worker/src/lib.rs, go/nemo_relay/*.go	Defines optimization contribution and summary types, extends annotated responses and intercept outcomes, and exposes the new contract surface through core codecs, scope mark params, FFI, Node, Python, plugin, worker, and Go bindings.
Recorder and lifecycle wiring crates/core/src/api/optimization.rs, crates/core/src/api/llm.rs, crates/core/src/api/runtime/state.rs, crates/core/src/api/runtime/subscriber_dispatcher.rs, crates/core/src/stream.rs, crates/core/tests/integration/middleware_tests.rs, crates/core/tests/integration/stream_tests.rs, crates/core/tests/integration/pipeline_tests.rs, crates/core/tests/unit/*	Implements the recorder, scopes request-intercept collection, emits optimization marks, finalizes summaries on LLM end and stream end, and covers the lifecycle with unit and integration tests.
Observability projections crates/core/src/observability/{atif,openinference,otel}.rs, crates/core/tests/unit/observability/*, crates/core/tests/unit/atif_tests.rs	Projects optimization summaries into ATIF, OpenInference, and OTEL attributes and updates the related fixtures and tests.
Contract fixtures and conformance tests crates/types/tests/*, crates/core/tests/*, crates/ffi/tests/*, crates/node/tests/*, crates/python/tests/*, crates/plugin/tests/*	Adds round-trip and coverage tests for optimization contracts, recorder behavior, and language/runtime bindings.
Estimated code review effort: 5 (Critical)	~120 minutes

Switchyard and CLI wiring

Layer / File(s)	Summary
Switchyard crate contracts and translation crates/switchyard/**	Adds the Switchyard crate root, routing contracts, request/response translation, and streaming transcoding implementation.
CLI Switchyard gateway and identity wiring crates/cli/src/{doctor,error,gateway,launcher,plugins,plugins/config_io,plugins/editor_model,server,session}.rs, crates/cli/Cargo.toml, crates/cli/tests/coverage/{gateway,launcher,session}_tests.rs, crates/cli/tests/switchyard_process_e2e.rs	Registers and validates Switchyard in the CLI, enriches request and tool metadata with routing identity headers, makes the launcher Hermes-aware, and adds the gateway process-boundary e2e test.
Retry-aware upstream failures and dispatch overrides crates/core/src/error.rs, crates/cli/src/{error,gateway}.rs, crates/ffi/src/error.rs, crates/core/src/plugin/dynamic/native.rs, crates/cli/tests/coverage/gateway_tests.rs	Adds structured upstream failure modeling, retry-aware gateway dispatch, effective dispatch overrides, and CLI/FFI/runtime status mapping for provider failures.
ATOF environment-variable header resolution crates/core/src/observability/atof.rs, crates/core/src/observability/plugin_component.rs, crates/cli/src/doctor.rs, crates/core/tests/unit/observability/{atof,plugin_component}_tests.rs	Adds header_env to endpoint configs and resolves headers from environment variables across HTTP/WebSocket/NDJSON transports, with CLI validation support and tests.
Workspace and gitignore wiring Cargo.toml, .gitignore	Adds the Switchyard crate to the workspace and ignores /artifacts/.

Switchyard examples, smoke tests, and process e2e

Layer / File(s)	Summary
Example configs and docs crates/switchyard/README.md, examples/switchyard/{plugins.toml,real-e2e-plugins.toml,inferencehub-stage-router-plugins.toml.in,hermes-ollama-plugins.toml,real-e2e-profiles.yaml,inferencehub-stage-router-profiles.yaml,hermes-ollama-profiles.yaml}	Adds Switchyard configuration examples, routing profiles, and crate documentation for local and remote backends.
Smoke/e2e helpers and runners examples/switchyard/{e2e-common.sh,fake_upstream.py,otel-collector.yaml,run-hermes-ollama-smoke.sh,run-inferencehub-stage-router-smoke.sh,run-real-e2e.sh}	Adds helper scripts, a fake upstream server, collector config, and Bash smoke/e2e runners that exercise Switchyard end to end.
Gateway process-boundary e2e crates/cli/tests/switchyard_process_e2e.rs	Adds an e2e test spawning the gateway process with fake decision/provider Axum servers verifying routing decisions, fallback on 503, and internal-header stripping.

Estimated code review effort: 5 (Critical) | ~120 minutes

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 44.96% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Title check	✅ Passed	Uses a valid Conventional Commits breaking-feat format and accurately summarizes the new Switchyard routing plugin.
Description check	✅ Passed	Includes the required overview, details, reviewer start, checklist, and related issues sections with substantive content.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

_{Comment @coderabbitai help to get the list of available commands.}

[github-actions]

Fern docs preview: https://nvidia-preview-pull-request-369.docs.buildwithfern.com/nemo/relay (​https://nvidia-preview-pull-request-369.docs.buildwithfern.com/nemo/relay​)

[bbednarski9]

reference arch:

flowchart TD
    S["Relay startup"] --> V["Validate configuration, URLs, bindings, and secret environment variables"]
    V --> C["Build HTTP clients"]
    C --> R["Relay becomes ready"]
    R -. "No startup probe" .-> SY["Switchyard server"]

    Q["LLM request"] --> D["POST Decision API"]
    D -->|Valid decision| P["Dispatch selected provider"]
    D -->|Unavailable, timeout, invalid response, or binding drift| F["Dispatch trusted same-protocol fallback"]

    E["ATOF event"] --> A["Best-effort POST"]
    A -->|Switchyard unavailable| L["Log failure; routing continues"]

Loading

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[willkill07]

As written, this will only work through the sidecar and not through any native integrations (like Hermes or LC Deep Agents).

I appreciate the non-breaking API changes for FFI / Python / Node

Since it's experimental and likely subject to change, I'm approving with minor cleanliness nits.