feat(latency-service): credential policy, request-type routing, metric attribution, response fidelity, and shared session-affinity store

.agents/skills/switchyard-lib-core/SKILL.md

@@ -39,11 +39,12 @@ right validation set. If the change is driven by a launcher need, also read
 | An OpenAI-compatible provider target such as NVIDIA Inference Hub or OpenRouter | Use the existing OpenAI-compatible backend/profile with `base_url`, `api_key`, and model id wiring. Add a new backend only when the provider has a real wire-format, auth, retry, or health contract that cannot fit that path. |
 | Direct Rust component bindings | Add concrete PyO3 classes under `crates/switchyard-py/src/component_bindings/`, keep config bindings near the component binding that consumes them, and expose them lazily from `switchyard_rust/components.py`. Do not keep growing `core_bindings.rs` or `switchyard_rust/core.py` with concrete component classes. |
 | Route YAML / model dispatch | Use `switchyard/cli/route_bundle.py` and `switchyard/lib/route_table_builders.py`. They build `RouteTable` entries from profile-backed runtimes and keep launchers plus `switchyard serve --routing-profiles` on one path. |
+| Shared/persistent session-affinity pins across workers or pod churn | Configure the latency route with `session_affinity: true` + `affinity_store: redis` + `affinity_store_url` (optional `affinity_store_ttl_seconds`, `affinity_key_prefix`). `SessionAffinity` keeps the Rust `SessionCache` as L1 and reads/writes through the `AffinityPinStore` L2 (`switchyard/lib/redis_pin_store.py`), fail-open behind a 0.1s socket timeout and a 3-failure/10s-cooldown circuit breaker (`switchyard_affinity_l2_breaker_open` gauge). Requires the `switchyard[affinity-redis]` extra. |
 | Stats / telemetry | Reuse `StatsRequestProcessor`, `StatsResponseProcessor`, `StatsLlmBackend`, and `StatsAccumulator`. A profile config should thread one accumulator through all three when stats are enabled. Do not write a parallel collector. |
 | A fixed-path endpoint contributed by per-route components | Set `Endpoint.register_once = True`; `build_switchyard_app(...)` mounts the first instance while still running every component's lifecycle. Leave the default `False` for configurable endpoint classes that may mount distinct instances. |
 | Per-endpoint attribution on `/metrics` for a Python backend that can't be wrapped by `StatsLlmBackend` | Set `ctx.selected_model = endpoint_id` before returning the response. Also set `ctx.backend_call_latency_ms = upstream_call_ms` so the response processor can compute routing overhead. `LatencyServiceLLMBackend.call` is the reference. |
 | State metrics on `/metrics` | Register a `PrometheusEmitter` via `switchyard.lib.endpoints.prometheus_emitter.register(...)` and unregister on `shutdown()`. This is for backend-owned state, not request-flow counters. |
-| Error-rate / retry-recovery counters | Use `switchyard.lib.endpoints.outcome_metrics`. FastAPI middleware records client outcomes. A retrying Python backend records each upstream attempt itself (and `record_retry_recovered()`) and sets `CTX_UPSTREAM_ATTEMPTS_RECORDED` so the endpoint skips its fallback — `LatencyServiceLLMBackend.call` is the reference. Single-attempt backends (Rust native / passthrough / multi) record nothing themselves; the endpoint fallback (`record_upstream_attempt_success` / `record_upstream_attempt_failure` in `upstream_error.py`, called from `dispatch_chat_request` and `handle_chain_exception`) counts their one attempt. Don't add a `model` label — these counters are layer-aggregate. Keep labels bounded. |
+| Error-rate / retry-recovery counters | Use `switchyard.lib.endpoints.outcome_metrics`. FastAPI middleware records client outcomes. A retrying Python backend records each upstream attempt itself (and `record_retry_recovered()`) and sets `CTX_UPSTREAM_ATTEMPTS_RECORDED` so the endpoint skips its fallback — `LatencyServiceLLMBackend.call` is the reference. Single-attempt backends (Rust native / passthrough / multi) record nothing themselves; the endpoint fallback (`record_upstream_attempt_success` / `record_upstream_attempt_failure` in `upstream_error.py`, called from `dispatch_chat_request` and `handle_chain_exception`) counts their one attempt. Don't add a `model` label — these counters are layer-aggregate. Keep labels bounded. For a per-model error breakdown, emit a backend-owned counter via the `PrometheusEmitter` instead (labels bounded to config-derived ids: the route id `config.route_model` + endpoint ids, else the `other` sentinel) — `LatencyServiceLLMBackend._render_prometheus_lines` exposes `switchyard_latency_upstream_attempts_total{requested_model,upstream_model,outcome,code}` next to the aggregate, leaving the shared counter model-free. |
 | Per-event error log | Use `switchyard.lib.endpoints.upstream_error_log.log_upstream_attempt_failure(...)` on the failure path. Events belong in logs/traces, not Prometheus sample timestamps. |
 | CLI launcher integration | Build one profile-backed `SwitchyardApp` with `build_tier_passthrough_switchyard(...)` for single-target mode, or merge route YAML with `load_route_bundle_table(...)`. Hand the result to `build_switchyard_app`. |
 | A new preset | Put preset helpers beside the profile config they produce, under `switchyard/lib/profiles/`. Presets should return typed config objects, not runnable chains. |

.secrets.baseline

@@ -98,10 +98,6 @@
       "path": "detect_secrets.filters.common.is_baseline_file",
       "filename": ".secrets.baseline"
     },
-    {
-      "path": "detect_secrets.filters.common.is_ignored_due_to_verification_policies",
-      "min_level": 2
-    },
     {
       "path": "detect_secrets.filters.heuristic.is_indirect_reference"
     },
@@ -387,7 +383,7 @@
         "filename": "switchyard/lib/proxy_context.py",
         "hashed_secret": "70ca3ed5e568a3c9180bbb4496f1a2072e4ef36f",
         "is_verified": false,
-        "line_number": 42
+        "line_number": 43
       }
     ],
     "switchyard/lib/request_metadata.py": [
@@ -593,35 +589,35 @@
         "filename": "tests/test_latency_service_llm_backend.py",
         "hashed_secret": "3acfb2c2b433c0ea7ff107e33df91b18e52f960f",
         "is_verified": false,
-        "line_number": 58
+        "line_number": 63
       },
       {
         "type": "Secret Keyword",
         "filename": "tests/test_latency_service_llm_backend.py",
         "hashed_secret": "35188bb026d29646649697a020aa4803df869146",
         "is_verified": false,
-        "line_number": 869
+        "line_number": 947
       },
       {
         "type": "Secret Keyword",
         "filename": "tests/test_latency_service_llm_backend.py",
         "hashed_secret": "8294c1af1a5c47187ddf7f15c6fd13f0d4619bec",
         "is_verified": false,
-        "line_number": 931
+        "line_number": 1009
       },
       {
         "type": "Secret Keyword",
         "filename": "tests/test_latency_service_llm_backend.py",
         "hashed_secret": "3922bcd29ff79bae723278599532a5cfd75e9b9f",
         "is_verified": false,
-        "line_number": 1090
+        "line_number": 1244
       },
       {
         "type": "Secret Keyword",
         "filename": "tests/test_latency_service_llm_backend.py",
         "hashed_secret": "d50bd504c3d7f5dae7d6bf0f06e18280954ba408",
         "is_verified": false,
-        "line_number": 1165
+        "line_number": 1319
       }
     ],
     "tests/test_latency_service_stats_metrics.py": [
@@ -761,7 +757,7 @@
         "filename": "tests/test_llm_client.py",
         "hashed_secret": "c77a24bfec49c5b2e2b22fab5369ff079fa4431a",
         "is_verified": false,
-        "line_number": 53
+        "line_number": 63
       }
     ],
     "tests/test_outcome_metrics.py": [
@@ -1101,7 +1097,7 @@
         "filename": "tests/test_upstream_error_passthrough.py",
         "hashed_secret": "48854c46907b8f99e4c4c711f3fb7336a93bc290",
         "is_verified": false,
-        "line_number": 91
+        "line_number": 95
       }
     ],
     "tests/test_user_config.py": [
@@ -1193,5 +1189,5 @@
       }
     ]
   },
-  "generated_at": "2026-07-06T23:24:20Z"
+  "generated_at": "2026-07-08T09:06:49Z"
 }

crates/switchyard-components/src/stats/usage.rs

@@ -47,16 +47,54 @@ pub fn openai_chat_usage_from_stream_event(event: &StreamEvent) -> Option<TokenU
 }
 
 /// Extracts OpenAI Responses streaming usage from an event.
+///
+/// Fidelity-preserving backends yield raw SSE frame *strings*
+/// (`StreamEvent::Text`) instead of decoded JSON events; those frames are
+/// parsed here so usage accounting survives verbatim passthrough.
 pub fn openai_responses_usage_from_stream_event(event: &StreamEvent) -> Option<TokenUsage> {
-    let StreamEvent::Json(value) = event else {
-        return None;
-    };
+    match event {
+        StreamEvent::Json(value) => usage_from_responses_value(value),
+        StreamEvent::Text(text) => sse_data_payloads(text)
+            .iter()
+            .find_map(usage_from_responses_value),
+    }
+}
+
+/// Reads `response.usage` from one decoded Responses stream event.
+fn usage_from_responses_value(value: &Value) -> Option<TokenUsage> {
     value
         .get("response")
         .and_then(|response| response.get("usage"))
         .and_then(usage_from_candidate)
 }
 
+/// Parses the JSON `data:` payload(s) out of a raw SSE frame string.
+///
+/// Per the SSE contract, a frame's `data:` lines join with newlines to form
+/// one payload; a single leading space after the colon is stripped. Comment
+/// frames, `[DONE]` sentinels, and non-JSON payloads yield nothing.
+fn sse_data_payloads(text: &str) -> Vec<Value> {
+    let mut payloads = Vec::new();
+    for block in text.split("\n\n") {
+        let data_lines: Vec<&str> = block
+            .split('\n')
+            .filter_map(|line| line.strip_prefix("data:"))
+            .map(|value| value.strip_prefix(' ').unwrap_or(value))
+            .collect();
+        if data_lines.is_empty() {
+            continue;
+        }
+        let data = data_lines.join("\n");
+        if data.trim() == "[DONE]" {
+            continue;
+        }
+        if let Ok(parsed) = serde_json::from_str::<Value>(&data) {
+            payloads.push(parsed);
+        }
+    }
+    payloads
+}
+
 /// Accumulates Anthropic streaming usage and commits once at `message_stop`.
 #[derive(Clone, Copy, Debug, Default)]
 pub struct AnthropicStreamUsage {

crates/switchyard-components/tests/stats_processors.rs

@@ -448,6 +448,47 @@ async fn streaming_response_is_forwarded_and_records_nested_responses_usage() ->
     Ok(())
 }
 
+// Raw SSE frame strings (verbatim Responses passthrough) still record usage.
+#[tokio::test]
+async fn raw_sse_frame_responses_stream_passes_through_and_records_usage() -> Result<()> {
+    let accumulator = StatsAccumulator::new();
+    let processor = StatsResponseProcessor::new(accumulator.clone());
+    let mut ctx = ProxyContext::new();
+    record_backend_selection(&mut ctx, ModelId::new("raw-frame-model")?);
+    ctx.insert(StatsRequestStart::now());
+
+    let events = vec![
+        StreamEvent::Text(
+            "event: response.output_text.delta\n\
+             data: {\"type\":\"response.output_text.delta\",\"delta\":\"hi\"}\n\n"
+                .to_string(),
+        ),
+        StreamEvent::Text(": keep-alive\n\n".to_string()),
+        StreamEvent::Text(
+            "event: response.completed\n\
+             data: {\"type\":\"response.completed\",\"response\":{\"usage\":\
+             {\"input_tokens\":8,\"output_tokens\":13,\
+             \"input_tokens_details\":{\"cached_tokens\":5}}}}\n\n"
+                .to_string(),
+        ),
+    ];
+    let stream_events = events.clone();
+    let stream = futures_util::stream::iter(stream_events.into_iter().map(Ok));
+    let response = ChatResponse::OpenAiResponsesStream(Box::pin(stream));
+
+    let processed = processor.process(&mut ctx, response).await?;
+    let drained = drain_responses_stream(processed).await?;
+
+    // Frames pass through byte-identical; usage still lands in the snapshot.
+    assert_eq!(drained, events);
+    let snapshot = accumulator.snapshot()?;
+    let model = model_stats(&snapshot, "raw-frame-model")?;
+    assert_eq!(model.prompt_tokens, 8);
+    assert_eq!(model.completion_tokens, 13);
+    assert_eq!(model.cached_tokens, 5);
+    Ok(())
+}
+
 // OpenAI streams should record the first real usage block only.
 #[tokio::test]
 async fn openai_chat_stream_records_first_usage_chunk_only_with_details() -> Result<()> {