ai(observability): wire the routing shadow to $ai_metric, and close out #10226's remaining sources

[JSONbored]

Part of epic #8286 (Phase 3 — AI observability). Follow-on to #10226, which closed on #10228's merge with two of its six named sources wired.

What actually landed under #10226

#10228 wired the first two deliverables and closed the issue:

• reviewer_vote_fail — per reviewer, at src/queue/ai-review-orchestration.ts
• judgment_agreement / judgment_sample_count — once per review, via judgmentAgreementMetrics

Live in PostHog since the 14:00Z image, so both are confirmed working end to end. But #10226 listed six sources and said "the remaining four follow", and the issue auto-closed on the linked-PR merge before they did. This issue carries the remainder — and narrows it, because three of the four turn out not to be $ai_metric material at all.

Three of the four are structurally ineligible, not merely unwired

capturePostHogAiMetric joins a measurement to the ambient OTel trace and drops the event when there is no trace:

const traceId = currentOtelTraceIds()?.trace_id;
if (!traceId) return;

That guard is deliberate — #10226 asked for it, so an orphan quality score cannot join to nothing and inflate counts. It also decides the scope here. Tracing the three remaining aggregate sources to their callers:

Source	Called from	Ambient AI trace?
loadGatePrecisionReport (src/services/gate-precision.ts)	src/mcp/server.ts:3736, src/review/maintainer-recap-wire.ts:196, src/review/ops-wire.ts:275	No — MCP tool handler, recap and ops reporting
loadPublicRulePrecision (src/review/public-rule-precision.ts)	src/api/routes.ts	No — public HTTP endpoint
buildEvalScoreRecordsFromRulePrecision (src/review/eval-score-records.ts)	src/api/routes.ts:966 (/v1/public/eval-scores)	No — public HTTP endpoint

None of these runs under a review's AI trace. They are corpus-level aggregates over a 90-day window, read on request by a reporting or serving path. Wiring capturePostHogAiMetric into any of them produces a call the helper's own guard discards on every invocation — instrumentation that provably never fires, which is the same "don't instrument ahead of traffic" call already made on #10200.

These are windowed rates, not per-review measurements. If gate and rule precision are wanted in PostHog they belong in a different shape — a periodic gauge keyed on the repo group, not a metric hung off a trace id that does not exist at the call site. That is a separate design question and deliberately not opened here.

Self-consistency is already covered

src/review/self-consistency.ts needs no separate emission. The orchestration call site already folds its samples into the agreement metric:

judgmentAgreementMetrics([...result.reviewerVotes, ...result.selfConsistencySamples])

The N=2–3 rotated-exemplar sampling that would produce a distinct self-consistency signal is explicitly not implemented (see the closing comment in src/review/judgment-agreement.ts) — it needs a per-review AI budget decision and a flag defaulting off, tracked under #8834. There is nothing to emit until that ships.

The one real deliverable: routing shadow

recordRoutingShadow (src/services/reviewer-routing.ts:135) runs at the same orchestration call site as the two metrics that already work, so it does have an ambient trace. It returns a RoutingShadowDecision | null naming the provider evidence-weighted routing would have preferred, against the providers the review actually used — and today that comparison reaches only the audit log and the recap.

As an $ai_metric it joins directly to the cost and model of the review that produced it, which is the whole point: whether the shadow's preference agrees with what ran is only interesting next to what that run cost.

Deliverables

• Emit a routing-shadow agreement metric from the existing recordRoutingShadow call site in src/queue/ai-review-orchestration.ts, using the decision it already returns — no second load of the track records.
• No event when recordRoutingShadow returns null (no decision) or throws — it is fail-safe by contract and must stay that way; the review path cannot be affected.
• Emit nothing when there is no ambient trace, inherited from capturePostHogAiMetric's existing guard rather than re-implemented.
• Tests covering both branches: shadow agrees with the providers that ran, and shadow prefers a different provider.

Not in scope

Gate precision, rule precision and eval score records, for the reason above. $ai_evaluation remains PostHog's output rather than ours (#10232 covers the evaluation side).

[linear-code]

JSO-7147