feat(service-detail): Spotlight section + per-op expansion (PR I)

[coccyx]

Summary

PR I — last of the v0.9.0 faceted-nav thread (siblings: #46, #47, #48, #49, #50). Brings the Honeycomb-BubbleUp-style differential view to Service Detail at two levels.

What changes for the user

1. Service-level Spotlight section

A new "Spotlight" section sits between the health charts and the Operations table. Selection = error spans of this service. Baseline = the rest of the time window.

Answers "why is this service unhealthy?" in one glance — surfaces overrepresented operations, pods, HTTP routes, RPC methods, etc.

2. Per-operation Spotlight expansion

Each Operations table row gets a chevron (▶). Clicking it expands a Spotlight strip scoped to that operation. Useful for drilling into a single op when its error rate or p95 stands out.

Click any value in either Spotlight to drill into the Traces page pre-seeded with the service / operation / lookback / filter.

Engineering wins

• Concurrency limit on streaming queries. Service Detail already fires ~15 queries of its own (RED metrics, time series, status mix, ops, instances, dependencies). Unconditionally fanning out 22 more Spotlight queries blew past the cluster's 20-concurrent-job ceiling and the tail returned 429s. The streaming helpers in `api/search.ts` now cap parallelism at 4 via a small semaphore. Per-attr streaming UX is unchanged — attrs still appear one by one, just paced.

• Curated attribute subset prop on ``. Embedded surfaces can pass an `attributes` prop with a tighter 8-attr list instead of the full 22 from `SPOTLIGHT_ATTRIBUTES`. Service Detail uses: `http.response.status_code`, `http.status_code`, `http.request.method`, `http.route`, `rpc.method`, `rpc.grpc.status_code`, `k8s.pod.name`, `response_flags` — the attributes most likely to differentiate failing vs healthy spans on a single service.

Screenshots

Service-level Spotlight on the frontend service (no 429s after concurrency fix):

`http.status_code 500` ranks at +99.1% in the failing-span selection.

Per-operation expansion with rpc.method, http.route, k8s.pod.name differentials:

The `k8s.pod.name` differential is the killer signal here — if one pod dominates the failure spans, that's the smoking gun.

Test plan

• `npx tsc --noEmit` — clean
• `npm run lint` — 0 errors
• `npm test` — 101/101 passing
• `npm run deploy` — packed + uploaded + provisioned on staging
• Playwright capture spec succeeds (service-level + per-op)
• Health charts populate cleanly (concurrency limit verified)

Validate on staging

1. Open Services → click any service (try `frontend` or `checkoutservice` for richest signal)
2. Verify the Spotlight section appears between the health charts and the Operations table
3. Confirm health charts populate (no 429 errors visible)
4. Click any value in the service-level Spotlight → navigates to Traces pre-filtered
5. In the Operations table, click any row's chevron — Spotlight expands inline
6. Click a value in the per-op Spotlight — Traces opens with service + operation + filter

v0.9.0 readiness

After this PR lands the v0.9.0 faceted-nav thread is complete:

• PR D (feat(faceted-nav): data layer — facet panel + Spotlight queries (PR D of 3) #46) — query helpers + wrappers ✅
• PR E (feat(faceted-nav): Spotlight engine + UI primitives (PR E of 3) #47) — engine + UI primitives ✅
• PR F (feat(faceted-nav): wire Spotlight + facet panel into Search page (PR F of 3) #48) — Search page integration ✅
• PR G (feat(traces): UX guidance + Spotlight as default tab (PR G) #49) — Traces UX guidance + Spotlight default ✅
• PR H (feat(errors): expandable rows with per-error Spotlight strip (PR H) #50) — Errors page Spotlight strip ✅
• PR I (this) — Service Detail Spotlight

Tag v0.9.0 once this merges and CI is green.

Session log

`docs/sessions/2026-05-28-service-spotlight.md`

🤖 Generated with Claude Code