feat: PoC Prometheus/Mimir streaming Search API over websocket

[itsmylife]

Important

This is an exploration / proof-of-concept, not a merge candidate. It is a large "monster" branch that we will not merge as-is — the delivery plan (how this gets split into small, reviewable PRs) will be tracked separately. Please review it as a design spike, not as final code.

What & why

Adds experimental support for Prometheus/Mimir's new NDJSON streaming search API (/api/v1/search/{metric_names,label_names,label_values}, gated upstream by --enable-feature=search-api, shipped in Prometheus 3.13.0).

Today autocomplete injects the user's typed text into match[] as a regex (and sometimes runs a client-side fuzzy filter). This PR routes typed text into the new search[] param instead, giving fuzzy, scored, server-side metric/label/value suggestions that stream in progressively as they are computed.

The whole feature is strictly gated behind a per-datasource enableSearchApi toggle and falls back to the existing labels/series clients whenever the toggle is off, the server is unsupported, or Grafana Live is unavailable — so default behavior is unchanged.

⚠️ This uses WebSockets (Grafana Live) — new to this datasource

The streaming transport is a persistent WebSocket via Grafana Live. This datasource has never used a socket-based transport before, so this is a significant new dependency and operational surface (Live must be enabled, websockets must be reachable through any proxies/LB, HA node-affinity considerations, etc.).

This is intentional for the streaming PoC, but we plan to introduce an alternative solution that does not require a socket (e.g. a plain HTTP streaming/chunked path) so the feature does not hard-depend on Grafana Live. That socket-free design will come as a follow-up — feedback on the transport choice is very welcome here.

Architecture

Rather than one Live channel per query, we open one long-lived bidirectional channel per browser session (search/<sessionNonce>) and run a request/response protocol over it:

Autocomplete UI → language_provider → resourceClient getter → SearchApiClient
   subscribe ONCE → getGrafanaLiveSrv()  (search/<uuid>)
   publish {requestId, slotId, endpoint, params} per debounced search
        └─ websocket ─→ Go RunStream loop (+ per-channel mailbox)
                          └─ client.QueryResource (raw *http.Response)
                               └─ Prometheus/Mimir /api/v1/search/*  (NDJSON)
        ←─ SendFrame(requestId, batch…, terminal) ─┘

• The frontend subscribes once at client construction and publishes {requestId, slotId, endpoint, params} per (debounced) search.
• The backend RunStream idles on an in-process mailbox, performs the upstream NDJSON read, and streams frames back down the same channel tagged with requestId.
• cancel-previous is scoped per slot (endpoint + slotId), so independent widgets/panels sharing one datasource instance don't cancel each other; a bounded number of slots run concurrently.
• The per-session nonce guarantees per-browser isolation (own channel, own RunStream, own mailbox) — frames never leak to org-mates.

Dev environment / how to test

The repo ships a ready-to-run Phase 0 environment: a dev Prometheus with the search API flag on, plus Grafana with Live enabled and two provisioned datasources (one with the toggle on, one off for comparison).

1. Build the backend (produces the dist/gpx_* binaries):

   mage -v

2. Build the frontend (or use yarn dev for watch mode):

   yarn install
   yarn build

3. Start the stack (Grafana + dev Prometheus via docker compose):

   yarn server   # == docker compose up --build

   This runs:

   • prom/prometheus:v3.13.1 with --enable-feature=search-api (pinned — the latest tag has a known bug that can resolve to a pre-search-api release).
   • Grafana with GF_LIVE_MAX_CONNECTIONS=100 (Live/websocket is required for the transport) and the externalized plugin loaded in place of core Prometheus.

4. Sanity-check the upstream API returns NDJSON ending in a trailer line:

   curl -g 'http://localhost:9090/api/v1/search/metric_names?limit=5'
   # → {"results":[...]} lines, then {"status":"success","has_more":false}

5. Exercise the feature in Grafana (http://localhost:3000):

   • Use the prometheus-search-api datasource (toggle on). Open Explore / the query builder and type in the metric combobox, label filters, metrics browser, and the metrics explorer modal — you should see fuzzy, scored suggestions stream in as you type.
   • Confirm the WebSocket: in browser devtools → Network → WS, you should see a single long-lived Live connection carrying search/<uuid> publish/frame traffic (not a new connection per keystroke).
   • Switch to the prometheus datasource (toggle off) and confirm the classic labels/series autocomplete behavior is unchanged.
   • Optional: disable Grafana Live / block the websocket and confirm autocomplete still works via the HTTP fallback.

What to focus a review on

The parts most worth scrutiny (highest risk / most novel):

1. Transport choice — the WebSocket/Grafana Live dependency (see the callout above). Is this acceptable, and does the planned socket-free alternative change how we'd want the abstraction to look now?
2. Security of the stream handlers — endpoint allowlist (SSRF protection: only metric_names/label_names/label_values), strict search/<uuid> channel-path validation, param allowlist + bounds, and the fact that forwarded per-request auth is rejected on the stream path (service-level datasource auth only). See pkg/promlib/stream.go + pkg/promlib/resource/search_stream.go.
3. Concurrency & lifecycle correctness — per-slot cancel-previous, bounded concurrent slots, mailbox startup race + reconnect idempotency, and cleanup on stream stop / InstanceDisposer. This is where subtle leaks/races would live.
4. NDJSON parsing edge cases — partial/chunked lines, oversized lines, pre-stream HTTP errors, mid-stream error line, and abrupt EOF (treated as clean completion).
5. Fallback + gating — that everything is a strict no-op when the toggle is off / server unsupported / Live unavailable, and drop-in methods never reject (resolve to []/partial snapshot).

Delivery plan

This branch is a proof of concept. It will not be merged in one shot. The plan for how to break it into small, independently reviewable PRs (backend stream handlers, frontend client, per-component UI wiring, dev-env, socket-free alternative, etc.) will be shared separately.