feat: PoC Prometheus/Mimir streaming Search API over HTTP

[itsmylife]

Important

This is an exploration / proof of concept, not a merge candidate. It demonstrates end-to-end Prometheus/Mimir Search API streaming over plain HTTP. The implementation still needs to be split into small, reviewable changes and refined before production use.

Parameter handling across autocomplete and discovery consumers is also provisional. Follow-up work must ensure each call site supplies the correct Search API parameters and semantics instead of adapting values intended for the legacy labels/series endpoints.

What & why

Adds opt-in support for the Prometheus/Mimir NDJSON streaming Search API:

• /api/v1/search/metric_names
• /api/v1/search/label_names
• /api/v1/search/label_values

The Search API provides fuzzy, scored, server-side suggestions and streams results progressively as they are computed.

This PoC integrates it into:

• Query builder metric selection
• Metrics Explorer, including progressive metadata rendering
• Monaco metric, label-name, and label-value completions
• Existing language-provider discovery consumers through the ResourceApiClient interface

The feature is gated by the per-data-source Search API setting. When enabled, upstream Search API errors are surfaced instead of silently falling back to legacy discovery. Default behavior is unchanged while the setting is disabled.

Architecture

This implementation uses a chunked HTTP resource request:

Autocomplete / Metrics Explorer
        ↓
SearchApiClient
        ↓ getBackendSrv().chunked()
Grafana data source resource endpoint
        ↓ streaming CallResource
promlib ExecuteSearch
        ↓
Prometheus/Mimir Search API
        ↑ NDJSON batches
Progressive UI updates

The backend forwards upstream bytes without buffering the complete response. The frontend parses arbitrary chunk boundaries, emits completed batches immediately, and accumulates the final result.

How this differs from the earlier PoC

The earlier streaming PoC (#248) used a persistent Grafana Live WebSocket with a custom request/response protocol, per-session channels, mailboxes, request IDs, and per-slot cancellation.

This implementation explores a simpler, socket-free design:

• Each search uses a standard chunked HTTP resource request.
• getBackendSrv().chunked() streams bytes directly to the frontend.
• SDK CallResource streaming forwards NDJSON without buffering.
• AbortController and request contexts propagate cancellation upstream.
• It requires no Grafana Live configuration, WebSocket connectivity, session channels, or backend mailbox lifecycle.
• Authentication, organization context, URL handling, and tracing remain closer to Grafana's normal HTTP request path.

The trade-off is one HTTP request per search instead of multiplexing searches over a persistent connection. This PR is intended to evaluate whether that simpler operational model is preferable before committing to a production architecture.

Transport behavior

The implementation:

• Uses the public getBackendSrv().chunked() API
• Preserves cancellation through the browser, Grafana, plugin, and upstream request
• Includes the active X-Grafana-Org-Id
• Retries once after a 401 through Grafana's login-ping flow
• Requests and incrementally decodes gzip upstream
• Bounds error-body and NDJSON line sizes
• Detects streams that end without a success trailer
• Preserves partial results after an allowed abrupt EOF
• Removes stale framing headers before forwarding responses

Known PoC gaps

The main remaining work is correct, context-specific parameter construction.

Several integrations currently reuse or adapt values intended for legacy labels/series endpoints. Before this becomes production-ready, each consumer must be audited for the correct use of:

• search[] versus match[]
• Label-name and label-value context
• Time ranges
• Limits and pagination/truncation behavior
• batch_size
• include_metadata
• Sorting behavior
• Query-builder label selectors and Monaco completion context

There are also some getBackendSrv().fetch() behaviors that chunked() cannot reproduce through the public plugin API, including device ID injection, fetch queueing, Query Inspector events, JWT URL-login headers, and automatic error toasts. These are considered low-impact for this discovery PoC but need an explicit product decision.

Dev environment / how to test

The development stack includes Prometheus v3.13.1 with --enable-feature=search-api, a data source with Search API enabled, and a classic data source for comparison.

1. Build the backend:

   mage -v

2. Build the frontend:

   yarn build

3. Start the stack:

   yarn server

4. Use the prometheus-search-api data source in Explore and the query builder. Exercise the metric combobox, Metrics Explorer, and Monaco metric/label completions.

5. Confirm results arrive progressively over a chunked HTTP resource request without a Grafana Live WebSocket.

6. Compare with the prometheus data source and confirm the classic discovery behavior remains unchanged when the toggle is off.

What to focus a review on

1. Whether chunked HTTP is the right transport abstraction
2. Cancellation and partial-stream behavior
3. NDJSON parsing and response-size safeguards
4. Compatibility with existing language-provider consumers
5. Parameter semantics at each UI integration point
6. How this PoC should be split into small, independently reviewable PRs

[bboreham]

Wanted to link to prometheus/proposals#74 so this work is discoverable from there.