Streaming labels values PR 3 - #15347
Merged
Merged
Conversation
tcp13equals2
added
the
changelog-not-needed
tcp13equals2
changed the title
4 tasks
tcp13equals2
force-pushed
the
streaming-labels-values-PR-2
branch
from
May 18, 2026 03:05
d0a2d1a to
7331937
Compare
tcp13equals2
force-pushed
the
streaming-labels-values-PR-3
branch
4 times, most recently
from
May 18, 2026 06:28
a50005a to
06b3a65
Compare
tcp13equals2
marked this pull request as ready for review
May 18, 2026 06:37
tcp13equals2
force-pushed
the
streaming-labels-values-PR-2
branch
from
May 19, 2026 01:25
fd387fc to
f239bd3
Compare
tcp13equals2
force-pushed
the
streaming-labels-values-PR-3
branch
from
May 19, 2026 02:25
a1d1b8a to
faf7f73
Compare
tcp13equals2
force-pushed
the
streaming-labels-values-PR-2
branch
from
May 20, 2026 07:01
f4e2ea0 to
f2c9f0a
Compare
tcp13equals2
force-pushed
the
streaming-labels-values-PR-3
branch
2 times, most recently
from
May 21, 2026 01:47
4f60313 to
8b402b3
Compare
tcp13equals2
force-pushed
the
streaming-labels-values-PR-2
branch
from
May 22, 2026 02:11
6f457cd to
bb35558
Compare
tcp13equals2
force-pushed
the
streaming-labels-values-PR-3
branch
from
May 22, 2026 02:18
8b402b3 to
46fff02
Compare
4 tasks
ldufr
previously approved these changes
May 22, 2026
Wires multiQuerier as the cross-source fan-out point for streaming label/value search. New file pkg/querier/multi_searcher.go declares a local mimirSearcher interface that matches the Mimir-extended Searcher signature (extra *streaminglabelvalues.Params argument — see spec invariant 9 for why params travel separately from the opaque hints.Filter). distributorQuerier and blocksStoreQuerier (added in PR #2) both satisfy it. Per-tenant clamping reuses the existing MaxLabelNamesLimit / MaxLabelValuesLimit ceilings — search returns label names/values just like the existing endpoints, so a separate per-tenant cap would be configuration churn for no real gain (spec § PR #3 left this open; "reuse" is the spec's preference). When the clamp fires, an extra warning-only SearchResultSet is added to the fan-out so the merge primitive surfaces the MaxLimitError via Warnings(), mirroring how the existing LabelNames/LabelValues path returns warnings alongside the result. Time-range short-circuit and child type-assertion failures return storage.EmptySearchResultSet / storage.ErrSearchResultSet so the merge composes cleanly without special-casing in callers. Reachable but unused — no HTTP route yet (PR #4). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…ough Symmetric with the existing LabelNames/LabelValues pass-through. Memory tracking is intentionally skipped on the search path: the streaming SearchResultSet is already capped by the per-tenant Limit applied at the merge layer in multiQuerier, so byte-counting each emitted Value would be redundant work on the hot path. When the inner querier does not implement mimirSearcher, SearchLabelNames/Values returns storage.ErrSearchResultSet with a descriptive message so callers can distinguish "no inner support" from a real iteration failure. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Straight pass-through to the wrapped querier when it implements the search interface, matching the existing LabelNames/LabelValues pass-through pattern. Prometheus's lazySearchResultSet already covers deferred init at the merge layer, so no extra wrapping is needed here. The mimirSearcher interface is declared locally as `searcher` in lazyquery.go to avoid an import cycle on pkg/querier; the Mimir extension to Prometheus's storage.Searcher (extra *streaminglabelvalues.Params argument — see spec invariant 9) means we cannot type-assert to the upstream storage.Searcher directly. When the wrapped querier does not implement search, SearchLabelNames/Values returns storage.ErrSearchResultSet so the absence of support surfaces as an iteration error rather than a panic. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…t risk Two small cleanups on the PR-3 surface: * fanOutSearch took a context.Context that it never used (both call sites already had ctx in scope and threaded it through their own closures). Drop the unused parameter. * lazyquery's local `searcher` interface is a manual copy of pkg/querier.mimirSearcher, declared locally to avoid an import cycle. The runtime type assertion silently falls back to "search not supported" if the two definitions ever drift. The existing comment noted the cycle but not the consequence; spell out the drift risk so a future signature change leaves a breadcrumb here. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
tcp13equals2
force-pushed
the
streaming-labels-values-PR-3
branch
from
May 22, 2026 05:22
46fff02 to
fabc599
Compare
ldufr
approved these changes
May 22, 2026
4 tasks
tcp13equals2
added a commit
that referenced
this pull request
May 28, 2026
#### What this PR does This is the 4th PR in the streaming labels/values search API stack. See prometheus/proposals#74. This PR builds upon #15233, #15301 and #15347 This PR adds three experimental NDJSON-streaming HTTP endpoints on the querier; * /api/v1/search/metric_names — searches values of the __name__ label * /api/v1/search/label_names — searches for label names * /api/v1/search/label_values — searches for values of a single label The HTTP endpoints have been placed behind a feature flag ( -querier.experimental-search-api-enabled ) and this defaults to false. Note - in a subsequent PR will be support for the metadata query param to decorate metric names, additional documentation, bechmarks and updates to dashboard mixins. #### Which issue(s) this PR fixes or relates to Fixes #<issue number> #### Checklist - [x] Tests updated. - [ ] Documentation added. - [x] `CHANGELOG.md` updated - the order of entries should be `[CHANGE]`, `[FEATURE]`, `[ENHANCEMENT]`, `[BUGFIX]`. If changelog entry is not needed, please add the `changelog-not-needed` label to the PR. - [ ] [`about-versioning.md`](https://github.com/grafana/mimir/blob/main/docs/sources/mimir/configure/about-versioning.md) updated with experimental features. <!-- CURSOR_SUMMARY --> --- > [!NOTE] > **Medium Risk** > New query surface can drive expensive label scans across ingesters/store-gateways; mitigated by default-off flag, limits, and streaming, but multi-tenant federation fan-out increases blast radius when enabled. > > **Overview** > Adds **experimental** Prometheus-style **streaming label/value search** on the querier: NDJSON endpoints at `/api/v1/search/{metric_names,label_names,label_values}`, gated by **`-querier.experimental-search-api-enabled`** (default off). When disabled, callers get **404** `feature_not_enabled`. > > The new **`search_handler`** layer parses query params (fuzz, sort, limits, `match[]` OR-union), opens a **`mimirSearcher`**, streams batched NDJSON with success/error trailers, and sets **`has_more`** using a limit+1 probe plus typed **`MaxLimitError`** clamp warnings. **`MaxLimitError`** is refactored so handlers can read which limit was enforced without parsing messages. > > Routes are wired in **`pkg/api`** and **`handlers.go`** with usage stats. **Tenant federation** **`merge_queryable`** fans out **`SearchLabelNames` / `SearchLabelValues`**, merges result sets, and special-cases synthetic **`__tenant_id__`** (and retain-prefix) label values like existing **`LabelValues`**. > > Changelog and config docs flag the feature as experimental. > > <sup>Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit bc74dda. Bugbot is set up for automated code reviews on this repo. Configure [here](https://www.cursor.com/dashboard/bugbot).</sup> <!-- /CURSOR_SUMMARY --> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
tcp13equals2
added a commit
that referenced
this pull request
May 28, 2026
#### What this PR does This is the fifth PR in the streaming label/value search API implementation. See also #15233, #15301, #15347, #15349. This PR focuses on adding support for the `include_metadata` enrichment on results from ` /api/v1/search/metric_names`. This feature allows for metric metadata to be included in the metric name results. This allows for a client to avoid needing to make a separate API call to retrieve metadata records. It should be noted that in Mimir, only the ingesters maintain metric metadata information - and only if it has been included in the remote write. This metadata information is only maintained in the ingesters for a short period of time (10 minutes). To maintain simplicity, this implementation only decorates in the most recent metadata record for a metric. This is regardless of any time range submitted in the search. In the future if Mimir was persist or maintain metadata for longer this implementation choice could be reviewed. No metadata is included if the search time range only requires store-gateways to full-fill the request. This is intentional since "old" metrics which have not been seen (metadata record pushed) by an ingester in the last 10 minutes will not have a metadata in Mimir. #### Which issue(s) this PR fixes or relates to Fixes #<issue number> #### Checklist - [x] Tests updated. - [ ] Documentation added. - [x] `CHANGELOG.md` updated - the order of entries should be `[CHANGE]`, `[FEATURE]`, `[ENHANCEMENT]`, `[BUGFIX]`. If changelog entry is not needed, please add the `changelog-not-needed` label to the PR. - [ ] [`about-versioning.md`](https://github.com/grafana/mimir/blob/main/docs/sources/mimir/configure/about-versioning.md) updated with experimental features. <!-- CURSOR_SUMMARY --> --- > [!NOTE] > **Low Risk** > Additive, flag-gated experimental API behavior with ingester-only enrichment; no auth or ingestion path changes. > > **Overview** > Adds optional **`include_metadata`** to the experimental streaming search API so **`/api/v1/search/metric_names`** can return **type**, **help**, and **unit** on each metric without a separate metadata call. > > The flag is parsed on the querier HTTP handler, carried on **`streaminglabelvalues.Params`**, and sent to ingesters as **`include_metadata`** on **`SearchLabelValues`** when searching **`__name__`**. Ingesters attach **`MetricMetadata`** per batch (most recent in-memory record per metric on that replica); the distributor maps wire metadata into **`storage.SearchResult`**, and the metric-names NDJSON builder emits the extra fields. **Store-gateway–only** searches stay un-enriched by design. > > Also tightens search plumbing: shared **`parseBoolParam`**, **`buildSearchHints`** rejects negative limits and clamps huge values, and dev **`mimir.yaml`** turns on **`experimental_search_api_enabled`**. > > <sup>Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit 1b2c64f. Bugbot is set up for automated code reviews on this repo. Configure [here](https://www.cursor.com/dashboard/bugbot).</sup> <!-- /CURSOR_SUMMARY --> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
tcp13equals2
added a commit
that referenced
this pull request
Jun 4, 2026
#### What this PR does This is the 6th PR in the streaming label/value search API implementation. See also #15233, #15301, #15347, #15349 and #15364. This PR adds a README.md to assist in documenting this implementation, benchmark test files and some enhancements found whilst running the newly added benchmarks. No changelog has been recorded since these are internal implementation, testing and documentation changes with no user facing modifications. #### Benchmarks: legacy `LabelValues` vs new `SearchLabelValues` Apple M4 Pro, `go test -benchmem -benchtime=5s -count=3`, medians of 3 runs reported. ##### Ingester (`pkg/ingester`) `BenchmarkIngester_LegacyVsSearchLabelValues` — in-process, no RPC framing. | Cardinality | Path | ns/op | B/op | allocs/op | Δ time | Δ B/op | Δ allocs | | ----------- | ------ | ----------: | ----------: | ---------: | -------- | -------- | -------- | | 4 199 | legacy | 211 794 | 90 418 | 4 209 | | | | | 4 199 | new | 216 027 | 223 578 | 21 | +2% | +147% | **−99.5%** | | 1 000 003 | legacy | 149 987 546 | 23 942 547 | 1 000 161 | | | | | 1 000 003 | new | 130 728 720 | 48 140 526 | 158 | **−13%** | +101% | **−99.98%** | **Observations** - At 4 199 values: parity on wall-clock, but allocations drop from 4 209 → 21 (≈200×). - At 1 000 003 values: new path is 13% faster, uses 2× the memory, but allocates **6 330× fewer objects** (158 vs 1 000 161). The allocation reduction is due to the batch buffer reuse and synchronous-send which make skipping the clone safe. ``` q, err := db.Querier(...) defer q.Close() ... rs := searcher.SearchLabelValues(ctx, ...) // result set holds yolo strings defer rs.Close() // return streamSearchResults(ctx, rs, stream.Send, ...) // synchronous; blocks until all batches sent ``` The increase in B/op is due to the wrapping of each record in a `SearchResult{Value, Score}`. ##### Store-gateway (`pkg/storegateway`) `BenchmarkBucketStoreSearchLabelValuesVsLabelValues` — bucket store, cold index cache (`worstCaseFetchedDataStrategy`). | Cardinality | Path | ns/op | B/op | allocs/op | Δ time | Δ B/op | Δ allocs | | ----------- | ------ | -------------: | ----------: | ----------: | -------- | ------ | -------- | | 1 000 | legacy | 21 346 956 | 13 931 796 | 61 668 | | | | | 1 000 | new | 21 368 149 | 14 004 859 | 61 681 | +0.1% | +0.5% | +0.02% | | 1 000 000 | legacy | 32 330 190 178 | 3 482 e6 | 51 001 391 | | | | | 1 000 000 | new | 29 567 861 292 | 3 569 e6 | 51 009 234 | **−9%** | +2.5% | +0.02% | **Observations** - At 1 000 values: parity. - At 1 000 000 values: new path is ~9% faster on time. Memory and allocation count are essentially unchanged because the bulk of the work at the store-gateway layer is in the shared TSDB postings/chunk walk. Although the same SearchResult wrapper is present it is lost in the other SG B/op noise. #### Which issue(s) this PR fixes or relates to Fixes #<issue number> #### Checklist - [x] Tests updated. - [x] Documentation added. - [ ] `CHANGELOG.md` updated - the order of entries should be `[CHANGE]`, `[FEATURE]`, `[ENHANCEMENT]`, `[BUGFIX]`. If changelog entry is not needed, please add the `changelog-not-needed` label to the PR. - [ ] [`about-versioning.md`](https://github.com/grafana/mimir/blob/main/docs/sources/mimir/configure/about-versioning.md) updated with experimental features. <!-- CURSOR_SUMMARY --> --- > [!NOTE] > **Low Risk** > No user-facing API or behavior changes beyond optional micro-optimizations on an experimental, flag-gated search path; risk is mainly benchmark/doc churn and regression in edge fast paths (single tenant/block, cancel lifecycle). > > **Overview** > Adds **`docs/internal/streaming-label-value-search/README.md`**, an internal guide for the experimental streaming label/value search stack (NDJSON HTTP endpoints, data flow, merge layers, wire batching, benchmarks). > > **Benchmark coverage** is added across ingester, distributor, store-gateway, tenant federation, and HTTP handler tests (including legacy vs new parity benches and distributor merge/dedup scenarios). > > **Performance tweaks** found while benchmarking: HTTP **`search_handler`** pools NDJSON batch envelopes, splits score vs no-score record types to avoid `*float64` allocs, and writes a prebuilt success trailer; **distributor** stores stream cancel funcs on `ingesterSearchResultSet` instead of per-ingester closures; **tenant federation** uses `tenantJobsForSearch` (skip map work when no id-label matcher) and a single-tenant job bypass; **store-gateway** skips `MergeSearchResultSets` when only one block matches. Ingester/store-gateway sources link to the new README. > > <sup>Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit 3990db9. Bugbot is set up for automated code reviews on this repo. Configure [here](https://www.cursor.com/dashboard/bugbot).</sup> <!-- /CURSOR_SUMMARY -->
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this PR does
This is the third PR in the streaming labels/values search API stack.
See prometheus/proposals#74.
This PR builds upon #15233 and #15301
This PR wires the querier-side multi-source fan-out that consumes the work from the above PRs.
Reachable but unused - there is no Mimir-internal caller from the HTTP layer — the new methods are exercised only by unit tests in this PR. The user-visible HTTP endpoints will come in a subsequent PR.
Which issue(s) this PR fixes or relates to
Fixes #
Checklist
CHANGELOG.mdupdated - the order of entries should be[CHANGE],[FEATURE],[ENHANCEMENT],[BUGFIX]. If changelog entry is not needed, please add thechangelog-not-neededlabel to the PR.about-versioning.mdupdated with experimental features.Note
Medium Risk
Adds new querier-level
SearchLabelNames/SearchLabelValuesfan-out and limit-clamping logic plus wrapper passthroughs; while covered by unit tests, it touches core query orchestration and could affect search correctness, ordering, and per-tenant limits.Overview
Adds
multiQuerier.SearchLabelNames/SearchLabelValuesto fan out streaming label/values search across distributor and block-store, then merge/deduplicate orderedstorage.SearchResultSets while enforcing per-tenantLimitviaMaxLabelNamesLimit/MaxLabelValuesLimit(including clamp warnings).Extends
memoryTrackingQuerierandlazyquery.LazyQuerierwith search pass-through methods that type-assert for search support and otherwise returnErrSearchResultSet, intentionally skipping additional memory tracking on the streaming search path. Comprehensive unit tests cover pass-through behavior, error propagation, ordering/dedup, empty-range short-circuit, and limit-clamp warnings.Reviewed by Cursor Bugbot for commit fabc599. Bugbot is set up for automated code reviews on this repo. Configure here.