Streaming labels values PR#6 by tcp13equals2 · Pull Request #15426 · grafana/mimir

tcp13equals2 · 2026-05-25T08:33:29Z

What this PR does

This is the 6th PR in the streaming label/value search API implementation.

See also #15233, #15301, #15347, #15349 and #15364.

This PR adds a README.md to assist in documenting this implementation, benchmark test files and some enhancements found whilst running the newly added benchmarks.

No changelog has been recorded since these are internal implementation, testing and documentation changes with no user facing modifications.

Benchmarks: legacy `LabelValues` vs new `SearchLabelValues`

                                                                                                                                                                                                                                                         Apple M4 Pro, `go test -benchmem -benchtime=5s -count=3`, medians of 3 runs reported.

Ingester (`pkg/ingester`)

                                                                                                                                                                                                                                                         `BenchmarkIngester_LegacyVsSearchLabelValues` — in-process, no RPC framing.

Cardinality	Path	ns/op	B/op	allocs/op	Δ time	Δ B/op	Δ allocs
4 199	legacy	211 794	90 418	4 209
4 199	new	216 027	223 578	21	+2%	+147%	−99.5%
1 000 003	legacy	149 987 546	23 942 547	1 000 161
1 000 003	new	130 728 720	48 140 526	158	−13%	+101%	−99.98%

Observations

At 4 199 values: parity on wall-clock, but allocations drop from 4 209 → 21 (≈200×).
At 1 000 003 values: new path is 13% faster, uses 2× the memory, but allocates 6 330× fewer objects (158 vs 1 000 161).

The allocation reduction is due to the batch buffer reuse and synchronous-send which make skipping the clone safe.

  q, err := db.Querier(...)                                                                                                                                                                                                              
  defer q.Close()                                                                                                                                                                                                                                    
  ...                                                                                                                                                                                                                                                        
  rs := searcher.SearchLabelValues(ctx, ...)            // result set holds yolo strings                                                                                                                                                              
  defer rs.Close()                                      //                                                                                                                                                                                              
  return streamSearchResults(ctx, rs, stream.Send, ...) // synchronous; blocks until all batches sent

The increase in B/op is due to the wrapping of each record in a SearchResult{Value, Score}.

Store-gateway (`pkg/storegateway`)

                                                                                                                                                                                                                                                         `BenchmarkBucketStoreSearchLabelValuesVsLabelValues` — bucket store, cold index cache (`worstCaseFetchedDataStrategy`).

Cardinality	Path	ns/op	B/op	allocs/op	Δ time	Δ B/op	Δ allocs
1 000	legacy	21 346 956	13 931 796	61 668
1 000	new	21 368 149	14 004 859	61 681	+0.1%	+0.5%	+0.02%
1 000 000	legacy	32 330 190 178	3 482 e6	51 001 391
1 000 000	new	29 567 861 292	3 569 e6	51 009 234	−9%	+2.5%	+0.02%

Observations

At 1 000 values: parity.
At 1 000 000 values: new path is ~9% faster on time. Memory and allocation count are essentially unchanged because the bulk of the work at the store-gateway layer is in the shared TSDB postings/chunk walk.

Although the same SearchResult wrapper is present it is lost in the other SG B/op noise.

Which issue(s) this PR fixes or relates to

Fixes #

Checklist

Tests updated.
Documentation added.
CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX]. If changelog entry is not needed, please add the changelog-not-needed label to the PR.
about-versioning.md updated with experimental features.

Note

Low Risk
No user-facing API or behavior changes beyond optional micro-optimizations on an experimental, flag-gated search path; risk is mainly benchmark/doc churn and regression in edge fast paths (single tenant/block, cancel lifecycle).

Overview
Adds docs/internal/streaming-label-value-search/README.md, an internal guide for the experimental streaming label/value search stack (NDJSON HTTP endpoints, data flow, merge layers, wire batching, benchmarks).

Benchmark coverage is added across ingester, distributor, store-gateway, tenant federation, and HTTP handler tests (including legacy vs new parity benches and distributor merge/dedup scenarios).

Performance tweaks found while benchmarking: HTTP search_handler pools NDJSON batch envelopes, splits score vs no-score record types to avoid *float64 allocs, and writes a prebuilt success trailer; distributor stores stream cancel funcs on ingesterSearchResultSet instead of per-ingester closures; tenant federation uses tenantJobsForSearch (skip map work when no id-label matcher) and a single-tenant job bypass; store-gateway skips MergeSearchResultSets when only one block matches. Ingester/store-gateway sources link to the new README.

^{Reviewed by Cursor Bugbot for commit 3990db9. Bugbot is set up for automated code reviews on this repo. Configure here.}

Re-introduces the dropped optimizations on top of the new PR-5 base: - search_handler: per-(endpoint × score) sync.Pool for batch envelopes; *WithScore record variants with non-pointer Score float64 to avoid the per-record *float64 heap allocation; defaultSuccessTrailer fast path; streamSearchNDJSON takes a pre-built envelope. - distributor_search: replace cancel closure with dual context.CancelCauseFunc fields on ingesterSearchResultSet. - tenantfederation/merge_queryable: tenantJobsForSearch fast path (zero-alloc when no id-label matcher) + single-job bypass in SearchLabelNames/SearchLabelValues. - storegateway/bucket_search: single-set fast path skipping MergeSearchResultSets wrapper alloc. Benchmarks re-added: handler Encoding / MetadataEncoding / OverHTTP + DefaultSuccessTrailer pinning test; distributor SearchLabelValues / SearchLabelNames; ingester SearchLabelValues / SearchLabelNames / LegacyVsSearchLabelValues; storegateway SearchLabelValues / SearchLabelNames / SearchLabelValuesVsLabelValues; tenantfederation SearchLabelValues.

ldufr · 2026-06-02T14:02:45Z

+  ┌──────────────────────────────────────────────────────────────────────┐
+  │ Tenant federation                                                    │
+  │   per-tenant fan-out; k-way merge across tenants                     │
+  │   single-tenant fast paths skip the merge wrapper                    │


Maybe not that important for an overview.

ldufr · 2026-06-02T14:32:24Z

+// newIngesterSearchResultSet stores the two cancel funcs as struct
+// fields so the constructor does not need to allocate a closure to
+// bundle them — Close calls both directly.


I don't understand why this is relevant? Is it about changing the type of cancel? Anyway, this comment probably can go.

knylander-grafana · 2026-06-02T22:48:42Z

@tcp13equals2 Would you like me to review the README? I don't always review internal docs as close, but I'm happy to if you would like.

tcp13equals2 · 2026-06-03T04:10:08Z

@tcp13equals2 Would you like me to review the README? I don't always review internal docs as close, but I'm happy to if you would like.

hey @knylander-grafana - sure that would be great. Thank you!

ldufr · 2026-06-03T07:43:24Z

+// a real httptest.NewServer + http.Client so chunked-encoding cost shows up
+// honestly. Gated by testing.Short(): the in-memory Encoding benchmark above
+// covers the encoding cost without paying the network round-trip tax.
+func BenchmarkSearchLabelNamesHandler_OverHTTP(b *testing.B) {


I don't know if this benchmark shows what is interesting about this new API. Just for fun, I benchmarked exactly the same test and it has slightly worst performance than the legacy API /api/v1/labels. That's not surprising, the way it's benchmarked.

It's maybe fine, but why do you want this benchmark?

Imo, if we include benchmark we could for instance show time to first byte (maybe with filter or not), but that's really what is interesting here.

Also that's arguably what matters the most, because if we mess-up something on the server where the querier waits for all results before streaming it back to the client, we should see a huge regression.

ldufr

Overall good, I didn't review too thoroughly the README, since @knylander-grafana
offered to have a look.

tcp13equals2 · 2026-06-04T01:40:05Z

I have merged this one and any feedback on the docs I will include in a subsequent PR. Thanks all for your reviews!

tcp13equals2 added the changelog-not-needed PRs that don't need a CHANGELOG.md entry label May 25, 2026

tcp13equals2 changed the title ~~Streaming labels values pr 6~~ Streaming labels values PR#6 May 25, 2026

tcp13equals2 force-pushed the streaming-labels-values-PR-5 branch from 8853ab6 to e6982ad Compare May 28, 2026 04:21

tcp13equals2 force-pushed the streaming-labels-values-PR-6 branch from 5b2fa36 to 217c905 Compare May 28, 2026 05:32

tcp13equals2 force-pushed the streaming-labels-values-PR-5 branch from e6982ad to 1b2c64f Compare May 28, 2026 06:34

Base automatically changed from streaming-labels-values-PR-5 to main May 28, 2026 07:27

tcp13equals2 added 4 commits May 28, 2026 15:36

Add AGENT.md files

040d6fd

make doc: prettier reformat AGENT.md files

c700cd1

Testing touching a file

191e343

tcp13equals2 force-pushed the streaming-labels-values-PR-6 branch from 0c26c56 to 191e343 Compare May 28, 2026 07:38

tcp13equals2 added 2 commits May 29, 2026 12:07

Refine documentation

090c745

Improving docs

f47cc95

tcp13equals2 marked this pull request as ready for review May 29, 2026 05:52

tcp13equals2 requested a review from a team as a code owner May 29, 2026 05:52

Update README.md

cc2a291

ldufr reviewed Jun 2, 2026

View reviewed changes

PR comments

82bfb41

ldufr reviewed Jun 3, 2026

View reviewed changes

PR feedback

3990db9

ldufr approved these changes Jun 3, 2026

View reviewed changes

tcp13equals2 merged commit fadce14 into main Jun 4, 2026
94 checks passed

tcp13equals2 deleted the streaming-labels-values-PR-6 branch June 4, 2026 01:35

Navigation Menu

Uh oh!

Streaming labels values PR#6 - #15426

Streaming labels values PR#6#15426
tcp13equals2 merged 9 commits into
mainfrom
streaming-labels-values-PR-6

tcp13equals2 commented May 25, 2026 •

edited by cursor Bot

Loading

Uh oh!

ldufr Jun 2, 2026

Uh oh!

tcp13equals2 Jun 3, 2026

Uh oh!

ldufr Jun 2, 2026

Uh oh!

tcp13equals2 Jun 3, 2026

Uh oh!

knylander-grafana commented Jun 2, 2026

Uh oh!

tcp13equals2 commented Jun 3, 2026

Uh oh!

Uh oh!

Uh oh!

ldufr Jun 3, 2026

Uh oh!

tcp13equals2 Jun 3, 2026

Uh oh!

ldufr left a comment

Uh oh!

Uh oh!

tcp13equals2 commented Jun 4, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

tcp13equals2 commented May 25, 2026 • edited by cursor Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What this PR does

Benchmarks: legacy LabelValues vs new SearchLabelValues

Ingester (pkg/ingester)

Store-gateway (pkg/storegateway)

Which issue(s) this PR fixes or relates to

Checklist

Uh oh!

ldufr Jun 2, 2026

Choose a reason for hiding this comment

Uh oh!

tcp13equals2 Jun 3, 2026

Choose a reason for hiding this comment

Uh oh!

ldufr Jun 2, 2026

Choose a reason for hiding this comment

Uh oh!

tcp13equals2 Jun 3, 2026

Choose a reason for hiding this comment

Uh oh!

knylander-grafana commented Jun 2, 2026

Uh oh!

tcp13equals2 commented Jun 3, 2026

Uh oh!

Uh oh!

Uh oh!

ldufr Jun 3, 2026

Choose a reason for hiding this comment

Uh oh!

tcp13equals2 Jun 3, 2026

Choose a reason for hiding this comment

Uh oh!

ldufr left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

tcp13equals2 commented Jun 4, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

tcp13equals2 commented May 25, 2026 •

edited by cursor Bot

Loading

Benchmarks: legacy `LabelValues` vs new `SearchLabelValues`

Ingester (`pkg/ingester`)

Store-gateway (`pkg/storegateway`)