loki: Use labels.Labels for entries instead of model.LabelSet

[kalleep]

Component(s)

Loki components

Background

Today loki.Entry stores model.LabelSet as its representation for labels, so that entries can be mutated as they move through the pipeline. But almost nothing in the pipeline actually wants a map.

Most of our loki.source.* components support relabeling and convert between labels.Labels and model.LabelSet, e.g. loki.source.api.

loki.relabel has this exact same pattern, converting in both directions on every cache miss.

loki.write needs to sort these labels in another slice to properly build the required label string before sending to Loki.

Our WAL implementation will convert to labels.Labels when writing and back to model.LabelSet after reading.

loki.process prematurely clones the entry because mutating it in place would be considered unsafe since we can fan out to multiple components. There is already a TODO on that line proposing copy-on-write so the copy only happens on the first stage that needs to modify labels.

And with the new batch structure we want to use for the pipeline changes described in #4953 we could have some potentially expensive clones / comparisons.

Proposal

In our code I have seen this comment about potentially using labels.Labels instead and this other comment around using some kind of copy on write mechanism.

Using labels.Labels would solve all of these issues for us. In places where we need mutations we would use labels.Builder, it's already used in all places where we do relabeling and it would solve the copy on write mentioned in loki.process since it would not allocate anything for labels if nothing has changed. We would also get much cheaper comparisons in our batch implementation since we use stringlabels.

So for this proposal we would change our Entry to

type Entry struct {
  Labels labels.Labels
  push.Entry
  created int64
}

We would need to update the places that do this conversion to just use labels directly. But the biggest change would be in loki.process. In this component we have another representation of entries. We would update this one to look like this:

type Entry struct {
  Extracted map[string]any
  Builder   *labels.Builder
  push.Entry
  created int64
}

With this structure we would get the copy-on-write semantics for labels, so we would only allocate if a stage actually mutated them. We still need to clone entries since we have other mutable fields like StructuredMetadata, but this would at least be a no-op for labels.

AI disclosure

• This issue was substantially generated with AI assistance.

Tip

_{React with 👍 if this issue is important to you.}

[thampiotr]

Sounds good to me! In terms of prioritisation, just wanted to check how important do you think it is to ship this before the new loki pipelines?

[kalleep]

I would say it's required before we turn on batching. I am doing pr:s updating components to implement Consumer interface for both non-batch and batching and then first step would be to just switch over to non-batch functions instead of channels, since batching requires bigger changes at sources.

So it would be something like this:

1. Complete all pr:s to implement the consumer interface for components without using it
2. Switch the pipeline over to use ConsumeEntry (non-batch).
3. Do the changes proposed in this issue
4. Switch over sources to Consume (batching)