Skip to content

POC: New structure for loki.process pipeline - #6872

Draft
kalleep wants to merge 7 commits into
mainfrom
kalleep/loki-process-consumer
Draft

POC: New structure for loki.process pipeline#6872
kalleep wants to merge 7 commits into
mainfrom
kalleep/loki-process-consumer

Conversation

@kalleep

@kalleep kalleep commented Aug 12, 2026

Copy link
Copy Markdown
Contributor

Most loki components are trivial to change in order to support the new loki pipeline we are moving too but loki.process is not.

We have a couple of different types of stages:

  1. Simple stages that just process entries and passes to next e.g. stage.static_labels
  2. Stages that can drop labels e.g. stage.drop
  3. Stages that can produce multiple entries, stage.split_json and stage.cri
  4. Stages that creates a sub pipeline, the only stage we have is stage.match
  5. Stages that can flush entries on it's own, the only one with this capability is stage.multiline

I am pretty sure the last one is the reason why this component today is built up by starting each stage in it's own goroutine and chain them with channels.

This is problematic for the loki pipeline changes since with this structure a batch / entry sent down the pipeline would be disconnected from the original caller and this would still cause the same head-of-line blocking that exists today.

What we really want is for the stages them self just be function calls, just like component are going to be. In this pr I have experimented a bit with a new stage / pipeline interface. The best way to check how it would be used is to check the added test TestPipeline2 where I have added these different kind of stages we have today (simple ports).

A couple of things I want to highlight:

  1. All stages that carries internal state now needs to be safe to call concurrently, for this POC made them safe with a sync.Mutex but we need to review this stage by stage.
  2. Since stage.multiline can flush entries on it's own we need an external trigger, this would most likely be up to the component to scan for nearest flush deadline from potentially several stages and call the flush function and forward the batch on it's own.
  3. For this POC I opted to supply a Emitter interface to every Process call, this allows a stage to emit multiple entries during on call. Another solution instead of this would be to change the signature of each stage to work on slices of entries instead and loop on each stage.

@kalleep
kalleep force-pushed the kalleep/loki-process-consumer branch from 70fae0a to 55ce994 Compare August 14, 2026 13:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant