Fix worker pool stuck on shutdown in certain cases - #4111
Merged
Conversation
kalleep
approved these changes
Aug 4, 2025
dehaansa
approved these changes
Aug 4, 2025
dehaansa
pushed a commit
that referenced
this pull request
Aug 6, 2025
* Fix worker pool stuck on shutdown in certain cases * Changelog (cherry picked from commit ee4ffd0)
Merged
kalleep
pushed a commit
that referenced
this pull request
Aug 7, 2025
* fix faro.receiver cors not allowing x-scope-orgid, traceparent headers (#4051) * fix faro.receiver cors not allowing x-scope-orgid, traceparent headers * sort values listed in Access-Control-Request-Headers header in test * fix faro.receiver readme to include allowed headers list * Update docs/sources/reference/components/faro/faro.receiver.md Co-authored-by: Clayton Cornell <131809008+clayton-cornell@users.noreply.github.com> --------- Co-authored-by: Clayton Cornell <131809008+clayton-cornell@users.noreply.github.com> (cherry picked from commit d481390) * Ensure cluster notifications are sent to remotecfg components (#4006) * Ensure cluster notifications are sent to remotecfg components * Update changelog * Address PR feedback * add some locking for a race condition in test (cherry picked from commit 1fa6b5a) * Fix references to `text_file` collector and lack of`dns` config conversion (#4024) * Fix references to collector * Ensure DNS config is appropriately converted for windows exporter * add changelog * Update docs/sources/reference/components/prometheus/prometheus.exporter.windows.md Co-authored-by: Clayton Cornell <131809008+clayton-cornell@users.noreply.github.com> * Update blurb about text_file/textfile in windows docs * Update docs/sources/reference/components/prometheus/prometheus.exporter.windows.md Co-authored-by: Clayton Cornell <131809008+clayton-cornell@users.noreply.github.com> --------- Co-authored-by: Clayton Cornell <131809008+clayton-cornell@users.noreply.github.com> (cherry picked from commit 12d94d3) * Update walqueue dependency for external_labels fix (#4133) (cherry picked from commit f8607d6) * Add metric missing features and remove unneeded validations (#4094) (cherry picked from commit 93a6365) * fix missed merge conflict * fix: stage.multiline now passes through structured metadata in loki.process (#4050) * fix: stage.multiline now passes through structured metadata in loki.process * fix: set correct expected assert value * Update internal/component/loki/process/stages/multiline.go Co-authored-by: Paulin Todev <paulin.todev@gmail.com> * refactor: implemented review feedback about how to clone, added needed import --------- Co-authored-by: Paulin Todev <paulin.todev@gmail.com> (cherry picked from commit f20c549) * Fix Windows installer values (#4125) * Fix Windows installer values Fix URLs appearing within quotes in the Windows installer. * Update Windows installer publisher Put the company name as the publisher, rather than the about URL. * Update CHANGELOG Add changes from PR. (cherry picked from commit 46a9d29) * Fix worker pool stuck on shutdown in certain cases (#4111) * Fix worker pool stuck on shutdown in certain cases * Changelog (cherry picked from commit ee4ffd0) * branch walqueue to skip prom3 in 1.10 patch * Remove unused remotecfg code unintentionally merged --------- Co-authored-by: Irina Marchuk <mar4ukira@gmail.com> Co-authored-by: Clayton Cornell <131809008+clayton-cornell@users.noreply.github.com> Co-authored-by: Marc Tudurí <marctc@protonmail.com> Co-authored-by: jan-mrm <67435696+jan-mrm@users.noreply.github.com> Co-authored-by: Paulin Todev <paulin.todev@gmail.com> Co-authored-by: Martin Costello <martin@martincostello.com> Co-authored-by: Piotr <17101802+thampiotr@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
PR Description
The worker pool, if given a component that is stuck in evaluation, would not properly shutdown and could lead to Alloy being stuck in k8s. This change was lifted from #4044 and has been discussed a bit there.
In this PR we introduce a timeout for current components evaluations to conclude within 5 seconds during shutdown. This should work for most cases as a) components typically don't evaluate this long - this is calling Update() function, b) There is no guarantee that Update() functions queued are executed during shutdown currently anyway. Stop() only waits for tasks that are currently being evaluated.
Which issue(s) this PR fixes
Notes to the Reviewer
Previous discussions:
PR Checklist