Fix metrics generator panic when write_relabel_configs is set - #6399
Merged
Conversation
carles-grafana
force-pushed
the
fix_config_panic
branch
2 times, most recently
from
February 6, 2026 10:17
3610c5d to
82e7dcc
Compare
carles-grafana
changed the title
carles-grafana
force-pushed
the
fix_config_panic
branch
from
February 6, 2026 10:27
82e7dcc to
3ad14a2
Compare
carles-grafana
marked this pull request as ready for review
February 6, 2026 10:27
carles-grafana
requested review from
electron0zero,
ie-pham,
javiermolinar,
joe-elliott,
mapno,
mattdurham,
mdisibio,
oleg-kozlyuk-grafana,
ruslan-mikhailov,
stoewer,
yvrhdn and
zalegrala
as code owners
February 6, 2026 10:27
| // mutate the shared input configs. Without this, concurrent | ||
| // watchOverrides goroutines from different tenants would race on | ||
| // the shared *relabel.Config objects. | ||
| output.WriteRelabelConfigs = copyRelabelConfigs(input.WriteRelabelConfigs) |
Contributor
There was a problem hiding this comment.
I don’t think the deep copy is needed here if we validate the base config once at startup. cfg.RemoteWrite is global (not tenant-specific), so we can initialize/validate its relabel configs when the metrics generator starts.
In generateTenantRemoteWriteConfigs, we only inject per-tenant headers. Doing validation once would simplify this path and avoid per-tenant allocations. Per-tenant path should just clone config as it is right now and inject headers.
Contributor
Author
There was a problem hiding this comment.
good point. The whole issue was that the Validate from Prometheus sets one field: https://github.com/carles-grafana/tempo/blob/fix_config_panic/vendor/github.com/prometheus/prometheus/model/relabel/relabel.go#L135
by moving it to startup it's much simpler. fixed
…a#6396) The Prometheus dependency upgrade from v0.304.2 to v0.307.3 (PR prometheus/prometheus#16928) moved label name validation from a global to a per-config NameValidationScheme field on relabel.Config. This field must be initialized by calling Validate() before use, otherwise relabeling panics with "Invalid name validation scheme requested: unset". Tempo constructs RemoteWriteConfig programmatically rather than going through Prometheus's config.Load() path, so Validate() was never called. Call RemoteWriteConfig.Validate(model.UTF8Validation) once at startup in storage.Config.Validate(), which initializes NameValidationScheme on all write relabel configs before any tenant storage instances are created. Also fix a pre-existing bug in watchOverrides where cached state was updated before ApplyConfig succeeded, preventing retries on failure.
carles-grafana
force-pushed
the
fix_config_panic
branch
from
February 6, 2026 11:27
b55b38d to
85e52e2
Compare
javiermolinar
approved these changes
Feb 6, 2026
javiermolinar
left a comment
javiermolinar
left a comment
Contributor
There was a problem hiding this comment.
LGTM
Have you tested it with our docker example?
Contributor
Author
|
yes, tested with docker successfully |
carles-grafana
added a commit
that referenced
this pull request
Feb 13, 2026
…#6399) The Prometheus dependency upgrade from v0.304.2 to v0.307.3 (PR prometheus/prometheus#16928) moved label name validation from a global to a per-config NameValidationScheme field on relabel.Config. This field must be initialized by calling Validate() before use, otherwise relabeling panics with "Invalid name validation scheme requested: unset". Tempo constructs RemoteWriteConfig programmatically rather than going through Prometheus's config.Load() path, so Validate() was never called. Call RemoteWriteConfig.Validate(model.UTF8Validation) once at startup in storage.Config.Validate(), which initializes NameValidationScheme on all write relabel configs before any tenant storage instances are created. Also fix a pre-existing bug in watchOverrides where cached state was updated before ApplyConfig succeeded, preventing retries on failure.
3 tasks
carles-grafana
added a commit
that referenced
this pull request
Feb 16, 2026
…#6399) (#6455) The Prometheus dependency upgrade from v0.304.2 to v0.307.3 (PR prometheus/prometheus#16928) moved label name validation from a global to a per-config NameValidationScheme field on relabel.Config. This field must be initialized by calling Validate() before use, otherwise relabeling panics with "Invalid name validation scheme requested: unset". Tempo constructs RemoteWriteConfig programmatically rather than going through Prometheus's config.Load() path, so Validate() was never called. Call RemoteWriteConfig.Validate(model.UTF8Validation) once at startup in storage.Config.Validate(), which initializes NameValidationScheme on all write relabel configs before any tenant storage instances are created. Also fix a pre-existing bug in watchOverrides where cached state was updated before ApplyConfig succeeded, preventing retries on failure.
mattdurham
pushed a commit
to mattdurham/tempo
that referenced
this pull request
Jun 18, 2026
…a#6396) (grafana#6399) The Prometheus dependency upgrade from v0.304.2 to v0.307.3 (PR prometheus/prometheus#16928) moved label name validation from a global to a per-config NameValidationScheme field on relabel.Config. This field must be initialized by calling Validate() before use, otherwise relabeling panics with "Invalid name validation scheme requested: unset". Tempo constructs RemoteWriteConfig programmatically rather than going through Prometheus's config.Load() path, so Validate() was never called. Call RemoteWriteConfig.Validate(model.UTF8Validation) once at startup in storage.Config.Validate(), which initializes NameValidationScheme on all write relabel configs before any tenant storage instances are created. Also fix a pre-existing bug in watchOverrides where cached state was updated before ApplyConfig succeeded, preventing retries on failure.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this PR does:
Fix metrics generator panic when write_relabel_configs is set (#6396)
The Prometheus dependency upgrade from v0.304.2 to v0.307.3 (PR
prometheus/prometheus#16928) moved label name validation from a global
to a per-config NameValidationScheme field on relabel.Config. This field
must be initialized by calling Validate() before use, otherwise
relabeling panics with "Invalid name validation scheme requested: unset".
Tempo constructs RemoteWriteConfig programmatically rather than going
through Prometheus's config.Load() path, so Validate() was never called.
Call RemoteWriteConfig.Validate(model.UTF8Validation) once at startup
in storage.Config.Validate(), which initializes NameValidationScheme on
all write relabel configs before any tenant storage instances are created.
Also fix a pre-existing bug in watchOverrides where cached state was
updated before ApplyConfig succeeded, preventing retries on failure.
Which issue(s) this PR fixes:
Fixes #6396
Checklist
CHANGELOG.mdupdated - the order of entries should be[CHANGE],[FEATURE],[ENHANCEMENT],[BUGFIX]