mimir-mixin: fix alert defs for compartmentalized deployments - #16289
Draft
pitasi wants to merge 12 commits into
Draft
mimir-mixin: fix alert defs for compartmentalized deployments#16289pitasi wants to merge 12 commits into
pitasi wants to merge 12 commits into
Conversation
pitasi
force-pushed
the
anto/fix-alerts-compartments
branch
from
August 3, 2026 15:29
7d63a05 to
9732e6e
Compare
pitasi
changed the title
Compartments scope workload and ring names with a "-rc-<id>" suffix but add no label to any metric, because a per-compartment label would change a metric's label set and collide with the same metric registered by another component in the same process. Alerts that must not mix compartments therefore have to parse the compartment out of an existing label. Add a single helper for that, so the regex lives in one place instead of being repeated by every alert that needs it. No alert uses it yet, so the compiled mixin is unchanged. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The compiled mixin is generated with compartments disabled, so nothing exercised the compartment code paths and a whole class of breakage went unnoticed: rules that hardcode a ring name, or that group workloads by stripping a trailing "-zone-<x>", stop working once compartments append "-rc-<id>" to those names. Add a test variant with compartments enabled and assert the invariants that generated alerts and rules must hold. The compactor scheduler is enabled too, as it is deployed per read compartment. Four assertions fail on purpose right now, one per bug they pin down: "ingester-partitions" ring matcher in alerts, and zone-only workload grouping in alerts, rules and the rollout progress dashboard. The following commits fix them. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
MimirFewerIngestersConsumingThanActivePartitions could not fire at all when compartments are enabled: each read compartment registers its own partition ring, named "ingester-partitions-rc-<id>", so the equality matcher on the bare ring name selected nothing and the left hand side of the comparison was always empty. Matching the suffixed name is not enough on its own. Partition IDs are derived from the ingester ordinal and therefore restart from 0 in every compartment, so comparing the largest compartment's partition count against the union of all the compartments' (colliding) partition IDs hides a shortfall: two compartments of two partitions each look like two partitions with four consumers. Both sides now carry a read_compartment label and are compared per compartment. A compartment that loses every consumer has no series on the right hand side, and vector matching would silently drop it from the comparison, so the right hand side defaults to zero consumers for every compartment the ring knows about. That case was also undetected without compartments, where it needed every ingester in the namespace to stop consuming; with compartments losing one compartment is enough, which is exactly what this alert should catch. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Every read compartment keeps the blocks of every tenant in its own bucket, with its own compactors maintaining that bucket's index, so a tenant reports one cortex_bucket_index_last_successful_update_timestamp_seconds per compartment. MimirBucketIndexNotUpdated aggregated those with "min", which keeps the freshest timestamp, so a compartment whose bucket index had gone stale was masked by any healthy compartment and queriers would start failing without a page. Group by the compartment as well, and name it in the alert message so it is clear which bucket to look at. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Each read compartment runs its own compactor scheduler over its own bucket, but MimirCompactorSchedulerNotCompletingJobs summed the completed jobs of every scheduler in the namespace before comparing to zero, so one scheduler completing jobs was enough to hide another that had completely stalled. Count the completed jobs per compartment instead, and keep the compartment on MimirCompactorSchedulerRepeatedJobFailure too, which fired without saying which scheduler was failing. These alerts are only generated when the compactor scheduler is enabled, so the compiled mixin is unchanged; the compartments mixin test enables it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
MimirIngesterInstanceHasNoTenants only fires if the cell holds more series than continuous testing alone would produce. On ingest storage that total is computed by deduplicating the zone replicas of a partition, which hold the same series, using the ingester ordinal as the identity. The ordinal restarts from 0 in every read compartment, so ingesters of different compartments, which hold different partitions and therefore different series, were deduplicated as if they were replicas of each other. The total came out roughly divided by the number of compartments and could stay under the threshold, suppressing the alert. Include the compartment in the identity so only genuine zone replicas are deduplicated. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
MimirRolloutStuck derives a rollout_group by stripping an optional trailing
"-zone-<x>" from the workload name. label_replace anchors its regex, so with
compartments the zone is no longer at the end ("ingester-zone-a-rc-0") and nothing
is stripped: every zone becomes a rollout group of its own.
That fires spuriously on every rollout longer than 30 minutes. The rollout-operator
updates one zone of a group at a time, so a zone waiting its turn sits at zero
updated replicas with nothing changing, which is indistinguishable from a stuck
rollout once it is looked at in isolation. Grouped correctly, the progress of the
zone being updated keeps the alert quiet.
Match the compartment suffix explicitly, so zones group into the rollout group the
rollout-operator actually manages ("ingester-rc-0"), which is also the value it
reports on rollout_operator_last_successful_group_reconcile_timestamp_seconds, so
MimirRolloutStuck and RolloutOperatorNotReconciling can be correlated again.
The regex lives in the config because the scaling rules and the rollout progress
dashboard group workloads the same way and are fixed next. Nothing changes without
compartments.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The two MimirIngesterTSDBWALCorrupted alerts tell apart a deployment spread over
several zones, where corruption reaching more than one zone risks unrecoverable
data loss, from a single-zone one, where any corruption does. Both inferred the
number of zones from the number of distinct ingester jobs.
With compartments each zone has one job per compartment ("ingester-zone-a-rc-0"),
which breaks that inference in both directions: a single-zone cell with two
compartments looks like it has two zones, so the single-zone variant can never
fire; and two corrupted ingesters in the same zone but different compartments look
like two corrupted zones, so the multi-zone critical fires even though every
compartment replicates across zones on its own and has only lost one replica.
Extract the zone from the job name and count zones instead. Without compartments
the job name is the zone, so nothing changes.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The scaling recording rules group a workload with the other workloads it is scaled together with by stripping a trailing "-zone-<x>", in eight places across actual_replicas, the CPU and memory usage and the resource requests. As in MimirRolloutStuck, that regex strips nothing once compartments move the zone away from the end of the name, so every zone of every compartment was reported as a scaling target of its own rather than one entry per compartment. Reuse the shared workload group regex, which keeps the current behaviour without compartments and groups the zones of a compartment together with them. All the rules that feed cpu_required_replicas_count and memory_required_replicas_count are changed together, so they keep matching each other. The baremetal variants don't group by zone and are left alone. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The scaling rules that size a component for a whole namespace, by sample rate or
active series, label their result with the bare component name. The Scaling
dashboard joins those to the actual replica count on the "deployment" label, which
with compartments is only ever recorded per compartment ("ingester-rc-0"), so the
join matched nothing and the sample_rate, active_series and *_limits rows silently
disappeared from the dashboard.
Record the total across a component's compartments next to the per-compartment
counts, so both kinds of rule find a counterpart: the namespace-wide ones match the
total, and the CPU and memory ones, which are derived per compartment, keep
matching the per-compartment counts.
Without compartments the selector matches nothing and no series is recorded, so
nothing changes.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The Rollout Progress dashboard folds Deployments and StatefulSets into a "workload" label and then strips a trailing "-zone-<x>" to show one row per group of workloads rolled out together. With compartments the zone is no longer at the end of the name, so nothing was stripped and the dashboard listed every zone of every compartment separately instead of one row per compartment. Reuse the shared workload group regex, the same one MimirRolloutStuck and the scaling rules use, so the dashboard rows line up with the rollout groups the rollout-operator manages. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Regenerates the alerts, recording rules and dashboards embedded in the Helm chart's metamonitoring output, after the compartment fixes in the preceding commits. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
pitasi
force-pushed
the
anto/fix-alerts-compartments
branch
from
August 11, 2026 09:55
9732e6e to
adb4853
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this PR does
Fix alerts definitions for compartments, in a backwards compatible way (= the definition should work both on non-compartmentalized and compartmentalized deployments).