You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
This repository was archived by the owner on Feb 9, 2024. It is now read-only.
On our team we wanted a way to centralize the management of both our monitors, but also the conditional notifications in the monitors' messages. This PR ties a team's severity notifications to the alert threshold. Instead of setting a monitor to have a certain severity, the message will have conditional blocks containing the appropriate notification channels based on the threshold of the alert. I also put all notifications into a is_recovery block so that alerts auto resolve as expected.
Nothing new is required in the config, and all fields are optional.
After some experimentation, I think monitors would still benefit from being marked as critical or info. While my changes add functionality to centralize threshold logic, some monitors are simply not important enough to warrant paging an on-call engineer at any threshold. So, I think the full solution should involve both the original severity tagging along with my functionality. I'm going to gauge interest in these changes before adding that however.
I'm envisioning the teams config would look like this:
The idea here is that on a critical alert we would first alert the team chat channel so that during business hours engineers would see the issue. Then, if the monitor goes critical the engineer on call would be paged. However, for monitors tagged info we have decided to not do anything with warnings and only send a nonintrusive chat message when the monitor alerts.
I'm also looking to get this for my team. Is there anything missing in order to merge this? It doesn't seem entirely backward compatible, but works MUCH better for our workflow.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for freeto subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Labels
None yet
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
On our team we wanted a way to centralize the management of both our monitors, but also the conditional notifications in the monitors' messages. This PR ties a team's severity notifications to the alert threshold. Instead of setting a monitor to have a certain severity, the message will have conditional blocks containing the appropriate notification channels based on the threshold of the alert. I also put all notifications into a
is_recoveryblock so that alerts auto resolve as expected.Nothing new is required in the config, and all fields are optional.
Example message: