🧩 Summary
We currently forward errors via custom Slack alerts to various channels, and rely on this to catch incidents in production. It's possible we could use off-the-shelf tools (e.g. Sentry or adding to our existing Grafana setup) to achieve the same thing and/or improve on it:
- There are some features of Sentry that we replicate, or would want to replicate:
- Labelling as warnings vs errors
- Grouping/rolling up similar errors
- Would be nice to have: The ability to mute types of error
- There are some gaps in the current system:
- Serious incidents (e.g. the site being down, pg-sync-service dying) are mixed in everything else in #update_tech-prod channel. Ideally we would have a PagerDuty style alert which ensures someone acknowledges it for serious issues
There are also benefits to have a simple custom system (e.g. it's free, it keeps everything in code). The scope of this issue is to investigate what is possible with off-the-shelf alternatives, how complex they would be, how much they would cost, etc, and raise this for review.
✅ Definition of done
We have investigated the possible alternatives, and agreed on whether to change our setup or keep it as-is.
📌 Notes / Designs
No response
🧩 Summary
We currently forward errors via custom Slack alerts to various channels, and rely on this to catch incidents in production. It's possible we could use off-the-shelf tools (e.g. Sentry or adding to our existing Grafana setup) to achieve the same thing and/or improve on it:
There are also benefits to have a simple custom system (e.g. it's free, it keeps everything in code). The scope of this issue is to investigate what is possible with off-the-shelf alternatives, how complex they would be, how much they would cost, etc, and raise this for review.
✅ Definition of done
We have investigated the possible alternatives, and agreed on whether to change our setup or keep it as-is.
📌 Notes / Designs
No response