Respond to Sentry Usage Alert
Overview
Operations Engineering run a Sentry usage job daily to check whether the Ministry of Justice (MoJ) Sentry quota for Errors, Transactions and Replays is exceeded.
The configuration for the alert is
- Period to check =
1 day- Usage threshold =
80%
Alerts are triggered for each category when usage exceeds the threshold for a given period. The alert provides some metrics for context and a link to the statistics in Sentry for further investigation as shown below:
Investigating Alerts
Click the link provided with the alert to open Sentry and display the quota usage for the alert time period in graph and table form, as shown below:
The link orders the table by
Accepted Descsince these are the events that affect quota usage
The shape of the graph in Sentry helps to determine the next step. The shape of graph largely falls into two types:
Intermittent Spikes
Intermittent spikes occur when for the majority of the time, usage seems normal - though there are abnormally high spikes in consumption.
These alerts can be ignored if it looks like the cause of the spikes have now subsided. Although this may lead to further alerts caused by the same project in the future, if the project is not properly configured to minimise the impact of spikes.
The projects spiking will
typicallybe at the top of the table.
Consistent High Usage
Consistent high usage occurs when the graph in Sentry has a business as usual look i.e. regular peaks and troughs that
reflect usage of a typical day.
In this case, consumption of the quota has presumably grown more organically over time. This is expected as more teams satrt using Sentry and applications gain more traffic.
The projects using the most quota will be at the top of the table.
Remediation
Read through the Sentry Service Documentation and ensure high usage projects are following the quota management guidance.
If a project is not following the guidance, contact the team to understand if there is any particular reason for their current configuration and assist them in implementing quota management controls.
If all high usage projects are following the guidance, raise this with Operations Engineering to discuss whether to:
- Increase the alerting threshold to minimise the number of alerts
- Buy additional Sentry quota to accommodate the increase in usage