Increased Error Rate
Symptoms
Section titled “Symptoms”- Message in prometheus-alerts Increased Error Rate Across Fleet
- PagerDuty Alert:
HighRailsErrorRate
Troubleshoot
Section titled “Troubleshoot”Kibana
Section titled “Kibana”- All 5xx statuses in rails
- All 5xx statuses by controller
- Check for abuse from a specific IP
- Check the triage overview dashboard for 5xx errors by backend.
- Check Sentry for new 500 errors or an uptick.
- If the problem persists send a channel wide notification in #development.