ContainerRegistryNotificationsPendingCountTooHigh
Overview
Section titled “Overview”- What does this alert mean?
- The number of pending outgoing notifications is too high.
- This can happen when notifications fail to be sent, typically seen if the following alerts happen:
- What factors can contribute?
- Increased load on the registry pods.
- Low available resources on the registry pods.
- What action is the recipient of this alert expected to take when it fires?
- Network transient errors should self-heal.
- Troubleshooting.
Services
Section titled “Services”- Service Overview
- Team that owns the service: Container Registry
Metrics
Section titled “Metrics”- Metric:
registry_notifications_pending_total.
Briefly explain the metric this alert is based on and link to the metrics catalogue. What unit is it measured in? (e.g., CPU usage in percentage, request latency in milliseconds)
- Dashboard URL focusing on the
Events queued per secondpanel. - The queue will grow while there are errors/failures.
Explain the reasoning behind the chosen threshold value for triggering the alert. Is it based on historical data, best practices, or capacity planning?
- If the gauage metric keeps increasing, it means we are not dispatching any events. Having a low threshold should signal issues early on, before we see failures like lack of resources.
Describe the expected behavior of the metric under normal conditions. This helps identify situations where the alert might be falsely firing.
- This metric should go up and down as pending events are queued and dispatched.
- Some peaks are expected during traffic peak times.
- The
Pending eventspanels should have a relatively low 2 digit number.
Alert Behavior
Section titled “Alert Behavior”Expected frequency of the alert. Is it a high-volume alert or expected to be rare?
- Should be rare.
Show historical trends of the alert firing e.g Kibana dashboard
- N/A (new alert)
Severities
Section titled “Severities”Guidance for assigning incident severity to this alert
s4
Who is likely to be impacted by this cause of this alert?
- Customers pushing/pulling images to the container registry.
Things to check to determine severity
- Service overview
- Escalate if service is degraded for a prolonged period of time.
Verification
Section titled “Verification”- Metric explorer
- Registry logs
registry-main/registry-overviewregistry-notifications/webhook-notifications-detailapi-main/api-overviewcloudflare-main/cloudflare-overview- Rails API logs.
Recent changes
Section titled “Recent changes”Recent changes
How to properly roll back changes
- Check the changelog in the MR that updated the registry.
- Review MRs included in the related release issue
- If any MR has the label ~cannot-rollback applied, a detailed description should exist in that MR.
- Otherwise, proceed to revert the commit and watch the deployment.
- Review the dashboards and expect the metric to go back to normal.
Troubleshooting
Section titled “Troubleshooting”- Registry troubleshooting
Possible Resolutions
Section titled “Possible Resolutions”Dependencies
Section titled “Dependencies”- Rails API
- Cloudflare/firewall rules