GKGCircuitBreakerRejectRateHigh
A circuit is open and shedding significant traffic, above 10 rejected calls per minute. Severity s3, fires after 5m.
Overview
Section titled “Overview”An open circuit breaker is rejecting more than 10 calls per minute, meaning
real traffic is being shed rather than served. Where GKGCircuitBreakerOpen
signals that a breaker tripped at all, this alert quantifies sustained user
impact: the dependency has been down long enough for meaningful load to pile
into the open breaker.
Services
Section titled “Services”- Orbit service overview
- Owner team:
context_systems
Metrics
Section titled “Metrics”The firing expression, on gkg_circuit_breaker_calls_rejected_total (emitted by the service, see the
metrics catalog):
sum by (env, environment, stage) (rate(gkg_circuit_breaker_calls_rejected_total[5m])) * 60 > 10Alert Behavior
Section titled “Alert Behavior”- Silence via Alertmanager filtered on the alertname.
- Rejections scale with incoming traffic, so this quantifies sustained user impact where GKGCircuitBreakerOpen only marks the event.
Severities
Section titled “Severities”- s3, no paging. Real traffic is being shed; treat the incident at the underlying dependency’s severity.
Verification
Section titled “Verification”sum by (service) (rate(gkg_circuit_breaker_calls_rejected_total[5m])) * 60The service label names the dependency being shed. Correlate with the
error-rate alerts for that dependency and with
gkg_circuit_breaker_state_transitions_total to see whether the breaker is
stuck open or flapping.
Recent changes
Section titled “Recent changes”- argocd-apps MRs touching
services/gkg(a merge is a deploy). - Production issues labeled Service::Orbit.
Troubleshooting
Section titled “Troubleshooting”| Symptom | Cause | Action |
|---|---|---|
| Sustained rejects, dependency confirmed down | Ongoing dependency outage | Restore the dependency; the breaker closes on its own probes |
| Rejects continue after the dependency recovered | The breaker’s probe never succeeds (bug) | File an issue with the breaker name; a pod restart is the stopgap |
Possible Resolutions
Section titled “Possible Resolutions”Same paths as GKGCircuitBreakerOpen: restore the dependency behind the named breaker. Because rejections here scale with incoming traffic, consider whether the callers retrying into the open breaker should back off; sustained high rejection with a recovered dependency means the breaker’s probe never succeeds, which is a bug worth filing with the breaker name.
Dependencies
Section titled “Dependencies”- Whichever dependency the
servicelabel names.
Escalation
Section titled “Escalation”Ask in #f_orbit_dev (context_systems). If user-facing error rates are
elevated at the same time, treat the incident at the dependency’s severity,
not this alert’s.
Definitions
Section titled “Definitions”- Alert rule (hand-maintained; keep the expr above in sync when tuning it).