GKGAuthorizationFailureRate
The Rails authorization exchange is failing, above 1 failure per minute; redaction callbacks are not completing. Severity s3, fires after 5m.
Overview
Section titled “Overview”Queries whose results need redaction are failing because the authorization exchange with Rails is not completing. In the query flow, GKG sends resource ids back over the Workhorse stream and Workhorse calls the Rails internal redaction endpoint with the user’s credentials; if Rails is slow, erroring, or unreachable, the query fails rather than returning unredacted rows.
Services
Section titled “Services”- Orbit service overview
- Owner team:
context_systems
Metrics
Section titled “Metrics”The firing expression, on gkg_query_pipeline_failed_total (emitted by the service, see the
metrics catalog):
sum by (env, environment, stage) (rate(gkg_query_pipeline_failed_total{failure_reason="authorization"}[5m])) * 60 > 1Alert Behavior
Section titled “Alert Behavior”- Silence via Alertmanager filtered on the alertname.
- Only queries whose results need redaction take this path; queries without redaction keep working while it fires.
Severities
Section titled “Severities”- s3, no paging. Impact is failed queries for results needing redaction.
Verification
Section titled “Verification”sum by (env, environment, stage) (rate(gkg_query_pipeline_failed_total{failure_reason="authorization"}[5m])) * 60Check GitLab webservice health (this is the Rails internal API, so Rails error rates and saturation apply) and the webserver logs for the redaction call errors. If the GitLab side is healthy, check whether the failures correlate with one user or namespace: an oversized authorization payload can fail deterministically.
Recent changes
Section titled “Recent changes”- argocd-apps MRs touching
services/gkg(a merge is a deploy). - Production issues labeled Service::Orbit.
Troubleshooting
Section titled “Troubleshooting”| Symptom | Cause | Action |
|---|---|---|
| GitLab webservice degraded overall | Rails-side incident | Resolves with the GitLab webservice; follow standard GitLab escalation |
| Failures pinned to one user or namespace | Oversized authorization payload failing deterministically | Capture the query shape from the logs and file an issue |
| Started with a GitLab deploy | Internal Orbit API regression in the monolith | Check recent changes to the Orbit internal API |
Possible Resolutions
Section titled “Possible Resolutions”- Rails degraded: this resolves when the GitLab webservice recovers; nothing to do on the GKG side. Queries not needing redaction keep working.
- Failures pinned to one query shape or namespace: capture it from the logs and file an issue; the redaction batch size may need tuning.
- Started with a GitLab deploy: the internal endpoint may have regressed; check recent changes to the Orbit internal API in the monolith.
Dependencies
Section titled “Dependencies”- GitLab Rails (JWT minting, internal Orbit API) answers the redaction callbacks.
- Workhorse carries the redaction round-trip.
Escalation
Section titled “Escalation”Ask in #f_orbit_dev (context_systems). If the Rails internal API itself
is down, follow the standard GitLab production escalation instead.
Definitions
Section titled “Definitions”- Alert rule (hand-maintained; keep the expr above in sync when tuning it).