GKGQueryingErrorRateHigh
Aggregate query error rate across all failure modes is above 5% for 5 minutes (the availability SLI). Severity s3, fires after 5m.
Overview
Section titled “Overview”The aggregate error rate across the whole query pipeline (compilation, security, execution) exceeds 5%. This is the availability SLI for querying: users are getting failed Orbit API responses. Compile errors caused by bad client input count toward this rate, so a single broken client with enough volume can trip it without any server-side fault.
Services
Section titled “Services”- Orbit service overview
- Owner team:
context_systems
Metrics
Section titled “Metrics”The firing expression, on gkg_query_pipeline_queries_total (emitted by the service, see the
metrics catalog):
sum by (env, environment, stage) (rate(gkg_query_pipeline_queries_total{status!="ok"}[5m])) / sum by (env, environment, stage) (rate(gkg_query_pipeline_queries_total[5m])) > 0.05Alert Behavior
Section titled “Alert Behavior”- Silence via Alertmanager filtered on the alertname.
- Client-caused compile errors count toward this rate, so a single broken high-volume client can trip it without a server fault.
Severities
Section titled “Severities”- s3, no paging. This is the availability SLI for querying: users are getting failed Orbit API responses.
Verification
Section titled “Verification”Break the rate down by failure mode to find what is actually failing:
sum by (status) (rate(gkg_query_pipeline_queries_total{status!="ok"}[5m]))compile_errordominating: likely a client problem, cross-checkGKGValidationFailedBurst.executiondominating: ClickHouse is failing queries, check the graph ClickHouse health and load.securitydominating: cross-checkGKGSecurityRejectedandGKGAuthFilterMissing.
Recent changes
Section titled “Recent changes”- argocd-apps MRs touching
services/gkg(a merge is a deploy). - Production issues labeled Service::Orbit.
Troubleshooting
Section titled “Troubleshooting”| Symptom | Cause | Action |
|---|---|---|
compile_error dominates the breakdown | Broken client input | Identify the client; cross-check GKGValidationFailedBurst |
execution dominates | ClickHouse failing queries | Follow GKGExecutionFailureRate |
authorization dominates | Rails redaction exchange failing | Follow GKGAuthorizationFailureRate |
security dominates | JWT problems | Follow GKGSecurityRejected |
Possible Resolutions
Section titled “Possible Resolutions”- Execution failures: reduce ClickHouse pressure (pause indexing pools) or scale the ClickHouse service; a heavy indexing burst degrading reads is the most common cause.
- Compile-error bursts from one caller: identify the client in the logs and fix or rate-limit it; the server needs no change.
- A fresh deploy lining up with the alert: roll back the image tag.
Dependencies
Section titled “Dependencies”- Graph ClickHouse (ClickHouse Cloud)
- GitLab Rails (JWT minting, internal Orbit API)
- Workhorse (result streaming).
Escalation
Section titled “Escalation”Ask in #f_orbit_dev (context_systems). If ClickHouse itself is unhealthy,
check the ClickHouse Cloud console before escalating further.
Definitions
Section titled “Definitions”- Alert rule (hand-maintained; keep the expr above in sync when tuning it).