Skip to content
Runbooks

Orbit query SLO alerts

Covers OrbitServiceOrbitQueryApdexSLOViolation, OrbitServiceOrbitQueryErrorSLOViolation, OrbitServiceOrbitQueryTrafficCessation and OrbitServiceOrbitQueryTrafficAbsent.

Overview

Section titled “Overview”

The webserver serves the graph over gRPC (KnowledgeGraphService). Every Orbit query a user runs through /api/v4/orbit/* ends up here.

Objectives:

  • 99% of valid queries finish without a server fault.
  • 95% of valid queries finish within 5 seconds.

These alerts are severity s3 and do not page. They post to #f_orbit_alerts.

Services

Section titled “Services”

Alerts

Section titled “Alerts”

OrbitServiceOrbitQueryErrorSLOViolation

Section titled “OrbitServiceOrbitQueryErrorSLOViolation”

Server faults use the error budget too fast. Users see failed queries. Server faults are ClickHouse errors (including the time and memory limits), timeouts, failed permission checks with Rails, content errors and compiler faults.

OrbitServiceOrbitQueryApdexSLOViolation

Section titled “OrbitServiceOrbitQueryApdexSLOViolation”

Too many queries take more than 5 seconds. Users see slow Orbit API responses. Timeouts record about 60 seconds, so they count as slow.

OrbitServiceOrbitQueryTrafficCessation / TrafficAbsent

Section titled “OrbitServiceOrbitQueryTrafficCessation / TrafficAbsent”

TrafficAbsent means the query metrics stopped arriving, for example after a rename or when the webserver pods stop exporting. TrafficCessation cannot fire at today’s traffic, because its guard needs more than 300 queries in 30 minutes.

Metrics

Section titled “Metrics”

The orbit_query SLI in metrics-catalog/services/orbit.jsonnet reads two metrics from the webserver pods:

  • gkg_query_pipeline_queries_total: valid queries have outcome!="caller_error", errors have outcome="server_error".
  • gkg_query_pipeline_duration_seconds: the le="5.0" bucket for valid queries. Schema calls (query_type="unknown") are left out, because they take about 10 ms.

The timer starts when the query pipeline starts and stops after the last result is sent. Caller errors (bad queries, cancels, disconnects, unreadable filter values, content over the row limit) are not valid events.

Traffic is low, about 200 queries per hour. The alerts use a 98% confidence bound and need at least 120 queries in a window. A 3-day window catches slow burns.

Alert Behavior

Section titled “Alert Behavior”
  • Silence via Alertmanager filtered on the alertname.
  • One caller that sends many slow queries can move the apdex. Check the slowest query types before you treat it as a service problem.

Severities

Section titled “Severities”
  • s3, no paging. Impact is slow or failed Orbit API queries for gitlab.com users.
  • If all queries fail, consider raising an incident at higher severity per the incident severity guide.

Verification

Section titled “Verification”

The alerts fire on these recording rules in the mimir-analytics-eventsdot datasource (Grafana Explore):

# Share of valid queries within 5 s
gitlab_component_apdex:ratio_1h{type="orbit", component="orbit_query", monitor="global"}


# Share of valid queries with a server fault
gitlab_component_errors:ratio_1h{type="orbit", component="orbit_query", monitor="global"}


# Server faults by status
sum by (status) (increase(gkg_query_pipeline_queries_total{env="orbit-prd", outcome="server_error"}[1h]))

Recent changes

Section titled “Recent changes”

Troubleshooting

Section titled “Troubleshooting”

Work through Order of checks. Webserver-specific signatures from Common signatures:

SymptomCauseAction
All query calls return HTTP 502 after ~15s, status and schema still work, logs show LazyLock instance has previously been poisonedBad image (missing analytics config), one panic poisons all queriesRoll back the image tag
Server faults with status="execution_error" and Code: 159 ... TIMEOUT_EXCEEDEDHeavy query hits the ClickHouse execution timeoutCheck ClickHouse load and query weight, not the pods
Server faults with status="timeout"The 60 s stream timeout, often a slow Rails permission checkCheck Rails internal API latency
First queries after a quiet period are slow, then recoverClickHouse Cloud idle_scaling waking upNo action, expected
Latency rises together with indexer write volumeShared graph ClickHouse, indexing burst degrades readsPause indexing pools (Pause and resume indexing)

Possible Resolutions

Section titled “Possible Resolutions”
  • Bad deploy: revert image.tag in argocd-apps (see Recent changes and rollback).
  • ClickHouse write pressure: pause the indexer pools, queries keep working while data goes stale.
  • Webserver OOM: scale memory as a stopgap, then report the query to context_systems.

Dependencies

Section titled “Dependencies”
  • Graph ClickHouse (ClickHouse Cloud) dominates query performance.
  • Workhorse and GitLab Rails sit in front: if they are down, traffic stops here even though Orbit is healthy. See Query flow.

Escalation

Section titled “Escalation”

Definitions

Section titled “Definitions”
Section titled “Related Links”