Skip to content
Runbooks

Orbit webserver SLO alerts

Covers OrbitServiceGkgWebserverApdexSLOViolation, OrbitServiceGkgWebserverErrorSLOViolation, OrbitServiceGkgWebserverTrafficCessation and OrbitServiceGkgWebserverTrafficAbsent.

Overview

Section titled “Overview”

The webserver serves the graph over gRPC (KnowledgeGraphService). Every Orbit query a user runs through /api/v4/orbit/* ends up here.

These alerts are severity s3 and do not page. They post to #f_orbit_alerts.

Services

Section titled “Services”

Alerts

Section titled “Alerts”

OrbitServiceGkgWebserverApdexSLOViolation

Section titled “OrbitServiceGkgWebserverApdexSLOViolation”

gRPC queries are slower than the 5s (satisfied) / 10s (tolerated) thresholds. Users see slow Orbit API responses.

OrbitServiceGkgWebserverErrorSLOViolation

Section titled “OrbitServiceGkgWebserverErrorSLOViolation”

The rate of non-OK gRPC status codes is rising. Users see failed queries.

OrbitServiceGkgWebserverTrafficCessation / TrafficAbsent

Section titled “OrbitServiceGkgWebserverTrafficCessation / TrafficAbsent”

The webserver stopped receiving queries. Either nobody can reach it (Workhorse, LB, or the pods) or the alert fired on a real usage gap. TrafficCessation means traffic recently dropped to zero, TrafficAbsent means it has been missing for longer.

Metrics

Section titled “Metrics”

The gkg_webserver SLI in metrics-catalog/services/orbit.jsonnet measures rpc_server_duration_seconds on the webserver pods. Errors are responses with a non-OK gRPC status code. Normal query latency is 2-5s, so the 5s/10s apdex thresholds sit just above the healthy range.

Under normal conditions the apdex stays above 0.99 and the error ratio below 5%.

Alert Behavior

Section titled “Alert Behavior”
  • Silence via Alertmanager filtered on the alertname.
  • Expected to be rare. The most common real trigger is ClickHouse load, not the pods.

Severities

Section titled “Severities”
  • s3, no paging. Impact is degraded or failed Orbit API queries for gitlab.com users.
  • If the whole query path is down (all queries fail), consider raising an incident at higher severity per the incident severity guide.

Verification

Section titled “Verification”

The alerts fire on these recording rules in the mimir-analytics-eventsdot datasource (Grafana Explore):

# Apdex (1 means all requests were fast enough)
gitlab_component_apdex:ratio_1h{type="orbit", component="gkg_webserver", monitor="global"}


# Error ratio
gitlab_component_errors:ratio_1h{type="orbit", component="gkg_webserver", monitor="global"}


# Request rate (traffic cessation fires when this drops to zero)
gitlab_component_ops:rate_1h{type="orbit", component="gkg_webserver", monitor="global"}

Recent changes

Section titled “Recent changes”

Troubleshooting

Section titled “Troubleshooting”

Work through Order of checks. Webserver-specific signatures from Common signatures:

SymptomCauseAction
All query calls return HTTP 502 after ~15s, status and schema still work, logs show LazyLock instance has previously been poisonedBad image (missing analytics config), one panic poisons all queriesRoll back the image tag
Queries fail with Code: 159 ... TIMEOUT_EXCEEDEDHeavy query hits the ClickHouse 30s execution timeoutCheck ClickHouse load and query weight, not the pods
First queries after a quiet period are slow, then recoverClickHouse Cloud idle_scaling waking upNo action, expected
Latency rises together with indexer write volumeShared graph ClickHouse, indexing burst degrades readsPause indexing pools (Pause and resume indexing)

Possible Resolutions

Section titled “Possible Resolutions”
  • Bad deploy: revert image.tag in argocd-apps (see Recent changes and rollback).
  • ClickHouse write pressure: pause the indexer pools, queries keep working while data goes stale.
  • Webserver OOM: scale memory as a stopgap, then report the unoptimized query to context_systems.

Dependencies

Section titled “Dependencies”
  • Graph ClickHouse (ClickHouse Cloud) dominates query performance.
  • Workhorse and GitLab Rails sit in front: if they are down, traffic ceases here even though Orbit is healthy. See Query flow.

Escalation

Section titled “Escalation”

Definitions

Section titled “Definitions”
Section titled “Related Links”