Orbit SDLC indexer SLO alerts

Covers OrbitServiceGkgIndexerSdlcApdexSLOViolation, OrbitServiceGkgIndexerSdlcErrorSLOViolation, OrbitServiceGkgIndexerSdlcTrafficCessation and OrbitServiceGkgIndexerSdlcTrafficAbsent.

Overview

The SDLC indexer pool consumes entity events (users, projects, MRs, pipelines, …) from NATS and writes graph data to the graph ClickHouse.

No user-facing errors: queries keep working, data goes stale. These alerts are severity s3 and do not page. They post to #f_orbit_alerts.

Services

Orbit service overview
Owner team: context_systems

Alerts

OrbitServiceGkgIndexerSdlcApdexSLOViolation

ETL handlers are slower than 2.5s (satisfied) / 5s (tolerated).

OrbitServiceGkgIndexerSdlcErrorSLOViolation

Handlers are failing. Failed messages are redelivered by JetStream, so errors also show up as nats_jetstream redeliveries.

OrbitServiceGkgIndexerSdlcTrafficCessation / TrafficAbsent

The pool stopped consuming messages. Either the pods are down, the NATS consumer is stuck, or nothing upstream (Siphon, dispatcher) is producing. TrafficCessation means traffic recently dropped to zero, TrafficAbsent means it has been missing for longer.

Metrics

The gkg_indexer_sdlc SLI in metrics-catalog/services/orbit.jsonnet measures gkg_etl_handler_duration_seconds for the entity.* handlers. SDLC handlers are light; their latency is bound by ClickHouse insert latency and NATS fetch rate, not pod resources.

Alert Behavior

Silence via Alertmanager filtered on the alertname.
A short error burst during a deploy or a ClickHouse hiccup self-heals through JetStream redelivery. Sustained violation means something is actually stuck.

Severities

s3, no paging. User impact is stale SDLC data in Orbit query results, not errors.

Verification

The alerts fire on these recording rules in the mimir-analytics-eventsdot datasource (Grafana Explore):

# Apdex (1 means all requests were fast enough)
gitlab_component_apdex:ratio_1h{type="orbit", component="gkg_indexer_sdlc", monitor="global"}

# Error ratio
gitlab_component_errors:ratio_1h{type="orbit", component="gkg_indexer_sdlc", monitor="global"}

# Request rate (traffic cessation fires when this drops to zero)
gitlab_component_ops:rate_1h{type="orbit", component="gkg_indexer_sdlc", monitor="global"}

Orbit Overview dashboard, gkg_indexer_sdlc row.
Indexer dashboard for per-handler duration and error split.
Indexing progress for a namespace: glab orbit remote graph-status --full-path gitlab-org.
Logs: Kibana, orbit index, filter json.mode: indexer.

Recent changes

argocd-apps MRs touching services/gkg.
Production issues labeled Service::Orbit.

Troubleshooting

The detailed procedures live in the KG repo’s SDLC indexing runbook: checkpoint inspection, force reindex, purging stuck subjects.

Symptom	Cause	Action
Errors on the ClickHouse write	Graph ClickHouse slow or unreachable	Check ClickHouse Cloud status and load first
Traffic ceased, pods healthy	Upstream dry: Siphon CDC or the dispatcher stopped producing	Check the Siphon dashboard and the dispatcher alerts page
One namespace never finishes	Checkpoint stuck or corrupted cursor	Inspect checkpoints
Same batch fails repeatedly	Poison message in redelivery loop	Remove stuck messages from NATS
Consumer lag grows, no errors	Throughput bound	Scale the pool: `indexer.pools.sdlc.replicas` in the ArgoCD values (Scale or restart)

NATS state can be inspected from a nats-box pod, see Troubleshoot NATS with nats-box.

Possible Resolutions

Scale the SDLC pool horizontally, it has no ephemeral-storage constraint.
Purge a stuck subject and force a reindex of the affected namespace (KG runbook above). Jobs queued during a pause are redelivered, nothing is lost, only delayed.

Dependencies

Graph ClickHouse (writes), NATS JetStream (queue), datalake ClickHouse via the dispatcher (job source). See What degrades if X fails.

Escalation

Ask in #f_orbit_dev (team channel for context_systems).
Generic SLO playbooks: ApdexSLOViolation, ErrorSLOViolation, TrafficAbsent.

Definitions

Alert rules (generated, do not edit).
Tune the SLI, then run make generate.