Orbit code indexer SLO alerts

Covers OrbitServiceGkgIndexerCodeApdexSLOViolation, OrbitServiceGkgIndexerCodeErrorSLOViolation, OrbitServiceGkgIndexerCodeTrafficCessation and OrbitServiceGkgIndexerCodeTrafficAbsent.

Overview

The code indexer pool downloads repository archives through the GitLab internal API, parses code and writes definitions, references and imports to the graph ClickHouse.

No user-facing errors: queries keep working, code data goes stale. These alerts are severity s3 and do not page. They post to #f_orbit_alerts.

Services

Orbit service overview
Owner team: context_systems

Alerts

OrbitServiceGkgIndexerCodeApdexSLOViolation

Code indexing tasks are slower than 10s (satisfied) / 30s (tolerated). The apdex target is lowered to 0.9 because task duration scales with repository size.

OrbitServiceGkgIndexerCodeErrorSLOViolation

Indexing tasks are failing. After the retry budget, failed tasks go to the dead letter queue.

OrbitServiceGkgIndexerCodeTrafficCessation / TrafficAbsent

The pool stopped consuming tasks. TrafficCessation means traffic recently dropped to zero, TrafficAbsent means it has been missing for longer.

Metrics

The gkg_indexer_code SLI in metrics-catalog/services/orbit.jsonnet measures gkg_etl_handler_duration_seconds for the code_indexing_task handler. Code indexing is CPU bound (parsing) and needs memory plus ephemeral disk for archives, so resource saturation shows up here first.

Alert Behavior

Silence via Alertmanager filtered on the alertname.
A single huge repository can drag the apdex down without anything being broken. Check whether failures concentrate on one project before treating it as a service problem.

Severities

s3, no paging. User impact is stale code data (definitions, references) in Orbit results and code tools.

Verification

The alerts fire on these recording rules in the mimir-analytics-eventsdot datasource (Grafana Explore):

# Apdex (1 means all requests were fast enough)
gitlab_component_apdex:ratio_1h{type="orbit", component="gkg_indexer_code", monitor="global"}

# Error ratio
gitlab_component_errors:ratio_1h{type="orbit", component="gkg_indexer_code", monitor="global"}

# Request rate (traffic cessation fires when this drops to zero)
gitlab_component_ops:rate_1h{type="orbit", component="gkg_indexer_code", monitor="global"}

Orbit Overview dashboard, gkg_indexer_code row.
Indexer dashboard.
Pod-level checks (OOM kills, evictions, restarts): see Status commands.
Logs: Kibana, orbit index, filter on the project id to follow one repository.

Recent changes

argocd-apps MRs touching services/gkg.
Production issues labeled Service::Orbit.

Troubleshooting

The detailed procedures live in the KG repo’s code indexing runbook: dead letter queue, stuck tasks, force reindex.

Symptom	Cause	Action
OOM kills on the code pool	A repository does not fit the memory limit	Raise the pool memory limit, see Indexers
Pod evictions for ephemeral storage	Archive larger than `tmpSizeLimit`	Raise `tmpSizeLimit` and the ephemeral-storage limit together (Indexers)
Archive downloads fail	Gitaly, Workhorse or the GitLab internal API is unhealthy	Check those services, see Code archive download flow
ClickHouse write fails with `Column '_deleted' is not presented ... (THERE_IS_NO_COLUMN)`	Known code bug, not config	No infra fix, report to `context_systems`
Same task fails repeatedly	Poison task in a redelivery loop	Message stuck in redelivery loop
Tasks vanish without completing	Dead-lettered after retries	Inspect the DLQ, replay after fixing the cause

Possible Resolutions

Scale the code pool (indexer.pools.code.replicas), capped by the gkg-pool node pool maximum, see Scalability.
Raise memory / ephemeral limits for large repositories.
Replay dead-lettered tasks once the underlying cause is fixed (KG runbook above).

Dependencies

Gitaly and Workhorse (archives), GitLab webservice (internal Orbit API), NATS JetStream (queue), graph ClickHouse (writes).

Escalation

Ask in #f_orbit_dev (team channel for context_systems).
Generic SLO playbooks: ApdexSLOViolation, ErrorSLOViolation, TrafficAbsent.

Definitions

Alert rules (generated, do not edit).
Tune the SLI, then run make generate.