OrbitSdlcDataOlderThan4h

Overview

At least one enabled namespace and SDLC entity pair was more than 4 hours behind the datalake for 15 minutes. Users read SDLC data (issues, merge requests, pipelines) that is more than 4 hours old for these namespaces.

This is a hard limit with no error budget. Severity s3, no paging, posts to #f_orbit_alerts.

Services

Orbit service overview
Owner team: context_systems

Metrics

The dispatcher records gkg_indexer_sdlc_freshness_lag_seconds for every pair every 5 minutes. The alert counts the samples above the le="14400.0" bucket in the last 10 minutes. See sdlc-freshness for what one sample means.

A schema migration under 4 hours cannot fire this alert. A migration that runs longer fires it for every namespace, which is also how a stuck migration shows up.

Verification

# Pairs more than 4 hours behind, by entity
sum by (entity) (increase(gkg_indexer_sdlc_freshness_lag_seconds_count{env="orbit-prd"}[10m]))
  - sum by (entity) (increase(gkg_indexer_sdlc_freshness_lag_seconds_bucket{env="orbit-prd", le="14400.0"}[10m]))

# Migration age in seconds, 0 when idle
max(gkg_schema_migrating_age_seconds{env="orbit-prd"})

Indexer dashboard, “Behind 1 h” tile and the freshness lag chart.

Troubleshooting

Symptom	Cause	Action
Every entity of one or few namespaces	The namespace cannot finish a sync, or never finished its first sync	Find the namespace in the indexer logs, check `glab orbit remote status` for it
Every namespace, migration age above 4 hours	Slow or stuck full-rebuild migration	Check the migration completion log line and knowledge-graph#1004
Every namespace, no migration	Indexing stopped for more than 4 hours	Check the indexer and dispatcher alerts, pods and NATS

Escalation

Ask in #f_orbit_dev (team channel for context_systems).

Definitions

Alert rule, then run make generate.