Orbit NATS server SLO alerts

Covers OrbitServiceNatsServerErrorSLOViolation, OrbitServiceNatsServerTrafficCessation and OrbitServiceNatsServerTrafficAbsent.

Overview

NATS is the message broker between Siphon, the dispatcher and the indexers. This SLI watches core NATS message flow.

These alerts are severity s3 and do not page. The SLI is tagged platform_insights, which has no SLO-alert Slack routing in services/teams.yml, so these alerts go to the default alert feed rather than a team channel. Disk saturation has its own paging alert (nats_disk_space, s2), with its runbook at docs/nats/operations.md.

Services

Orbit service overview
NATS runbook
SLI owner team: platform_insights (the Siphon/DIP platform team, also behind the NATS cluster). The Orbit indexers consuming NATS are owned by context_systems.

Alerts

OrbitServiceNatsServerErrorSLOViolation

Slow consumers are rising. A slow consumer is a client that cannot keep up with the messages NATS is sending it; NATS drops it after a buffer limit.

OrbitServiceNatsServerTrafficCessation / TrafficAbsent

No messages are flowing at all. Either the producers stopped (dispatcher, Siphon) or the cluster itself is unhealthy. TrafficCessation means traffic recently dropped to zero, TrafficAbsent means it has been missing for longer.

Metrics

The nats_server SLI in metrics-catalog/services/orbit.jsonnet uses the NATS varz metrics (message in-rate, slow consumer count) from the Prometheus exporter sidecar.

Slow consumers should be zero in normal operation. Any sustained non-zero rate means a consumer is undersized or stuck.

Alert Behavior

Silence via Alertmanager filtered on the alertname.
Slow consumers usually track indexer health: an underscaled or crashing indexer pool shows up here before anywhere else.

Severities

s3, no paging. User impact is indexing delay (stale data), queries are unaffected.

Verification

The alerts fire on these recording rules in the mimir-analytics-eventsdot datasource (Grafana Explore):

# Error ratio
gitlab_component_errors:ratio_1h{type="orbit", component="nats_server", monitor="global"}

# Request rate (traffic cessation fires when this drops to zero)
gitlab_component_ops:rate_1h{type="orbit", component="nats_server", monitor="global"}

NATS dashboard for message rates, slow consumers, JetStream state.
Cluster state: kubectl --context gke_gl-orbit-prd_us-east1_orbit-prd -n nats get statefulset,pod,pvc.
Stream and consumer detail from a nats-box pod: Troubleshoot NATS with nats-box.

Recent changes

argocd-apps MRs for services nats and gkg.
Production issues labeled Service::Orbit.

Troubleshooting

Symptom	Cause	Action
Slow consumers rising, indexer pods restarting	Indexer pool unhealthy or undersized	Fix or scale the indexers, not NATS. See indexer SDLC / indexer code pages
Traffic ceased, NATS pods healthy	Producers stopped: dispatcher down or Siphon CDC dry	See the dispatcher page and the Siphon dashboard
NATS pod down, cluster degraded	Node or PVC problem	Standard kube triage; JetStream replicates within the cluster, consumers reconnect
Disk usage growing	Lagging consumers accumulate persisted stream data	Scale the indexers. For emergency disk mitigation see docs/nats/operations.md

Possible Resolutions

Scale or fix the consuming indexer pools; NATS itself rarely needs scaling (see NATS).
mTLS certificate problems surface as connection errors on Orbit pods; certificates are cert-manager issued, check their age and events.

Dependencies

Producers: Siphon CDC, the dispatcher. Consumers: the indexer pools. If NATS is fully down, all indexing stops and NATS KV (locks and progress) is unavailable, so graph-status degrades. See What degrades if X fails.

Escalation

Ask in #g_analytics_platform_insights (platform_insights, owns the NATS layer) or #f_orbit_dev (context_systems, owns the Orbit consumers).
Generic SLO playbooks: ErrorSLOViolation, TrafficAbsent.

Definitions

Alert rules (generated, do not edit).
Tune the SLI, then run make generate.