Orbit NATS JetStream SLO alerts

Covers OrbitServiceNatsJetstreamErrorSLOViolation, OrbitServiceNatsJetstreamTrafficCessation and OrbitServiceNatsJetstreamTrafficAbsent.

Overview

JetStream is the persistence layer on the NATS cluster: it stores the indexing job streams and the KV buckets (locks, indexing progress).

These alerts are severity s3 and do not page. The SLI is tagged platform_insights, which has no SLO-alert Slack routing in services/teams.yml, so these alerts go to the default alert feed rather than a team channel.

Services

Orbit service overview
NATS runbook
SLI owner team: platform_insights (the Siphon/DIP platform team, also behind the NATS cluster). The Orbit indexers consuming NATS are owned by context_systems.

Alerts

OrbitServiceNatsJetstreamErrorSLOViolation

Message redeliveries are rising. A redelivery means a consumer took a message but did not acknowledge it in time, so JetStream handed it out again. This is how indexer failures self-heal, but a rising rate means work is failing repeatedly.

OrbitServiceNatsJetstreamTrafficCessation / TrafficAbsent

No JetStream messages are moving. TrafficCessation means traffic recently dropped to zero, TrafficAbsent means it has been missing for longer.

Metrics

The nats_jetstream SLI in metrics-catalog/services/orbit.jsonnet uses JetStream stream and consumer metrics (total messages, redeliveries) from the Prometheus exporter.

A low background redelivery rate is normal during deploys and pod restarts. Sustained growth is not.

Alert Behavior

Silence via Alertmanager filtered on the alertname.
Redeliveries spike during indexer rollouts (in-flight messages get redelivered). A spike that lines up with a deploy and settles within minutes is expected.

Severities

s3, no paging. User impact is indexing delay. A redelivery loop on one subject can also starve other work on the same consumer.

Verification

The alerts fire on these recording rules in the mimir-analytics-eventsdot datasource (Grafana Explore):

# Error ratio
gitlab_component_errors:ratio_1h{type="orbit", component="nats_jetstream", monitor="global"}

# Request rate (traffic cessation fires when this drops to zero)
gitlab_component_ops:rate_1h{type="orbit", component="nats_jetstream", monitor="global"}

NATS dashboard, JetStream panels: redeliveries, ack pending, consumer lag.
Per-stream and per-consumer state from nats-box: Check NATS stream health.
Cross-check which handler is failing on the indexer dashboard; redeliveries are the flip side of handler errors.

Recent changes

argocd-apps MRs for services nats and gkg. Stream, consumer or subject config changes are the usual suspects.
Production issues labeled Service::Orbit.

Troubleshooting

Symptom	Cause	Action
Redeliveries concentrated on one subject	Poison message: a task that fails on every attempt	Find the failing handler in the logs, then purge the subject or wait for dead-lettering (code tasks)
Redeliveries across all subjects	Consumers too slow: ClickHouse insert latency or undersized pools	Check ClickHouse first, then scale the indexer pools
Consumer stopped consuming after a config change	JetStream does not apply config changes to existing streams or buckets in place	Delete the stream or KV bucket and restart the pod so Orbit recreates it (Common signatures)
Ack pending grows, disk follows	Consumers lagging, persisted messages accumulate	Scale indexers; disk has its own paging alert (`nats_disk_space`) with runbook docs/nats/operations.md

Dead letter queue procedures (inspect, replay, purge) are in the KG repo’s code indexing runbook.

Possible Resolutions

Remove or dead-letter the poison message, then replay once the cause is fixed.
Scale the consuming pool or relieve ClickHouse pressure (Pause and resume indexing).

Dependencies

Consumers: indexer pools. Producers: dispatcher, Siphon. Storage: JetStream file storage on the NATS PVCs, backed up by the Velero schedule velero-nats-backup.

Escalation

Ask in #g_analytics_platform_insights (platform_insights, owns the NATS layer) or #f_orbit_dev (context_systems, owns the Orbit consumers).
Generic SLO playbooks: ErrorSLOViolation, TrafficAbsent.

Definitions

Alert rules (generated, do not edit).
Tune the SLI, then run make generate.