Skip to content
Runbooks

CiOrchestrationServicePipelinesFinishedRequestRateDrop

Overview

Section titled “Overview”

This alert compares the rate at which CI pipelines finish with status success against a baseline: the median of the same rate at the same time of week over the three prior weeks. It fires when the ratio of the two stays below 0.5 for 15 minutes, meaning half the usual number of pipelines or fewer are succeeding.

It is generated by the metrics catalog’s request rate drop alerts, configured with requestRateDropAlertConfig on the pipelines_finished SLI.

It watches successes rather than all completions. When pipelines fail in bulk, failed pipelines take the place of successful ones and the total barely moves. A drop in successes catches both that and a stall where pipelines stop completing at all.

Impact

Section titled “Impact”

Pipelines are not completing successfully for many users. Either they are stuck (running or pending without progress) or they are failing.

Contributing Factors

Section titled “Contributing Factors”
  • Pipeline creation has dropped, so fewer pipelines reach completion (see CiOrchestrationServicePipelinesCreatedRequestRateDrop)
  • A site-wide outage or a degraded dependency (Redis, PostgreSQL, Gitaly) failing jobs or stalling processing
  • Pipeline processing workers stalled or erroring, so pipelines do not transition to a completed status
  • Runners not picking up jobs, or jobs failing on runner infrastructure
  • A deploy or feature flag change on the processing or job execution path
  • A legitimate traffic change: a holiday, a large customer pausing CI
Section titled “Related alerts”
AlertConditionSustainedSeverity
CiOrchestrationServicePipelinesFinishedRequestRateDropSuccessful completions below 0.5 of their baseline, or absent while other statuses report15ms3, Slack
CiOrchestrationServicePipelinesCreatedRequestRateDropPer-source creation rate below 0.5 of its baseline15ms3, Slack
CiOrchestrationServicePipelinesFinishedTrafficCessationTotal completion rate is exactly zero~35ms3, Slack
CiOrchestrationServicePipelinesFinishedTrafficAbsentNo pipelines_finished_total series at all30ms3, Slack
Pipeline processing SLO violationsLatency or errors in the pipeline processing workerss2, pages
CiOrchestrationServiceJobInfraFailureRatioErrorSLOViolationJobs failing for infrastructure reasonss2, pages

If the creation drop alert is firing too, start there: completions follow creation. If a paging alert is firing, treat it as authoritative and this alert as supporting detail.

Not covered:

  • A drop in failed or canceled completions. They swing too much from day to day for a ratio threshold.
  • Failures limited to a few projects or customers, which do not move the global rate.

Thresholds

Section titled “Thresholds”

The threshold of 0.5 comes from replaying gprd from 2026-07-31 to 2026-11-05 (!12057). It fired for the two incidents in that window where successes stayed below 0.5 of their baseline for 15 minutes or more, production#23018 and production#23194. It did not fire for two shorter or shallower drops: production#23261, where successes fell to 0.11 for about 10 minutes, and production#23191, where they only fell to 0.55. The only other firing was a Sidekiq metrics gap on 2026-08-18 (see Alert Behavior). The lowest ratio on a normal day was 0.59, on US Labor Day.

The ratio in Grafana Explore shows the last 30 days.

Alert Behavior

Section titled “Alert Behavior”
  • Severity s3, Slack-only (#s_verify_alerts), no paging.
  • Detection takes roughly 15–20 minutes from the start of a drop. A pipeline finishes some time after it is created, so a stall shows up here later than in the creation rate.
  • Sidekiq metrics gaps fire this alert. pipelines_finished_total is mostly emitted by Sidekiq, and a little by API and web. If Sidekiq metrics stop reaching Mimir, the series stays present near zero, which looks the same as a real stall. Check for that first (troubleshooting step 1).
  • An absent success series counts as zero while other statuses are still reported. When the whole SLI is missing, this alert does not fire and CiOrchestrationServicePipelinesFinishedTrafficAbsent covers it.
  • Silencing during a planned event that suppresses CI traffic: match on alertname="CiOrchestrationServicePipelinesFinishedRequestRateDrop", not on component=pipelines_finished alone, which also silences the traffic cessation and traffic absent alerts.
  • Holidays: the median tolerates one unusual week, not several. Expect firings over the late-December holidays; consider a planned silence for that window.
  • Daylight saving changes shift that region’s traffic by an hour against the baseline for about two weeks, and the morning ramp reads lower than usual.

Troubleshooting

Section titled “Troubleshooting”

The Slack alert links to Grafana Explore with the current rate and its baseline as two queries. The Pipeline Processing section of the Pipeline Observability dashboard has the surrounding context.

1. Rule out a metrics gap

Section titled “1. Rule out a metrics gap”
sum by (type) (sli_aggregations:pipelines_finished_total:rate_5m{env="gprd"})

If the sidekiq series is missing while api and web remain, Sidekiq metrics are not reaching Mimir. Check up{type="sidekiq", env="gprd"}; if that is missing too, it is a monitoring problem, not a CI one.

2. Check whether creation dropped too

Section titled “2. Check whether creation dropped too”
sum(sli_aggregations:pipelines_created_total:rate_5m{env="gprd"})

If creation dropped at the same time, follow the creation drop runbook.

3. Failing or stuck?

Section titled “3. Failing or stuck?”
sum by (status) (sli_aggregations:pipelines_finished_total:rate_5m{env="gprd"})

4. Check recent deployments and feature flags

Section titled “4. Check recent deployments and feature flags”

5. Rule out a legitimate traffic change

Section titled “5. Rule out a legitimate traffic change”

Compare against the same hour on prior weeks (the baseline series) before escalating.

Escalation

Section titled “Escalation”

Escalate when:

  • The drop persists beyond 30 minutes with no identified cause
  • The ratio is below ~0.2 (near-total loss of successful pipelines)
  • Customers report pipelines failing or not finishing

This alert does not page. Raise it in #production and declare an incident yourself if needed.

Support Channels

Section titled “Support Channels”
  • #s_verify_alerts (primary)
  • #g_pipeline-execution (owning group)
  • #production

Dependencies

Section titled “Dependencies”
  • Sidekiq: runs the pipeline processing workers and emits most of the metric
  • Runners: execute the jobs whose results complete the pipeline
  • PostgreSQL (CI): pipeline and job state
  • Redis: Sidekiq queuing

Definitions

Section titled “Definitions”
Section titled “Related Links”