Skip to content
Runbooks

CiOrchestrationServicePipelinesCreatedRequestRateDrop

Overview

Section titled “Overview”

This alert compares the CI pipeline creation rate, per source (push, merge_request_event, parent_pipeline), against a baseline: the median of the same rate at the same time of week over the three prior weeks. It fires when the ratio of the two stays below 0.5 for 15 minutes, meaning that source is creating half of its usual pipelines or fewer.

It is generated by the metrics catalog’s request rate drop alerts, configured with requestRateDropAlertConfig on the pipelines_created SLI.

The pipelines_created SLI has no apdex or error ratio, and the traffic cessation alert only fires when the total rate is exactly zero. This alert covers a drop in one source while the others continue.

It is based on the number of pipelines created, not on the health of any one worker or dependency, and therefore also catches failures that produce no errors or latency on the creation path.

Impact

Section titled “Impact”

Pipelines are likely failing to start for some or all users on that source.

Contributing Factors

Section titled “Contributing Factors”
  • Saturation of the urgent-ci-pipeline Sidekiq shard, starving MergeRequests::CreatePipelineWorker
  • Sidekiq concurrency limits throttling the pipeline creation workers
  • Application errors in the pipeline creation chain
  • Gitaly or database degradation slowing creation
  • A deploy or feature flag change on the creation path
  • A legitimate traffic change: a holiday, a large customer pausing CI
Section titled “Related alerts”
AlertConditionSustainedSeverity
CiOrchestrationServicePipelinesCreatedRequestRateDropPer-source rate below 0.5 of its baseline, or absent while other sources report15ms3, Slack
CiOrchestrationServicePipelinesCreatedTrafficCessationTotal rate is exactly zero~35ms2, pages
CiOrchestrationServicePipelinesCreatedTrafficAbsentNo pipelines_created_total series at all30ms2, pages

If one of the paging alerts fires, treat it as authoritative and this alert as supporting detail. The service_ops_out_of_bounds_* anomaly alerts do not cover this SLI (serviceAggregation: false excludes it from gitlab_service_ops).

Not covered by any of these:

  • A single cluster going quiet (about 20% of volume) is too small for the drop alert. Check per-cluster reporting (troubleshooting step 3) if you suspect it.
  • Any drop in the other sources (schedule, pipeline, api, trigger and the rest). They are too bursty or too low-volume for a ratio threshold. Only a drop to zero across all sources is caught.

Thresholds

Section titled “Thresholds”

The threshold (0.5 for all three sources) comes from replaying five months of gprd data. It caught every pipeline creation incident in that window. The only other firings were for parent_pipeline, on Saturdays and US holidays. If those turn out to be noisy, 0.35 removes them and detects a drop in push alone later.

The ratio in Grafana Explore shows the last 30 days.

Alert Behavior

Section titled “Alert Behavior”
  • Severity s3, Slack-only (#s_verify_alerts), no paging yet. It is the candidate for promotion to paging once its Slack firings show it is quiet outside real incidents.
  • Detection takes roughly 15–20 minutes from the start of a drop.
  • An absent source counts as zero while pipelines_created_total is still reported for other sources. The counter is created lazily per source, so a source that stops creating pipelines after a restart has no series, and this alert still fires. When the whole SLI is missing, for example during a Mimir ingestion gap, this alert does not fire and CiOrchestrationServicePipelinesCreatedTrafficAbsent covers it.
  • Silencing during a planned event that suppresses CI traffic (deploy freeze, maintenance window): match on alertname="CiOrchestrationServicePipelinesCreatedRequestRateDrop". Do not match on component=pipelines_created alone: that also silences the paging traffic cessation and traffic absent alerts.
  • Holidays: the median tolerates one unusual week, not several. Expect firings over the late-December holidays; consider a planned silence for that window.
  • Daylight saving changes (EU: last Sunday of March and October; US: second Sunday of March, first Sunday of November) shift that region’s traffic by an hour against the baseline for about two weeks. The morning ramp then reads lower than usual against the baseline.

Troubleshooting

Section titled “Troubleshooting”

The Slack alert links to Grafana Explore with the current rate and its baseline as two queries. The Pipeline Creation section of the Pipeline Observability dashboard has the surrounding context. To compare by hand:

gitlab_component_ops_anomaly:rate_5m{env="gprd", type="ci-orchestration", component="pipelines_created"}
/
gitlab_component_ops_anomaly:rate_5m:baseline{env="gprd", type="ci-orchestration", component="pipelines_created"}

1. Check the Sidekiq concurrency limit queue

Section titled “1. Check the Sidekiq concurrency limit queue”
sidekiq_concurrency_limit_queue_jobs{worker="MergeRequests::CreatePipelineWorker", env="gprd"}

This is normally zero. A backlog here means the concurrency limit is throttling pipeline creation.

2. Check the urgent-ci-pipeline Sidekiq shard

Section titled “2. Check the urgent-ci-pipeline Sidekiq shard”

On the Sidekiq Shard Detail dashboard, check saturation, queue depth and scaling for urgent-ci-pipeline. High queue depth with a flat dequeue rate means worker starvation.

3. Check whether one cluster is responsible

Section titled “3. Check whether one cluster is responsible”
sum by (cluster) (sli_aggregations:pipelines_created_total:rate_5m{env="gprd"})

If one cluster reports zero or disappears while others remain steady, check both its scrape/exporter health and its pipeline-creation workers. This signal alone cannot distinguish a telemetry failure from a cluster-local creation outage.

4. Check recent deployments and feature flags

Section titled “4. Check recent deployments and feature flags”

5. Rule out a legitimate traffic change

Section titled “5. Rule out a legitimate traffic change”

Compare against the same hour on prior weeks (the baseline series above) before escalating.

When querying manually, use sum(rate(...)), never rate(sum(...)): the number of reporting series changes as Sidekiq scales.

Escalation

Section titled “Escalation”

Escalate when:

  • The drop persists beyond 30 minutes with no identified cause
  • The ratio is below ~0.2 for any source (near-total loss that the paging cessation alert will not catch)
  • Customers report missing pipelines

This alert does not page. Raise it in #production and declare an incident yourself if needed.

Support Channels

Section titled “Support Channels”
  • #s_verify_alerts (primary)
  • #g_pipeline-execution / #g_pipeline-authoring (owning groups)
  • #production

Dependencies

Section titled “Dependencies”
  • Sidekiq (urgent-ci-pipeline shard): runs the pipeline creation chain
  • PostgreSQL (CI): pipeline and job record creation
  • Gitaly: .gitlab-ci.yml reads during creation
  • Redis: Sidekiq queuing and the concurrency limit queue

Definitions

Section titled “Definitions”
Section titled “Related Links”