Request Rate Drop Alerts

The traffic cessation alerts fire when an SLI’s request rate is exactly zero or its series disappears. They do not fire on a partial drop, such as one pipeline source stopping while the others continue. An SLI with only a request rate also has no apdex or error ratio that burns error budget.

requestRateDropAlertConfig adds an alert for that case. The SLI’s request rate is compared with its median rate at the same time of week over the previous weeks (three by default). The alert fires when the ratio is below a threshold for the configured duration. Comparing against the same time of week means the regular daily and weekly cycle does not fire the alert. Taking the median means a single unusual week (an incident, a holiday, a deploy freeze) does not distort the baseline.

This differs from the service ops rate anomaly detection (service_ops_out_of_bounds_lower_5m), whose threshold is three standard deviations of the rate over one week. For a strongly diurnal signal that spread is wide enough that a near-complete stop does not cross it. The per-SLI drop alert uses a ratio instead.

Configuration

    pipelines_created: {
      userImpacting: true,
      requestRate: rateMetric(counter='pipelines_created_total'),
      significantLabels: ['source'],

      requestRateDropAlertConfig: {
        by: 'source',
        selector: { source: { oneOf: ['push', 'merge_request_event', 'parent_pipeline'] } },
        threshold: 0.5,
        tenants: ['gitlab-gprd'],
      },
    },

Field	Default	Meaning
by	null	One of the SLI’s significantLabels. The rate and baseline are split by this label and each value is alerted on separately. Without it, the SLI total is used.
selector	{}	Selector hash applied to the source metric, to limit the alert to some values of by or to exclude noisy ones.
threshold	required	Ratio of current rate to baseline below which the alert fires, between 0 and 1.
for	'15m'	How long the ratio must be below the threshold.
severity	's3'	s1/s2 page, s3/s4 go to Slack.
burnRate	'5m'	Rate window. Must be one of the SLI’s source burn rates (5m, 30m, 1h).
baselineOffsets	['1w', '2w', '3w']	Offsets whose median forms the baseline.
minimumBaselineRate	0	Branches whose baseline is at or below this rate are ignored. A value with no history cannot fire.
tenants	null (all)	Restrict the rules to these Mimir tenants, for thresholds that only hold for one environment’s traffic volume.

Generated rules

For each opted-in SLI one rule group is generated per tenant, in mimir-rules/<tenant>/<service>/...-request-rate-drop-alerts.yml:

• gitlab_component_ops_anomaly:rate_<burnRate>: the current rate, summed over env, environment and the by label, with type, component, tier and the SLI’s static labels attached.
• gitlab_component_ops_anomaly:rate_<burnRate>:baseline: the median of the same aggregation at each of the baselineOffsets. The offsets are applied to the source aggregation, not to the recorded rate. The baseline is therefore complete as soon as the rules are deployed.
• <Service>Service<Sli>RequestRateDrop: fires when rate / baseline < threshold. An absent rate series is treated as zero and a complete stop fires this alert as well. The alert has alert_class="request_rate_drop", sli_type="ops", slo_alert="no" and the SLI’s routing labels.

Choosing thresholds

Thresholds are ratios. The same value is a different amount of traffic for a high-volume signal with a flat daily curve and a bursty low-volume one. Replay the ratio over a few weeks of production data before picking one:

sum by (source) (sli_aggregations:pipelines_created_total:rate_5m{env="gprd"})
/
quantile(0.5,
  label_replace(sum by (source) (sli_aggregations:pipelines_created_total:rate_5m{env="gprd"} offset 1w), "p", "1w", "", "")
  or
  label_replace(sum by (source) (sli_aggregations:pipelines_created_total:rate_5m{env="gprd"} offset 2w), "p", "2w", "", "")
  or
  label_replace(sum by (source) (sli_aggregations:pipelines_created_total:rate_5m{env="gprd"} offset 3w), "p", "3w", "", "")
) without (p)

One threshold is enough in practice: for pipeline creation, 0.5 fires on the same incidents for push, merge request and parent pipelines as separate thresholds did. The difference between sources is which ones are noisy, and that is what selector is for. Cron-driven sources such as scheduled pipelines dip below any threshold two to three times a week and are better left to the traffic cessation alerts. Daylight-saving changes shift the baseline by an hour for up to three weeks. The morning rise in traffic then comes an hour early or late against the baseline and the ratio is closer to the threshold. Leave a margin for that.