`sidekiq_queueing` apdex violation

Summary

This alert is triggered when jobs are being picked up by a Sidekiq-worker later than the target set based on the urgency of a worker.

high-urgency workloads need to start execution 10s after scheduling
low-urgency workloads need to start execution 60s after scheduling
throttled workloads have no queueing target

These targets are defined in Gitlab::Metrics::SidekiqSlis.

An alert will fire if more than 0.1% of jobs don’t start within their set target.

Affected stage groups

While the shard alert fires, a companion SidekiqQueueingStageGroupAffected alert fires once for each stage group that owns at least 10% of the shard’s over-target jobs over the last 30 minutes. It carries the stage_group and product_stage labels, so the incident for the shard can list the affected teams without anyone reading Kibana first.

It is an s4 alert without a pager label. It never pages and is sent only to incident.io, not to #alerts or the stage groups’ Slack channels.
It resolves when the shard alert resolves, or when the stage group’s share drops below 10%.
Its kibana_dashboard_link annotation opens the customer impact dashboard filtered to the shard.
Jobs in the not_owned feature category have no stage group and are never reported.

Debugging

Check inflight workers for a specific shard: https://dashboards.gitlab.net/d/sidekiq-shard-detail/sidekiq3a-shard-detail?orgId=1&viewPanel=11
- A specific worker might be running a large amount of jobs.
Check apdex attribution per worker: https://dashboards.gitlab.net/d/sidekiq-main/sidekiq3a-overview?orgId=1&from=now-6h&to=now&timezone=utc&var-PROMETHEUS_DS=mimir-gitlab-gprd&var-environment=gprd&var-stage=main&var-shard=catchall&viewPanel=panel-110
- If several workers are affected, it’s usually a symptom of pgbouncer connection pool saturation, or other database resources. Check the connection saturation per worker graph to see if any worker is predominantly holding connections in extended period.
- If only a single worker is affected, the worker itself might be throttled by concurrency limit. Throttled jobs will spend queueing time in a separate concurrency limit queue, thus affecting the shard’s queue apdex. Check sidekiq-concurrency-limit for more info.
Check started jobs for a specific queue: https://log.gprd.gitlab.net/app/r/s/v28cQ
- A specific worker might be enqueueing a lot of jobs.
Latency of job duration: https://log.gprd.gitlab.net/app/r/s/oZnYz
- We might be finishing jobs slower, so we get queue build up.
Throughput: https://dashboards.gitlab.net/d/sidekiq-shard-detail/sidekiq3a-shard-detail?orgId=1&var-PROMETHEUS_DS=mimir-gitlab-gprd&var-environment=gprd&var-stage=main&var-shard=catchall&viewPanel=panel-17&from=now-6h/m&to=now/m&timezone=utc
- If there is a sharp drop of a specific worker it might have slowed down.
- If there is a sharp increase of a specific worker it’s saturating the queue.

Customer impact

The customer impact dashboard in Kibana answers who was affected and what they experienced. Select the shard from the alert and the alert’s time range; every panel can be narrowed further by clicking a cell or with the controls at the top.

Delayed jobs by feature category shows which stage groups’ features slowed down. The table at the bottom of the dashboard maps each feature_category to its group and Grafana stage group dashboard, and is generated from services/stage-group-mapping-with-overrides.jsonnet. Use it to fill in “Affected teams” on the incident and to phrase the status page update. Each group dashboard has an informational Sidekiq queueing row (collapsed, it doesn’t count against the error budget) with the group’s top delayed workers and the product features each of its feature categories covers, so the update can name the features users noticed.
Namespaces: total vs delayed jobs separates victims from causers. One namespace dwarfing the rest in total jobs is flooding the shard (see Remove jobs with certain metadata). Delays spread proportionally across namespaces point at capacity.
Delayed jobs by originating user action (root_caller_id) tells you which user-facing request enqueued the delayed jobs, for example runners finishing jobs (PUT /api/:version/jobs/:id), pushes (post_receive), or MR creation.
Namespace x worker x caller is the per-customer answer for support tickets, e.g. “auto-merge was delayed by about 2 minutes between 00:00 and 02:00 UTC” when MergeRequests::ProcessAutoMergeFromEventWorker shows up for that namespace.

A job counts as delayed when scheduling_latency_s exceeds its urgency’s target. Only done/fail log lines carry these fields.

The same questions can be answered from the command line with esquery, for example:

esquery esql --query 'FROM pubsub-sidekiq-inf-gprd*
  | WHERE @timestamp > NOW() - 1 hour AND json.shard.keyword == ?shard
    AND json.job_status.keyword == "done"
    AND json.scheduling_latency_s > json.target_scheduling_latency_s
  | STATS n = COUNT(*) BY json.meta.root_namespace.keyword, json.class.keyword
  | SORT n DESC | LIMIT 20' --param shard=catchall

The dashboard source lives in elastic/kibana-dashboards.

Resolution

Increase Capacity

You can increase the maxReplicas for the specific shard.

Things to keep in mind:

If we run more concurrent jobs it might add more pressure to downstream services (Database, Gitaly, Redis)
Check whether it makes sense to increase capacity, the bottleneck could be elsewhere, most likely a connection pool being saturated.
Check if this was a sudden spike or if it’s sustained load.

Project authorization refresh fan-out

If there is a fan-out of AuthorizedProjectsWorker / AuthorizedProjectUpdate::* jobs, project_authorizations writes have peaked, or the urgent-authorized-projects shard is saturated, see Project authorization refresh fan-out.

New Worker

It could be that this is a new worker that started running hopefully behind a feature flag that we can turn off.

Drop worker jobs

Drop all jobs, be sure that droping the jobs is safe and won’t leave the application in a weird state.

Mail queue

If the queue is all in mailers and is in the many tens to hundreds of thousands it is possible we have a spam/junk issue problem. If so, refer to the abuse team for assistance, and also https://gitlab.com/gitlab-com/runbooks/-/snippets/1923045 for some spam-fighting techniques we have used in the past to clean up. This is in a private snippet so as not to tip our hand to the miscreants. Often shows up in our gitlab public projects but could plausibly be in any other project as well.

Get queues using sq.rb script

sq is a command-line tool that you can run to assist you in viewing the state of Sidekiq and killing certain workers. To use it, first download a copy:

curl -o /tmp/sq.rb https://gitlab.com/gitlab-com/runbooks/-/raw/master/scripts/sidekiq/sq.rb

To display a breakdown of all the workers, run:

sudo gitlab-rails runner /tmp/sq.rb

Remove jobs with certain metadata from a queue (e.g. all jobs from a certain user)

We currently track metadata in sidekiq jobs, this allows us to remove sidekiq jobs based on that metadata.

Interesting attributes to remove jobs from a queue are root_namespace, project and user. The admin Sidekiq queues API can be used to remove jobs from queues based on these medata values.

For instance:

curl --request DELETE --header "Private-Token: $GITLAB_API_TOKEN_ADMIN" https://gitlab.com/api/v4/admin/sidekiq/queues/post_receive?user=reprazent&project=gitlab-org/gitlab

Will delete all jobs from post_receive triggered by a user with username reprazent for the project gitlab-org/gitlab.

Check the output of each call:

It will report how many jobs were deleted. 0 may mean your conditions (queue, user, project etc) do not match anything.
This API endpoint is bound by the HTTP request time limit, so it will delete as many jobs as it can before terminating. If the completed key in the response is false, then the whole queue was not processed, so we can try again with the same command to remove further jobs.