Runaway Sidekiq backlog saturating redis-sidekiq

This runbook is the entry point when a worker’s backlog grows faster than it drains and the resulting jobs flood Sidekiq and redis-sidekiq.

It does not replace Disabling Sidekiq workers; it tells you which parts of it to use and in what order.

Symptoms

Sidekiq queueing apdex collapses on one or more shards, and one shard shows a growing backlog.
A redis-sidekiq memory alert (redis_memory) fires or memory on one redis-sidekiq shard is climbing.
A worker shows a high enqueue rate or a large number of concurrently running jobs, sometimes enqueueing further jobs for other workers.
Database or PgBouncer saturation alerts, which can start before the Redis alert.

What makes this different from other worker incidents

Jobs that are already running keep running and keep enqueueing follow-up jobs. Deferring or dropping only affects jobs that have not started.
Deferring keeps the backlog in Redis, so it does not relieve Redis memory. Jobs deferred automatically by the database health check also wait in Redis.
Dropping discards the work, which can leave data in an inconsistent state. The owning team and the IMOC both need to approve it.

Steps

Identify the trigger and the workers.
- sidekiq: Worker Detail: enqueue rate, queue size, execution time per worker.
- Sidekiq logs: group by meta.user, meta.caller_id and meta.feature_category to find what is causing the jobs (see the Logs section of the Sidekiq survival guide).
- Enqueue rate and skipped jobs: sidekiq_enqueued_jobs_total and sidekiq_jobs_skipped_total (see the redis_memory runbook).
Bring in the owning team and the IMOC at the same time as you start mitigating. See Find the owning team.
Check the Redis shard using the decision gate. Map the worker to its Redis shard using the deployment to shard table.
Choose defer or drop using the gate and the mitigation comparison table.
- Memory and CPU well below their soft SLOs: defer, and keep watching Redis memory.
- At or near the soft SLO: start evaluating a drop with the owning team and the IMOC. Do not drop yet.
- At the hard SLO (the redis_memory alert pages): drop, with approval from the owning team and the IMOC.
Remove what is already queued for the trigger, if needed, with the admin API. Time-box it.
If Redis is degraded (HTTP 500s from the admin API, Redis CPU close to saturation, clients dropping), follow Redis is degraded. If ChatOps is unavailable, use the Rails console commands in the incident quick reference.
Stop the source. If the enqueues come from a single caller (see meta.caller_id and meta.user), check whether it is still active and engage the owning team or Support if it needs to be stopped or slowed down.
Clean up. Remove the run_sidekiq_jobs_* and drop_sidekiq_jobs_* flags when done (/chatops gitlab run feature list --match run_sidekiq_jobs, and the same for drop_sidekiq_jobs), and plan for any data left in an inconsistent state.

What each mitigation does

Mitigation	Stops new work from running	Frees Redis memory	Stops jobs that are already running	Needs healthy Redis
Defer flag `run_sidekiq_jobs_*`	Yes, when a job is picked up	No, the backlog stays in Redis	No	Yes, jobs must be picked up and rescheduled
Drop flag `drop_sidekiq_jobs_*`	Yes, when a job is picked up	Yes, gradually, as jobs are picked up (see the notes below)	No	Yes, jobs must be picked up
Admin API queued-job removal	n/a	Yes, for matching jobs in the queue and concurrency-limit buffer only	No	Yes, returns 500 when Redis is degraded
Redis node reset	n/a	n/a	n/a	Restores Redis so the other options work

Notes:

Deferred jobs wait in the scheduled set of the Redis store named in the worker’s routing rule (the main redis-sidekiq if none), which can differ from the shard its queue is on. Check that shard too.
The admin API does not remove deferred jobs. The drop flag discards them when they come due.
For workers with a concurrency_limit, excess jobs are buffered before the flags are checked, so the drop flag only applies once they are released. See Backlog that outlives its trigger.
Flags only take effect when a job is picked up, so nothing changes while processing is stalled.

Removing queued jobs for a user or worker with the admin API

Use the admin Sidekiq queues API (full procedure: Remove jobs with certain metadata):

curl --request DELETE --header "Private-Token: $GITLAB_API_TOKEN_ADMIN" "https://gitlab.com/api/v4/admin/sidekiq/queues/<queue>?user=<username>&worker_class=<WorkerClass>"

<queue> is the shard queue from the worker’s routing rule (routingRules), not the per-worker queue. Queues are shared, so always add worker_class.
Filters match by exact string equality against the job’s metadata, and unknown keys are ignored. Use the string-valued keys: user (username), project, root_namespace, caller_id, feature_category, worker_class. Numeric ids such as user_id won’t match, because stored ids are integers. Avoid gl_user_id and gl_root_namespace_id.
Repeat while the response has completed: false (30 s limit per call).
Removes jobs from the queue and concurrency-limit buffer only, not deferred jobs in the scheduled set or running jobs. Set the drop flag as well.
Needs a healthy Redis. After two HTTP 500s or timeouts, stop and go to Redis is degraded.

Redis is degraded

Symptoms: the admin API returns HTTP 500, Redis CPU is close to saturation, Redis connected clients drop sharply, or Redis nodes and their Sentinels stop responding.

Set the drop flag now, with approval from the owning team and the IMOC, using ChatOps or the Rails console (see the incident quick reference), so that jobs are discarded as soon as they can be picked up.
Escalate to the redis-sidekiq owner (see the service catalog) and the Redis survival guide for SREs. Unresponsive nodes can also block Sentinel failover, so recovery may need a node reset.
Once Redis responds, remove the queued jobs for the affected user or worker with the admin API (above).
Watch the redis-sidekiq: Overview memory and connection panels and the Skipped Jobs panel.