Sidekiq Concurrency Limit

SidekiqConcurrencyLimitQueueBacklogged Alert

This alert fires when a Sidekiq worker has accumulated too many jobs in the Concurrency Limit queue (>100,000 jobs for more than 1 hour).

These jobs are queued in Redis Cluster SharedState, so large amount of jobs could saturate Redis Cluster SharedState memory if left untreated.

Option 1: Increase Worker Concurrency Limit (Preferred)

If the worker can safely handle more concurrent jobs:

1. Locate the worker definition in the codebase
2. Check current concurrency limit setting from the dashboard or the worker class definition.
3. Create an MR to increase the limit to an appropriate value based on:
   • Current processing rate
   • System resource constraints

Option 2: Temporarily Disable Concurrency Limit

As a last resort, sidekiq_concurrency_limit_middleware feature flag can be disabled to help clear the backlogs faster without waiting for deployment as in Option 1. Note that this FF affects all workers globally and disabling on worker level is not supported.

1. Check the Worker Concurrency Detail dashboard to ensure no other workers have significant backlog
2. Disable the feature flag using:

/chatops run feature set sidekiq_concurrency_limit_middleware false --ignore-feature-flag-consistency-check

3. Monitor the queue size to confirm it’s draining
4. Important: Re-enable the feature flag once the backlog is cleared:

/chatops run feature set sidekiq_concurrency_limit_middleware true --ignore-feature-flag-consistency-check

When the feature flag is disabled:

• Up to 5000 jobs will be immediately processed from the queue
• New jobs will execute immediately without throttling

Post-Incident Tasks

1. Create an issue to properly address the root cause if Option 2 was used
2. Update monitoring thresholds if needed
3. Document any findings about the worker’s behavior under load

Useful Dashboards

• sidekiq: Worker Concurrency Detail
• Redis Cluster SharedState metrics
• Sidekiq: Overview

References

• Concurrency limit worker attribute