Skip to content
Runbooks

DuoWorkflowSessionCreditsSyncBacklogPersistent

Overview

Section titled “Overview”

This alert fires when the v2 Duo Agent Platform session credits sync has more work than it can finish. The concurrency limit queue of Ai::DuoWorkflows::SyncNamespaceSessionCreditsWorker has not been empty once in the last 24 hours, so groups left over from one tick are still waiting when the next tick starts, for at least 4 ticks in a row.

How the sync works:

  • Every 6 hours, Ai::DuoWorkflows::SyncSessionCreditsCronWorker selects the top-level groups that used Duo Agent Platform in the last 3 days and have the duo_workflow_session_credits_sync_v2 feature flag on.
  • It schedules one Ai::DuoWorkflows::SyncNamespaceSessionCreditsWorker job per group, spread over the first 5.5 hours of the tick.
  • At most 20 of those jobs run at once (the worker’s concurrency limit). Jobs that cannot start wait in the concurrency limit queue, and a healthy tick drains it before the next one.

Nothing is lost while the queue is backlogged: each job resumes from its group’s cursor. But the groups at the back of the queue sync later and later, so their credit usage falls behind.

When it fires, find out why the sync cannot keep up (see Troubleshooting), and raise the concurrency limit or fix the slow dependency.

Services

Section titled “Services”

Metrics

Section titled “Metrics”

The alert reads sidekiq_concurrency_limit_queue_jobs{worker="Ai::DuoWorkflows::SyncNamespaceSessionCreditsWorker"}: the number of jobs waiting in the worker’s concurrency limit queue, exported by ConcurrencyLimitSampler in Sidekiq. It is a count of jobs.

The SLO is an empty queue at the start of every tick, and the alert is meant to fire after 4 ticks in a row (more than 24 hours). The tick start time is set per instance, so the rule cannot pick out tick starts. It checks instead that the queue has not been empty once in 24 hours. This is a conservative proxy: it can miss a backlog that drains briefly in the middle of a tick and refills before the next one. The exact measurement is namespace_jobs_buffered on the cron’s done line (see Verification).

  • max_over_time(...[5m]) bridges the sampler’s lease handoff: every 5 minutes the old holder reports 0 for a moment.
  • The count guard needs most of 24 hours of data, so a series that just appeared cannot fire early.

Normally the queue is empty most of the time and grows for a while during a tick, only when more than 20 jobs are due at once.

Alert Behavior

Section titled “Alert Behavior”
  • The alert is s3: it does not page, and goes to #g_analytics_optimize.
  • It is expected to be rare: it means the number of active groups outgrew the concurrency limit, or jobs got much slower.
  • It clears as soon as the queue empties once, even if the next tick backlogs again; it then needs another full 24 hours to fire.
  • During a CustomersDot outage each job fails on its first request and ends, so the queue still drains and the alert does not fire. The error rate alert covers that case.
  • A CustomersDot rate limit makes jobs wait inside the job, up to their 120-second budget, before they stop with stop_reason: rate_limited. If it hits many groups at once, the jobs get slower and the queue can back up.

Severities

Section titled “Severities”
  • Impact: customers of GitLab.com top-level groups that use Duo Agent Platform see session credit usage that is hours behind. Billing data in CustomersDot is not affected.
  • Incident severity: usually S4. Raise to S3 if the queue keeps growing from tick to tick, or if customers report stale credit usage.

Verification

Section titled “Verification”

Queue size, bridged over the sampler handoff:

max by (env, environment) (
  max_over_time(sidekiq_concurrency_limit_queue_jobs{env="gprd", worker="Ai::DuoWorkflows::SyncNamespaceSessionCreditsWorker"}[5m])
)

Running jobs against the limit:

max by (env) (sidekiq_concurrency_limit_current_concurrent_jobs{env="gprd", worker="Ai::DuoWorkflows::SyncNamespaceSessionCreditsWorker"})
max by (env) (sidekiq_concurrency_limit_current_limit{env="gprd", worker="Ai::DuoWorkflows::SyncNamespaceSessionCreditsWorker"})

Dashboard: Sidekiq worker concurrency detail.

Kibana, pubsub-sidekiq-inf-gprd index, cron done lines, one per tick:

json.class: "Ai::DuoWorkflows::SyncSessionCreditsCronWorker" and json.job_status: "done"

Under json.extra.ai_duo_workflows_sync_session_credits_cron_worker.*:

  • namespace_jobs_buffered: jobs still in the queue when the tick started. This is the SLO measurement itself; it should be 0.
  • namespaces_enqueued: jobs scheduled by this tick.

Namespace job done lines, for the job durations and stop reasons:

json.class: "Ai::DuoWorkflows::SyncNamespaceSessionCreditsWorker" and json.job_status: ("done" or "fail")

Recent changes

Section titled “Recent changes”

Troubleshooting

Section titled “Troubleshooting”
  1. Check namespace_jobs_buffered on the last few cron done lines. Growing from tick to tick means the sync falls further behind every tick; flat means it is just over capacity.
  2. Compare namespaces_enqueued across ticks. A step up usually follows a feature flag rollout step.
  3. Check how long the namespace jobs take (json.duration_s on the done lines). Most should finish well under the 120-second budget; many at 120 seconds with stop_reason: over_time mean groups with large backlogs, for example right after a rollout step, when each new group reads 90 days of sessions.
  4. Check whether jobs are slow because of a dependency: CustomersDot response times, or ClickHouse insert latency.

The capacity of one tick is about 20 × 360 minutes ÷ average job duration, since the queue keeps draining until the next tick starts. At the 120-second worst case that is about 3,600 jobs.

Possible Resolutions

Section titled “Possible Resolutions”
  • Rollout step added more groups than one tick can sync: hold the rollout until the backlog drains. A new group’s first syncs are the slowest, so it often drains on its own within a day or two.
  • Steady growth in active groups: raise the worker’s concurrency limit (concurrency_limit in SyncNamespaceSessionCreditsWorker). Agree with the Fulfillment team first, because every job makes CustomersDot calls (see INC-13873).
  • A slow dependency: fix or escalate it; the backlog drains once jobs are fast again.

Dependencies

Section titled “Dependencies”
  • CustomersDot GraphQL sessions query (job duration).
  • ClickHouse (job duration).
  • Redis concurrency limit queue (memorystore-redis-concurrency-limit).

Escalation

Section titled “Escalation”
  • group::optimize in #g_analytics_optimize.
  • Raising the concurrency limit or CustomersDot slowness: the Fulfillment team in #g_monetization_platform.

Definitions

Section titled “Definitions”
Section titled “Related Links”