SidekiqServiceDuoWorkflowSessionCreditsSyncErrorSLOViolation

Overview

This alert fires when too many jobs of the v2 Duo Agent Platform session credits sync fail. The sync copies credit usage from CustomersDot into ClickHouse, so a failing sync means customers see stale or missing credit usage.

How the sync works:

Every 6 hours, Ai::DuoWorkflows::SyncSessionCreditsCronWorker selects the top-level groups that used Duo Agent Platform in the last 3 days and have the duo_workflow_session_credits_sync_v2 feature flag on.
It enqueues one Ai::DuoWorkflows::SyncNamespaceSessionCreditsWorker job per group, spread over the first 5.5 hours of the tick, with a concurrency limit of 20.
Each job pages through the sessions CustomersDot changed since the group’s cursor, writes them to ClickHouse, and advances the cursor.

A known failure (CustomersDot, ClickHouse, Redis) does not raise for Sidekiq to retry. The job logs why and the next tick resumes from the cursor, so no data is lost. An unexpected exception does raise and Sidekiq retries it, and each attempt counts as an error. Either way, credit data falls behind until the cause is fixed.

When it fires, find which failure reason dominates, then act on that cause (see Troubleshooting).

Services

Sidekiq service overview
Owner: group::optimize, Slack #g_analytics_optimize

Metrics

The alert is based on the duo_workflow_session_credits_sync application SLI, defined in metrics-catalog/gitlab-slis/library.libsonnet and attached to the sidekiq service in metrics-catalog/services/sidekiq.jsonnet.

gitlab_sli_duo_workflow_session_credits_sync_total: one per namespace job that attempted a sync.
gitlab_sli_duo_workflow_session_credits_sync_error_total: the ones that failed.

What counts:

Success: the job drained its window (stop_reason: completed), or ran out of its 120-second budget after making progress (stop_reason: over_time).
Error: any other failure, including exceptions nobody rescued.
Not measured: jobs skipped because their namespace is paused (stop_reason: backoff), jobs stopped by a CustomersDot rate limit longer than their remaining budget (stop_reason: rate_limited, resumed from the stored page token on the next tick), namespaces with no CustomersDot subscription (no_cdot_subscription_found), fields CustomersDot has not deployed (undefined_field), self-managed instances without a billable license (no_billable_license), and jobs interrupted by a Sidekiq shutdown.

The SLO is a 1% error ratio (errorRatio: 0.99), with the standard multi-window burn-rate alerts. The sample gate is lowered to 100 jobs in each alert window (1h and 6h), because there is only one job per active group every 6 hours. Until about 550 groups are synced per tick, only the 6h window has enough jobs to alert.

Under normal conditions the error ratio stays near 0, with traffic in bursts during the first 5.5 hours of each 6-hour tick and none for the remaining 30 minutes.

Alert Behavior

The alert is s3: it does not page, and goes to #g_analytics_optimize.
It is expected to be rare. It fires on a CustomersDot outage or a ClickHouse or Redis problem, or on a code change that breaks the sync.
During a CustomersDot outage every job keeps calling CustomersDot and fails, so the alert keeps firing for as long as the outage lasts. Nothing pauses the sync for server or connection errors.
Only the CustomersDot kill switch (failure_reason: disabled) pauses a namespace, and only that one, for 2 hours. Its jobs then return stop_reason: backoff and are not measured. The cron still enqueues every namespace on every tick.
Silence it only while a known CustomersDot incident is open.

Severities

Impact: customers of GitLab.com top-level groups that use Duo Agent Platform see stale or missing session credit usage. Billing data in CustomersDot is not affected; only GitLab’s copy of it falls behind.
Usually S4. Raise to S3 if every job is failing for more than one tick (6 hours), or if customers report it.

Verification

Failure rate by reason:

sum by (reason) (rate(duo_workflow_session_credits_v2_fetch_failures_total{env="gprd"}[1h]))

This counter also includes no_cdot_subscription_found, undefined_field, no_billable_license and rate_limited, which the SLI does not measure.

SLI error ratio:

sum(rate(gitlab_sli_duo_workflow_session_credits_sync_error_total{env="gprd"}[1h]))
/
sum(rate(gitlab_sli_duo_workflow_session_credits_sync_total{env="gprd"}[1h]))

Kibana, pubsub-sidekiq-inf-gprd index, namespace job done lines. Jobs that raised are logged with job_status: "fail":

json.class: "Ai::DuoWorkflows::SyncNamespaceSessionCreditsWorker" and json.job_status: ("done" or "fail")

Fields under json.extra.ai_duo_workflows_sync_namespace_session_credits_worker.*: stop_reason, failure_reason, root_namespace_id, rows_written, pages_fetched, calls_made, cursor_before, cursor_after, cursor_lag_s, cursor_reseeded, cdot_sync_lag_s.

Cron done lines, with namespaces_enumerated, namespaces_enqueued, namespaces_flag_off, namespace_jobs_buffered and stop_reason under json.extra.ai_duo_workflows_sync_session_credits_cron_worker.*:

json.class: "Ai::DuoWorkflows::SyncSessionCreditsCronWorker" and json.job_status: "done"

Kibana, pubsub-application-inf-gprd index, application log warnings:

json.message: "Failed to sync session credits"

It carries json.error_type (the failure reason), json.error_message and json.gl_root_namespace_id.

Sentry: errors of Ai::DuoWorkflows::SyncNamespaceSessionCreditsWorker, with root_namespace_id and reason in the extra data. CustomersDot response errors are reported by Gitlab::SubscriptionPortal::SubscriptionUsageClient instead.

Recent changes

Recent changes to the session credits code
Feature flag changes: search #production or the feature flag log for duo_workflow_session_credits_sync_v2.
To stop the sync entirely, turn the flag off: /chatops gitlab run feature set duo_workflow_session_credits_sync_v2 false. Queued jobs check the flag and stop without calling CustomersDot.

Troubleshooting

Find the dominant failure_reason, with the Prometheus query or the Kibana done lines above.
Check whether failures are spread across many namespaces or come from a few. Break the done lines down by root_namespace_id.
Act on the cause:

`failure_reason`	Likely cause	Action
`server_error`, `connection_error`, `error`, `response_error`	CustomersDot is down or degraded	Check `#g_monetization_platform` and `#production` for a CustomersDot incident. Nothing pauses the sync, so every job keeps failing until CustomersDot recovers. If the outage is long, or the retries add load to CustomersDot, turn the sync off with the feature flag (see Recent changes) and back on once it recovers.
`unavailable`	CustomersDot shed the sessions query (its query timeout)	If it is one namespace, it is a heavy subscription and only that namespace is behind. If many, contact the Fulfillment team in `#g_monetization_platform`.
`disabled`	CustomersDot turned on its kill switch for the field	Confirm with Fulfillment. Each affected namespace is paused for 2 hours, then retried; a success does not lift the pause early. See Paused namespaces.
`rate_limited`	CustomersDot throttled the job for longer than its remaining budget	Not measured by the SLI. The next tick resumes from the stored page token. Act only if it is widespread on every tick: then the sync is too heavy for CustomersDot, so contact Fulfillment.
`clickhouse_error`	ClickHouse is down, or rejected the insert	Check ClickHouse health and the ClickHouse runbook.
`invalid_rows`	CustomersDot sent a session GitLab cannot parse: a bad `updatedAt`, a `workflowId` with no numeric id, or non-numeric credits	The namespace fails every tick until fixed. Find the session in Sentry, and contact Fulfillment in `#g_monetization_platform` and group::optimize.
`redis_error`	Redis SharedState is unavailable	Check the `redis` service.
`malformed_response`	CustomersDot returned `hasNextPage` without an `endCursor`, or a null `sessions` connection	Contact Fulfillment in `#g_monetization_platform`; the namespace stays stuck until fixed.
`missing_root_namespace`	A job was enqueued without a namespace	A code bug; file an issue for group::optimize.
(empty, `stop_reason: error`)	An unexpected exception escaped the job	Check Sentry for the exception. These jobs are logged with `job_status: "fail"`.

The concurrency limit (20 jobs) does not cause errors, but it can hide them. If more groups are active than the limit can sync in one tick, jobs wait in the buffer past the next tick: the cron logs a non-zero namespace_jobs_buffered, and SidekiqConcurrencyLimitQueueGrowing may fire for this worker. Fewer jobs then run per window, which can drop the SLI below its sample gate so the alert goes quiet.

Paused namespaces

Only the CustomersDot kill switch pauses a namespace: that namespace alone, for 2 hours, after which its next job tries again. Check and clear one from a Rails console, passing the root namespace id as a string (instance on self-managed):

Ai::DuoWorkflows::SessionCredits::SyncPause.reason(source: '<root_namespace_id>')  # :disabled while paused, nil otherwise
Ai::DuoWorkflows::SessionCredits::SyncPause.clear!(source: '<root_namespace_id>')  # only once CustomersDot has turned the kill switch off

To find paused namespaces, look for stop_reason: backoff in the namespace done lines.

One namespace

Last complete sync, in Kibana (pubsub-sidekiq-inf-gprd index). Sort by time, newest first; the top hit is the last complete sync:

json.class: "Ai::DuoWorkflows::SyncNamespaceSessionCreditsWorker"
and json.job_status: "done"
and json.extra.ai_duo_workflows_sync_namespace_session_credits_worker.root_namespace_id: <root_namespace_id>
and json.extra.ai_duo_workflows_sync_namespace_session_credits_worker.stop_reason: "completed"

Drop the stop_reason filter and use json.job_status: ("done" or "fail") to see every run of the namespace, which usually shows why it is behind: over_time means it made progress without finishing its window, backoff that it was skipped, and error comes with a failure_reason.

Kibana keeps about 7 days of logs. For an older last sync, or a namespace that has never completed one, use the freshness record in a Rails console. It keeps up to 90 days:

Ai::DuoWorkflows::SessionCredits::SyncFreshness.status(<root_namespace_id>)
# => { last_completed_at:, last_seen_in_scope_at:, in_scope_since: }

last_completed_at is nil when no complete sync was recorded, and in_scope_since is when the namespace’s current stay in scope began.

Sync cursor, in a Rails console:

cursor = Ai::DuoWorkflows::SessionCredits::Cursor.new(<root_namespace_id>)
cursor.stored_time    # CustomersDot updatedAt of the newest synced session
cursor.stale?         # true when the cursor fell out of CustomersDot's 90-day window
cursor.position_time  # where the next sync starts reading

Avoid cursor.after_token here: when the stored token has expired it deletes it, as the next job would.

Dependencies

CustomersDot GraphQL sessions query.
ClickHouse (duo_workflow_session_enrichments, sync_cursors, siphon_duo_workflows_workflows).
Redis SharedState (page tokens, namespace pauses).

Escalation

group::optimize in #g_analytics_optimize.
CustomersDot problems: the Fulfillment team in #g_monetization_platform.

Definitions

SLI definition and SLO, sample gate and routing.
Recalibrate the sample gate (minimumSamplesForMonitoring) once the v2 sync is rolled out to every group.
Update the template used to format this playbook