Skip to content
Runbooks

DeletionScheduledBacklogOverdueRecords / DeletionScheduledBacklogOldestAgeTooHigh

Overview

Section titled “Overview”

One or more projects or groups have been scheduled for deletion for longer than they should be and still have not been removed. Deletions are backing up instead of completing.

Two alerts share this runbook:

  • DeletionScheduledBacklogOverdueRecords (primary) — the number of records overdue for deletion is above zero. This is the main signal and is not skewed by a single stuck record.
  • DeletionScheduledBacklogOldestAgeTooHigh (backstop) — the oldest record awaiting deletion is older than the age threshold.

Common causes:

  • A deletion worker (ProjectDestroyWorker, GroupDestroyWorker, AdjournedProjectDeletionWorker, AdjournedGroupDeletionWorker) keeps failing or rolling back, often on a statement timeout.
  • The Sidekiq deletion queue is paused, starved, or backed up.
  • One record can’t be deleted because of a data problem, and it blocks the rest of the backlog behind it.

Background

Section titled “Background”

This alert exists because of INC-13111, where deletions were silently broken on GitLab.com for over several days. A stuck record looks exactly like one that is simply queued, so nothing paged. We alert on how overdue the backlog is, not how big it is, so any future deletion outage can only last so long before someone is notified.

Services

Section titled “Services”
  • Team that owns this alert: Organizations (Tenant Scale section, groups_and_projects feature category).
  • Underlying platform: the Sidekiq service, which runs the deletion workers and the sampler that emits these metrics.

Metrics

Section titled “Metrics”

Both metrics are emitted by the DeletionScheduledBacklogSampler, which runs in Sidekiq every 5 minutes and is emitted into the gitlab-gprd, gitlab-gstg, and gitlab-pre Mimir tenants (gitlab-org/gitlab!252036). Both projects and groups are covered, because scheduling either for deletion sets namespace_details.deletion_scheduled_at.

  • gitlab_deletion_scheduled_backlog_overdue_count — number of records whose deletion_scheduled_at is past their expected hard-deletion date (the deletion_adjourned_period setting plus a 2-day grace period).
  • gitlab_deletion_scheduled_backlog_oldest_age_seconds — age of the oldest record still awaiting deletion in seconds. Normally low, dropping as records are deleted.

Thresholds:

  • Overdue count: alert when > 0, sustained for 2 hours.
  • Oldest age: alert when older than 32 days (the default deletion_adjourned_period of 30 days plus a 2-day grace period), sustained for 30 minutes.

Both are starting values and will be tuned before they can page — see Definitions.

Alert Behavior

Section titled “Alert Behavior”
  • Both alerts clear automatically once the backlog drains below their thresholds.
  • Silencing is appropriate only during a deliberate, tracked pause of deletion processing (for example, an open Change Request).
  • These should be rare alerts. Under normal operation the backlog drains continuously and the overdue count sits at zero.

Severities

Section titled “Severities”
  • Default severity: s3, Slack-only (no pager label) while the thresholds are being tuned. Add the pager: pagerduty label (and raise the severity) to make it page once the thresholds are settled.
  • This is not an outage — deletions are delayed, not lost. Customers may open Zendesk tickets about groups or projects that “won’t delete.”
  • Raise the severity if the overdue count keeps climbing. That means the whole pipeline is stalled, not just one record blocking it.

Verification

Section titled “Verification”
  • Prometheus, on the affected environment at dashboards.gitlab.net:
    • max(gitlab_deletion_scheduled_backlog_overdue_count)
    • max(gitlab_deletion_scheduled_backlog_oldest_age_seconds)
  • Kibana: search Sidekiq logs for the failing deletion worker classes (ProjectDestroyWorker, GroupDestroyWorker, AdjournedProjectDeletionWorker, AdjournedGroupDeletionWorker) for exceptions or repeated rollbacks.

Recent changes

Section titled “Recent changes”
  • Check for recent deploys or database changes affecting the deletion services, and for open Change Requests that pause deletion processing.

Troubleshooting

Section titled “Troubleshooting”

  1. Check the deletion workers in Kibana. Look for exceptions, statement timeouts, or repeated rollbacks in ProjectDestroyWorker / GroupDestroyWorker and their adjourned-deletion cron workers.

  2. Check the Sidekiq deletion queues are running and not paused or starved.

  3. Look for a single blocking record. Query for the oldest record still in the backlog and inspect it for a data problem that fails deletion:

    Namespace::Detail
      .where.not(deletion_scheduled_at: nil)
      .order(:deletion_scheduled_at)
      .first

  4. Re-drive deletion for the affected record(s) once the underlying cause is fixed, and confirm the overdue count returns to zero.

Escalation

Section titled “Escalation”
  • Primary contact: #g_organizations Slack channel (team handbook).
  • For database-level blockers (statement timeouts, integrity errors preventing deletion), escalate to #database_operations.

Definitions

Section titled “Definitions”
Section titled “Related Links”