Skip to content
Runbooks

DatabasePartitionSyncStalled

Overview

Section titled “Overview”

This alert fires when 5 or more partitioned tables have been missing at least one partition for 12 hours, in one environment.

GitLab keeps partitions ahead of time for its partitioned tables: daily tables 28 days ahead, monthly tables 6 months ahead, and CI tables one partition ahead. The dynamic partition sync (Gitlab::Database::Partitioning.sync_partitions) creates them. A healthy sync creates whatever is missing within one run, so a growing list of tables with missing partitions means the sync has stopped. That happens in two ways:

  • The sync is disabled. partition_manager_sync_partitions (or disallow_database_ddl_feature_flags) was turned off, for example during an incident or maintenance, and has not been turned back on.
  • The sync keeps failing. Each run tries and fails, for example because the ATTACH PARTITION for the next CI partition cannot get its lock on p_ci_builds (INC-15918).

Nothing breaks straight away, which is why this alert does not page. But every day the sync stays stalled uses up headroom: once a daily table runs out of its 28 days, inserts into it fail. Old partitions also stop being detached, so disk use grows.

When it fires, find out which of the two cases it is, and make sure there is an owner and a date for getting the sync running again.

Services

Section titled “Services”

Metrics

Section titled “Metrics”
  • db_partitions_missing: a Rails gauge, one series per partitioned table (table label). It is the number of partitions the table’s partitioning strategy expects but cannot find.
  • Database::PartitionManagementWorker (every 6 hours) and Database::DropDetachedPartitionsWorker set it after each run (partition_monitoring.rb). They report even when the sync is disabled.
  • The gauge lives on the Sidekiq pod that ran the worker, and the series ends when that pod is replaced. The alert takes max_over_time over 7 hours to bridge the gap until the next run reports.
  • Threshold. Over the 30 days before INC-15918, gprd never had more than 3 tables missing a partition at once. Those are usually tables that rotate quickly, such as p_ci_finished_pipeline_ch_sync_events, and they catch up on the next run. Within hours of disabling the sync in INC-15918 the count was 12, and 25 a day later. gstg had one spike to 22 tables in the same period, which cleared after about 9 hours. The 12-hour for keeps spikes like that from firing.
  • For a daily table, the missing count is roughly the number of days the sync has been stalled. When it gets close to 28, inserts into that table are about to fail.

Alert Behavior

Section titled “Alert Behavior”
  • Expected to be rare. It fires when the sync is disabled for longer than about half a day, or when a partition keeps failing to attach.
  • If the sync was disabled on purpose and there is a plan to re-enable it, you can silence the alert until the planned date. Do not silence it with no end date: the alert is the reminder that the flag is still off.

Severities

Section titled “Severities”
  • Default s4. There is no customer impact while headroom remains.
  • Raise it if any daily table is close to running out (missing count above about 20), or if the next CI partition is needed soon (see CI partition sync lock contention).

Verification

Section titled “Verification”

List the tables that are missing partitions, and how many each:

max by (table) (max_over_time(db_partitions_missing{env="gprd", type="sidekiq"}[7h])) > 0

Check the flags:

Terminal window
/chatops gitlab run feature get partition_manager_sync_partitions
/chatops gitlab run feature get disallow_database_ddl_feature_flags

In Kibana (Rails logs), search for:

  • Skipping sync of dynamic postgres partitions: the sync is disabled, and the log names the flag.
  • Failed to create / detach partition(s): the sync runs but fails. The log names the table and the exception, for example AttemptsExhaustedError.

Recent changes

Section titled “Recent changes”
  • Flag changes made with chatops are recorded in the feature flag log. The most recent issue for partition_manager_sync_partitions shows who changed it and when.
  • To roll back a deliberate disable, re-enable the flag as described in CI partition sync lock contention.

Troubleshooting

Section titled “Troubleshooting”
  1. The sync is disabled. Find who disabled it and why, usually in an incident or a change request, and agree a re-enable date. For the CI tables, follow the re-enable steps in CI partition sync lock contention.
  2. The sync runs but fails for the CI tables (p_ci_builds and the tables with a foreign key to it): follow CI partition sync lock contention. Check the patroni-ci rails_primary_sql Apdex as well. Repeated attach attempts can queue CI writes.
  3. The sync runs but fails for other tables. The failure log names the table and the exception. A lock timeout points to contention on that table (PostgreSQL locking). Raise anything else with the team that owns the table.

Possible Resolutions

Section titled “Possible Resolutions”
  • INC-15918 / production#23392: partition 118 could not attach to p_ci_builds. The sync was disabled to stop the retries, after which the missing count kept growing across every database.

Dependencies

Section titled “Dependencies”
  • Sidekiq: the gauge comes from Sidekiq workers, so the alert goes quiet if they stop running.
  • The ops feature flags partition_manager_sync_partitions and disallow_database_ddl_feature_flags.

Escalation

Section titled “Escalation”
  • Database Operations in #g_database_operations.
  • For the partitioning framework itself, the Database Frameworks team.

Definitions

Section titled “Definitions”
Section titled “Related Links”