Skip to content
Runbooks

PatroniClusterPaused

Overview

Section titled “Overview”

The Patroni cluster has been paused (maintenance mode) for 15 minutes. While it is paused there is no automatic failover for any reason, and the watchdog is inactive. If the primary fails now, nothing promotes a replica, and the cluster stays without a writable primary until someone acts.

A pause is expected during planned maintenance and during the Primary unresponsive or no Leader runbook. This alert is for the pause that outlives its reason: maintenance that finished without a resume, or a recovery that was handed over without one.

First steps

Section titled “First steps”

  1. Find out who paused it, and why. Look for an open change request or incident for this cluster, and check #g_database_operations and the incident channels.

  2. On any member, confirm the pause and check that there is a Leader:

    Terminal window
    sudo gitlab-patronictl list

    The output shows Maintenance mode: on while paused.

  3. Decide from what you found:

    • A change request or incident covers the pause: acknowledge the alert and confirm with its owner that they plan to resume. For a pause meant to last longer than a few hours, silence the alert for the planned window, naming the change request or incident in the silence.
    • The recovery runbook is in progress: acknowledge the alert. The runbook resumes at Step 3, item 8, and stays paused on purpose in Step 4. Don’t resume it yourself.
    • Nothing covers it: treat it as a leftover pause, and resume once the checks below pass.

Resume safely

Section titled “Resume safely”

⚠️ Never resume while there is no Leader. A replica will promote immediately, and every write that only the old primary has will be lost. If there is no Leader, follow Primary unresponsive or no Leader instead.

  1. sudo gitlab-patronictl list shows exactly one Leader, and the replicas are streaming.

  2. Nobody is working on the cluster: no open change request or incident needs the pause.

  3. Resume, then re-enable chef on every member, if it was disabled for the pause:

    Terminal window
    sudo gitlab-patronictl resume --wait
    knife ssh 'roles:<cluster-role>' 'sudo chef-client-enable'

  4. sudo gitlab-patronictl list no longer shows Maintenance mode: on. The alert clears within a scrape interval.

Alert behaviour

Section titled “Alert behaviour”
  • for: 15m, so a short planned pause, such as a Consul upgrade, can finish first.
  • Reads patroni_is_paused from Patroni’s REST API on port 8009. Every member reports the cluster-wide pause, so the alert still fires while some members are down.
  • Notifies without paging for now (s3). gstg goes to #alerts-nonprod, and gprd to incident.io, #s_database_excellence_alerts and #production. Severity and paging will be revisited once firings have been reviewed in production#23070.
  • A chef run on any member replaces the Patroni configuration with the role’s, which removes the pause. That’s why maintenance disables chef first, and why chef has to be re-enabled after you resume.
  • If the input series disappears, this alert can’t fire, and PatroniPauseAlertNoData fires instead.
Section titled “Related links”