Skip to content
Runbooks

PatroniIsDown

Overview

Section titled “Overview”
  • This alert fires when the patroni process is no longer running on a Patroni node. Patroni manages the local PostgreSQL instance, so if Patroni is gone PostgreSQL is usually down on that host too, and the node can neither serve traffic nor participate in failover.
  • The alert is not scoped to a single cluster. The expression matches on the process group name only, so it fires for any Patroni node in the environment: patroni (main), patroni-ci, patroni-registry and patroni-sec. Always read the type and fqdn labels first to establish which cluster and node are affected.
  • Contributing factors include an OOM kill or crash of the Patroni process, a failed or partially-applied chef-client converge, a node being rebuilt or decommissioned while still being scraped, and a host that has become unresponsive.
  • On receiving this alert, confirm whether the node is a leader or a replica, confirm whether the cluster still has a healthy leader, and then decide between restarting Patroni and removing the node from service.

Services

Section titled “Services”

The service overview dashboards above each contain a collapsed patroni process stats row, which is the panel that corresponds directly to this alert’s metric.

Metrics

Section titled “Metrics”
  • The alert is based on namedprocess_namegroup_num_procs, exported by process_exporter. It is a gauge: the number of running processes in a named process group. The patroni group is configured per node under default_attributes.process_exporter.config.process_names in the Chef role.
  • The threshold is a straight == 0 — zero patroni processes — held for: 5m. The 5 minute window is there to ride out an ordinary Patroni restart (a gitlab-patronictl restart or a converge-triggered restart completes well inside it), so a firing alert means the process has genuinely not come back.
  • Under normal conditions this series sits at a small non-zero value on every Patroni node and the alert expression returns no results at all. A node dropping to 0 with the rest of the cluster unaffected points at that host; several nodes in one cluster dropping together points at a cluster-wide or fleet-wide cause such as a bad converge.
  • Note that if the node stops being scraped entirely the series disappears rather than going to zero, so this alert goes quiet and PatroniScrapeFailures fires instead. Treat the two as a pair.

Alert Behavior

Section titled “Alert Behavior”
  • Severity s1, pages via PagerDuty, routed to the database_automation team.
  • This alert is expected to be rare. It is not maintenance-aware: it will fire for a node that is intentionally stopped, so create a silence in Alertmanager before stopping Patroni, rebuilding a replica, or decommissioning a cluster.
  • Node rebuilds are a known false-positive path. The pg-replica-rebuild playbooks disable chef-client and stop Patroni while the node is still registered for scraping, which is enough to fire this alert. See production-engineering#25699 for a related case where the same playbooks left state behind.

Severities

Section titled “Severities”
  • Default to s1 and downgrade only once you have established the blast radius.
  • Who is impacted depends on the cluster and the node’s role:
    • A leader down on patroni (main) or patroni-ci affects nearly all customers until failover completes.
    • A replica down reduces read capacity and removes failover headroom, but is usually not customer-facing on its own.
    • patroni-registry affects container registry users; patroni-sec affects security-feature workloads rather than the main application path.
  • Check how many nodes in the cluster are affected, and whether the cluster still has a leader — a cluster with a healthy leader and one missing replica is materially less severe than a cluster with no leader.
  • Review the Incident Severity handbook page to confirm.

Verification

Section titled “Verification”

Confirm the alert against the metric, then against the node itself.

  • Which nodes are reporting zero Patroni processes:
namedprocess_namegroup_num_procs{groupname="patroni", env="gprd"} == 0
  • The same series over time for the affected cluster, to see when it dropped and whether it is flapping:
namedprocess_namegroup_num_procs{groupname="patroni", env="gprd", type="<type from the alert>"}
  • Cross-check that the node is still being scraped at all. If this returns 0 the node is unreachable and this is a host problem rather than a Patroni process problem:
up{fqdn="<fqdn from the alert>", env="gprd"}
  • Then on the node:
Terminal window
ssh <fqdn from the alert>
sudo systemctl status patroni
sudo journalctl -u patroni --since '-30min'
sudo gitlab-patronictl list

gitlab-patronictl list can be run from any surviving member of the cluster and is the fastest way to see whether a leader is still present.

Recent changes

Section titled “Recent changes”

Troubleshooting

Section titled “Troubleshooting”

Work from cluster health inwards to the individual node.

  1. Establish the cluster’s state with sudo gitlab-patronictl list on a surviving member. If there is no Leader, that is the priority — the cluster is not accepting writes.
  2. Establish whether the affected node is reachable. If SSH and up{} both fail, treat it as a host failure and follow PatroniScrapeFailures.
  3. If the node is reachable, check whether Patroni was killed or failed to start:
Terminal window
sudo journalctl -u patroni --since '-1h' --no-pager
sudo dmesg -T | grep -i 'killed process'
  1. Check whether chef-client has been disabled on the node, which would indicate planned maintenance that should have been silenced. Chef runs are disabled with chef-client-disable and re-enabled with chef-client-enable (see Disable Chef runs on a VM), and the state is exported as chef_client_enabled:
chef_client_enabled{fqdn="<fqdn from the alert>", env="gprd"} == 0
  1. Confirm the disk has not filled, which will stop both PostgreSQL and Patroni:
Terminal window
df -h /var/opt/gitlab

Possible Resolutions

Section titled “Possible Resolutions”
  • If Patroni was killed but the host is otherwise healthy, restart it and watch it rejoin: sudo systemctl start patroni then watch -n 30 'gitlab-patronictl list'. Expect the member to move through starting into streaming.
  • If the node was intentionally stopped for maintenance, silence the alert and record the silence on the change issue. Do not restart Patroni on a node that is mid-rebuild.
  • If the cluster has lost its leader and will not elect one, follow the failover guidance in Patroni cluster management and escalate — do not force a promotion without a second pair of eyes.
  • If the host is unresponsive, remove it from service and rebuild rather than trying to recover Patroni in place.

Dependencies

Section titled “Dependencies”
  • Consul — Patroni uses Consul for leader election and service registration. A Consul agent problem on the node can prevent Patroni from starting or staying healthy. See Consul commands for Patroni.
  • PostgreSQL — a PostgreSQL instance that cannot start (corrupt state, full disk, bad configuration) will keep Patroni crash-looping.
  • Chef — Patroni’s configuration is managed by chef-client; a failed or partial converge can leave the service stopped or misconfigured.
  • process_exporter — this alert’s metric source. If process_exporter is running but its configuration no longer matches the Patroni process, the count can read zero while Patroni is in fact healthy. Verify with systemctl status patroni before acting.

Escalation

Section titled “Escalation”
  • Slack channel: #g_database_operations
  • Alert traffic for this team lands in #s_database_excellence_alerts
  • For anything meeting the infra-dev escalation criteria, use /devoncall <incident_url> in Slack.

Definitions

Section titled “Definitions”
Section titled “Related Links”