PatroniPauseAlertNoData

Overview

patroni_is_paused has been missing for the cluster for 30 minutes, so PatroniClusterPaused can’t fire for it. A cluster left paused would go unnoticed until the series is back.

The series comes from Patroni’s REST API /metrics on port 8009, scraped by the patroni job in argocd/apps services/prometheus-scrape-configs.

First steps

1. Check whether the targets are scraped at all:

   up{job="scrapeConfig/monitoring/prometheus-agent-patroni", type="<type>"}

   • No series: the scrape job is missing or no longer matches the nodes. Check that the job is enabled for the environment, and that the nodes still carry the GCE labels gitlab_com_service=postgres and gitlab_com_type=<type>. A cluster renamed in its type label, as at a major-version cutover, needs the scrape job and this alert updated together.
   • 0 on every node: Prometheus can’t reach port 8009. Check that Patroni is running (sudo systemctl status patroni) and listens on 0.0.0.0:8009, and that the firewall allows the GKE Prometheus agents to reach it.

2. While the alert is firing, check for a pause by hand on any member: sudo gitlab-patronictl list | grep 'Maintenance mode'.

Alert behaviour

• for: 30m, so a scrape gap during a node rebuild or a Prometheus agent restart doesn’t fire it.
• Doesn’t page (s3).

Related links

• Alert definition
• Link to edit this playbook