PatroniLeadershipAlertsNoData
Overview
Section titled “Overview”One of the inputs to PatroniNoHealthyPrimary or PatroniNoReplicaStreaming has been missing for 10 minutes. Until it’s back, the paging alert that depends on it can’t fire for this cluster. This alert doesn’t mean the database is unhealthy.
| Missing series | Comes from | Paging alert it blinds |
|---|---|---|
consul_catalog_service_node_healthy{service_id="<type>-master"} | consul_exporter, querying the Consul servers | PatroniNoHealthyPrimary |
pg_replication_is_replica{type="<type>", shard="default"} | postgres_exporter on each Patroni node | PatroniNoReplicaStreaming |
Common causes
Section titled “Common causes”- The consul_exporter pod is down. It currently runs as a single pod, so every Patroni cluster fires at once. Check the pod and its scrape target.
- The cluster was renamed. A major-version cutover can change the Consul service name or the
typelabel, which comes from the VM’s GCE labels in config-mgmt (for examplepatroni-ci-v18). Both selectors are built from the cluster’s metrics-catalogtypeinconsulSelectorandpostgresSelector(libsonnet/alerts/patroni-leadership-alerts.libsonnet). Restore the original names on the cluster, or update those selectors to match. - postgres_exporter is down on every node of the cluster.
First steps
Section titled “First steps”- The alert’s
missinglabel names the absent series. If both are missing, two alerts fire. - If it’s
consul_catalog_service_node_healthy, checkupfor the consul_exporter job and the exporter’s logs. - If it’s
pg_replication_is_replica, checkup{env="<env>", type="<type>"}for the cluster’s scrape targets, and whether the cluster’stypelabel has changed. - Until the data is back, watch the cluster directly:
sudo gitlab-patronictl list.