Skip to content
Runbooks

WalReceiverNotConnected

Overview

Section titled “Overview”

This alert fires when the WAL receiver’s replication slot on the current primary has not been in use for 5 minutes. In other words, the receiver is not copying WAL from the primary.

While it fires, new writes on the primary have no second copy outside the cluster.

Common causes:

  • the receiver’s service or VM is down;
  • after a failover, the receiver is still waiting on the old primary, because that machine is stuck but nobody has fenced it yet (the receiver only leaves a node that may still be writing once it is fenced);
  • the replication password changed and the receiver was not updated;
  • the receiver can’t reach the primary (network, or the Consul name doesn’t resolve).

The WAL receiver is a small VM per Patroni cluster, outside the cluster, that keeps a live copy of the cluster’s WAL with pg_receivewal. If the primary’s upload to GCS stops, as in INC-14518, the WAL still exists on the receiver. It is set up by the gitlab-patroni::wal_receiver recipe.

Services

Section titled “Services”

Metrics

Section titled “Metrics”
  • pg_replication_slots_active{slot_name=~"walrecv(_.*)?"} on the primary (from postgres_exporter). The slot exists on every member, because it is a permanent Patroni slot, so the alert only looks at the primary (pg_replication_is_replica == 0).

Alert Behavior

Section titled “Alert Behavior”
  • Fires after 5 minutes. Clears as soon as the receiver is copying from the current primary again.
  • A planned switchover can make it fire briefly, until the receiver has moved to the new primary (normally within a minute).
  • Severity s3: Slack only, no page. In gprd it goes to #s_database_excellence_alerts and #alerts; in gstg to #alerts-nonprod, so staging firings can be reviewed before production.

Verification

Section titled “Verification”

On the receiver:

Terminal window
sudo systemctl status walrecv.service                 # is the receiver running?
sudo journalctl -u walrecv --since "-1h" --no-pager | tail -30   # what it did last
cat /var/opt/gitlab/walrecv/node                       # the IP of the primary it follows
ls /var/opt/gitlab/walrecv/fenced/                     # fenced nodes (normally empty)
cat /var/opt/gitlab/walrecv-metrics/walrecv.prom       # the numbers behind these alerts

On the primary:

Terminal window
sudo gitlab-patronictl list
sudo gitlab-psql -c "SELECT slot_name, active, wal_status, pg_size_pretty(pg_current_wal_lsn() - restart_lsn) AS kept FROM pg_replication_slots WHERE slot_name LIKE 'walrecv%';"

What the receiver’s log tells you:

Log lineMeaning
<ip> unreachable, not fenced, stayingIt is waiting on a stuck old primary. See “Possible Resolutions”
<ip>: password refused, stayingThe replication password changed
no primary in Consul yetThe Consul name master.<cluster>.service.consul doesn’t resolve
GAP ALARMIts slot was lost; see WalReceiverGapDetected

Recent changes

Section titled “Recent changes”
  • Check recent chef-repo MRs for the receiver’s role, and for wal-g upgrades on the cluster.
  • Check recent production change requests for failovers or Patroni changes on the cluster.

Possible Resolutions

Section titled “Possible Resolutions”
  • The service is down: sudo systemctl restart walrecv. It continues with the node saved in /var/opt/gitlab/walrecv/node.
  • It waits on a stuck old primary: first confirm that the old primary can no longer write (its VM is stopped, or its Postgres processes are gone). Stopping the VM is enough: the receiver sees it in GCE and moves by itself. Only if the VM can’t be stopped, but Postgres is confirmed gone, create the fenced file on the receiver: sudo -u gitlab-psql touch /var/opt/gitlab/walrecv/fenced/<old primary's IP>. Never fence a node that may still be writing. Full procedure: production#23275 (page 1, steps 3–4).
  • The password changed: run chef on the receiver too.
  • The slot was lost: follow WalReceiverGapDetected.

Escalation

Section titled “Escalation”

If you can’t find and fix the cause with the steps above, raise it in #g_database_operations.

Section titled “Related Links”