Skip to content
Runbooks

WalReceiverFencedButStillPrimary

Overview

Section titled “Overview”

This alert fires when the WAL receiver found a node that is marked as fenced but still answers as a primary.

“Fenced” means someone told the receiver that the old primary is finished: its VM is stopped (the receiver checks GCE), or a file fenced/<its IP> exists on the receiver. If that node still answers as a primary, it may still be accepting writes. The receiver therefore keeps reading from it, and logs a warning every 10 seconds.

So the fence was not done correctly: either the old primary is still running, or a fenced file was created for a healthy primary by mistake.

The WAL receiver is a small VM per Patroni cluster, outside the cluster, that keeps a live copy of the cluster’s WAL with pg_receivewal. If the primary’s upload to GCS stops, as in INC-14518, the WAL still exists on the receiver. It is set up by the gitlab-patroni::wal_receiver recipe.

Services

Section titled “Services”

Metrics

Section titled “Metrics”
  • walrecv_fenced_but_primary_last_timestamp_seconds: the last time the receiver logged this warning. The alert fires while that was less than 15 minutes ago.

Alert Behavior

Section titled “Alert Behavior”
  • Fires at once, and stays while the receiver keeps logging the warning. Clears 15 minutes after the last warning.
  • Severity s2: pages the DBO (Database Operations) on-call. In gprd it also goes to #s_database_excellence_alerts and #production; in gstg only to #alerts-nonprod, without a page.

Verification

Section titled “Verification”

On the receiver:

Terminal window
sudo systemctl status walrecv.service                 # is the receiver running?
sudo journalctl -u walrecv --since "-1h" --no-pager | tail -30   # what it did last
cat /var/opt/gitlab/walrecv/node                       # the IP of the primary it follows
ls /var/opt/gitlab/walrecv/fenced/                     # fenced nodes (normally empty)
cat /var/opt/gitlab/walrecv-metrics/walrecv.prom       # the numbers behind these alerts

On the primary:

Terminal window
sudo gitlab-patronictl list
sudo gitlab-psql -c "SELECT slot_name, active, wal_status, pg_size_pretty(pg_current_wal_lsn() - restart_lsn) AS kept FROM pg_replication_slots WHERE slot_name LIKE 'walrecv%';"
Terminal window
sudo journalctl -u walrecv --since "-30min" --no-pager | grep "marked fenced"

Recent changes

Section titled “Recent changes”
  • Check recent chef-repo MRs for the receiver’s role, and for wal-g upgrades on the cluster.
  • Check recent production change requests for failovers or Patroni changes on the cluster.

Possible Resolutions

Section titled “Possible Resolutions”
  • If that node should be stopped (it is an old primary): stop it for real, for example by stopping its VM. Until it is stopped, it may still accept writes.
  • If the fenced file was created for a healthy primary by mistake: remove it: sudo rm /var/opt/gitlab/walrecv/fenced/<IP>.
  • Never leave a fenced file for a node that is a live primary. See production#23275 (page 1, steps 4 and 12).

Escalation

Section titled “Escalation”

A machine that should be stopped may still be accepting writes, so two primaries may exist. Declare an incident, and escalate to the DBO (Database Operations) on-call with /incident escalate in the incident channel (manual escalation).

Section titled “Related Links”