Skip to content
Runbooks

WalReceiverSlotRetainingTooMuchWal

Overview

Section titled “Overview”

This alert fires when the primary keeps more than 25% of its max_slot_wal_keep_size for the WAL receiver’s slot, because the receiver hasn’t copied that WAL yet.

When the limit is reached, Postgres drops the slot’s reservation: the receiver’s copy then gets a gap, and the receiver needs manual steps. Until then, the kept WAL also uses disk space on the primary.

The usual cause is that the receiver is not connected (see WalReceiverNotConnected), or is too slow to keep up (disk or network on the receiver).

The WAL receiver is a small VM per Patroni cluster, outside the cluster, that keeps a live copy of the cluster’s WAL with pg_receivewal. If the primary’s upload to GCS stops, as in INC-14518, the WAL still exists on the receiver. It is set up by the gitlab-patroni::wal_receiver recipe.

Services

Section titled “Services”

Metrics

Section titled “Metrics”
  • pg_replication_slots_restart_lsn_bytes{slot_name=~"walrecv(_.*)?"}: how much WAL the slot keeps on the primary.
  • pg_settings_max_slot_wal_keep_size_bytes: that cluster’s own limit (for example 3,640 GiB on gprd main).
  • Only the primary is checked (pg_replication_is_replica == 0).

Alert Behavior

Section titled “Alert Behavior”
  • Fires after 10 minutes above 25% of the limit. Clears when the receiver catches up.
  • On gprd main, 25% of the limit is about 900 GiB, which takes hours to build up even at peak write rate, so there is time to act.
  • Severity s3: Slack only, no page. In gprd it goes to #s_database_excellence_alerts and #alerts; in gstg to #alerts-nonprod, so staging firings can be reviewed before production.

Verification

Section titled “Verification”

On the receiver:

Terminal window
sudo systemctl status walrecv.service                 # is the receiver running?
sudo journalctl -u walrecv --since "-1h" --no-pager | tail -30   # what it did last
cat /var/opt/gitlab/walrecv/node                       # the IP of the primary it follows
ls /var/opt/gitlab/walrecv/fenced/                     # fenced nodes (normally empty)
cat /var/opt/gitlab/walrecv-metrics/walrecv.prom       # the numbers behind these alerts

On the primary:

Terminal window
sudo gitlab-patronictl list
sudo gitlab-psql -c "SELECT slot_name, active, wal_status, pg_size_pretty(pg_current_wal_lsn() - restart_lsn) AS kept FROM pg_replication_slots WHERE slot_name LIKE 'walrecv%';"

Recent changes

Section titled “Recent changes”
  • Check recent chef-repo MRs for the receiver’s role, and for wal-g upgrades on the cluster.
  • Check recent production change requests for failovers or Patroni changes on the cluster.

Possible Resolutions

Section titled “Possible Resolutions”
  • Fix the receiver first: see WalReceiverNotConnected.
  • Do not drop the slot to free space unless you decide to give up the receiver’s copy: dropping it means the receiver’s copy gets a gap. If the slot is lost or must be dropped, follow production#23275 (page 4, “receiver off too long”).

Escalation

Section titled “Escalation”

If you can’t find and fix the cause with the steps above, raise it in #g_database_operations.

Section titled “Related Links”