Skip to content
Runbooks

WalReceiverGapDetected

Overview

Section titled “Overview”

This alert fires when the WAL receiver has found a gap: WAL is missing between its own files and the point where the primary will send WAL from.

The receiver checks this each time it starts copying a new timeline (a new folder tl<number>). If the primary’s slot starts later than the last file the receiver has, for example because the slot was lost while the receiver was off too long, the WAL in between is missing. The receiver then logs GAP ALARM and writes a marker file GAP-<time> in /var/opt/gitlab/walrecv/.

Impact: the receiver’s copy has a hole. A restore that relied on the receiver’s files would stop at the gap without an error. The missing files are normally in GCS.

The WAL receiver is a small VM per Patroni cluster, outside the cluster, that keeps a live copy of the cluster’s WAL with pg_receivewal. If the primary’s upload to GCS stops, as in INC-14518, the WAL still exists on the receiver. It is set up by the gitlab-patroni::wal_receiver recipe.

Services

Section titled “Services”

Metrics

Section titled “Metrics”
  • walrecv_gap_markers: the number of GAP-* marker files on the receiver.

Alert Behavior

Section titled “Alert Behavior”
  • Fires after 2 minutes. Clears only when the GAP-* marker is removed, after the gap is filled.
  • Severity s3: Slack only, no page. In gprd it goes to #s_database_excellence_alerts and #alerts; in gstg to #alerts-nonprod, so staging firings can be reviewed before production.

Verification

Section titled “Verification”

On the receiver:

Terminal window
sudo systemctl status walrecv.service                 # is the receiver running?
sudo journalctl -u walrecv --since "-1h" --no-pager | tail -30   # what it did last
cat /var/opt/gitlab/walrecv/node                       # the IP of the primary it follows
ls /var/opt/gitlab/walrecv/fenced/                     # fenced nodes (normally empty)
cat /var/opt/gitlab/walrecv-metrics/walrecv.prom       # the numbers behind these alerts

On the primary:

Terminal window
sudo gitlab-patronictl list
sudo gitlab-psql -c "SELECT slot_name, active, wal_status, pg_size_pretty(pg_current_wal_lsn() - restart_lsn) AS kept FROM pg_replication_slots WHERE slot_name LIKE 'walrecv%';"

The GAP ALARM line in the receiver’s log says which file the receiver has up to, and where the slot starts:

Terminal window
sudo journalctl -u walrecv --no-pager | grep "GAP ALARM" | tail -3
ls /var/opt/gitlab/walrecv/GAP-*

Recent changes

Section titled “Recent changes”
  • Check recent chef-repo MRs for the receiver’s role, and for wal-g upgrades on the cluster.
  • Check recent production change requests for failovers or Patroni changes on the cluster.

Possible Resolutions

Section titled “Possible Resolutions”
  • Copy the missing WAL files from GCS into the receiver’s folder (wal-g wal-fetch), check that the files now follow each other without a hole, and then remove the GAP-* marker. Full procedure: production#23275 (page 4).
  • Find out why the slot was lost (the receiver was off too long, or WalReceiverSlotRetainingTooMuchWal was ignored).

Escalation

Section titled “Escalation”

If you can’t find and fix the cause with the steps above, raise it in #g_database_operations.

Section titled “Related Links”