Skip to content
Runbooks

WalReceiverDiskFilling

Overview

Section titled “Overview”

This alert fires when the WAL receiver has less than 15% free space on /var/opt/gitlab, where it keeps its WAL copy. If the disk fills, the receiver stops copying WAL.

The receiver keeps about 1 day of WAL: its cleanup runs every 8 hours and deletes files older than 1 day only when GCS has exactly the same file. So the disk fills when:

  • the cleanup is not deleting (dry run, failing, or not running; see the cleanup alerts);
  • the primary’s upload to GCS has stopped, so GCS doesn’t have the files: then the receiver is doing its job, and these files may be the only copy;
  • files were kept after a split brain, because they differ from GCS;
  • the cluster writes much more WAL than usual.

The WAL receiver is a small VM per Patroni cluster, outside the cluster, that keeps a live copy of the cluster’s WAL with pg_receivewal. If the primary’s upload to GCS stops, as in INC-14518, the WAL still exists on the receiver. It is set up by the gitlab-patroni::wal_receiver recipe.

Services

Section titled “Services”

Metrics

Section titled “Metrics”
  • node_filesystem_avail_bytes / node_filesystem_size_bytes for /var/opt/gitlab on machines that publish walrecv_heartbeat_timestamp_seconds (that is, WAL receivers).

Alert Behavior

Section titled “Alert Behavior”
  • Fires after 10 minutes below 15% free. Clears when space is freed or the disk is grown.
  • Severity s3: Slack only, no page. In gprd it goes to #s_database_excellence_alerts and #alerts; in gstg to #alerts-nonprod, so staging firings can be reviewed before production.

Verification

Section titled “Verification”

On the receiver:

Terminal window
sudo systemctl list-timers walrecv-cleanup.timer --no-pager      # when it last ran and runs next
sudo journalctl -u walrecv-cleanup --since "-1d" --no-pager | grep -E "ERROR|deleted or would delete"
cat /var/opt/gitlab/walrecv-metrics/walrecv_cleanup.prom          # the numbers behind these alerts
df -h /var/opt/gitlab

Also check whether the primary’s upload to GCS works (the walgBackupDelayed alert, and ls /var/opt/gitlab/walrecv/tl*/ | tail against the newest file in GCS).

Recent changes

Section titled “Recent changes”
  • Check recent chef-repo MRs for the receiver’s role, and for wal-g upgrades on the cluster.
  • Check recent production change requests for failovers or Patroni changes on the cluster.

Possible Resolutions

Section titled “Possible Resolutions”
  • The cleanup fails or doesn’t run: see WalReceiverCleanupNotSucceeding and WalReceiverCleanupKeepingFiles.
  • The primary’s upload to GCS stopped: do not delete the receiver’s files; they may be the only copy. Fix the upload, or follow the hand-over and restore procedures in production#23275. If needed, grow the receiver’s data disk (data_disk_size of the receiver’s module in config-mgmt).
  • Files kept after a split brain: remove them only after the incident is closed and those writes are recovered or written off (production#23275, page 2).

Escalation

Section titled “Escalation”

If you can’t find and fix the cause with the steps above, raise it in #g_database_operations.

Section titled “Related Links”