Skip to content
Runbooks

WalReceiverCleanupKeepingFiles

Overview

Section titled “Overview”

This alert fires when the last cleanup run on the WAL receiver could not confirm more than 10 files against GCS, and so kept them.

The cleanup deletes a file only when GCS has exactly the same file. For most files it compares the size and the CRC32C fingerprint that GCS stores, after squeezing the receiver’s copy with wal-g exactly as the primary did. “Check failed” means it couldn’t check the file; “different” means the fingerprints don’t match.

The most common cause is that wal-g on the receiver no longer matches wal-g on the primaries (a different version, or a different compression setting). Then nothing matches, nothing is deleted, and with 1-day retention the receiver’s disk fills fast. Other causes: GCS unreachable, or wrong credentials.

The WAL receiver is a small VM per Patroni cluster, outside the cluster, that keeps a live copy of the cluster’s WAL with pg_receivewal. If the primary’s upload to GCS stops, as in INC-14518, the WAL still exists on the receiver. It is set up by the gitlab-patroni::wal_receiver recipe.

Services

Section titled “Services”

Metrics

Section titled “Metrics”
  • walrecv_cleanup_files{result="check_failed"} and walrecv_cleanup_files{result="different"}: files the last cleanup run kept for these reasons.

Alert Behavior

Section titled “Alert Behavior”
  • Fires 1 minute after a cleanup run reports more than 10 such files. Clears after a run with 10 or fewer.
  • Severity s3: Slack only, no page. In gprd it goes to #s_database_excellence_alerts and #alerts; in gstg to #alerts-nonprod, so staging firings can be reviewed before production.

Verification

Section titled “Verification”

On the receiver:

Terminal window
sudo systemctl list-timers walrecv-cleanup.timer --no-pager      # when it last ran and runs next
sudo journalctl -u walrecv-cleanup --since "-1d" --no-pager | grep -E "ERROR|deleted or would delete"
cat /var/opt/gitlab/walrecv-metrics/walrecv_cleanup.prom          # the numbers behind these alerts
df -h /var/opt/gitlab

Compare wal-g on the receiver and on the cluster’s primary:

Terminal window
/opt/wal-g/bin/wal-g --version
cat /etc/wal-g.d/env/WALG_COMPRESSION_METHOD

Recent changes

Section titled “Recent changes”
  • Check recent chef-repo MRs for the receiver’s role, and for wal-g upgrades on the cluster.
  • Check recent production change requests for failovers or Patroni changes on the cluster.

Possible Resolutions

Section titled “Possible Resolutions”
  • wal-g differs: make the receiver’s role use the same wal-g role and the same walg_attr settings as its cluster’s Patroni role, and run chef on the receiver. The next cleanup run then deletes the files that GCS has. See production#23275 (page 6, “wal-g upgrade or compression change”).
  • GCS unreachable or credentials: fix access with the same key wal-g uses (/etc/wal-g.d/env/GOOGLE_APPLICATION_CREDENTIALS).
  • Files that really differ after a split brain are counted separately (result="timeline_switch_different") and are not part of this alert.

Escalation

Section titled “Escalation”

If you can’t find and fix the cause with the steps above, raise it in #g_database_operations.

Section titled “Related Links”