Skip to content
Runbooks

WalReceiverCleanupNotSucceeding

Overview

Section titled “Overview”

This page covers two alerts:

  • WalReceiverCleanupNotSucceeding: the cleanup on the WAL receiver has not finished successfully for more than 9 hours, although it runs every 8 hours.
  • WalReceiverCleanupNeverSucceeded: the receiver has been running for more than 9 hours, but its cleanup has never finished successfully.

In both cases nothing is deleted, so the receiver’s disk fills.

Common causes:

  • the cleanup stops with an error, for example WALG_COMPRESSION_METHOD is '...', but the checksum check needs brotli (the receiver’s wal-g settings don’t match the cluster’s);
  • the timer walrecv-cleanup.timer is not active;
  • the cleanup crashes.

The WAL receiver is a small VM per Patroni cluster, outside the cluster, that keeps a live copy of the cluster’s WAL with pg_receivewal. If the primary’s upload to GCS stops, as in INC-14518, the WAL still exists on the receiver. It is set up by the gitlab-patroni::wal_receiver recipe.

Services

Section titled “Services”

Metrics

Section titled “Metrics”
  • walrecv_cleanup_last_success_timestamp_seconds: when the cleanup last finished successfully. A run that stops with an error does not update it. It is 0 until the first run succeeds; both alerts ignore the 0, so a failed first run fires “never succeeded”, not “not succeeding”.
  • walrecv_heartbeat_timestamp_seconds: shows that the receiver itself is running (used for “never succeeded”).

Alert Behavior

Section titled “Alert Behavior”
  • WalReceiverCleanupNotSucceeding fires after 9 hours without a successful run (plus 10 minutes). WalReceiverCleanupNeverSucceeded fires when there has been no successful run in the first 9 hours. Both clear after a successful run.
  • Severity s3: Slack only, no page. In gprd it goes to #s_database_excellence_alerts and #alerts; in gstg to #alerts-nonprod, so staging firings can be reviewed before production.

Verification

Section titled “Verification”

On the receiver:

Terminal window
sudo systemctl list-timers walrecv-cleanup.timer --no-pager      # when it last ran and runs next
sudo journalctl -u walrecv-cleanup --since "-1d" --no-pager | grep -E "ERROR|deleted or would delete"
cat /var/opt/gitlab/walrecv-metrics/walrecv_cleanup.prom          # the numbers behind these alerts
df -h /var/opt/gitlab

Recent changes

Section titled “Recent changes”
  • Check recent chef-repo MRs for the receiver’s role, and for wal-g upgrades on the cluster.
  • Check recent production change requests for failovers or Patroni changes on the cluster.

Possible Resolutions

Section titled “Possible Resolutions”
  • Fix the cause in the cleanup’s log (the ERROR line says what is wrong). For a compression mismatch, see production#23275 (page 6).
  • If the timer isn’t active: sudo systemctl enable --now walrecv-cleanup.timer.
  • Start one run by hand and check its summary: sudo systemctl start walrecv-cleanup.service, then sudo journalctl -u walrecv-cleanup -n 5 --no-pager.

Escalation

Section titled “Escalation”

If you can’t find and fix the cause with the steps above, raise it in #g_database_operations.

Section titled “Related Links”