WAL receiver: wal-g upgrade or compression change

Use this when wal-g is upgraded, or its settings (such as the compression) change, on a Patroni cluster that has a WAL receiver.

Who: whoever makes the wal-g change, in the same chef-repo MR.

Words used

Word	Meaning
Receiver	The cluster’s WAL receiver VM.
Cleanup	A job on the receiver (walrecv-cleanup.timer, every 8 hours) that deletes its files once GCS has an identical copy.
Compression	wal-g compresses each WAL file before uploading it. Our clusters use brotli.
CRC32C	A checksum that GCS keeps for every file. The cleanup compares it with its own.

Why

To check a file, the cleanup compresses its own copy with the receiver’s wal-g, and compares the result with the file in GCS (size and CRC32C). This only matches when the receiver’s wal-g has the same version and settings as the primary’s. If they differ, nothing is lost: the cleanup keeps every file it can’t confirm. But the receiver keeps only 1 day of WAL, so its disk fills fast, and the WalReceiverCleanupKeepingFiles alert fires. If the compression is not brotli, the cleanup stops with an error, and WalReceiverCleanupNotSucceeding fires.

Steps

1. In the same MR that changes wal-g for the cluster, change the receiver’s role the same way:

   • the wal-g role in its run_list (for example role[gprd-walg-3_0_5-2204] in gprd);
   • the gitlab_walg.walg_attr settings (walg_compression_method, gcs_normalize_prefix, gcs_max_retries).

2. After the MR is merged, run chef on the receiver too: sudo chef-client.

3. Check that both have the same version. On the primary and on the receiver: /opt/wal-g/bin/wal-g --version.

4. After the next cleanup run, check that it confirmed files. On the receiver:

   sudo journalctl -u walrecv-cleanup --since -9h | tail -5
   grep -v '^#' /var/opt/gitlab/walrecv-metrics/walrecv_cleanup.prom

   walrecv_cleanup_files{result="deleted"} must be more than 0, and result="different" and result="check_failed" must be 0 or close to it.

(to test in gstg) This page has not been tried with a real wal-g upgrade yet.