WAL receiver: the receiver was off too long and its slot is lost

Use this when the WAL receiver was stopped or cut off for so long that the primary stopped keeping WAL for it. The receiver can’t continue from where it stopped.

Who: the DBRE on call.

Words used

Word	Meaning
WAL	Postgres’s log of every change, in files of 16 MB.
Receiver	The cluster’s WAL receiver VM. Its files are in `/var/opt/gitlab/walrecv/tl<timeline>/`.
Slot	A replication slot on the primary, named in the receiver’s role (`wal_receiver.slot`). It makes the primary keep WAL until the receiver has it. It is a permanent Patroni slot, so Patroni creates it on every member.
`max_slot_wal_keep_size`	The most WAL the primary keeps for a slot. Past it, the slot is lost: its WAL is deleted. This protects the primary’s disk.
Gap	WAL files the receiver never got. They are still in GCS, uploaded by the primary.
Gap marker	A file `/var/opt/gitlab/walrecv/GAP-<time>` that the receiver creates when it finds a gap. The `WalReceiverGapDetected` alert fires while one exists.

How to know

On the primary:

sudo gitlab-psql -c "select slot_name, active, wal_status, restart_lsn from pg_replication_slots where slot_name like 'walrecv%'"

wal_status is lost, or restart_lsn is empty. On the receiver, sudo journalctl -u walrecv --since -1h shows pg_receivewal failing to start from the slot.

Steps

Drop the slot on the primary. Patroni creates it again within about 25 seconds, starting at the primary’s current position:
Terminal window
```
sudo gitlab-psql -c "select pg_drop_replication_slot('<slot>')"
```
Check the slot is back: run the query from “How to know” again after 30 seconds.
Move the receiver’s current folder aside, and restart the receiver. Otherwise the receiver asks the primary for the WAL after its newest file, which no longer exists. On the receiver:
Terminal window
```
sudo systemctl stop walrecv
sudo mv /var/opt/gitlab/walrecv/tl<newest> /var/opt/gitlab/walrecv-before-slot-loss-<date>
sudo systemctl start walrecv
```
It creates a new tl<timeline> folder and starts from the new slot. Check with pgrep -a -x pg_receivewal and the primary’s query (active is t).
Work out the gap by hand. The receiver can’t always see it: once its current folder is moved away, it often has nothing left to compare with, so don’t rely on a GAP ALARM in its log. The gap is from the newest file in the moved folder to the file where the new slot starts:
Terminal window
```
sudo ls /var/opt/gitlab/walrecv-before-slot-loss-<date> | grep -v partial | tail -1   # on the receiver: the last file it has
sudo gitlab-psql -AtX -c "select pg_walfile_name(restart_lsn) from pg_replication_slots where slot_name = '<slot>'"   # on the primary: where the new slot starts
```
Every file between those two must be in GCS: cd /tmp && sudo -u gitlab-psql /usr/bin/envdir /etc/wal-g.d/env /opt/wal-g/bin/wal-g st ls wal_005/.

The receiver doesn’t need these files back: a restore reads from GCS any file the receiver doesn’t have (WAL receiver: restore with the receiver’s files). If one is missing in GCS too, that WAL exists nowhere else: raise it in the incident.
If a gap marker was created (/var/opt/gitlab/walrecv/GAP-*), remove it once GCS has the missing files, so the alert clears: sudo rm /var/opt/gitlab/walrecv/GAP-*.
Delete the moved folder after a day, once GCS has its files (the receiver’s cleanup doesn’t look at it).

Then find out why the receiver was off for so long, and why WalReceiverNotConnected didn’t get attention before the slot was lost.

Patroni’s error

With Patroni 3.3.0, when the slot was lost Patroni logged TypeError: '>' not supported between instances of 'int' and 'NoneType' once, in _ensure_physical_slots, and did not create the slot again by itself. gprd runs Patroni 3.3.4, which has the same code line. (to test in gstg) The steps above work either way, because they drop the slot by hand.