WAL receiver: the replication password changed

Use this when the password of the replication user (gitlab-replicator) changes for a Patroni cluster that has a WAL receiver.

Who: whoever changes the password, as part of the same change.

Words used

Word	Meaning
Receiver	The cluster’s WAL receiver VM. It connects to the primary as the replication user.
pgpass file	/var/opt/gitlab/walrecv.pgpass on the receiver. It holds the replication password. Chef writes it from the cluster’s gitlab-patroni secret.

Why

The receiver reads the password from its pgpass file, and chef only rewrites that file on a chef run on the receiver. A chef run on the members doesn’t change the receiver. Until chef runs on the receiver, as soon as it reconnects (for example after a failover), its connection fails with password authentication failed for user "gitlab-replicator" every 5 seconds, it copies nothing, and WalReceiverNotConnected fires after 5 minutes.

Steps

1. Change the password in the cluster’s gitlab-patroni secret, as for the members (see PostgreSQL role credential rotation).

2. Run chef on the receiver, right after the members:

   sudo chef-client

3. Check that it reconnected. On the receiver, sudo journalctl -u walrecv --since -5min no longer shows password authentication failed, and pgrep -a -x pg_receivewal shows it running. On the primary, the receiver’s slot is active.