Skip to content
Runbooks

WalReceiverMetricsMissing

Overview

Section titled “Overview”

This alert fires when the WAL receiver is not publishing its metrics: walrecv_heartbeat_timestamp_seconds is missing on a walrecv-* node, or it is more than 10 minutes old. The receiver rewrites it every 10 seconds.

While this fires, the other receiver alerts that use the receiver’s own numbers (WalReceiverDiskFilling, WalReceiverGapDetected, WalReceiverFencedButStillPrimary, WalReceiverCleanupKeepingFiles, WalReceiverCleanupNotSucceeding) can’t fire. The slot alerts still work, because they use the primary’s metrics.

Common causes:

  • node_exporter’s text-file folder (collector.textfile.directory) isn’t set, so chef didn’t link the receiver’s metric files (chef logs a warning: node_exporter has no collector.textfile.directory);
  • the links in node_exporter’s folder to /var/opt/gitlab/walrecv-metrics/ are missing or broken;
  • the receiver’s script is stuck, so it stops rewriting the file;
  • node_exporter itself is down.

The WAL receiver is a small VM per Patroni cluster, outside the cluster, that keeps a live copy of the cluster’s WAL with pg_receivewal. If the primary’s upload to GCS stops, as in INC-14518, the WAL still exists on the receiver. It is set up by the gitlab-patroni::wal_receiver recipe.

Services

Section titled “Services”

Metrics

Section titled “Metrics”
  • walrecv_heartbeat_timestamp_seconds: written by the receiver every 10 seconds, in /var/opt/gitlab/walrecv-metrics/walrecv.prom.
  • node_uname_info{fqdn=~"walrecv-.*"}: shows that node_exporter is running on a receiver, so a missing heartbeat can be told apart from a missing node.

Alert Behavior

Section titled “Alert Behavior”
  • Fires about 20 minutes after the last heartbeat (the query waits until it is 10 minutes old, then for: 10m adds 10 more), or after 10 minutes if the heartbeat is missing altogether. Clears as soon as the heartbeat is fresh again.
  • Severity s3: Slack only, no page. In gprd it goes to #s_database_excellence_alerts and #alerts; in gstg to #alerts-nonprod, so staging firings can be reviewed before production.

Verification

Section titled “Verification”

On the receiver:

Terminal window
grep heartbeat /var/opt/gitlab/walrecv-metrics/walrecv.prom; date +%s       # the heartbeat should be within 10 s of now
ls -l /opt/prometheus/node_exporter/metrics/ | grep walrecv                  # both links should point to /var/opt/gitlab/walrecv-metrics/
systemctl is-active walrecv node_exporter
sudo journalctl -u walrecv --since "-15min" --no-pager | tail -5

Recent changes

Section titled “Recent changes”
  • Check recent chef-repo MRs for the receiver’s role, and for node_exporter changes.
  • Check recent production change requests for changes on the receiver.

Possible Resolutions

Section titled “Possible Resolutions”
  • If the links are missing, run chef on the receiver: sudo chef-client. If chef warns that node_exporter has no text-file folder, fix the node_exporter settings in the receiver’s role.
  • If the heartbeat in the file is old but the service is active, the script is stuck: sudo systemctl restart walrecv, then check that the heartbeat moves.
  • If node_exporter is down: sudo systemctl restart node_exporter.

Escalation

Section titled “Escalation”

If you can’t find and fix the cause with the steps above, raise it in #g_database_operations.

Section titled “Related Links”