Prometheus Dead Man's Snitch

Symptoms

Prometheus SnitchHeartBeat is an always-firing alert. It’s used as an end-to-end test of Prometheus through the Alertmanager.

Possible checks

• Make sure the SnitchHeartBeat alert is firing
• Make sure the SnitchHeartBeat alert is not silenced.
• Check Alertmanager config to see if routes are pointing to correct snitch by searching for dead_mans_snitch_
• Check the Prometheus and Alertmanager logs to make sure they are communicating properly with https://deadmanssnitch.com/.

Setting up a snitch

1. Create a new snitch in the config-mgmt repo https://ops.gitlab.net/gitlab-com/gl-infra/config-mgmt/-/blob/main/environments/dmsnitch/main.tf?ref_type=heads
2. Update the json array snitchChannels in the alertmanager Vault secret, which alertmanager.jsonnet uses.