Skip to content
Runbooks

Pingdom

How to configure checks in Pingdom

Section titled “How to configure checks in Pingdom”

  • Pingdom checks are configured in the pingdom/pingdom.yml file

  • On non-master branches, the dryrun_pingdom_checks CI job will (partially) validate the pingdom.yml file and will display changes that will be made via the Pingdom API (see Dry-run Mode for details).

  • When changes to this file are merged to master, the deploy_pingdom_checks GitLab CI will execute and make the actual changes:

  • This job will perform 3 tasks:

    • It will insert any new checks that have been added to the file. These checks will be prefixed with check:
    • It will remove any checks in Pingdom with the prefix check: that are not in this file
    • It will update any other checks from pingdom/pingdom.yml

  • Note: the pingdom script uses the check: prefix to signal that the check is managed by the script. Any checks on Pingdom that use the check: prefix but are not on the master branch will be deleted the next time the master pipeline is executed

pingdom.yml Details

Section titled “pingdom.yml Details”

The pingdom.yml is structured as follows:

unique_tag: "pingdom-automated" # This is a tag which only checks in this file should include
defaults:
  timeout_ms: 2000              # The default timeout in milliseconds
  resolution_minutes: 5         # The amount of time before raising an alert
integrations:                   # This maps from the names we use in this document to the IDS pingdom needs
  - name: pagerduty
    id: 65172                   # This can be found in the URL when editing an integration
checks:
  # Each check has the following structure
  - url: https://gitlab.com/gitlab-org/gitlab-ce/
    timeout_ms: 5000            # The timeout for this check
    notify_when_restored: true  # Send an alert when service is restored
    check_type: http            # Set to "tcp" to use a TCP check type. Other values are ignored, and http checks are created.
    check_port: 443             # Port to be used during TCP checks. Otherwise ignored.
    tags:                       # Any additional tags to add to the check
      - gitaly
      - database
    teams:                      # Teams to associate the check with. See Pingdom for a list of teams
      - Infrastructure
    integrations:
      - pagerduty

Dry-run Mode

Section titled “Dry-run Mode”

When run with the --dry-run, the script will execute in dry-run mode, which will display changes but without executing them against the Pingdom API.

It is always safe to run.

Running Locally

Section titled “Running Locally”

You can run the script locally, so long as you provide the following environment variables, which can be found in Pingdom and 1Password.

export PINGDOM_USERNAME="[email protected]"
export PINGDOM_PASSWORD="..."
export PINGDOM_APPKEY="..."
export PINGDOM_ACCOUNT_EMAIL="..."

Extracing availability metrics from Pingdom for further analysis

Section titled “Extracing availability metrics from Pingdom for further analysis”

We can extract Pingdom availability metrics to CSV format using the ./pingdom/generate-availability-stats.sh script.

Currently, this script can be executed manually. As a next step we should automate this process.

Terminal window
export PINGDOM_APP_KEY="..."
export PINGDOM_PASSWORD="..."
export PINGDOM_ACCOUNT_EMAIL="..."


$ ./pingdom/generate-availability-stats.sh


#Check,Date,Availability
"check:https://gitlab.com/","2018-10-24",1
"check:https://gitlab.com/","2018-10-25",1
"check:https://gitlab.com/","2018-10-26",1
"check:https://gitlab.com/","2018-10-27",1

This will generate raw data in a CSV format. This data can then be analysed further using Google Sheets or other analysis tools.

An example of this analysis is Pingdom Availability Statistics spreadsheet, which pivots the data to generate availability KPI values for GitLab.com.

PagerDuty integration

Section titled “PagerDuty integration”

PagerDuty to Pingdom integration is used for the Uptime checks and alert notifications.

The pingdom integration is configured in the PagerDuty for GitLab Production Service with a webhook. The webhook is attached to the Uptime checks in the Pingdom. The Uptime checks are configured in the pingdom.yml file and updated with CI.

Pingdom Exporter

Section titled “Pingdom Exporter”

We have pingdom exporter deployed in OPS through the helmfiles.

To limit the API requests made by the exporter we filter the checks picked up by the exporter.

It only queries checks which have the pingdom-exporter-check tag.