GitalyVersionMismatch

Overview

Two or more different version of Gitaly are running in a stage for longer than 60 minutes. This should not be the case as gitaly nodes are expected to run on the same version. It’s considered a high-severity issue that requires immediate attention.

During a deployment, two distinct versions of Gitaly may be running alongside one another, but this should not be the case for more than 60m.

Services

Service Overview
Team that owns the service: Core Platform:Gitaly Team
Label: gitlab-com/gl-infra/production~“Service::Gitaly”

Metrics

This expression is measuring the count of deployed Gitaly versions in gprd environment in different stages. The counter per stage should be equal to 1 under normal conditions
Alert definition

Alert Behavior

This alert should be rare, but if it’s triggered, needs to be investigated immediately

Severities

This alert might create S1 incidents.
There might be some gitlab.com users impact
Review Incident Severity Handbook page to identify the required Severity Level

Verification

Gitaly Service Overview dashboard
Verify that each stage has only one version of Gitaly deployed

Recent changes

Closed production issues for Gitaly Service

Troubleshooting

Check recent deployments or rollbacks in gprd environment
Identify the prevailing Gitaly version in the gprd environment
Identify the nodes having different Gitaly version deployed in gprd environemnt (replace version with the version found in the previous step)
Verify that chef-client run normally on the affected nodes (replace fqdn for the affected nodes). It is expected to run every 30 minutes

Possible Resolutions

Previous GitalyVersionMismatch incidents

Dependencies

failed chef-client runs or failed deployments (rollbacks) can cause a gitaly node to fail to apply a required version

Escalation

For escalation contact the following channels:

#g_gitaly

Alternative slack channels: