Skip to content
Runbooks

ZoektVersionBehindGitLab

Overview

Section titled “Overview”

The Zoekt pods in an environment have run a version other than the one the deployed GitLab expects for 14 days. GitLab expects the version in GITLAB_ZOEKT_VERSION, which Renovate bumps within about a day of each indexer release. Production Zoekt is upgraded by hand, so it can fall behind without anyone noticing.

A Zoekt that is behind can be missing indexer and webserver fixes, or schema changes that the current GitLab release relies on. Search keeps working, so this is not urgent, but the gap grows with every release.

Alert behavior

Section titled “Alert behavior”

The rule compares two series:

  • Running version: the image tag of the zoekt-webserver containers, from kube-state-metrics’ kube_pod_container_info, with the leading v removed.
  • Expected version: search_zoekt_expected_version_info{version="..."}, exported every minute by Search::Zoekt::MetricsUpdateCronWorker from GITLAB_ZOEKT_VERSION (ee/app/services/search/zoekt/metrics_service.rb).

It fires for a running version that matches no expected version in the same environment for 14 days. The timer is per running version: it starts when that version stops matching and resets when Zoekt moves to another version. While a GitLab deploy is in progress, old and new Sidekiq pods both report, and a running version matching either does not fire.

The rule stays silent when the expected-version metric is absent, for example before it is deployed or where Zoekt indexing is disabled. The expected side looks back 1 hour, so a short gap in the metric does not reset the 14-day timer.

Severity s4, non-paging, user_impacting: no.

First response

Section titled “First response”

  1. Open the Zoekt overview dashboard, expand the Versions row, and compare the running images with the expected by GitLab line in Running and expected Zoekt versions.

  2. Upgrade Zoekt, following Deploying to GitLab.com in the indexer README:

    1. Check that the CNG gitlab-zoekt image for the expected version exists. Renovate opens that MR in CNG.
    2. Update gstg.zoekt-versions.yaml in k8s-workloads/gitlab-com, and check staging.
    3. Update gprd.zoekt-versions.yaml.

  3. If there is a reason to stay on the running version, for example a known regression in the newer release, record it in an issue and silence the alert for the environment until the fix ships.

Escalation

Section titled “Escalation”

Not a paging alert. It is posted to #g_global_search_alerts through the global_search team label. Raise the upgrade with the Global Search team.

Definitions

Section titled “Definitions”
Section titled “Related links”