Skip to content
Runbooks

ZoektPushIndexingLagHigh

Overview

Section titled “Overview”

More than 10% of Zoekt push indexing tasks (index_repo) have taken more than 5 minutes from their scheduled time to completion, and this has held for an hour. New commits are slow to become searchable in exact code search. Search itself keeps serving the existing index, so results are only missing recent changes.

This catches indexing lag in the range of minutes to an hour. ZoektTaskQueueNotDraining and the zoekt_tasks apdex SLO only react once tasks approach the 2-hour APDEX_THRESHOLD_S deadline.

Alert behavior

Section titled “Alert behavior”

The source is the gitlab_search_zoekt_task_duration_seconds histogram, recorded by Rails when a node reports a task done (ee/lib/gitlab/metrics/zoekt_tasks_slis.rb). It measures the time between a task’s perform_at and its completion.

The rule computes, over a 30-minute window, the share of index_repo tasks outside the le="300" bucket. 300 is an exact DURATION_BUCKETS edge, so no quantile interpolation is involved. It fires when that share is above 10% for an hour, and only while at least one index_repo task per second completes, so a quiet period cannot fire on a handful of slow tasks.

The rule is fleet-wide rather than per node: lag usually comes from shared task pickup, and ZoektTaskQueueNotDraining already names a node that is falling behind.

How the threshold was chosen

Section titled “How the threshold was chosen”

Backtested against gprd from 2026-10-15 to 2026-10-22:

RuleFirings in 7 days
Share over 5 minutes > 5%, for 30m15
Share over 5 minutes > 10%, for 30m9
Share over 5 minutes > 10%, for 1h (this rule)4
Share over 5 minutes > 20%, for 1h0

The four firings were all during measured indexing degradations: one on 2026-10-16, and three during a system-wide force reindex burst on 2026-10-21. Ordinary push bursts did not fire.

Severity s4, non-paging, user_impacting: no.

First response

Section titled “First response”

  1. Open the observability dashboard and look at the Push indexing tasks waiting over 5 minutes panel, next to Task processing queue size. Check whether one node or the whole fleet is behind.

  2. Check the task mix. In the Indexing request rate panel, a jump in force_index_repo or delete_repo usually means a system-wide event: a schema-version reindex, a node eviction or a rebalancing. Those drain on their own; watch that the share trends down.

  3. Check whether one root namespace dominates. In Kibana, count Search::Zoekt::IndexingTaskWorker jobs with json.job_status: done in pubsub-sidekiq-inf-gprd*, grouped by json.meta.root_namespace. One namespace holding a large share points at a single heavy actor.

  4. No action is a valid outcome when the cause is a known system-wide reindex and the share is falling.

Escalation

Section titled “Escalation”

Not a paging alert. It is posted to #g_global_search_alerts through the global_search team label. If lag keeps growing, or one namespace dominates, raise it with the Global Search team.

Definitions

Section titled “Definitions”
Section titled “Related links”