Skip to content
Runbooks

ZoektSearchApdexBurn

Overview

Section titled “Overview”

Exact code searches are taking longer than their apdex target. Over the last hour, fewer than 99.9% of zoekt searches completed inside 15.52 seconds.

Two sourced numbers:

How this differs from the ZoektService* SLO alerts

Section titled “How this differs from the ZoektService* SLO alerts”

There are two distinct search latency SLIs and confusing them wastes triage time:

AlertMeasuresEmitted by
This alertThe Rails-side global_search SLI filtered search_type="zoekt" — end-to-end, including Rails work around the Zoekt callGitLab Rails (web/api)
ZoektServiceZoektSearchingApdexSLOViolationzoekt_search_duration_seconds — the Zoekt binary’s own view of its search handlerthe zoekt-webserver container

This alert firing while the service SLO alert is quiet means the latency is in Rails, not in Zoekt — result rendering, permission filtering, the gateway hop. The reverse means Zoekt itself is slow.

Alert behavior

Section titled “Alert behavior”

Fires when the 1-hour apdex ratio is below 0.999, sustained 30 minutes, and there is traffic. The traffic clause is explicit in the rule rather than relying on NaN comparison semantics; traffic cessation is deliberately not this alert’s job — ZoektServiceZoektSearchingTrafficAbsent and ...TrafficCessation already cover it.

Severity s3, alert_type: symptom, user_impacting: yes.

Note this alert has no environment annotation, unlike the node alerts: it reads Rails-emitted counters, which carry env rather than environment. An environment annotation here would render empty.

First response

Section titled “First response”

  1. Split by scope and level. The observability dashboard breaks apdex out by search_level / search_scope. A regression confined to global level is a very different problem from one across all scopes.

  2. Compare against the Rails-side latency panel (row 1, p50/p90/p99 of http_zoekt_requests_duration_seconds) and against the zoekt overview dashboard’s own search duration. That comparison localises the latency to Rails or to Zoekt, per the table above.

  3. Check node health. Searches routed to a node that is struggling — high CPU throttling, memory pressure, mmap exhaustion — are the common cause. The Memory and CPU rows of the zoekt overview dashboard cover this, and ZoektContainersRestartsDueToOOM may also be firing.

  4. Check whether nodes went offline recently. Losing nodes concentrates search load on the survivors. See ZoektNodesOffline.

  5. Look at Kibana for slow-search detail: Zoekt health dashboard. Search errors and slow searches land in production_json.log and zoekt.log.

  6. Mitigation of last resort: turn Zoekt searching off, leaving indexing on. Searches then fall back to Advanced Search, which is slower for code but healthy. Admin > Settings > Search > Exact code search > Enable searching — see enabling/disabling Zoekt search. To scope it to one namespace instead:

    /chatops gitlab run feature set --group=root-group-path disable_zoekt_search_for_saas true --production

Escalation

Section titled “Escalation”

Escalate to the Global Search team. This is a user-facing symptom alert, so if searches are visibly broken rather than slow, treat it as an incident and consider the search-disable mitigation above while diagnosing.

Definitions

Section titled “Definitions”

Known limitation

Section titled “Known limitation”

This alert reads the raw gitlab_sli_global_search_apdex_* counters rather than the application_sli_aggregation:global_search:apdex:* recording rules. The recorded aggregation exists, but is only emitted for the api and web services, and the matching error-rate aggregation does not exist at all because the global_search SLI is declared apdex-only. Moving both this alert and ZoektSearchErrorRateHigh onto recorded series is a follow-up on the Zoekt observability epic.

Section titled “Related links”