ZoektRolloutCapacityExhausted

Overview

Zoekt rollout could not plan a replica for at least one namespace in the last hour because no Zoekt node has room for it. Those namespaces get no exact code search (searches fall back to advanced search or basic search) until capacity is added.

The alert counts planning failures from Search::Zoekt::PlanningService, exported by Sidekiq as search_zoekt_rollout_planning_failures_total. Each rollout run counts a namespace once when its planning fails with one of these errors:

Error type	Logged as	Meaning
`index_limit_exceeded`	`Replica reached maximum index limit (N)`	The namespace needs more indices than one replica allows, because the free space left on each node is too small.
`node_unavailable`	`No node can accommodate project <id> with size <bytes>`	No node has room for a project.
`node_unavailable`	`Every node with capacity for project <id> ... already holds an index for this namespace`	Nodes with room exist, but each already holds an index for this namespace.
`node_unavailable`	`No nodes available to create an empty replica`	No node is free to hold an empty replica.

Adding Zoekt nodes resolves all of them.

Alert behavior

Fires when increase(search_zoekt_rollout_planning_failures_total[1h]) > 0, which can be a single failure. There is no for.

A namespace that fails gets last_rollout_failed_at set and is skipped by rollout until zoekt_rollout_retry_interval passes (1d by default). So the counter usually moves once per namespace per day, and the alert resolves after an hour even though the fleet is still full. A resolved alert is not a fix.

Rails creates the series at 0 when each Sidekiq process boots, so increase() catches the first failure. The one known gap: a process that hits a failure before its first scrape is first seen at 1, and that failure is missed.

Severity s4, non-paging, user_impacting: yes.

First response

Find the failed namespaces. In Kibana, on the pubsub-sidekiq-inf-gprd* data view, search:
```
json.class.keyword:"Search::Zoekt::RolloutService" and json.zoekt_rollout_plan.failures.errors.details:*
```
json.zoekt_rollout_plan.failures.namespace_id lists them.
Count all blocked namespaces, not only the ones in this alert:
```
Search::Zoekt::EnabledNamespace.with_missing_indices.with_rollout_blocked.count
```
gitlab-rake gitlab:zoekt:info shows the same number as rollout blocked.
Check fleet capacity. On the observability dashboard, look at storage used and unclaimed storage per node.
Add nodes. See when to add a Zoekt node. Blocked namespaces are retried automatically once zoekt_rollout_retry_interval has passed.

Escalation

Not a paging alert. A capacity increase is a production change request raising replicas in k8s-workloads/gitlab-com, coordinated with the Global Search team.

Definitions

libsonnet/alerts/zoekt-alerts.libsonnet
Generated rules: gprd, gstg
Unit tests: test/mimir-rules/zoekt-alerts_test.yml