Elastic Nodes Disk Space Saturation
Get cluster health
Section titled “Get cluster health”Using the dev console or the API Console get the health of the cluster.
Check each value and see if this gives you an indication that something is wrong with the cluster.
Unassigned shards
Section titled “Unassigned shards”If the cluster health indicates that there is a number of unassigned shards, this can lead up to the hot nodes disk being filled up because ILM is not able to roll over indencies properly.
Using the dev console or the API Console get an overview ilm.
GET _all/_ilm/explain
. To filter out for the ones with errors withGET _all/_ilm/explain?only_errors
.GET _cluster/allocation/explain
. Which provides an explanation for why the shard is unassigned.
Find which lifecycle an index is using
Section titled “Find which lifecycle an index is using”GET _cat/indices/pubsub-gitaly-inf-gprd-*
: Get all gitaly indencies.GET pubsub-gitaly-inf-gprd-007382/_settings
: Getindex.lifecycle
.GET _ilm/policy/gitlab-infra-high-ilm-policy
: Get policy details.