Skip to content
Runbooks

component_saturation_slo_out_of_bounds:kube_persistent_volume_claim_inodes

Overview

Section titled “Overview”
  • This alert means that a Kube persistent volume is running out of inodes — the number of files it can hold — which is independent of how much disk space is free.
  • A volume can exhaust inodes while still reporting plenty of free space. df -h will look healthy while df -i is at 100%.
  • This is usually caused by a workload accumulating a very large number of small files: per-branch or per-MR scratch clones, dependency/module caches, WAL or index files.
  • This affects the pod(s) mounting the volume. Once inodes are exhausted, any attempt to create a file fails with ENOSPC (“no space left on device”) even though space remains, which is a misleading error to debug from cold.
  • The recipient needs to identify which PVC is saturated, then decide between cleaning up accumulated files, bounding the growth, or growing the volume. Prefer bounding the growth — see Possible Resolutions.

Services

Section titled “Services”

Metrics

Section titled “Metrics”
  • This alert is based on kubelet_volume_stats_inodes_used / kubelet_volume_stats_inodes.
  • The soft SLO is at 85% and the hard SLO is at 90%.
  • Example Grafana Query

Alert Behavior

Section titled “Alert Behavior”
  • The alerting rule aggregates by env, environment and shard, so the alert itself does not name the PVC. The saturation detail dashboard does break it down per-claim (resourceLabels: ['persistentvolumeclaim']) — use the dashboard or the raw kubelet query above to identify it.
  • Growth is often a step function rather than a drift: each step is one unit of work (a plan, a build, a compaction) creating a batch of files. Reading the steps and correlating them to timestamps on disk is usually the fastest route to the cause.
  • This alert can be silenced if there is a plan in place to resolve the issue. Generally it should be resolved instead of silenced. Note that a capacity silence on an unbounded-growth problem tends to recur — see INC-8304.

Severities

Section titled “Severities”
  • Incidents involving this alert are likely S3 or S4 — the service is still up, and the alert fires before exhaustion.
  • This is not normally a user impacting alert. Assess what the owning workload does: for internal tooling (Atlantis, CI) the impact is blocked automation rather than customer traffic.

Verification

Section titled “Verification”

  • Grafana Dashboard (ops)

  • Confirm on the pod itself, which also tells you whether it is inode-bound or space-bound:

    Terminal window
    kubectl -n <namespace> exec <pod> -- df -i <mount>   # inodes
    kubectl -n <namespace> exec <pod> -- df -h <mount>   # bytes

Recent changes

Section titled “Recent changes”

Troubleshooting

Section titled “Troubleshooting”

  1. Identify the PVC from the dashboard or the raw kubelet query, remembering to use the tenant matching the alert’s env. To list every claim over a threshold:

    topk(15, kubelet_volume_stats_inodes_used / kubelet_volume_stats_inodes) > 0.8

  2. Confirm it is inodes, not bytes with df -i and df -h on the owning pod. If bytes are also high, treat it as kube_persistent_volume_claim_disk_space instead.

  3. Find where the inodes are. With GNU coreutils available, one pass lands on the directory holding them, since --separate-dirs counts each directory’s own entries rather than its subtree:

    Terminal window
    du --inodes --separate-dirs --one-file-system <mount> | sort -h | tail -20

    Many images ship busybox, whose du has no --inodes and whose find has no -printf. There, count per top-level directory and descend into whichever dominates:

    Terminal window
    for d in <mount>/*; do printf "%10s %s\n" "$(find "$d" -xdev | wc -l)" "$d"; done

  4. Establish whether it is accumulation or undersized provisioning. Divide used bytes by used inodes to get the average file size. On a volume formatted at the ext4 default of one inode per 16 KiB, inodes run out once byte utilisation reaches roughly average file size / 16 KiB — so a 4 KiB average exhausts inodes at ~25% full, and an 8 KiB average at ~50%. Compute it from the observed average rather than assuming a fraction; it tells you how much byte headroom will still be free when writes start failing, which is what makes ENOSPC confusing.

  5. Correlate the steps to timestamps on disk. ls -ld on the candidate directories maps each step in the graph to the work that created it.

Possible Resolutions

Section titled “Possible Resolutions”

  • Bound the growth. If the volume accumulates scratch data that is safely regenerable, add a garbage collector rather than capacity. This is the durable fix.

  • Clean up accumulated files, once you have confirmed what is safe to remove. Regenerable caches and scratch clones usually are; check for in-flight work first. chmod -R u+rwX the tree first if removal hits Permission denied — a file cannot be unlinked without write permission on its parent directory, and Terraform copies module sources with their original permissions.

  • Grow the PVC — k8s-pvc-resize. Growing an ext4 volume adds block groups and therefore inodes, so this does raise the inode budget. Persist the new size in GitOps or it will revert.

  • Previous incidents:

    • INC-13112 — Atlantis per-MR workspaces accumulating; .terraform/modules copied once per module block
    • INC-8304 — same PVC resized 10Gi to 20Gi without identifying the cause; recurred five months later
    • INC-7429 — Atlantis PVC in ops, disk space rather than inodes

Dependencies

Section titled “Dependencies”
  • No other dependencies can cause this alert.

Escalation

Section titled “Escalation”
  • Slack Channel: #g_runway
  • For Atlantis instances specifically, the workload owner is the team owning the repository that instance serves.

Definitions

Section titled “Definitions”
Section titled “Related Links”