PatroniKernelHungTask

Overview

The kernel’s hung-task detector has reported tasks on a Postgres node that have been in uninterruptible sleep (state D) for longer than kernel.hung_task_timeout_secs (check it with sysctl kernel.hung_task_timeout_secs; it was 30 seconds on gstg in October 2026). That almost always means they’re waiting on IO that isn’t completing: a hung disk, or a filesystem journal (jbd2) stuck behind one.

The alert reads the node’s serial console, which Compute Engine captures outside the guest, so it still fires when the disk holding journald and the exporters has stopped. This is how the INC-14518 boot disk hang showed itself (incident review), and at the time only the serial console kept the evidence.

It watches every node that reports pg_replication_is_replica, whatever its type: the Patroni clusters, and also archive, delayed and PITR replicas and upgrade clusters. A node that isn’t in recovery reports as primary, so a promoted PITR node shows up as one.

The role label is the node’s last role as reported by its own postgres_exporter. Backup nodes (shard="backup") are included, unlike most Patroni alerts: a hung backup node stalls backups.

First steps

Save the serial console now, before anyone resets the VM. The buffer rolls over, and a reset loses what the guest held in memory:
```
gcloud compute instances get-serial-port-output <instance> --project=<project> --zone=<zone> > <instance>-serial.log
grep -E -A3 'blocked for more than' <instance>-serial.log
```
<project> is gitlab-production for env="gprd" and gitlab-staging-1 for env="gstg". <instance> is the instance_name on the alert. The zone is on the instance in the GCP console, or from gcloud compute instances list --filter="name=<instance>". Attach the file to the incident or issue.

The task names say what is stuck: jbd2/<device> is the ext4 journal of that device, and postgres, consul or node_exporter show who’s waiting on it.
If role is primary, check for PatroniPrimaryDiskStalled and PatroniPrimaryDiskDegraded on the same node, and whether it still responds. If it doesn’t, or there’s no Leader, follow Primary unresponsive or no Leader: recover it without losing data. Do not fail over before it has shown that the old primary isn’t writing.
If you can still SSH in, capture the stacks of every blocked task. Write to /dev/shm, which is memory-backed, so the capture doesn’t wait on the hung disk itself:
```
ps -eo pid,stat,wchan:32,comm | awk 'NR == 1 || $2 ~ /^D/' | sudo tee /dev/shm/blocked.txt
for pid in $(ps -eo pid=,stat= | awk '$2 ~ /^D/ {print $1}'); do
  echo "== $pid $(cat /proc/$pid/comm)"; sudo cat /proc/$pid/stack
done | sudo tee /dev/shm/blocked-stacks.txt
echo w | sudo tee /proc/sysrq-trigger >/dev/null && sudo dmesg -T | tail -n 500 | sudo tee /dev/shm/sysrq-w.txt >/dev/null
```
echo w > /proc/sysrq-trigger works as root whatever kernel.sysrq is set to. Its output goes to the kernel log at KERN_INFO, which dmesg shows but the serial console doesn’t (see Alert behaviour). Copy the files off with scp, or paste them into the incident. If a command hangs, the binary itself is on the stuck disk: leave it and rely on the serial console.
If role is replica and the node is otherwise healthy, it’s not urgent. Check its replication lag, and raise an issue with the stacks so the DBO team can decide whether to rebuild the node.

Why this signal

On a Patroni node, journald, Consul, Patroni and every exporter run from the boot disk. When that disk hangs, guest-side logs and metrics stop together. The kernel keeps running from memory and writes its hung-task reports to the serial console, which Compute Engine reads from the hypervisor.

The series is a Cloud Logging log-based metric, kernel_hung_task, which counts serial console lines like:

INFO: task jbd2/nvme0n1p1-8:412 blocked for more than 30 seconds.

# Hung-task reports for the alerting node (replace <instance>)
stackdriver_gce_instance_logging_googleapis_com_user_kernel_hung_task{env="gprd", instance_name="<instance>"}

The _Default log sink in gitlab-production and gitlab-staging-1 excludes every gce_instance log, serial console included, so these lines aren’t in Logs Explorer. Log-based metrics are calculated from both included and excluded logs, so the alert sees them anyway.

Alert behaviour

Non-paging (s3) until there’s a baseline. The serial console has never been stored, so we don’t yet know how often healthy Patroni nodes report hung tasks. The paging alerts for a hung primary disk are PatroniPrimaryDiskStalled and PatroniPrimaryDiskDegraded.
Fires on the first report, and resolves 10 minutes after the last one.
Can resolve while the node is still hung. With kernel.hung_task_warnings = 10, the Ubuntu default, the kernel stops reporting after ten tasks per boot and logs Future hung task reports are suppressed. A resolve doesn’t mean the IO recovered.
The serial console carries the header line, not the stack. Ubuntu sets kernel.printk = 4 4 1 7 (/etc/sysctl.d/10-console-messages.conf), so only messages at KERN_ERR and above reach the console. The header is KERN_ERR, but the kernel prints the blocked task’s stack at KERN_INFO. Use step 3 for stacks.
Can’t fire unless the kernel_hung_task log-based metric exists in the project. The stackdriver-exporter already collects every logging.googleapis.com metric (argocd/apps services/stackdriver-exporter).

Escalation

If the node is the primary and isn’t responding, this is an outage for the cluster: declare an incident, and escalate to the DBO (Database Operations) on-call with /incident escalate in the incident channel (manual escalation). Otherwise, raise it in #g_database_operations.