Skip to content
Runbooks

PatroniArchiveReplicaNoData

Overview

Section titled “Overview”

No node of a DR archive replica type has reported pg_replication_lag for 30 minutes. Until one does, there is no archive copy of that cluster.

The archive replicas are standby clusters that replay WAL from the GCS archive with no delay (types postgres-archive, patroni-ci-archive, patroni-sec-archive, patroni-registry-archive). They exist only in gprd and serve no traffic.

The series disappears when:

  • Postgres is down, for example because the data volume filled during replay,
  • the VM is stopped, or
  • postgres_exporter is down.

For example, the sec archive (postgres-sec-dr-archive-v18-01-db-gprd) stopped on 2026-11-04 at 04:25 UTC with could not extend file ... No space left on device. A stale 6.1T data18.bak.* directory left over from a July re-init had filled the 16T data volume. Nothing alerted, and it was found five days later (production#23452).

First steps

Section titled “First steps”

  1. Find the cluster’s archive node and whether it’s running:

    Terminal window
    gcloud compute instances list --project gitlab-production --filter="labels.type=<type>"

  2. On the node, check Patroni, Postgres and the exporter:

    Terminal window
    sudo gitlab-patronictl list
    sudo sv status postgres_exporter
    sudo gitlab-psql -Atc "select pg_is_in_recovery(), now() - pg_last_xact_replay_timestamp()"

  3. Check free space. A full data volume stops Postgres, and a full /var/log usually follows:

    Terminal window
    df -h /var/opt/gitlab /var/log
    ls -ld /var/opt/gitlab/postgresql/data*

    Anything next to the live data directory, such as data18.bak.*, is a leftover from a re-init.

  4. If Postgres won’t start, read the end of the Postgres log:

    Terminal window
    sudo grep -a -E 'FATAL|PANIC|No space left|restore_command|forked off' /var/log/gitlab/postgresql/postgresql.csv | tail -20

  5. If the node is being replaced or rebuilt on purpose, silence the alert for that type until the replacement reports, and link the change issue.

Recovery

Section titled “Recovery”

Recovery after a long outage is a change, not an on-call fix: the replica has to replay every WAL segment since it stopped, and the primary keeps generating more. Open a change issue and follow production#23452 as a template. It covers freeing space, restarting replay in place, and measuring whether replay can catch up. If it can’t, rebuild from a recent disk snapshot of the cluster’s backup replica (Restoring with a disk-snapshot), so only WAL written since the snapshot has to be replayed.

Alert behaviour

Section titled “Alert behaviour”
  • One alert per type. Each archive cluster has one node.
  • for: 30m: a restart or short VM maintenance rides through it. Over the 30 days before 2026-11-04, the longest gap on any archive type was 8 minutes.
  • s3: Slack only, no page.

Escalation

Section titled “Escalation”

If a cluster will be without its archive copy for more than a working day, raise it in #g_database_operations.

Section titled “Related links”