Skip to content

Troubleshooting `data-server-rebuild-ansible` Pipeline Failures

The data-server-rebuild-ansible pipeline periodically rebuilds the Patroni “data analytics” replica VMs in the gitlab-db-benchmarking GCP project from fresh production snapshots. Each shard (main, ci, sec, test) refreshes the data disk on a corresponding patroni-*-data-analytics-01-db-db-benchmarking VM in us-east1-c and is then promoted out of standby so analytics queries hit recent-ish data.

These are internal data-analytics replicas, not customer-facing. Failures typically map to severity-4 and do not affect production traffic. There is no alerting tied to this pipeline; incidents are filed manually after a failed pipeline is noticed.

The pipeline runs three playbooks in order, against a target VM that is expected to be already running:

StagePlaybookWhere it runsWhat it does
replacediskdisableservices.ymlon the VMWait for SSH (1200s timeout), stop patroni/pgbouncer
replacediskgcpdiskreplace.ymlon localhostValidate snapshot size, then swap …-data disk from snapshot and restart the VM
postscriptrebuild.ymlon the VMWait for SSH, reset patroni state, start postgres, promote out of standby

Only gcpdiskreplace.yml ever stops or starts the VM. No playbook is designed to cold-start a VM that is already TERMINATED when the pipeline begins. disableservices.yml’s first task is wait_for on port 22 with a 1200s (20-minute) timeout, so if the VM is off, every retry waits 20 minutes before the pipeline gives up.

The shard-to-VM and shard-to-source-snapshot mappings live in vars.yml under instance_names: and snapshot_source_disks:. Look them up directly rather than relying on a copy here:

Terminal window
# From a checkout of data-server-rebuild-ansible:
yq '.instance_names, .snapshot_source_disks' vars.yml

Common error messages in the failing job log, mapped to a cause section below:

  • Timeout when waiting for search string OpenSSH in <hostname>:22 — the 20-minute SSH wait failed (cold-start; see Target VM is TERMINATED).
  • Target GCE VM '…' is in state 'TERMINATED' (expected: RUNNING) — preflight refusing to wait for SSH (same cause; the fast-fail variant after MR !100).
  • data_disk_size ... is smaller than snapshot or Could not read diskSizeGb for snapshot. A size check failed before anything was stopped or deleted (see Disk size checks fail).
  • Task '<name>' failed: followed by the data disk state and next steps. The disk swap failed partway and the rescue ran (see Rescue message after a failed disk swap).
  • Could not translate host name "master.patroni.service.consul" to address — patroni stuck trying to reach a deleted primary (see Patroni stuck on standby’s missing primary DNS).
  • WAL-replay timeout from wait for "<host>" startup scripts to finish — see Other failure modes.

This is the most common case. The VM was previously stopped (a manual action, a failed prior pipeline mid-flight, GCP maintenance, or zone resource exhaustion at the moment of start) and nothing has restarted it since. The playbook then sits at wait for host to be up for 20 minutes per attempt.

For a recent example see the production incident gitlab-com/gl-infra/production#21938.

Terminal window
gcloud --project=gitlab-db-benchmarking compute instances describe \
<instance-name> --zone=us-east1-c \
--format='value(status,lastStartTimestamp,lastStopTimestamp)'

If status is TERMINATED (or STOPPING/STAGING), this is the cause.

First check the data disk. A previous replacedisk attempt may have left it detached or still restoring, and a VM started without its data disk boots into an emergency shell that GCE still reports as RUNNING:

Terminal window
gcloud --project=gitlab-db-benchmarking compute disks describe \
<instance-name>-data --zone=us-east1-c --format='value(status,users)'

If the disk is READY but has no users, attach it before starting the VM:

Terminal window
gcloud --project=gitlab-db-benchmarking compute instances attach-disk \
<instance-name> --disk=<instance-name>-data \
--device-name=persistent-disk-1 --zone=us-east1-c

If it is CREATING or RESTORING, wait for READY. The preflight failure message prints both commands, and the first attempt’s replacedisk log holds the rescue message with the full next steps.

Once the disk is READY and attached, start the VM:

Terminal window
gcloud --project=gitlab-db-benchmarking compute instances start \
<instance-name> --zone=us-east1-c

The boot disk and …-log disk persist across stops; only …-data is recreated by the pipeline. Starting the VM is non-destructive: its old data disk (if still attached) will be wiped and replaced as part of gcpdiskreplace.yml.

After the VM reaches RUNNING, what to run depends on how far the previous attempt got. Read the rescue message in the first attempt’s replacedisk log. If it says the old disk is gone and the swap is done, start a new pipeline with DEPLOY=true, SKIP_REPLACEDISK=true and the right DB_SHARD. Retrying replacedisk in that state would delete the new disk and redo the swap. If it says the old disk was never deleted, or there is no rescue message because the pipeline never reached gcpdiskreplace.yml, retry the failing replacedisk job. Its detach and delete steps are idempotent (failed_when guards on "is not attached to instance" and "was not found").

NOTE: Retrying the failing job alone will not start the VM. The pipeline has no logic to start a TERMINATED VM, so each retry hits the same 1200s SSH timeout until the VM is up.

Since MR !110 the pipeline reads diskSizeGb from the source snapshot and uses it as the data disk size. The hand-maintained snapshot_source_disk_size map no longer exists, so a production disk resize needs no change in vars.yml. This replaced the Requested disk size cannot be smaller than the snapshot size failure, which used to happen after the old disk had already been deleted (INC-14493).

Three checks run before the VM is stopped. Each one fails the play with nothing destroyed. The message ends with a reminder that disableservices.yml has already stopped patroni and pgbouncer, so the replica is offline until the pipeline is re-run.

  • Could not read diskSizeGb for snapshot .... The gcloud compute snapshots describe call failed or returned something that is not a positive integer. Check that the snapshot still exists and that the CI service account can read it.
  • data_disk_size '...' must be a whole number followed by GB and data_disk_size ... is smaller than snapshot. Only reachable when someone passed -e data_disk_size=... on the command line, which outranks the derived value. Drop the override or fix it.

Fix the cause above and re-run the replacedisk job. No disk was changed, so the full pipeline is safe to retry.

The stop, detach, delete, create, wait, attach and start steps run inside a block. If any of them fails, the rescue checks the data disk’s state, attaches it and starts the VM only when the disk is READY, and then either continues the play or fails it with a message that names the failed task, the disk state, the exact gcloud commands to run, and one of three closing instructions:

  • The old disk is gone. Finish the recovery by hand, wait for the serial console to show Bootstrap finished in, then run a new pipeline with DEPLOY=true, SKIP_REPLACEDISK=true and the right DB_SHARD. That runs only ansible-deploy. Do not retry replacedisk, it would redo the swap.
  • The old disk was never deleted. Fix the cause and retry replacedisk. Do not run ansible-deploy alone, the data has not been refreshed.
  • Unknown. The delete reported an error and the disk state could not be read. Check the disk with the disks describe command in the message. If it is READY and attached, treat it as never deleted. If it is gone or not READY, treat the swap as done.

Two things to know when reading the job:

  • replacedisk has retry: 2, so GitLab re-runs it twice on its own. If the rescue could not recover, read the first attempt’s log. When the VM is not RUNNING, the retries stop at the preflight, whose message does not repeat the rescue steps.
  • If the rescue did recover, the play continues and the pipeline goes green with rescued=1 in the recap. Nothing to do.

The repository README has the current wording and the SKIP_REPLACEDISK usage.

Patroni stuck on standby’s missing primary DNS

Section titled “Patroni stuck on standby’s missing primary DNS”

A snapshot taken from a standby cluster may reference a primary (master.patroni.service.consul) that does not exist in db-benchmarking DNS. Postgres fails to start and patroni restarts it in a tight loop until monitoring until ready times out.

rebuild.yml already handles this by promoting out of standby (gitlab-patronictl edit-config -s standby_cluster=null --force). If that promotion task itself fails, run it manually on the target VM:

Terminal window
ssh data-analytics-refresh@<instance>.c.gitlab-db-benchmarking.internal \
"sudo gitlab-patronictl edit-config -s standby_cluster=null --force"

Then re-run the ansible-deploy job to continue from wait for "<host>" state to be running.

Less common, but historically observed:

  • ZONE_RESOURCE_POOL_EXHAUSTED on start: wait for capacity to free, or retry. To list recent occurrences in the audit log:

    Terminal window
    gcloud logging read \
    'protoPayload.status.message="ZONE_RESOURCE_POOL_EXHAUSTED" AND protoPayload.methodName="v1.compute.instances.start"' \
    --project=gitlab-db-benchmarking --freshness=30d --limit=20 \
    --format='value(timestamp,protoPayload.resourceName)'
  • WAL-replay timeout during monitoring until ready: Postgres is replaying a large amount of WAL after restoring from snapshot. Bump patroni_ready_timeout[<shard>] in vars.yml if persistent, or wait it out and retry.

  • Snapshot not READY: the wait for snapshot to be ready task retries 30× at 60s. If a snapshot is still being taken in production this can happen; usually self-resolves on retry.

List all rebuild VMs and their current state:

Terminal window
gcloud compute instances list --project=gitlab-db-benchmarking \
--filter="name~analytics-01-db-db-benchmarking" \
--format="table(name,zone,status,lastStartTimestamp)"