Troubleshooting `data-server-rebuild-ansible` Pipeline Failures
The data-server-rebuild-ansible
pipeline periodically rebuilds the Patroni “data analytics” replica VMs in the
gitlab-db-benchmarking GCP project from fresh production snapshots. Each
shard (main, ci, sec, test) refreshes the data disk on a corresponding
patroni-*-data-analytics-01-db-db-benchmarking VM in us-east1-c and is
then promoted out of standby so analytics queries hit recent-ish data.
These are internal data-analytics replicas, not customer-facing. Failures typically map to severity-4 and do not affect production traffic. There is no alerting tied to this pipeline; incidents are filed manually after a failed pipeline is noticed.
Mental Model
Section titled “Mental Model”The pipeline runs three playbooks in order, against a target VM that is expected to be already running:
| Stage | Playbook | Where it runs | What it does |
|---|---|---|---|
replacedisk | disableservices.yml | on the VM | Wait for SSH (1200s timeout), stop patroni/pgbouncer |
replacedisk | gcpdiskreplace.yml | on localhost | Validate snapshot size, then swap …-data disk from snapshot and restart the VM |
postscript | rebuild.yml | on the VM | Wait for SSH, reset patroni state, start postgres, promote out of standby |
Only gcpdiskreplace.yml ever stops or starts the VM. No playbook is designed
to cold-start a VM that is already TERMINATED when the pipeline begins.
disableservices.yml’s first task is wait_for on port 22 with a 1200s
(20-minute) timeout, so if the VM is off, every retry waits 20 minutes before
the pipeline gives up.
The shard-to-VM and shard-to-source-snapshot mappings live in
vars.yml
under instance_names: and snapshot_source_disks:. Look them up directly
rather than relying on a copy here:
# From a checkout of data-server-rebuild-ansible:yq '.instance_names, .snapshot_source_disks' vars.ymlSymptoms
Section titled “Symptoms”Common error messages in the failing job log, mapped to a cause section below:
Timeout when waiting for search string OpenSSH in <hostname>:22— the 20-minute SSH wait failed (cold-start; see Target VM is TERMINATED).Target GCE VM '…' is in state 'TERMINATED' (expected: RUNNING)— preflight refusing to wait for SSH (same cause; the fast-fail variant after MR !100).data_disk_size ... is smaller than snapshotorCould not read diskSizeGb for snapshot. A size check failed before anything was stopped or deleted (see Disk size checks fail).Task '<name>' failed:followed by the data disk state and next steps. The disk swap failed partway and the rescue ran (see Rescue message after a failed disk swap).Could not translate host name "master.patroni.service.consul" to address— patroni stuck trying to reach a deleted primary (see Patroni stuck on standby’s missing primary DNS).- WAL-replay timeout from
wait for "<host>" startup scripts to finish— see Other failure modes.
Target VM is TERMINATED at pipeline start
Section titled “Target VM is TERMINATED at pipeline start”This is the most common case. The VM was previously stopped (a manual
action, a failed prior pipeline mid-flight, GCP maintenance, or zone
resource exhaustion at the moment of start) and nothing has restarted it
since. The playbook then sits at wait for host to be up for 20 minutes
per attempt.
For a recent example see the production incident gitlab-com/gl-infra/production#21938.
Identify
Section titled “Identify”gcloud --project=gitlab-db-benchmarking compute instances describe \ <instance-name> --zone=us-east1-c \ --format='value(status,lastStartTimestamp,lastStopTimestamp)'If status is TERMINATED (or STOPPING/STAGING), this is the cause.
Resolution
Section titled “Resolution”First check the data disk. A previous replacedisk attempt may have left it
detached or still restoring, and a VM started without its data disk boots into
an emergency shell that GCE still reports as RUNNING:
gcloud --project=gitlab-db-benchmarking compute disks describe \ <instance-name>-data --zone=us-east1-c --format='value(status,users)'If the disk is READY but has no users, attach it before starting the VM:
gcloud --project=gitlab-db-benchmarking compute instances attach-disk \ <instance-name> --disk=<instance-name>-data \ --device-name=persistent-disk-1 --zone=us-east1-cIf it is CREATING or RESTORING, wait for READY. The preflight failure
message prints both commands, and the first attempt’s replacedisk log holds
the rescue message with the full next steps.
Once the disk is READY and attached, start the VM:
gcloud --project=gitlab-db-benchmarking compute instances start \ <instance-name> --zone=us-east1-cThe boot disk and …-log disk persist across stops; only …-data is
recreated by the pipeline. Starting the VM is non-destructive: its old data
disk (if still attached) will be wiped and replaced as part of
gcpdiskreplace.yml.
After the VM reaches RUNNING, what to run depends on how far the previous
attempt got. Read the rescue message in the first attempt’s replacedisk
log. If it says the old disk is gone and the swap is done, start a new
pipeline with DEPLOY=true, SKIP_REPLACEDISK=true and the right
DB_SHARD. Retrying replacedisk in that state would delete the new disk
and redo the swap. If it says the old disk was never deleted, or there is no
rescue message because the pipeline never reached gcpdiskreplace.yml,
retry the failing replacedisk job. Its detach and delete steps are
idempotent (failed_when guards on "is not attached to instance" and
"was not found").
NOTE: Retrying the failing job alone will not start the VM. The
pipeline has no logic to start a TERMINATED VM, so each retry hits the
same 1200s SSH timeout until the VM is up.
Disk size checks fail
Section titled “Disk size checks fail”Since MR !110 the pipeline reads diskSizeGb from the source
snapshot and uses it as the data disk size. The hand-maintained
snapshot_source_disk_size map no longer exists, so a production disk resize
needs no change in vars.yml. This replaced the
Requested disk size cannot be smaller than the snapshot size failure, which
used to happen after the old disk had already been deleted
(INC-14493).
Three checks run before the VM is stopped. Each one fails the play with
nothing destroyed. The message ends with a reminder that
disableservices.yml has already stopped patroni and pgbouncer, so the
replica is offline until the pipeline is re-run.
Could not read diskSizeGb for snapshot .... Thegcloud compute snapshots describecall failed or returned something that is not a positive integer. Check that the snapshot still exists and that the CI service account can read it.data_disk_size '...' must be a whole number followed by GBanddata_disk_size ... is smaller than snapshot. Only reachable when someone passed-e data_disk_size=...on the command line, which outranks the derived value. Drop the override or fix it.
Resolution
Section titled “Resolution”Fix the cause above and re-run the replacedisk job. No disk was changed,
so the full pipeline is safe to retry.
Rescue message after a failed disk swap
Section titled “Rescue message after a failed disk swap”The stop, detach, delete, create, wait, attach and start steps run inside a
block. If any of them fails, the rescue checks the data disk’s state, attaches
it and starts the VM only when the disk is READY, and then either continues
the play or fails it with a message that names the failed task, the disk state,
the exact gcloud commands to run, and one of three closing instructions:
- The old disk is gone. Finish the recovery by hand, wait for the serial
console to show
Bootstrap finished in, then run a new pipeline withDEPLOY=true,SKIP_REPLACEDISK=trueand the rightDB_SHARD. That runs onlyansible-deploy. Do not retryreplacedisk, it would redo the swap. - The old disk was never deleted. Fix the cause and retry
replacedisk. Do not runansible-deployalone, the data has not been refreshed. - Unknown. The delete reported an error and the disk state could not be
read. Check the disk with the
disks describecommand in the message. If it isREADYand attached, treat it as never deleted. If it is gone or notREADY, treat the swap as done.
Two things to know when reading the job:
replacediskhasretry: 2, so GitLab re-runs it twice on its own. If the rescue could not recover, read the first attempt’s log. When the VM is notRUNNING, the retries stop at the preflight, whose message does not repeat the rescue steps.- If the rescue did recover, the play continues and the pipeline goes green
with
rescued=1in the recap. Nothing to do.
The repository README
has the current wording and the SKIP_REPLACEDISK usage.
Patroni stuck on standby’s missing primary DNS
Section titled “Patroni stuck on standby’s missing primary DNS”A snapshot taken from a standby cluster may reference a primary
(master.patroni.service.consul) that does not exist in db-benchmarking
DNS. Postgres fails to start and patroni restarts it in a tight loop until
monitoring until ready times out.
rebuild.yml already handles this by promoting out of standby
(gitlab-patronictl edit-config -s standby_cluster=null --force). If that
promotion task itself fails, run it manually on the target VM:
ssh data-analytics-refresh@<instance>.c.gitlab-db-benchmarking.internal \ "sudo gitlab-patronictl edit-config -s standby_cluster=null --force"Then re-run the ansible-deploy job to continue from wait for "<host>" state to be running.
Other failure modes
Section titled “Other failure modes”Less common, but historically observed:
-
ZONE_RESOURCE_POOL_EXHAUSTEDonstart: wait for capacity to free, or retry. To list recent occurrences in the audit log:Terminal window gcloud logging read \'protoPayload.status.message="ZONE_RESOURCE_POOL_EXHAUSTED" AND protoPayload.methodName="v1.compute.instances.start"' \--project=gitlab-db-benchmarking --freshness=30d --limit=20 \--format='value(timestamp,protoPayload.resourceName)' -
WAL-replay timeout during
monitoring until ready: Postgres is replaying a large amount of WAL after restoring from snapshot. Bumppatroni_ready_timeout[<shard>]invars.ymlif persistent, or wait it out and retry. -
Snapshot not READY: the
wait for snapshot to be readytask retries 30× at 60s. If a snapshot is still being taken in production this can happen; usually self-resolves on retry.
Useful commands
Section titled “Useful commands”List all rebuild VMs and their current state:
gcloud compute instances list --project=gitlab-db-benchmarking \ --filter="name~analytics-01-db-db-benchmarking" \ --format="table(name,zone,status,lastStartTimestamp)"