Restore/Backup Runway-managed Cloud SQL

This document is meant for (1) disaster-recovery purposes or (2) change management issues with high risk of data corruption/loss (e.g. snapshot migration from an external Cloud SQL instance).

Backup

All Runway-managed Cloud SQL instances will have periodic backups. These backups are tested daily through restore and data validation jobs.

Service owners may want to manually create a backup before performing a critical data migration. They may do so using a change management issue and enlisting the help from SREs.

Creating manual backup

gcloud sql backups create --async --project PROJECT_NAME --instance=INSTANCE_NAME --description="<insert description here>"

Listing existing backups

gcloud sql backups list --project PROJECT_NAME --instance INSTANCE_NAME

Listing backups during outages

During an outage, you can only view backups for that instance using a wildcard (-) and grep for the instance.

gcloud sql backups list --project PROJECT_NAME --instance - | grep INSTANCE_NAME

Restore

Restoring operations should be performed through a change management issue and approved by the engineer-on-call.

There are two ways to restore. Start from the clone path. The in-place restore is a last resort because it destroys data.

	Clone to a new instance	In-place restore from backup
Command	`cloudsql-dr` pipeline (runs `gcloud sql instances clone`)	`gcloud sql backups restore`
Granularity	Any timestamp within the PITR window	Only the moment a backup was taken
Effect on production	None — production keeps running	Overwrites production data
Reversible	Yes	No

Pick the in-place restore only when no current data is worth keeping. If any writes since the recovery point still matter, use the clone path — an in-place restore discards them.

Default path: clone to a new instance

This creates a new instance from a point in time. Production is untouched, so it is safe to run while you are still working out what went wrong.

Confirm the most recent point you can recover to. The oldest is bounded by the instance’s transaction_log_retention_days and retained_backups_count:

gcloud sql instances get-latest-recovery-time SOURCE_INSTANCE --project=PROJECT_NAME

1. Create the clone

The clone is created by the cloudsql-dr pipeline on ops.gitlab.net. It does not run on gitlab.com.

Open a new pipeline: https://ops.gitlab.net/gitlab-com/gl-infra/platform/runway/provisioner/-/pipelines/new, on the default branch.

Set these five variables:

Variable	Value
`RUN_CLOUDSQL_DR`	`true`. Without it the jobs are not created.
`ENVIRONMENT`	`staging` or `production`
`SOURCE_INSTANCE`	the instance to recover, e.g. `runway-db-secrets-manager-gke`
`CLONE_INSTANCE`	a new name, e.g. `runway-db-smgke-dr`. It must start with `runway-db-`, be at most 33 characters, and must not end in `-restore-instance`. The job checks this before it starts.
`POINT_IN_TIME`	the time to recover to, in UTC, in this format: `YYYY-MM-DDTHH:MM:SS.sssZ`. The `Z` at the end means UTC. For example `2026-08-20T14:00:00.000Z` is 14:00 UTC on 20 August 2026. It must fall inside the recovery window confirmed above.

Run the pipeline, then click play on cloudsql-dr:authenticate. This job is manual.
cloudsql-dr:clone starts on its own once authenticate finishes. Wait for it to pass.

In testing, the clone took 15m55s for a 19 MB database. Larger databases take longer. Example run: https://ops.gitlab.net/gitlab-com/gl-infra/platform/runway/provisioner/-/pipelines/6204057. The job and script are .gitlab/ci/cloudsql-dr.yml and scripts/cloudsql-dr-clone.sh.

The pipeline always needs a POINT_IN_TIME. To clone the source’s current consistent state, use the latest recovery time from the check above. If the data is already damaged, you get the damage back.

The clone is a new instance. It has a new instance connection name and its own PSC service attachment, but no consumer endpoint, DNS record or IAM bindings. Those are keyed to the instance name in the Runway provisioner. The clone exists, but nothing can reach it yet.

2. Import the clone into Runway

Do this for both paths below. Importing brings the clone under Terraform, so from then on Terraform manages it like any other instance. It is also what makes the clone reachable.

Open an MR in the provisioner with two changes:

Add an entry for the clone to config/runtimes/gke/cloud-sql/managed.yml. Copy the source’s values so the plan stays clean.

If the clone is temporary (Extract & retire), set prevent_db_deletion: false so it can be removed later in one MR. If the clone will become production (Cutover & keep), leave that line out so deletion protection stays on.

- name: CLONE_INSTANCE
  provider: GCP
  db_version: "POSTGRES_18"
  region: us-east1
  zone: us-east1-b
  env_configuration:
    ENVIRONMENT:
      instance_type: <same as source>
      disk_size_gb: <same as source>
      prevent_db_deletion: false
      backup_configuration:
        transaction_log_retention_days: 7
        retained_backups_count: 7
  labels:
    <same as source>

Add a one-off import block to import.tf:

import {
  to = module.gcp_gke.module.cloudsql_instances["CLONE_INSTANCE-ENVIRONMENT-us-east1"].google_sql_database_instance.default
  id = "projects/PROJECT_NAME/instances/CLONE_INSTANCE"
}

Check the Atlantis plan before you apply. It must show 1 to import and only in-place changes for the clone. If the plan wants to destroy or recreate the instance, stop. That would wipe the recovered data.

Apply takes about 5 minutes. It builds the PSC endpoint, the DNS record and the IAM for the clone. Example: provisioner!1864.

What happens next depends on why you cloned:

Extract & retire: pull the lost data out of the clone, put it back into production, then tear the clone down.
Cutover & keep: promote the clone to be the production instance. The workload is repointed to it, and the old instance is kept for rollback.

Extract & retire

Use this when only part of the data is lost, for example after an accidental DELETE, UPDATE or DROP TABLE, and the rest of production is fine. Production is never repointed and keeps serving.

3. Connect to the clone. Use Cloud SQL Studio as the postgres user. DBRE can get the password from 1Password. Check that you are on the clone and that it is a copy of the source:

SELECT current_database(),
       (SELECT system_identifier FROM pg_control_system()) AS system_identifier,
       pg_postmaster_start_time() AS started;

system_identifier should match the source.

4. Copy the lost data back. Agree with the service owner exactly which rows are missing, then copy them back into production.

5. Retire the clone. Open one MR that removes the clone’s entry from managed.yml and its block from import.tf. The plan should destroy the clone, its PSC endpoint and its DNS record, and nothing else. Example teardown: provisioner!1862.

Cutover & keep

Use this when the whole database is bad and nothing written since the recovery point is worth keeping, for example after a bad migration, a DROP DATABASE or corruption.

IMPORTANT: this method will cause data loss of all transactions that occurred in the database after POINT_IN_TIME

3. Add monitoring for the new instance. The Grafana dashboard and SLIs are keyed on the instance name, so the clone has none yet, and Terraform cannot create them. Open a runbooks MR that copies the source’s files and replaces the source instance name with the clone name, for example runway-db-secrets-manager-gke → runway-db-smgke-dr:

metrics-catalog/services/CLONE_INSTANCE.jsonnet: copy the source’s file. Change one line: local type = 'SOURCE_INSTANCE'; becomes local type = 'CLONE_INSTANCE';.
metrics-catalog/services/all.jsonnet: don’t copy anything. Add one new line directly below the source’s import line:
```
import 'SOURCE_INSTANCE.jsonnet',
import 'CLONE_INSTANCE.jsonnet',
```
dashboards/CLONE_INSTANCE/main.dashboard.jsonnet: copy the source’s file. Change serviceDashboard.overview('SOURCE_INSTANCE') to serviceDashboard.overview('CLONE_INSTANCE').
services/service-catalog.yml: copy the source’s whole - name: SOURCE_INSTANCE block and paste it below. In the copy, change five things: name, friendly_name, label (which must be unique in the file), the instance name inside the logging permalink, and primary_grafana_dashboard to CLONE_INSTANCE-main/CLONE_INSTANCE-overview.
docs/CLONE_INSTANCE/README.md: copy the source’s file. Replace the source instance name with the clone name everywhere, and change the heading and label to match the new friendly_name and label.

Then generate the alerting and SLI rules. make generate builds them from the metrics-catalog file you just added. Without them CI fails and the dashboard has no data. From the root of the runbooks repo:

mise trust
mise install
mise exec -- make generate
git status --short

Expect one new directory, mimir-rules/runway/CLONE_INSTANCE/, with 13 files. Commit only that directory with the rest of the MR. On macOS, make generate needs nproc, which comes from brew install coreutils. Example: runbooks!11637.

Merge this MR before the cutover. Dashboards only reach Grafana from master, so the dashboard does not exist until the MR merges. After it merges, the dashboard is at https://dashboards.gitlab.net/d/CLONE_INSTANCE-main/CLONE_INSTANCE3a-overview. Select the right environment in the dropdown.

4. Give the workload access to the clone. Add the clone to the workload’s postgres_instances in config/runtimes/gke/workloads.yml. This grants the workload’s service account roles/cloudsql.client and roles/cloudsql.instanceUser on the clone. The grant only applies in environments where the clone has an env_configuration entry. Example: provisioner!1912.

Merge this before the repoint, or the workload cannot connect.

5. Stop writes. Agree a write freeze with the service owner. For secrets-manager-gke, this means pausing the Sidekiq workers that provision and deprovision secrets managers.

6. Repoint the workload. The service owner changes the instance connection name in the service’s Runway config to PROJECT_NAME:us-east1:CLONE_INSTANCE, then deploys. Example: secrets-manager-runway!30.

7. Check that the workload moved. In Grafana Explore, with the mimir-runway datasource:

sum by (type) (stackdriver_cloudsql_database_cloudsql_googleapis_com_database_postgresql_num_backends{type=~"CLONE_INSTANCE|SOURCE_INSTANCE", database="DATABASE_NAME", env="ENV"})

Connections should drop to zero on the source and appear on the clone. Then ask the service owner to confirm the service is healthy.

8. Clean up later. Keep the old instance until you are sure you will not roll back. Then remove it and its monitoring files in a follow-up MR. Also remove the clone’s block from import.tf. The instance itself stays in managed.yml.

Last resort: in-place restore from backup

This is destructive and cannot be undone. It overwrites all data on the target instance, and every write between the backup and the restore is permanently lost. It also restarts the instance and drops existing connections. Only use it when no current data is worth keeping.

Find the backup to restore from:

gcloud sql backups list --project PROJECT_NAME --instance INSTANCE_NAME

Before you restore, save the current backup configuration. You will compare against it afterwards:

gcloud sql instances describe INSTANCE_NAME --project PROJECT_NAME \
  --format='yaml(settings.backupConfiguration)' > before-restore-backupconfig.yaml

IMPORTANT: Point of No Return

Proceeding will completely overwrite the existing instance.
All current backups and restore points will be permanently lost.

Restore it. gcloud asks you to confirm before it starts:

gcloud sql backups restore BACKUP_ID --project PROJECT_NAME --restore-instance=INSTANCE_NAME

What to expect. In testing, the restore took about 8.5 minutes. During that time the database restarted three times over about 3.5 minutes, and dropped every connection each time. Treat the application as down for that whole window.

After the restore, check three things:

The backup configuration is unchanged. Compare it with the file you saved:

gcloud sql instances describe INSTANCE_NAME --project PROJECT_NAME \
  --format='yaml(settings.backupConfiguration)' > after-restore-backupconfig.yaml
diff before-restore-backupconfig.yaml after-restore-backupconfig.yaml

You should see no output, and pointInTimeRecoveryEnabled should still be true. In both tests the restore did not reset the configuration.

Point-in-time recovery has restarted.
```
gcloud sql instances get-latest-recovery-time INSTANCE_NAME --project=PROJECT_NAME
```
Look at earliestRecoveryTime. It should be just before the time the restore finished. In testing, the restore finished at 11:49:15 UTC and the output was:
```
earliestRecoveryTime: 2026-09-09T11:48:49Z
latestRecoveryTime: 2026-09-09T12:03:26Z
```
The recovery window now starts at the moment of the restore. You cannot clone to any point before it. This is why an in-place restore cannot be undone.
Terraform agrees with the instance. Run a plan in the provisioner. It should show no backup_configuration changes.

If any of these checks fail, engage @dbre at #g_database_operations and #g_runway before closing the change management issue.