Secrets Manager GKE Database (Postgres) Service
- Service Overview
- Alerts: https://alerts.gitlab.net/#/alerts?filter=%7Btype%3D%22runway-db-secrets-manager-gke%22%2C%20tier%3D%22db%22%7D
- Label: gitlab-com/gl-infra/production~“Service::RunwayDBSecretsManager”
Logging
Section titled “Logging”Summary
Section titled “Summary”runway-db-secrets-manager-gke is the Runway-managed Cloud SQL for PostgreSQL instance behind the secrets-manager-gke OpenBao service.
For the service itself, see the secrets-manager-gke runbook.
OpenBao owns two tables in the openbao database, openbao_kv_store and openbao_ha_locks.
The leadership lock lives in openbao_ha_locks, so this database is also OpenBao’s leader election.
Instance and backup settings for both environments live in config/runtimes/gke/cloud-sql/managed.yml.
Durability
Section titled “Durability”Automated backups run daily at 11:00 UTC and the seven most recent are kept — a count, not a seven-day window.
Point-in-time recovery is on, with seven days of transaction logs in Cloud Storage.
On-demand backups are kept until someone deletes them and never consume one of the seven slots.
Restore validation
Section titled “Restore validation”Runway restores the newest backup into runway-db-secrets-manager-gke-restore-instance daily and runs SELECT COUNT(*), NOW() FROM openbao_kv_store HAVING COUNT(*) > 0; against it.
That proves the backup restores and the table came back holding rows.
It does not prove that any row decrypts, because OpenBao takes no part in the check.
Both alerts are s4 and route to Runway, so neither one pages us, and no restore-validation alert covers staging.
See Cloud SQL Restore Pipeline Troubleshooting.
Pipeline history is in runway-db-secrets-manager-gke-production-us-east1, under the private cloudsql_backups_validation group on ops.gitlab.net.
Ask in #f_runway if it returns 404.
Restoring from a backup
Section titled “Restoring from a backup”Follow Restore/Backup Runway-managed Cloud SQL. The restore runs from a Runway provisioner MR that an SRE approves, so line that approval up before you scale OpenBao down — otherwise the service sits at zero pods for the length of a review.
- Scale OpenBao down to zero pods.
- Restore the PostgreSQL instance.
- Scale OpenBao back up to two pods.
Ask in #f_runway to run scale down and scale up command as write access to the cluster is required.
Afterward confirm the pods unsealed, per Healthy startup baseline.
Consistency with the Rails main database
Section titled “Consistency with the Rails main database”OpenBao derives its namespace paths from Rails main database IDs, and neither side stores a cross-reference to the other.
Restore both databases to the same point in time.
Otherwise Rails keeps offering namespaces that are no longer there, CI jobs receive paths that resolve to nothing, and the rotation reminder job deletes its own rows once the identifier it reads back from OpenBao stops matching.
No task reconciles the two.
Ask in #sec-secrets-manager-openbao if that is not possible.
Security/Compliance
Section titled “Security/Compliance”A direct SELECT gives you row counts and sizes and nothing else, because namespaces and mounts appear only as UUIDs and KVv2 key names are encrypted or hashed.
See Security/Compliance for the barrier and KMS setup.
Access needed for a restore
Section titled “Access needed for a restore”The engineer performing the restore must have Developer access to the Runway provisioner project to open the required MR.