CustomersDot service recovery (Puma, NGINX, redeploy)
This runbook provides end-to-end, step-by-step recovery guidance for customers.gitlab.com (CustomersDot) during an incident: restarting Puma and NGINX, recovering from a failed Puma restart, and manually redeploying the latest commit (including how and when to bypass the 2-hour staging wait).
Before you start: access
Section titled “Before you start: access”All of the procedures below require SSH access to the CustomersDot VMs.
- Primary access is via Teleport — see SSH access in the overview runbook.
- If Teleport denies access, follow Teleport access denied: fallback below before attempting any restart or redeploy.
The VMs are named customers-NN-inf-prdsub (production) and
customers-NN-inf-stgsub (staging). List them with:
# Productiongcloud --project gitlab-subscriptions-prod compute instances list --filter="name~customers-.*-inf-prdsub"
# Staginggcloud --project gitlab-subscriptions-staging compute instances list --filter="name~customers-.*-inf-stgsub"Log in to a VM via Teleport:
tsh login --proxy=production.teleport.gitlab.nettsh ssh customers-01-inf-prdsubTeleport access denied: fallback
Section titled “Teleport access denied: fallback”Responders during a recent incident were blocked by Teleport access denials with no documented fallback. Work through these steps in order.
1. Confirm you have the required Teleport role
Section titled “1. Confirm you have the required Teleport role”SSH to the CustomersDot VMs requires the Teleport role that grants the
customersdot / production node login. SREs are expected to have this through
their standard role set.
For more details on what roles to have to access the CustomersDot VMs, check out the Teleport documentation in the CustomersDot project.
Check which roles and logins your current Teleport session has:
tsh statustsh login --proxy=production.teleport.gitlab.net # refresh if your cert is stale/expiredIf you are missing the required role, request it just-in-time:
# Request access to the node you need, then share the request ID with an approver.tsh request create --resource /production.teleport.gitlab.net/node/<node-uuid># ortsh request create --roles <role-name>Find out about the <node-uuid> to use for a given CustomersDot Production VM with the
following command:
tsh ls -v | grep 'customers.*prdsub'Follow the Teleport approval workflow to get the request approved.
2. Check the unknown user case
Section titled “2. Check the unknown user case”If Teleport authenticates but the VM rejects you with
Failed to launch: user: unknown user <name>, your Okta local part does not
match your Chef data bag id. Follow
Troubleshooting: unknown user.
This is a per-user fix (new Chef data bag entry) and is not a fast
incident-time path — move to the break-glass path below if you need access now.
3. Break-glass SSH (Teleport unavailable)
Section titled “3. Break-glass SSH (Teleport unavailable)”If Teleport itself is down or cannot be made to grant access in time, use the IAP break-glass procedure documented in the overview runbook: Break-glass procedure for SSH access.
In short: file the firewall-rule MR for the relevant environment, then:
gcloud --project <PROJECT_ID> compute ssh <USERNAME>@<VM_NAME> --tunnel-through-iap
# Example (staging):gcloud --project gitlab-subscriptions-staging compute ssh <USERNAME>@customers-01-inf-stgsub --tunnel-through-iap4. If you still cannot get access
Section titled “4. If you still cannot get access”Escalate via the
Fulfillment escalation process for incidents or outages
and in #customersdot_errors. A Fulfillment engineer or an SRE with existing
access can run the recovery steps while your access is being granted.
Restarting Puma
Section titled “Restarting Puma”Puma serves the Rails web requests behind NGINX. Restart it when health checks or liveness probes are failing but the VM is otherwise reachable.
-
SSH to the affected VM (see access above).
-
Check the current state of Puma before changing anything:
Terminal window sudo systemctl status puma -
Restart Puma:
Terminal window sudo systemctl restart puma -
Confirm it came back up and is listening:
Terminal window sudo systemctl status pumasudo ss -ltnp | grep -i puma # confirm it is listening on its socket/port -
Tail the application log to confirm it is serving requests without errors:
Terminal window tail -f /home/customersdot/CustomersDot/current/log/production.log -
Confirm the instance is healthy again on the per-instance dashboard and logs.
Puma fails to restart
Section titled “Puma fails to restart”A common cause is that Puma cannot re-establish its database connections (for example after a DB blip), so a plain restart or reload leaves it in a failed or flapping state.
-
Look at why it failed:
Terminal window sudo systemctl status puma --no-pager -lsudo journalctl -u puma --since "15 min ago" --no-pagertail -n 200 /home/customersdot/CustomersDot/current/log/production.log -
If the process is wedged, stop it fully, confirm nothing is left, then start it again (a full stop/start clears a bad in-memory/DB-connection state that a reload does not):
Terminal window sudo systemctl stop puma# confirm no lingering puma processesps aux | grep -i '[p]uma'# if any remain, remove the stale pid filesls -l /home/customersdot/CustomersDot/current/tmp/pids/sudo pkill -9 -f '[p]uma'# check for stale pid files left behind, and remove them if presentls -l /home/customersdot/CustomersDot/current/tmp/pids/# sudo rm -f /home/customersdot/CustomersDot/current/tmp/pids/puma.pid# restartsudo systemctl start puma -
If it still fails to start, check for the common root causes:
- Database unreachable — confirm the Cloud SQL instance is healthy and
connectivity works. Open the DB console with
gitlab-db(see Database and Rails console access). - Bad or partial deploy — the
currentsymlink may point at a broken release. Redeploy the last known-good commit — see Manually redeploying the latest commit. - Disk full — a full disk prevents Puma from writing pids/logs. Follow Disk space alerts.
- Database unreachable — confirm the Cloud SQL instance is healthy and
connectivity works. Open the DB console with
-
If Puma cannot be recovered on an instance, take that instance out of the load-balancer target pool so it stops receiving traffic while you investigate (see the target-pool
remove-instancescommands in Scaling VMs → Vertical Scaling), and shift recovery to the remaining healthy VMs.
Restarting NGINX
Section titled “Restarting NGINX”NGINX terminates TLS and proxies to Puma. It also enforces some custom rate limit (see NGINX config for Production on CustomersDot Ansible).
-
SSH to the affected VM.
-
Validate the config before restarting — a restart with a broken config will take the web server down:
Terminal window sudo nginx -t -
If the config is valid, reload (zero-downtime) or restart:
Terminal window # Preferred: reload without dropping connectionssudo systemctl reload nginx# If a reload is not enough (e.g. NGINX itself is unhealthy):sudo systemctl restart nginx -
Confirm it is up and check the error log:
Terminal window sudo systemctl status nginxsudo tail -n 100 /var/log/nginx/error.log -
Confirm recovery on the per-instance dashboard and logs.
Manually redeploying the latest commit
Section titled “Manually redeploying the latest commit”Deploys normally flow automatically: a pipeline on the staging (default)
branch deploys to Staging, then to Production after a 3-hour delay
(see deploy-production job on CustomersDot).
During an incident you often need to deploy immediately.
When to bypass the staging wait
Section titled “When to bypass the staging wait”Trigger an immediate production deploy (skip the 3-hour wait) when:
- A fix for the active incident has already deployed to and been verified on Staging, and waiting 3 hours would prolong customer impact; or
- You need to redeploy the last known-good commit to production to roll back a bad release.
Do not skip the wait to push an unverified change straight to production during an incident unless the change itself is the agreed recovery action.
-
Identify the commit/pipeline to deploy:
- For a roll-forward fix: the latest
staging-branch pipeline that has passed and deployed to Staging. - For a rollback: the pipeline of the last known-good commit (find it in the project’s pipeline history / deploy environment history).
- For a roll-forward fix: the latest
-
In customers-gitlab-com: CI/CD → Pipelines, open the target pipeline.
-
Follow the manual deployment documentation for the exact invocation.
-
Watch the deploy complete, then verify:
Terminal window # On a VM, confirm the current release moved and services are healthyls -l /home/customersdot/CustomersDot/currentsudo systemctl status puma nginx -
Confirm recovery on the per-instance dashboard and logs and on the CustomersDot overview dashboard.
Per-instance metrics and logs
Section titled “Per-instance metrics and logs”Use these to confirm whether a specific VM has recovered.
Dashboards
Section titled “Dashboards”- CustomersDot overview dashboard — service-wide; use the instance/fqdn template variable to focus on a single VM.
- “Mimir - Fulfillment Platform” Explore screen on Grafana
- On the VM (per instance):
- NGINX:
/var/log/nginx/(access.log,error.log) - Application (Rails, Puma):
/home/customersdot/CustomersDot/current/log/production.log - Sidekiq:
/home/customersdot/CustomersDot/current/log/ - systemd:
sudo journalctl -u puma/sudo journalctl -u nginx/sudo journalctl -u sidekiq
- NGINX:
- Cloud Logging (filter by instance): the VM logs ship to GCP Stackdriver —
see Logs in the overview. Add a
resource.labels.instance_id/labels."compute.googleapis.com/resource_name"filter to scope to a single VM.
Related documentation
Section titled “Related documentation”- CustomersDot overview — access, SSH/Teleport, logs, metrics, deployments
- CustomersDot deployments — automatic staging → production flow and the 2-hour wait
- Scaling CustomersDot VMs — provisioning, target pools, access and permissions
- Disk space alerts
- customersdot-ansible deployment docs
- Fulfillment escalation process