Skip to content

CustomersDot service recovery (Puma, NGINX, redeploy)

This runbook provides end-to-end, step-by-step recovery guidance for customers.gitlab.com (CustomersDot) during an incident: restarting Puma and NGINX, recovering from a failed Puma restart, and manually redeploying the latest commit (including how and when to bypass the 2-hour staging wait).

All of the procedures below require SSH access to the CustomersDot VMs.

The VMs are named customers-NN-inf-prdsub (production) and customers-NN-inf-stgsub (staging). List them with:

Terminal window
# Production
gcloud --project gitlab-subscriptions-prod compute instances list --filter="name~customers-.*-inf-prdsub"
# Staging
gcloud --project gitlab-subscriptions-staging compute instances list --filter="name~customers-.*-inf-stgsub"

Log in to a VM via Teleport:

Terminal window
tsh login --proxy=production.teleport.gitlab.net
tsh ssh customers-01-inf-prdsub

Responders during a recent incident were blocked by Teleport access denials with no documented fallback. Work through these steps in order.

1. Confirm you have the required Teleport role

Section titled “1. Confirm you have the required Teleport role”

SSH to the CustomersDot VMs requires the Teleport role that grants the customersdot / production node login. SREs are expected to have this through their standard role set.

For more details on what roles to have to access the CustomersDot VMs, check out the Teleport documentation in the CustomersDot project.

Check which roles and logins your current Teleport session has:

Terminal window
tsh status
tsh login --proxy=production.teleport.gitlab.net # refresh if your cert is stale/expired

If you are missing the required role, request it just-in-time:

Terminal window
# Request access to the node you need, then share the request ID with an approver.
tsh request create --resource /production.teleport.gitlab.net/node/<node-uuid>
# or
tsh request create --roles <role-name>

Find out about the <node-uuid> to use for a given CustomersDot Production VM with the following command:

Terminal window
tsh ls -v | grep 'customers.*prdsub'

Follow the Teleport approval workflow to get the request approved.

If Teleport authenticates but the VM rejects you with Failed to launch: user: unknown user <name>, your Okta local part does not match your Chef data bag id. Follow Troubleshooting: unknown user. This is a per-user fix (new Chef data bag entry) and is not a fast incident-time path — move to the break-glass path below if you need access now.

If Teleport itself is down or cannot be made to grant access in time, use the IAP break-glass procedure documented in the overview runbook: Break-glass procedure for SSH access.

In short: file the firewall-rule MR for the relevant environment, then:

Terminal window
gcloud --project <PROJECT_ID> compute ssh <USERNAME>@<VM_NAME> --tunnel-through-iap
# Example (staging):
gcloud --project gitlab-subscriptions-staging compute ssh <USERNAME>@customers-01-inf-stgsub --tunnel-through-iap

Escalate via the Fulfillment escalation process for incidents or outages and in #customersdot_errors. A Fulfillment engineer or an SRE with existing access can run the recovery steps while your access is being granted.

Puma serves the Rails web requests behind NGINX. Restart it when health checks or liveness probes are failing but the VM is otherwise reachable.

  1. SSH to the affected VM (see access above).

  2. Check the current state of Puma before changing anything:

    Terminal window
    sudo systemctl status puma
  3. Restart Puma:

    Terminal window
    sudo systemctl restart puma
  4. Confirm it came back up and is listening:

    Terminal window
    sudo systemctl status puma
    sudo ss -ltnp | grep -i puma # confirm it is listening on its socket/port
  5. Tail the application log to confirm it is serving requests without errors:

    Terminal window
    tail -f /home/customersdot/CustomersDot/current/log/production.log
  6. Confirm the instance is healthy again on the per-instance dashboard and logs.

A common cause is that Puma cannot re-establish its database connections (for example after a DB blip), so a plain restart or reload leaves it in a failed or flapping state.

  1. Look at why it failed:

    Terminal window
    sudo systemctl status puma --no-pager -l
    sudo journalctl -u puma --since "15 min ago" --no-pager
    tail -n 200 /home/customersdot/CustomersDot/current/log/production.log
  2. If the process is wedged, stop it fully, confirm nothing is left, then start it again (a full stop/start clears a bad in-memory/DB-connection state that a reload does not):

    Terminal window
    sudo systemctl stop puma
    # confirm no lingering puma processes
    ps aux | grep -i '[p]uma'
    # if any remain, remove the stale pid files
    ls -l /home/customersdot/CustomersDot/current/tmp/pids/
    sudo pkill -9 -f '[p]uma'
    # check for stale pid files left behind, and remove them if present
    ls -l /home/customersdot/CustomersDot/current/tmp/pids/
    # sudo rm -f /home/customersdot/CustomersDot/current/tmp/pids/puma.pid
    # restart
    sudo systemctl start puma
  3. If it still fails to start, check for the common root causes:

  4. If Puma cannot be recovered on an instance, take that instance out of the load-balancer target pool so it stops receiving traffic while you investigate (see the target-pool remove-instances commands in Scaling VMs → Vertical Scaling), and shift recovery to the remaining healthy VMs.

NGINX terminates TLS and proxies to Puma. It also enforces some custom rate limit (see NGINX config for Production on CustomersDot Ansible).

  1. SSH to the affected VM.

  2. Validate the config before restarting — a restart with a broken config will take the web server down:

    Terminal window
    sudo nginx -t
  3. If the config is valid, reload (zero-downtime) or restart:

    Terminal window
    # Preferred: reload without dropping connections
    sudo systemctl reload nginx
    # If a reload is not enough (e.g. NGINX itself is unhealthy):
    sudo systemctl restart nginx
  4. Confirm it is up and check the error log:

    Terminal window
    sudo systemctl status nginx
    sudo tail -n 100 /var/log/nginx/error.log
  5. Confirm recovery on the per-instance dashboard and logs.

Deploys normally flow automatically: a pipeline on the staging (default) branch deploys to Staging, then to Production after a 3-hour delay (see deploy-production job on CustomersDot).

During an incident you often need to deploy immediately.

Trigger an immediate production deploy (skip the 3-hour wait) when:

  • A fix for the active incident has already deployed to and been verified on Staging, and waiting 3 hours would prolong customer impact; or
  • You need to redeploy the last known-good commit to production to roll back a bad release.

Do not skip the wait to push an unverified change straight to production during an incident unless the change itself is the agreed recovery action.

  1. Identify the commit/pipeline to deploy:

    • For a roll-forward fix: the latest staging-branch pipeline that has passed and deployed to Staging.
    • For a rollback: the pipeline of the last known-good commit (find it in the project’s pipeline history / deploy environment history).
  2. In customers-gitlab-com: CI/CD → Pipelines, open the target pipeline.

  3. Follow the manual deployment documentation for the exact invocation.

  4. Watch the deploy complete, then verify:

    Terminal window
    # On a VM, confirm the current release moved and services are healthy
    ls -l /home/customersdot/CustomersDot/current
    sudo systemctl status puma nginx
  5. Confirm recovery on the per-instance dashboard and logs and on the CustomersDot overview dashboard.

Use these to confirm whether a specific VM has recovered.

  • On the VM (per instance):
    • NGINX: /var/log/nginx/ (access.log, error.log)
    • Application (Rails, Puma): /home/customersdot/CustomersDot/current/log/production.log
    • Sidekiq: /home/customersdot/CustomersDot/current/log/
    • systemd: sudo journalctl -u puma / sudo journalctl -u nginx / sudo journalctl -u sidekiq
  • Cloud Logging (filter by instance): the VM logs ship to GCP Stackdriver — see Logs in the overview. Add a resource.labels.instance_id / labels."compute.googleapis.com/resource_name" filter to scope to a single VM.