gcp

GCP Load Balancer

GCP provides automatic SSL Certificates. Where possible, we use this feature to automatically generate and renew certificates.

For legacy certificates, or services behind a CDN, there is a manual procedure for updating them.

Google-managed certificates

GCP load balancers support automatic provisioning and renewal of free certificates backed by Let’s Encrypt.

Kubernetes

Load balancers provisioned by GKE’s ingress controller (ingress-gce) can be instructed to provision Google-managed certificates by creating a ManagedCertificate resource and binding it to an Ingress with an annotation. See docs for details.

If migrating from a non-Google-managed certificate, you must ensure that both the new managed and old self-managed certificate continue to be served by the Load balancer’s frontend to avoid downtime. This is done in 2 phases:

1. Create a ManagedCertificate and bind it to an Ingress that already has Ingress.spec.tls configured.
2. In the GCP console, you should see a “PROVISIONING” managed cert, in addition to the existing certificate.
3. Eventually, the cert will finish provisioning, and some time later, the LB will switch to serving the new cert with zero downtime. This can take up to 30 minutes.
4. You can now remove Ingress.spec.tls in a subsequent deployment.

See migration docs.

Example:

• Managed certificate provisioning in a Helm chart: https://gitlab.com/gitlab-org/charts/plantuml/-/merge_requests/17
• That same helm chart being used to provision a new managed cert, without removing the custom one: https://gitlab.com/gitlab-com/gl-infra/k8s-workloads/gitlab-helmfiles/-/merge_requests/193
• Finally, removing the custom certificate (Ingress.spec.tls is what is being manipulated by Helm): https://gitlab.com/gitlab-com/gl-infra/k8s-workloads/gitlab-helmfiles/-/merge_requests/194

Terraform

Create a google_compute_managed_ssl_certificate, and bind it to a google_compute_target_https_proxy (or the analogous resource for TCP load balancing).

When migrating from a self-managed cert to a managed one, you must not remove the old certificate from the google_compute_target_https_proxy.ssl_certificates array until the new certificate is done provisioning, and the LB has switched over to serving the new certificate. This can take up to 30 minutes.

See migration docs.

This will likely have to be orchestrated through our Terraform modules. For an example of cutting over to a new managed cert with zero downtime without making successive, and possibly breaking, module changes, see the following manual interventions and MRs:

• https://gitlab.com/gitlab-com/gl-infra/production/-/issues/2238
• https://ops.gitlab.net/gitlab-com/gl-infra/config-mgmt/-/merge_requests/1771
• https://ops.gitlab.net/gitlab-com/gl-infra/terraform-modules/google/https-lb/-/merge_requests/7 (although note that this contains a small error that was fixed in a follow-on: Google doesn’t like trailing-dot FQDNs in this context).

Replacement of non-Google-managed certificates

1. Obtain the new certificate from SSLMate. You will not be able to obtain a backup from via GCP!

2. Get the current private key from vault and save it to your machine. (preferably on a tmpfs)

   • vault kv get -field key.pem shared/env/gprd/sslmate/$DOMAIN

3. Determine the expiration date of the new certificate: < certificate.pem openssl x509 -noout -text | grep 'Not After'

4. Upload the certificate (chain) and private key from your local machine:

   gcloud --project=${project} compute ssl-certificates create ${CN}-exp-${EXP_DATE_ISO8601} --certificate certificate.pem --private-key key.pem

   • Replace any periods (.) with dashes (-).
   • For wildcards certificates (e.g. *.ops.gitlab.net) use wildcard-ops-gitlab-net etc.
   • The expiration date should be formatted in ISO8601 (e.g. 2019-12-31) and converted to UTC if necessary.

5. The certificate will then be available at the following resource route: projects/${project}/global/sslCertificates/${certificate} (you may have to specify that resource path to the relevant Terraform module).

6. Edit the Terraform for the Load Balancer to point at the new certificate, as seen in this example. It might help to do a string replace. To find old certificates in the project run gcloud --project=${project} compute ssl-certificates list

7. Once applied, it may take 10 minutes or longer for your changes to be seen live.

Cleanup

Validate the new certificate is working before doing this!

1. Delete the old certificate from GCP, to de-clutter:

   gcloud --project=gitlab-ops compute ssl-certificates delete wildcard-ops-gitlab-net

2. Delete the private key (and certificate) from your local machine.

Rollback

If you did not clean up:

1. Revert the Merge Request for Terraform changes
2. Apply it and wait.

If you already cleaned up and did not re-key the certificate:

1. Go to https://crt.sh/?q=*.ops.gitlab.net (replacing the query with the CN of the certificate you are looking for)

2. Copy the crt.sh ID for the old certificate (check via the expiration date)

3. Visit https://crt.sh/?d=<ID you copied> to retrieve a copy of the leaf certificate.

4. Get the required certificate chain.

   1. Open https://crt.sh/?id=<ID you copied>
   2. Scroll down and you should see the certificate contents
   3. Click on Issuer: (CA ID: xxxx)
   4. In the Certificates list click on the first crt.sh ID
   5. click on Download Certificate: PEM to get the intermediate certificate.
   6. Repeat until you see a Root CA in the Certificates list. This means you reached the root of trust.

5. Combine all the above certificates into one PEM.

6. Use this PEM as the rollback certificate.

If you already cleaned up and did re-key the certificate.

1. You’re out of luck. Depending on what the issue is, you might try:

   1. Re-installing the new certificate.
   2. Re-keying again
   3. Getting a Certificate from a new CA.