Skip to content

Gitlab Certificate Run Books

This is an overview of certificates, where they are used and how they can be replaced in their service.

Currently we have multiple ways of deploying certificates. Please see the Management and Details columns to find the management process and other details to edit according to that documentation.

  • COMODO has renamed to Sectigo, those names might get used interchangeably in this document. Any Certificate that is listed as issued by COMODO will in the future be issued by Sectigo.

  • Our primary certificate source is SSLMate.

    • Using the above link it is possible to retrieve the current certificate file for each CN listed there.
    • Those files are permanent links to the public chain of the certificate. The key is not part of that chain.
    • Some tasks require the commandline tool (available via https://sslmate.com/help/install and brew install sslmate).

Before buying a new certificate, please check if it is possible to use an automated Let’s Encrypt certificate for your purpose!

Terminal window
sslmate buy '<domain>' --auto-renew --approval=dns --key-type=ecdsa

Use sslmate help buy for additional options if needed.

But with the above example, this will purchase a certificate that expires after 1 year, will auto renew itself using DNS, and is of key type ecdsa.

We’ve chosen to go with ecdsa for it’s improvements:

  • Smaller size
  • Improved TLS handshake speed
  • Stronger algorithm

We have Route53 integration with SSLMate so the DNS authorization will be done automatically.

The sslmate client will download the key and certificate to the directory in which the command was executed.

Make sure to add the new certificate to the list below!

Add the private key to vault:

Terminal window
cat $DOMAIN.key.pem | vault kv put shared/env/gprd/sslmate/$DOMAIN key.pem=-

For older certificates we may not have renewal properly configured. Let’s change that:

Terminal window
sslmate edit '<domain>' --approval=dns
sslmate renew '<domain>'

This will change the existing approval method to our fancy DNS integration, and then force a renew. You can then download the certificate:

Terminal window
sslmate download '<domain>'

Note that sslmate may complain that you won’t have the key in your ${CWD}. This is fine as we should have the key on minimally on a server, but may also exist inside of 1Password, and even better, inside a Chef vault.

The SSL certificate for status.gitlab.com is no longer managed using SSLMate, meaning there should not be any manual steps to do once renewal time comes. :party:

Instead, we use a dedicated certificate from AWS as detailed in the status.io docs. This means Amazon Certificate Manager should handle renewals automatically ahead of the expiry date.

The Amazon certificate is provisioned through the use of a CNAME record for domain validation that must remain in our DNS zone permanently.

Should we ever decide to change back to a customer-provided certificate for whatever reason, we would likely need to contact status.io support to request a switch back to the custom certificate setting.

See this issue for more context on why we opted for an Amazon certificate.

If a certificate auto-renews but we have lost the private key, generate a new one (and CSR) using SSLMate’s web UI. Download the private key and upload to vault:

Terminal window
cat $DOMAIN.key.pem | vault kv put shared/env/gprd/sslmate/$DOMAIN key.pem=-

Wait for Chef to converge, or force a convergence.

Use a tool such as https://www.sslshopper.com/ssl-checker.html or https://www.ssllabs.com/ssltest/index.html to verify that the certificate is live, working, and fully valid.

We utilize Prometheus blackbox to regularly check endpoints and send us alerts when those endpoints go down as well as validate and alert us at a threshold when those certificates are going to expire.

See SSL certificate expiration rules.

The list of hosts that will have their certificate expiry scraped by Prometheus is defined in this file. By adding a server there, you will receive alerts when there are less than 30 days remaining until certificate expiration.

Safe execution of a update involving Chef nodes

Section titled “Safe execution of a update involving Chef nodes”
  1. Preparation:

    1. Check the state of chef-client:

      Terminal window
      knife ssh "roles:${chef_role} AND chef_environment:${chef_env}" "systemctl status chef-client"`
    2. Stop Chef on the HAProxy fleet that serves the certificate in question:

      Terminal window
      knife ssh "role:<role name>" "sudo service chef-client stop"
  2. Do the change according to the table below

  3. Slowly roll out Chef

    1. Inspect changes that would be applied on one of the nodes:

      Terminal window
      sudo chef-client --why-run
    2. Force a chef-run on one of the nodes for verification. You should be able to simply run sudo chef-client and see the updated certificate in the output.

    3. Use openssl to verify the correct certificate is in place:

      Terminal window
      openssl s_client -connect ${NODE IP ADDRESS}:443 -servername ${HOSTNAME} </dev/null 2>/dev/null | openssl x509 -noout -text

      Check for dates and general information to match, such as the CN and SANs.

    4. Restart Chef on the nodes from the first step.

      Terminal window
      knife ssh "role:${chef_role} AND chef_environment:${chef_env}" "sudo service chef-client start"

Certificates currently managed by the GitLab Infrastructure team

Section titled “Certificates currently managed by the GitLab Infrastructure team”
DomainsIssuerCommentsManagementDetails
about-src.gitlab.com, *.about-src.gitlab.com, about.gitlab-review.app, *.about.gitlab-review.appCOMODO RSA Domain Validation Secure Server CAabout-src.gitlab.com is no longer used, but the CN of the cert / The other SANs are used for GitLab review apps for www-gitlab-comChef Vaultdata bag: about-gitlab-com, item: _default, fields: ssl_certificate, ssl_key
about.gitlab.comGlobalSign CloudSSL CA - SHA256 - G3CDN Certificate for about.gitlab.comCloudflareAuto-renewed shared certificate
canary.gitlab.comSectigo RSA Domain Validation Secure Server CACanary direct accessGKMSitem: frontend-loadbalancer gprd, fields: gitlab-haproxy.ssl.canary_crt, gitlab-haproxy.ssl.canary_key
ce.gitlab.comSectigo RSA Domain Validation Secure Server CARedirect to CE repo, hosted on about-src., no CDNChef Vaultdata bag: about-gitlab-com, item: _default, fields: [ce.gitlab.com][ssl_certificate], [ce.gitlab.com][ssl_key]
chef.gitlab.comCOMODO RSA Domain Validation Secure Server CAChef serverChef Server-
contributors.gitlab.comGlobalSign CloudSSL CA - SHA256 - G3Redirect to gitlab.biterg.io, hosted on CloudflareCloudflareAuto-renewed shared certificate
customers.gitlab.comSectigo RSA Domain Validation Secure Server CACustomer managementChef Vaultdata bag: customers-gitlab-com, item: _default, fields: ssl_certificate, ssl_key
dashboards.gitlab.netLet’s Encrypt Authority X3Internal grafanaAutomated GCP Load Balancerproject: gitlab-ops
dev.gitlab.orgCOMODO RSA Domain Validation Secure Server CADev instanceChef Vaultdata bag: dev-gitlab-org, item: _default, fields: ssl.certificate, ssl.private_key
docs.gitlab.comLet’s Encrypt Authority X3-Automated (GitLab Pages managed)
dr.gitlab.comSectigo RSA Domain Validation Secure Server CADisaster recovery instanceGKMSitem: frontend-loadbalancer dr, fields: gitlab-haproxy.ssl.gitlab_crt, gitlab-haproxy.ssl.gitlab_key
ee.gitlab.comSectigo RSA Domain Validation Secure Server CARedirect to EE repo, hosted on about-src., no CDNChef Vaultdata bag: about-gitlab-com, item: _default, fields: [ee.gitlab.com][ssl_certificate], [ee.gitlab.com][ssl_key]
forum.gitlab.comLet’s Encrypt Authority X3Hosted by discourse.comForum
gitlab.comCloudflare Inc ECC CA-3, COMODO RSA Domain Validation Secure Server CADuhAutomated Cloudflare and GKMSitem: frontend-loadbalancer gprd, fields: gitlab-haproxy.ssl.gitlab_crt, gitlab-haproxy.ssl.gitlab_key
gitlab.orgGlobalSign CloudSSL CA - SHA256 - G3Redirect to about.gitlab.com, hosted on CloudflareCloudflareAuto-renewed shared certificate
hub.gitlab.comSectigo RSA Domain Validation Secure Server CARedirects to https://lab.github.com/ (https://gitlab.com/gitlab-com/gl-infra/reliability/-/issues/6667), hosted on about-src, no CDNChef Vaultdata bag: about-gitlab-com, item: _default, fields: [hub.gitlab.com][ssl_certificate], [hub.gitlab.com][ssl_key]
jobs.gitlab.comSectigo RSA Domain Validation Secure Server CARedirects to https://about.gitlab.com/jobs/, Hosted on CloudflareCloudflareAuto-renewed shared certificate
license.gitlab.comSectigo RSA Domain Validation Secure Server CAChef Vaultdata bag: license-gitlab-com, item: _default, fields: [gitlab-packagecloud][ssl_certificate], [gitlab-packagecloud][ssl_key]
log.gprd.gitlab.netLet’s Encrypt Authority X3Automated GCP Load Balancerproject: gitlab-ops ops-proxy
next.gitlab.comLet’s Encrypt Authority X3-Automated (GitLab Pages managed)
nonprod-log.gitlab.net, www.nonprod-log.gitlab.net, log.gstg.gitlab.net, log.dr.gitlab.net, log.pre.gitlab.net, log.ops.gitlab.netLet’s Encrypt Authority X3Non prod logsAutomated GCP Load Balancerproject: gitlab-ops ops-nonprod-proxy
ops.gitlab.netCloudflare Inc ECC CA-3Ops instanceCloudflareitem: gitlab-omnibus-secrets ops, fields: omnibus-gitlab.ssl.certificate, omnibus-gitlab.ssl.private_key
packages.gitlab.comCOMODO RSA Domain Validation Secure Server CAPackageCloud instanceChef Vaultdata bag: packages-gitlab.com, item: _default, fields: ssl.certificate, ssl.private_key
pre.gitlab.comCOMODO RSA Domain Validation Secure Server CAPrerelease instanceGKMSitem: frontend-loadbalancer pre, fields: gitlab-haproxy.ssl.gitlab_crt, gitlab-haproxy.ssl.gitlab_key
prod.pages-check.gitlab.netCOMODO RSA Domain Validation Secure Server CAGitLab pages checkAutomated (GitLab Pages managed)
prometheus-01.us-east1-c.gce.gitlab-runners.gitlab.netCOMODO RSA Domain Validation Secure Server CAChef Vaultdata bag: gitlab-runners-prometheus-gce-us-east1-c, item: ci-prd, fields: gitlab-oauth2-proxy.nginx.ssl_certificate, gitlab-oauth2-proxy.nginx.ssl_key
prometheus-01.us-east1-d.gce.gitlab-runners.gitlab.netCOMODO RSA Domain Validation Secure Server CAChef Vaultdata bag: gitlab-runners-prometheus-gce-us-east1-d, item: ci-prd, fields: gitlab-oauth2-proxy.nginx.ssl_certificate, gitlab-oauth2-proxy.nginx.ssl_key
prometheus.gitlab.comSectigo RSA Domain Validation Secure Server CAChef Vaultdata bag: gitlab-oauth2-proxy-prometheus, item: prd, fields: ssl_certificate, ssl_key
registry.gitlab.comSectigo RSA Domain Validation Secure Server CAGKMSitem: frontend-loadbalancer gprd, fields: gitlab-haproxy.ssl.registry_crt, gitlab-haproxy.ssl.registry_key
registry.ops.gitlab.netSectigo RSA Domain Validation Secure Server CAGKMSitem: gitlab-omnibus-secrets ops, fields: omnibus-gitlab.ssl.registry_certificate, omnibus-gitlab.ssl.registry_private_key
registry.pre.gitlab.comSectigo RSA Domain Validation Secure Server CAGKMSitem: frontend-loadbalancer pre, fields: gitlab-haproxy.ssl.registry_crt, gitlab-haproxy.ssl.registry_key
registry.staging.gitlab.comCOMODO RSA Domain Validation Secure Server CAGKMSitem: frontend-loadbalancer gstg, fields: gitlab-haproxy.ssl.registry_crt, gitlab-haproxy.ssl.registry_key
sentry.gitlab.netCOMODO RSA Domain Validation Secure Server CAChef hybridcert role: ops-infra-sentry cert field: default_attributes.gitlab-sentry.ssl_certificate, key data bag: gitlab-sentry, key items: _default and prd, key fields: gitlab-sentry.ssl_key
snowplow.trx.gitlab.netSectigo RSA Domain Validation Secure Server CAAutomated AWS (not terraform managed)
staging.gitlab.com, auth.staging.gitlab.com, geo.staging.gitlab.com, gstg.gitlab.comCloudflare Inc ECC CA-3, COMODO RSA Domain Validation Secure Server CAStaging instanceAutomated Cloudflare and GKMSitem: frontend-loadbalancer gstg, fields: gitlab-haproxy.ssl.gitlab_crt, gitlab-haproxy.ssl.gitlab_key
staging.pages-check.gitlab.netCOMODO RSA Domain Validation Secure Server CAGitLab pages checkautomated (GitLab Pages managed)
status.gitlab.comSectigo RSA Domain Validation Secure Server CAstatus.ioStatus.io
support.gitlab.comLet’s Encrypt Authority X3General zendeskZenDesk
user-content.staging.gitlab-static.netSectigo ECC Domain Validation Secure Server CAGCP Load Balancerproject: gitlab-production
version.gitlab.comSectigo RSA Domain Validation Secure Server CAChef Vaultdata bag: version-gitlab-com, item: _default, fields: ssl_certificate, ssl_key
*.ci-gateway.int.gstg.gitlab.netSectigo RSA Domain Validation Secure Server CAGKMSGKMS item: frontend-loadbalancer gstg, fields: gitlab-haproxy.ssl.ci_gateway_crt, gitlab-haproxy.ssl.ci_gateway_key; project: gitlab-staging-1
*.gitlab.ioC=BE, O=GlobalSign nv-sa, CN=AlphaSSL CA - SHA256 - G2GitLab Pages from SSLMateHCVaultPath: /v1/k8s/data/env/gprd/ns/gitlab/pages/tls
*.gprd.gitlab.netSectigo RSA Domain Validation Secure Server CAGCP Load Balancer & GKMSGKMS item: frontend-loadbalancer gprd, fields: gitlab-haproxy.ssl.internal_crt, gitlab-haproxy.ssl.internal_key; project: gitlab-production
*.gstg.gitlab.comSectigo RSA Domain Validation Secure Server CAGCP Load Balancerproject: gitlab-staging
*.staging.gitlab.ioSectigo RSA Domain Validation Secure Server CAGSTG GitLab Pages from SSLMateHCVaultPath: /v1/k8s/data/env/gstg/ns/gitlab/pages/tls
*.gstg.gitlab.netLet’s Encrypt Authority X3Automated GCP Load BalancerGCP project: gitlab-staging
*.ops.gitlab.netLet’s Encrypt Authority X3Automated GCP Load Balancerproject: gitlab-ops
*.ops.gitlab.netCloudFlare Inc ECC CA-2Automated Cloudflare
*.pre.gitlab.netLet’s Encrypt Authority X3Automated GCP Load Balancerproject: gitlab-pre
*.pre.gitlab.ioSectigo RSA Domain Validation Secure Server CAPRE GitLab Pages from SSLMateHCVaultPath: /v1/k8s/data/env/pre/ns/gitlab/pages/tls
*.qa-tunnel.gitlab.infoSectigo RSA Domain Validation Secure Server CAQA TunnelChef Vaultdata bag: gitlab-qa-tunnel, item: ci-prd, fields: "gitlab-qa-tunnel".ssl_certificate, "gitlab-qa-tunnel".ssl_key
*.staging.gitlab.netCloudFlare Inc ECC CA-2Automated Cloudflare

Certificates managed by other GitLab teams

Section titled “Certificates managed by other GitLab teams”
DomainsIssuerCommentsManagementOwner
workspaces.gitlab.devSectigo RSA Domain Validation Secure Server CAGitLab workspaces for internal use onlySSLMate, manually updated in GKMSEngineering Productivity team

Defunct certificates (dead hosts, no longer used, etc)

Section titled “Defunct certificates (dead hosts, no longer used, etc)”
DomainsIssuerValid untilComments
alerts.gitlab.comSectigo RSA Domain Validation Secure Server CA2020-06-24T23:59:59Active certificate, but not rolled out to the CN host.
allremote.orgSectigo RSA Domain Validation Secure Server CA2020-06-08T23:59:59Page 404s with HTTP, and NET::ERR_CERT_COMMON_NAME_INVALID on HTTPS. Is a gitlab.io page.
canary.staging.gitlab.comCOMODO RSA Domain Validation Secure Server CA2019-09-06T23:59:59Connection to host times out
canary.staging.gitlab.comSectigo RSA Domain Validation Secure Server CA2020-09-06T23:59:59
convdev.ioSectigo RSA Domain Validation Secure Server CA2020-05-30T23:59:59Current certificate, but not rolled out
dr.gitlab.comSectigo RSA Domain Validation Secure Server CACancelled disaster recovery instance
enable.gitlab.comLet’s Encrypt Authority X32019-10-14T21:09:02Site is a 404
geo1.gitlab.comCOMODO RSA Domain Validation Secure Server CA2019-11-02T23:59:59Does not resolve
geo2.gitlab.comCOMODO RSA Domain Validation Secure Server CA2019-11-15T23:59:59Does not resolve
gprd.gitlab.comCOMODO RSA Domain Validation Secure Server CA2020-02-06T23:59:59Does not resolve
gstg.gitlab.comSectigo RSA Domain Validation Secure Server CA2020-04-11T23:59:59Does not resolve
log.gitlap.comSectigo RSA Domain Validation Secure Server CA2020-06-02T23:59:59Replaced by log.gitlab.net
monitor.gitlab.netAmazon Server CA 1BNo longer in use
monkey.gitlab.netCOMODO RSA Domain Validation Secure Server CA2020-02-27T23:59:59Does not resolve
next.gitlab.comSectigo RSA Domain Validation Secure Server CA-Replaced with auto-renewed Let’s Encrypt certificate (GitLab pages)
next.staging.gitlab.comSectigo RSA Domain Validation Secure Server CA2020-02-22T23:59:59Does not work as of now, but should be fixed to work in the future (via LE certificate)
performance-lb.gitlab.netSectigo RSA Domain Validation Secure Server CA2020-05-17T23:59:59Does not resolve
plantuml.pre.gitlab.comSectigo RSA Domain Validation Secure Server CANo longer in use
prod-log.gitlab.netSectigo RSA Domain Validation Secure Server CA2020-08-30T23:59:59Initially used for the production logs cluster, we later decided to use log.gprd.gitlab.net instead
prod.geo.gitlab.comSectigo RSA Domain Validation Secure Server CA2020-08-14T23:59:59Does not resolve
prometheus-01.nyc1.do.gitlab-runners.gitlab.netCOMODO RSA Domain Validation Secure Server CA2019-11-06T23:59:59Times out
prometheus-2.gitlab.comSectigo RSA Domain Validation Secure Server CA2020-06-25T23:59:59Times out
prometheus-3.gitlab.comSectigo RSA Domain Validation Secure Server CA2020-06-25T23:59:59Times out
prometheus-app-01.gitlab.netCOMODO RSA Domain Validation Secure Server CA2020-02-16T23:59:59Times out
prometheus-app-02.gitlab.netCOMODO RSA Domain Validation Secure Server CA2020-02-16T23:59:59Times out
redash.gitlab.comCOMODO RSA Domain Validation Secure Server CAHosted on version.gitlab.com. Redash is no longer chef managed.
registry.gke.gstg.gitlab.comLet’s Encrypt Authority X32019-09-24T17:49:51Was retrieved, but is not used. Verified by jarv
registry.gke.pre.gitlab.comLet’s Encrypt Authority X32019-08-26T16:53:33Same as registry.gke.gstg.gitlab.com
registry.gke.staging.gitlab.comLet’s Encrypt Authority X32019-09-24T18:07:16Same as registry.gke.gstg.gitlab.com
runners-cache-5.gitlab.comSectigo RSA Domain Validation Secure Server CA2020-06-07T23:59:59Does not resolve
sentry-infra.gitlap.comSectigo RSA Domain Validation Secure Server CA2020-05-26T23:59:59Connection refused
sync.geo.gitlab.comSectigo RSA Domain Validation Secure Server CA2020-08-17T23:59:59Does not resolve
*.ce.gitlab-review.appCOMODO ECC Domain Validation Secure Server CA2019-10-03T23:59:59Time out
*.ce.gitlab-review.appSectigo ECC Domain Validation Secure Server CA2020-10-03T23:59:59Time out
*.design.gitlab.comSectigo RSA Domain Validation Secure Server CA2020-04-27T23:59:59Site uses a LE cert generated by gl pages. Wildcard is not installed. This cert is dead
*.dr.gitlab.netSectigo ECC Domain Validation Secure Server CA2020-01-23T23:59:59Does not resolve
*.ee.gitlab-review.appCOMODO ECC Domain Validation Secure Server CA2019-10-03T23:59:59Times out
*.eks.helm-charts.winSectigo RSA Domain Validation Secure Server CA2020-04-01T23:59:59Does not resolve
*.gitlab-review.appCOMODO RSA Domain Validation Secure Server CA2019-09-10T23:59:59Does not resolve
*.gprd.gitlab.comCOMODO RSA Domain Validation Secure Server CA-No longer in used (was used before gitlab.net)
*.helm-charts.winCOMODO RSA Domain Validation Secure Server CA2019-11-08T23:59:59Times out
*.k8s-ft.winCOMODO RSA Domain Validation Secure Server CA2019-11-08T23:59:59Times out
*.pre.gitlab.comCOMODO RSA Domain Validation Secure Server CA-Not required
*.separate-containers.partyCOMODO RSA Domain Validation Secure Server CA2019-11-08T23:59:59Does not resolve
*.single.gitlab.comCOMODO RSA Domain Validation Secure Server CA2019-09-12T23:59:59Does not resolve
*.single.gitlab.comSectigo RSA Domain Validation Secure Server CA2020-09-12T23:59:59
*.gstg.gitlab.ioSectigo RSA Domain Validation Secure Server CA2020-06-25T23:59:59Gitlab pages on staging, was not updated on hosts, is it still used? There IS *.staging.gitlab.io which is working
*.testbed.gitlab.netLet’s Encrypt Authority X3Automated GCP Load Balancer
DomainsIssuerValid untilComments
federal-support.gitlab.comSectigo RSA Domain Validation Secure Server CA2020-05-22T23:59:59US Federal Zendesk instance
federal-support.gitlab.comLet’s Encrypt Authority X32019-09-29T18:11:39
learn.gitlab.comSectigo RSA Domain Validation Secure Server CA2020-05-30T23:59:59Redirects to https://gitlab.lookbookhq.com/users/sign_in
page.gitlab.comCloudFlare, Inc.Redirect to about. (Non infra managed as CF renews automagically)
saml-demo.gitlab.infoSectigo RSA Domain Validation Securchef_hybride Server CA2020-05-18T23:59:59
saml-demo.gitlab.infoLet’s Encrypt Authority X32019-10-23T19:46:50
shop.gitlab.comLet’s Encrypt Authority X32019-10-09T19:47:35Swag shop
shop.gitlab.comCloudFlare, Inc.
translate.gitlab.comLet’s Encrypt Authority X32019-10-04T02:12:34GitLab translation site
www.meltano.comCOMODO RSA Domain Validation Secure Server CA2019-09-07T23:59:59Maybe mananaged by infra?
*.cloud-native.winCOMODO RSA Domain Validation Secure Server CA2019-11-08T23:59:59Looks like a k8s cluster