Skip to content
Runbooks

Container Registry CDN

The GitLab Container Registry CDN is a Google Application Load Balancer that caches requests made to the Registry backend bucket.

The registry is the one responsible for generating pre-signed URLs and redirecting clients to CDN. This only applies to HEAD/GET requests against the /v2/<name>/blobs/<digest> endpoint.

Possible Checks

Section titled “Possible Checks”

If it is believed that there is an issue with the Registry CDN:

  • Check the Registry Storage Overview dashboard to ensure that the CDN is caching requests and serving 200 status codes
  • Ensure that there is a valid certificate associated with the load balancer, the certificate is Google managed and issued by LetsEncrypt.

Each Registry bucket has a sample image that can be used to test that signed URLs are working properly, to generate a signed URL with the gcloud command line:

gcloud --project gitlab-production compute sign-url \
  "https://cdn.registry.gitlab-static.net/cdn-test/three-cats.jpg" \
  --key-name gprd-registry-cdn \
  --expires-in 2880m \
  --key-file /tmp/gprd-key-file

Where /tmp/gprd-key-file is the base64 encoded key value that can be read fetched from GKMS secrets (see below).

Alerting

Section titled “Alerting”

There are two BlackBox probes for the Staging and Production CDN endpoints:

These were added so that we can validate the CDN endpoint and certificate. If this alert fires, check to be sure the health object exists in the bucket /cdn-test/health.

This object was copied manually using gsutil and is a text file containing the string OK:

echo OK > /tmp/health
env=gprd
gsutil -h "Content-Type:text/html" cp /tmp/health gs://gitlab-$env-registry/cdn-test/health

Secret Key and Key Rotation

Section titled “Secret Key and Key Rotation”

Overview

Section titled “Overview”

The CDN is configured with a secret key that is used by the registry to generate signed URLs. This key is configured in Terraform, and is configured as a Kubernetes secret. The secret is sourced from GKMS.

The key is written to the configuration file for the Registry service (/etc/docker/registry/middleware.storage/0/private-key) which can be inspected in the Registry pod if you need to confirm the value. The path is configured in the Registry configuration file /etc/docker/registry/config.yml

...
middleware:
  storage:
  - name: googlecdn
    options:
      baseurl: cdn.registry.pre.gitlab-static.net
      ipfilteredby: gcp
      keyname: pre-registry-cdn
      privatekey: /etc/docker/registry/middleware.storage/0/private-key
...

The contents of /etc/docker/registry/middleware.storage/0/private-key should be the base64 encoded value of the key.

Rotation

Section titled “Rotation”

In order to rotate a key, you should first create a new random_password resource, and associate it with a new google_compute_backend_bucket_signed_url_key

Important: Do not remove the existing random_password and google_compute_backend_bucket_signed_url_key, until the new one is in place. No resources should be removed in the first Terraform apply.

After the new key is associated with the CDN:

  • Verify the new key by generating a signed URL (see above).
  • Wait at least 15 minutes which is the maximum lifetime of signed URLs generated by Registry
  • Find the base64 encoded key_value by inspecting the Terraform state after applying. The key should be a valid base64 string, decoded it will be a random 16 character password.
  • Update the key in the GKMS vault gitlab-omnibus-secrets for the appropriate environment. See the docs for how to update secrets. The key name is in the registry block, and named _k8s_cdn_private_key (key is the base64 encoded key):
      "registry": {
        "_k8s_cdn_private_key": "<key_value>",
...

Once the key has been fully deployed, you can remove the old key by removing the old key resource from Terraform.