Container Registry CDN

The GitLab Container Registry CDN is a Google Application Load Balancer that caches requests made to the Registry backend bucket.

The registry is the one responsible for generating pre-signed URLs and redirecting clients to CDN. This only applies to HEAD/GET requests against the /v2/<name>/blobs/<digest> endpoint.

Possible Checks

If it is believed that there is an issue with the Registry CDN:

Check the Registry Storage Overview dashboard to ensure that the CDN is caching requests and serving 200 status codes
Ensure that there is a valid certificate associated with the load balancer, the certificate is Google managed and issued by LetsEncrypt.

Each Registry bucket has a sample image that can be used to test that signed URLs are working properly, to generate a signed URL with the gcloud command line:

gcloud --project gitlab-production compute sign-url \
  "https://cdn.registry.gitlab-static.net/cdn-test/three-cats.jpg" \
  --key-name gprd-registry-cdn \
  --expires-in 2880m \
  --key-file /tmp/gprd-key-file

Where /tmp/gprd-key-file is the base64 encoded key value that can be read from Vault (under env/{{ $env }}/ns/gitlab/registry/cdn inside k8s engine).

Alerting

There are two BlackBox probes for the Staging and Production CDN endpoints:

These were added so that we can validate the CDN endpoint and certificate. If this alert fires, check to be sure the health object exists in the bucket /cdn-test/health.

This object was copied manually using gsutil and is a text file containing the string OK:

echo OK > /tmp/health
env=gprd
gsutil -h "Content-Type:text/html" cp /tmp/health gs://gitlab-$env-registry/cdn-test/health

Secret Key and Key Rotation

Overview

The CDN is configured with a secret key that is used by the Registry to generate signed URLs. This key is provisioned by Terraform, and is configured as a Kubernetes secret, which in turn is sourced from Vault.

To get the key initially (i.e. when a brand-new module is created), run terraform console in config-mgmt locally (see instructions on how run terraform locally) and execute nonsensitive(module.registry-cdn.url_signing_key).

The key is written to the configuration file for the Registry service (/etc/docker/registry/middleware.storage/0/private-key) which can be inspected in the Registry pod if you need to confirm the value. The path is configured in the Registry configuration file /etc/docker/registry/config.yml

...
middleware:
  storage:
  - name: googlecdn
    options:
      baseurl: cdn.registry.pre.gitlab-static.net
      ipfilteredby: gcp
      keyname: pre-registry-cdn
      privatekey: /etc/docker/registry/middleware.storage/0/private-key
...

The contents of /etc/docker/registry/middleware.storage/0/private-key should be the base64 encoded value of the key.

Rotation

In order to rotate a key, in the module provisioning the CDN, follow these steps:

Set create_new_key to true then apply the change
Grab the new key from the outputs (see Overview above) and verify the new key by generating a signed URL (see above)
Wait at least 15 minutes which is the maximum lifetime of signed URLs generated by Registry
Set the key in Vault (under env/{{ $env }}/ns/gitlab/registry/cdn inside k8s engine) and create a new MR to create a new version for the secret in gitlab-com and set it where appropriate
Once the key has been propagated, set remove_old_key to true and apply the change
Remove old K8s secret definition as it is no longer needed

To rotate an already rotated key, follow these steps:

Set remove_old_key to false (or remove the line altogether) then apply the change
Grab the new key from the outputs (see Overview above, output named new_url_signing_key) and verify the new key by generating a signed URL (see above)
Wait at least 15 minutes which is the maximum lifetime of signed URLs generated by Registry
Set the key in Vault (under env/{{ $env }}/ns/gitlab/registry/cdn inside k8s engine) and create a new MR to create a new version for the secret in gitlab-com and set it where appropriate
Once the key has been propagated, set create_new_key to false (or remove the line altogether) and apply the change
Remove old K8s secret definition as it is no longer needed