CI Artifacts CDN

The GitLab CI Artifacts CDN is a Google Application Load Balancer that caches requests made to the Artifacts backend bucket.

Possible Checks

If it is believed that there is an issue with the Artifacts CDN:

Check the Artifacts Storage Overview dashboard
Ensure that there is a valid certificate associated with the load balancer, the certificate is Google managed and issued by LetsEncrypt.

Each Artifacts bucket has a sample image that can be used to test that signed URLs are working properly, to generate a signed URL with the gcloud command line:

gcloud --project gitlab-production compute sign-url "https://cdn.registry.gitlab-static.net/cdn-test/three-cats.jpg" --key-name gprd-registry-cdn --expires-in 20y --key-file /tmp/gprd-key-file

Where /tmp/gprd-key-file is the base64 encoded key value that can be read fetched from GKMS secrets (see below).

Alerting

There are two BlackBox probes for the Staging and Production CDN endpoints:

If this alert fires, check to be sure the health object exists in the bucket /cdn-test/health.

This object was copied manually using gsutil and is a text file containing the string OK:

echo OK > /tmp/health
env=gprd
gsutil -h "Content-Type:text/html" cp /tmp/health gs://gitlab-$env-registry/cdn-test/health

Secret Key and Key Rotation

Overview

The CDN is configured with a secret key that is used by the registry to generate signed URLs. This key is configured in Terraform, and is configured as a Kubernetes secret, sourced from Vault.

Rotation

In order to rotate a key, you should first create a new random_password resource, and associate it with a new google_compute_backend_bucket_signed_url_key

Important: Do not remove the existing random_password and google_compute_backend_bucket_signed_url_key, until the new one is in place. No resources should be removed in the first Terraform apply.

After the new key is associated with the CDN:

Verify the new key by generating a signed URL (see above).
Wait at least 15 minutes which is the maximum lifetime of signed URLs generated by Artifacts
Find the base64 encoded key_value by inspecting the Terraform state after applying. The key should be a valid base64 string, decoded it will be a random 16 character password.
To rotate the key in Vault and creating a new secret version for the application, follow the instructions in the vault usage doc for rotating secrets.