Skip to content
Runbooks

CI Artifacts CDN

The GitLab CI Artifacts CDN is a Google Application Load Balancer that caches requests made to the Artifacts backend bucket.

Possible Checks

Section titled “Possible Checks”

If it is believed that there is an issue with the Artifacts CDN:

  • Check the Artifacts Storage Overview dashboard
  • Ensure that there is a valid certificate associated with the load balancer, the certificate is Google managed and issued by LetsEncrypt.

Each Artifacts bucket has a sample image that can be used to test that signed URLs are working properly, to generate a signed URL with the gcloud command line:

gcloud --project gitlab-production compute sign-url "https://cdn.artifacts.gitlab-static.net/cdn-test/three-cats.jpg" --key-name gprd-artifacts-cdn --expires-in 20y --key-file /tmp/gprd-key-file

Where /tmp/gprd-key-file is the base64 encoded key value that can be read from Vault (under env/{{ $env }}/ns/gitlab/artifacts-cdn inside k8s engine).

Alerting

Section titled “Alerting”

There are two BlackBox probes for the Staging and Production CDN endpoints:

If this alert fires, check to be sure the health object exists in the bucket /cdn-test/health.

This object was copied manually using gsutil and is a text file containing the string OK:

echo OK > /tmp/health
env=gprd
gsutil -h "Content-Type:text/html" cp /tmp/health gs://gitlab-$env-artifacts/cdn-test/health

Secret Key and Key Rotation

Section titled “Secret Key and Key Rotation”

Overview

Section titled “Overview”

The CDN is configured with a secret key that is used by Artifacts to generate signed URLs. This key is provisioned by Terraform, and is configured as a Kubernetes secret, which in turn is sourced from Vault.

To get the key initially (i.e. when a brand-new module is created), run terraform console in config-mgmt locally (see instructions on how run terraform locally) and execute nonsensitive(module.artifacts-cdn.url_signing_key).

Rotation

Section titled “Rotation”

In order to rotate a key, in the module provisioning the CDN, follow these steps:

  1. Set create_new_key to true then apply the change
  2. Grab the new key from the outputs (see Overview above) and verify the new key by generating a signed URL (see above)
  3. Wait at least 15 minutes which is the maximum lifetime of signed URLs generated by Artifacts
  4. Set the key in Vault (under env/{{ $env }}/ns/gitlab/artifacts-cdn inside k8s engine) and create a new MR to create a new version for the secret in gitlab-com and set it where appropriate
  5. Once the key has been propagated, set remove_old_key to true and apply the change
  6. Remove old K8s secret definition as it is no longer needed

To rotate an already rotated key, follow these steps:

  1. Set remove_old_key to false (or remove the line altogether) then apply the change
  2. Grab the new key from the outputs (see Overview above, output named new_url_signing_key) and verify the new key by generating a signed URL (see above)
  3. Wait at least 15 minutes which is the maximum lifetime of signed URLs generated by Artifacts
  4. Set the key in Vault (under env/{{ $env }}/ns/gitlab/artifacts-cdn inside k8s engine) and create a new MR to create a new version for the secret in gitlab-com and set it where appropriate
  5. Once the key has been propagated, set create_new_key to false (or remove the line altogether) and apply the change
  6. Remove old K8s secret definition as it is no longer needed