Dependency proxy for containers runbook
Summary
Section titled “Summary”The Dependency Proxy for GitLab Container Registry acts as a local proxy for frequently accessed Docker images from Docker Hub.
Note: For general Package Registry architecture, troubleshooting, and operational procedures, see the main Package Registry runbook.
v2/:group_id/dependency_proxy/containers/:image/manifests/*tagv2/:group_id/dependency_proxy/containers/:image/blobs/:shav2/:group_id/dependency_proxy/containers/:image/blobs/:sha/upload/authorizev2/:group_id/dependency_proxy/containers/:image/blobs/:sha/uploadv2/:group_id/dependency_proxy/containers/:image/manifests/*tag/upload/authorizev2/:group_id/dependency_proxy/containers/:image/manifests/*tag/upload
Observability
Section titled “Observability”Troubleshooting
Section titled “Troubleshooting”Service Changes
Section titled “Service Changes”Common Operations
Section titled “Common Operations”Test Docker pull through proxy
Section titled “Test Docker pull through proxy”docker pull $GITLAB_HOST:$PORT/$GROUP_PATH/dependency_proxy/containers/library/alpine:latest
Credential Rotation
Section titled “Credential Rotation”Docker Organization Access Token (OAT) for Dependency Proxy
Section titled “Docker Organization Access Token (OAT) for Dependency Proxy”GitLab’s Dependency Proxy uses a Docker Organization Access Token (OAT) to authenticate
pull-through requests to Docker Hub on behalf of users. This token is stored in 1Password
as “Docker gitlabci OAT” and is configured as a CI/CD variable (GITLAB_DEPENDENCY_TOKEN)
in the gitlab-org and gitlab-com groups.
If this token expires, CI pipelines across both groups will fail with authentication errors when pulling images through the Dependency Proxy.
Symptoms of an expired token
Section titled “Symptoms of an expired token”- CI jobs fail with errors like
Error response from daemon: unauthorized: authentication requiredorGITLAB_DEPENDENCY_TOKEN fails to authenticate - Failures are widespread across multiple projects in
gitlab-organdgitlab-com - The Dependency Proxy Dashboard shows a spike in errors
Manual rotation procedure
Section titled “Manual rotation procedure”-
Log in to Docker Hub with the GitLab organization credentials (find them in 1Password under the
gitlaborg vault). -
Navigate to Organization Settings → Access Tokens (or use the Docker Hub OAT API).
-
Create a new Organization Access Token with
Public Repo Read-onlyscope. -
Copy the new token value.
-
Update the CI/CD variable
GITLAB_DEPENDENCY_TOKENin both:https://gitlab.com/gitlab-org→ Settings → CI/CD → Variableshttps://gitlab.com/gitlab-com→ Settings → CI/CD → Variables
-
Update the token value in 1Password under “Docker gitlabci OAT”.
-
Revoke the old OAT via the Docker Hub UI or API.
-
Verify by re-running a previously failing pipeline, or by running:
Note: You must be authenticated to
gitlab.combefore pulling through the Dependency Proxy. An unauthenticated pull will fail regardless of whether the OAT rotation succeeded.Terminal window docker login gitlab.comdocker pull gitlab.com/gitlab-org/dependency_proxy/containers/library/alpine:latest
Automation
Section titled “Automation”Automatic rotation via the Docker Hub API is tracked in production-engineering#28867. Until that is implemented, rotation must be done manually following the steps above.
References
Section titled “References”Service Catalog
Section titled “Service Catalog”- Service label:
Service::Dependency Proxy - Metrics catalog entry:
dependency_proxycomponent in thewebservice - Severity: s3 (does not page SREs directly; alerts roll up to the
webservice) - Feature category:
virtual_registry - Owner team:
group::container registry - Slack:
#g_container-registry
Primary Contacts
Section titled “Primary Contacts”- Primary Owner: Container Registry team
- Slack Channel:
#g_container-registry(GitLab internal) - Escalation:
#productionfor active incidents