overview
GCS Bucket for GitLab.com Omnibus Packages
Section titled “GCS Bucket for GitLab.com Omnibus Packages”Overview
Section titled “Overview”To reduce our dependency on packages.gitlab.com, we sync all Omnibus packages to a bucket in GCP that can be used by Deployer for Omnibus installations.
Buckets
Section titled “Buckets”There are two buckets used internally for storing Omnibus packages, gitlab-com-pkgs-builds and gitlab-com-pkgs-release:
gitlab-com-pkgs-builds: Used for Omnibus branch builds or all builds that are not tagged in the Omnibus pipelinegitlab-com-pkgs-release: Used for Omnibus release builds, all builds that are tagged including auto-deploy and official self-managed releases
Configuration
Section titled “Configuration”Configuration of the bucket is done in Terraform in the gitlab-com-pkgs environment.
There is one service account [email protected] that has a key set as a CI variable GITLAB_COM_PKGS_SA_FILE in the omnibus-gitlab pipeline CI variables on dev.gitlab.org.
For Deployer, access is granted using the service account terraform@<account>.iam.gserviceaccount.com which is the service account associated to all VMs that require the Omnibus package for installations.
Deployer first checks to see if a package is available in the gitlab-com-pkgs-release bucket, if it isn’t, we fallback to packages.gitlab.com for installation.
The logic to use the bucket for installation can be disabled, by removing the DEB_INSTALL_ENABLE env variable in CI variables for Deployer.
Troubleshooting
Section titled “Troubleshooting”Packages are not available for download
Section titled “Packages are not available for download”If a package is not available for download, it is likely that rsync job that copies packages to the bucket didn’t run or failed in some way.
Check omnibus jobs for the corresponding version tag.
The rsync happens in the package stage (e.g.: rsync job).
Look for the build-package-sync section, there you should see the following in the job output:
GCS Sync: Activating service accountActivated service account credentials for: [[email protected]]GCS Sync: Copying pkg/ contents to gitlab-com-pkgsBuilding synchronization state......Package cleanup
Section titled “Package cleanup”Because these buckets are only used internally, all packages older than 1 year will be deleted. Additionally, we change the storage class for older packages which is configured in Terraform.