Skip to content
Runbooks

PackageCloud (packages.gitlab.com) Service

Logging

Section titled “Logging”

Troubleshooting Pointers

Section titled “Troubleshooting Pointers”

Metrics / Monitoring

Section titled “Metrics / Monitoring”

You can watch metrics in the Package Cloud folder on dashboards.gitlab.net. Specific dashboards of interest are:

  1. Watching Rainbows / Resque Saturation in the Kubernetes Overiew section.
  2. CPU Load per pod and pod counts in HPA in the Saturation Details section.

When you run something like /chatops run publish --security 16.6.2 you should see pipelines on dev.gitlab.org start and a corresponding load increase on Package cloud. Timing of those commoands can then be watched from those dashboards.

Support Requests

Section titled “Support Requests”

We are entitled to product support from Packagecloud. To open a new support request email: [email protected]

To escalate a support request reach out to the Reliability General Team.

Updating the License Key

Section titled “Updating the License Key”

  1. Get the new license (most likely this will come from Packagecloud support).

  2. Update the license in vault (same license key for both environments):

  3. Trigger a rolling restart:

    Terminal window
    kubectl -n packagecloud rollout restart deployment/packagecloud-toolbox
    kubectl -n packagecloud rollout restart deployment/packagecloud-resque
    kubectl -n packagecloud rollout restart deployment/packagecloud-rainbows
    kubectl -n packagecloud rollout restart deployment/packagecloud-web

Credentials

Section titled “Credentials”

Package key

Section titled “Package key”

The package key is used for the pre-release repo which is used for all GitLab deployments that pull packages from the gitlab/pre-release repository.

We do not let the wider community pull from this repo because GitLab.com production and non-production environments use it for testing security updates and unreleased builds before they are released.

Key rotation

Section titled “Key rotation”

Follow this process to rotate the pre-release token for an environment:

  1. Visit https://packages.gitlab.com/gitlab/pre-release/tokens

  2. Under Custom Master Tokens, click Revoke on the token that corresponds to the environment you want to rotate the token for. This will render the token unusable by deleting the token!

  3. Click Create Master Token and enter the environment name in the Master Token Name field

  4. Copy the new token value

  5. Visit https://vault.gitlab.net/ui/

  6. Go to the following secrets path for the environment in question: env/<env>/shared/gitlab-omnibus-secrets

  7. Click on Create new version

  8. Click on the JSON toggle

  9. Look for the path omnibus-gitlab.package.key and update the value with the new token.

NOTE: when the key gets rotated, the source file does not get updated before the first apt upgrade. This will cause chef runs to fail unless it is manually deleted (issue).

The following knife command can workaround the issue:

Terminal window
knife ssh -C10 "recipes:omnibus-gitlab\\:\\:default" "sudo rm -f /etc/apt/sources.list.d/gitlab_pre-release.list"

Running chef-client once the file has been deleted should yield a successful run.