Skip to content
Runbooks

Auto DevOps

Auto DevOps is a reusable CI/CD configuration. Auto DevOps can be used in place of a .gitlab-ci.yml, or as an addition to an existing .gitlab-ci.yml.

Auto DevOps bugs typically cause pipeline failures, but, in extreme cases, can also result in broken user applications.

Auto DevOps is vendored together with the GitLab Rails application code. Fixes to Auto DevOps configuration always require a full rollout of the GitLab Rails application. But in the interim, bad image updates to Auto Build and Auto Deploy can be temporarily overridden using instance level CI/CD variables.

Quickly mitigate a bad update to Auto Build or Auto Deploy images

Section titled “Quickly mitigate a bad update to Auto Build or Auto Deploy images”

A bad image update can be reverted without a full rollout of the Rails application code, by setting instance-level CI/CD variables.

Symptoms

Section titled “Symptoms”
  • Numerous reports of failing Auto Build jobs following an update to AUTO_BUILD_IMAGE_VERSION
  • Numerous reports of failing Auto Deploy jobs following an update to AUTO_DEPLOY_IMAGE_VERSION or DAST_AUTO_DEPLOY_IMAGE_VERSION

Pre-checks

Section titled “Pre-checks”
  • Verify that the affected component has been recently updated
  • Reproduce the problem or otherwise confirm it. If in doubt, contact #f_autodevops on Slack (internal link) for help.

Resolution

Section titled “Resolution”
  1. Identify a good image version:
    • AUTO_DEPLOY_IMAGE_VERSION and DAST_AUTO_DEPLOY_IMAGE_VERSION:
      • If the problem was caused by an update, use the previously version in Jobs/Deploy.gitlab-ci.yml.
      • In rare cases, a fix may need to be implemented in auto-deploy-image. For example, if a hardcoded remote goes permanently offline, then every released version will be equally broken.
    • AUTO_BUILD_IMAGE_VERSION:
      • If the problem was caused by an update, use the previously used version in Jobs/Build.gitlab-ci.yml.
      • In rare cases, a fix may need to be implemented in auto-build-image (e.g. hardcoded remote going offline).
  2. Set instance CI/CD variables to override the default versions of the affected image.
    • Navigate to instance level CI/CD settings and expand the Variables section.
    • The version overrides should not be protected, and not be masked
    • For Auto Deploy, set AUTO_DEPLOY_IMAGE_VERSION and DAST_AUTO_DEPLOY_IMAGE_VERSION to the chosen version
    • For Auto Build, set AUTO_BUILD_IMAGE_VERSION to the chosen version
  3. Open an MR against the CI/CD template with the fix
  4. Once updated code fully rolled out to gprd, unset the CI/CD variables.
    • Navigate to instance level CI/CD settings and expand the Variables section.
    • For Auto Deploy, remove AUTO_DEPLOY_IMAGE_VERSION and DAST_AUTO_DEPLOY_IMAGE_VERSION
    • For Auto Build, remove AUTO_BUILD_IMAGE_VERSION

Post-checks

Section titled “Post-checks”
  • If you were able to reproduce the problem, you should be able to reproduce the fix
  • Double check that the overrides have been removed from the Variables section in thethe override has instance level CI/CD settings

Rollback

Section titled “Rollback”