Skip to content
Runbooks

Cell Provisioning and De-Provisioning

Prerequisites

Section titled “Prerequisites”

How to Provision a new Cell

Section titled “How to Provision a new Cell”

ringctl uses adaptive deployment strategy, i.e it takes reference from the exisiting tenant_model from a target_ring and replace just the necessary fields.

[!note] For most cases you would want to take reference from the ring 0.

Provisioning Steps

Section titled “Provisioning Steps”

  1. Create a new TENANT_MODEL from the target ring:

    Terminal window
    ringctl cell provision --dry-run --ring 0 --amp_environment <cellsdev|cellsprod> --cell_id <cell_id>
Example Run
Terminal window
ringctl cell provision --dry-run --ring 0 --amp_environment cellsdev --cell_id 134
Terminal window
[DRY-RUN] CreateCommit - action: create - path: rings/cellsdev/-1/c01jpmm6vgv82xqkf9.json
{
  "$schema": "https://gitlab-com.gitlab.io/gl-infra/gitlab-dedicated/tenant-model-schema/v1.62.0/tenant-model.json",
  "amp_gcp_project_id": "amp-b6f1",
  "audit_logging": false,
  "aws_account_id": "211125640907",
  "backup_region": "eu-west1",
  "byod": {
    "instance": "staging.gitlab.com"
  },
  "cells": {
    "cell_id": 134
  },
  "cloud_provider": "gcp",
  "cloudflare_waf": {
    "enabled": true,
    "migration_stage": "COMPLETE",
    "proxied": "NOT_PROXIED"
  },
  "dns_aws_account_id": "211125640907",
  "external_smtp_parameters": {
    "authentication": "login",
    "domain": "mg.staging.gitlab.com",
    "from": "[email protected]",
    "host": "smtp.mailgun.org",
    "pool": true,
    "port": 2525,
    "reply": "[email protected]",
    "starttls": true,
    "tls": false,
    "username": "[email protected]"
  },
  "gcp_oidc_audience": "//iam.googleapis.com/projects/1002415312824/locations/global/workloadIdentityPools/gitlab-pool-oidc-amp-1290/providers/gitlab-jwt-amp-1290",
  "gcp_onboarding_state_region": "us-east1",
  "gcp_project_id": "cell-c01jpmm6vgv82xqkf9",
  "gitlab_version": "17.7.0",
  "instrumentor_version": "v16.620.1",
  "internal_reference": "cell-c01jpmm6vgv82xqkf9",
  "managed_domain": "cell-c01jpmm6vgv82xqkf9.gitlab-cells.dev",
  "perform_qa": true,
  "prerelease_version": "17.10.202503180906-6a387d336a7.c1c3e017782",
  "primary_region": "us-east1",
  "reference_architecture": "ra3k_v3",
  "reference_architecture_overlays": [],
  "sandbox_account": false,
  "service_account_impersonation_members": [],
  "site_regions": [
    "us-east1"
  ],
  "tenant_id": "c01jpmm6vgv82xqkf9",
  "use_gar_for_prerelease_image": true
}

  1. Create a Merge Request (MR) in cells/tissue and get it merged:

    • Use the TENANT_MODEL JSON output from the previous command
    • Important: Always place the cell initially in the quarantine ring (-1 folder) regardless of the target ring
    • Ensure the filename matches <tenant_id>.json

  2. Trigger the Instrumentor Stages for cell provisioning:

    Terminal window
    ./ringctl cell deploy -e <cellsdev|cellsprod> <tenant_id> --only-gitlab-upgrade=false

    [!note] This command will output a link to the cell pipeline which you can use to track the deployment progress. The pipeline will run all the Instrumentor stages to create your cell.

    [!warning] The prepare stage might fail on the initial run with an error like:

    Error when reading or editing KMSKeyRing "projects/cell-c01jp4m0711mwrhb8j/locations/us-east1/keyRings/gld-single-region": googleapi: Error 403: Google Cloud KMS API has not been used in project 9065488916 before or it is disabled. Enable it by visiting <https://console.developers.google.com/apis/api/cloudkms.googleapis.com/overview?project=9065488916> then retry. If you enabled this API recently, wait a few minutes for the action to propagate to our systems and retry.

    This is expected as the cell project was just created and Google is still enabling the API. Retry the job after approximately 10 minutes to resolve this issue.

  3. Access your new cell:

    • Once the pipeline completes successfully, access your cell through the domain specified in the managed_domain field of the TENANT_MODEL
    • Example: cell-c01jpmm6vgv82xqkf9.gitlab-cells.dev

Moving Cells to Rings

Section titled “Moving Cells to Rings”

Moving Cells between Rings is a logical operation that only impacts when that Cell will receive patches. Cells in Quarentine (-1) Ring do not recieve any automation. All other Rings will receive operations in order.

Moving A Cell into Quarantine

Section titled “Moving A Cell into Quarantine”

  1. Using ringctl:

    Terminal window
    % ringctl cell quarantine <cell-id> --ring <ring-id> -b <branch-name>
Example Run
Terminal window
% ringctl cell quarantine c01j2t2v563b55mswz --ring 0 -b jts/test
time=2025-04-01T15:49:04.424-04:00 level=WARN msg="DELIVERY_METRICS_URL or DELIVERY_METRICS_TOKEN not set, disabling metrics" DELIVERY_METRICS_URL=""
time=2025-04-01T15:49:04.424-04:00 level=WARN msg="using default text icons; please select your preferred set of icons and store the value in the ringctl.yml file"
time=2025-04-01T15:49:04.530-04:00 level=INFO msg="tissue client initialized" instance=https://ops.gitlab.net branch=jts/test dry_run=false amp=cellsdev project=gitlab-com/gl-infra/cells/tissue version=dev-g92b52bf-dirty local=false
time=2025-04-01T15:49:04.864-04:00 level=WARN msg="branch not found, searching in default branch" branch=jts/test default_branch=main
New branch: jts/test
You can open a merge request visiting
      https://ops.gitlab.net/gitlab-com/gl-infra/cells/tissue/-/merge_requests/new?merge_request%5Bsource_branch%5D=jts%2Ftest
time=2025-04-01T15:49:06.670-04:00 level=INFO msg="cell operation" ring=-1 amp=cellsdev action=move url=https://ops.gitlab.net/gitlab-com/gl-infra/cells/tissue/-/commit/6ac6e3fca1b34cbc1022c966e4f7a2028bf5899e
time=2025-04-01T15:49:06.671-04:00 level=INFO msg="Successfully quarantined cell" cell_id=c01j2t2v563b55mswz
  1. Open the Merge Request using the provided log output containing a link
  2. Obtain review, approval, and merge the MR.

Moving a Cell out of Quarantine

Section titled “Moving a Cell out of Quarantine”

This is currently a manual process.

  1. Interrogate the target cell configuration and the target Ring destination
  2. Validate the version of Instrumentor is the same.
  3. Validation the configuration of the cell is the same.
  4. Validate the version of GitLab is installed, is the same.
  5. If any of the above differ, create a patch to address any concerns.
  6. If the target cell is in a sane state, create an MR which moves the JSON file from the -1 directory to the target Ring directory
  7. Obtain review, approval and merge the MR.

How to De-Provision a Cell

Section titled “How to De-Provision a Cell”

  1. Ensure the target cell is in the quarantine ring:

    • If not already there, create an MR to move the cell definition to the -1 folder (quarantine ring)
    • Get the MR approved and merged

  2. Trigger the tear-down pipeline:

    Terminal window
    ringctl cell deprovision --cell <tenant_id> --ring -1 -e <cellsdev|cellsprod>

    [!note] This command will output a link to the cell pipeline which you can use to track the tear-down progress.

  3. Remove the cell definition:

    • After the tear-down pipeline completes successfully, create an MR to delete the cell definition file from cells/tissue
    • Get the MR approved and merged