Patch Cell's Tenant Model

Overview

Changes to a Cell’s TENANT_MODEL are applied through a JSON6902 patch. Patches are applied from inner rings to outer rings, starting with ring 0 and proceeding outward sequentially.

If a patch application fails at any ring, the entire process halts and requires manual intervention.

Note

Patches are not applied to the quarantine ring (-1). To update the TENANT_MODEL of a Cell in the quarantine ring, open a Merge Request and update it directly.

Prerequisites

• Configure ringctl in your environment
• Ensure you have access to cells/tissue

Patch Operations

The following patch operations are available using ringctl:

Operation	Description	Parameters
add	Add a new field to the TENANT_MODEL	Target path, Value
replace	Replace a field’s value	Target JSON path, Value
remove	Remove a field	JSON path
move	Move a field to a new location	Source path, Destination path
copy	Copy a field to a new location	Source path, Destination path

Creating a Patch

Use the ringctl patch create subcommand with the desired operation.

Basic Examples

Update a single value:

ringctl patch create replace /instrumentor_version v16.xxx.x -e <cellsdev|cellsprod> --related-to "<issue_url>"

Combine multiple operations in one patch:

ringctl patch create add /use_gar_for_prerelease_image true replace /instrumentor_version v16.xxx.x --related-to <issue_id> --priority <3> -e <cellsdev|cellsprod>

Add a complex JSON structure:

ringctl patch create add "/byod" '{"instance": "gitlab.com"}' --related-to <issue_id> --priority <3> -e <cellsdev|cellsprod>

The command will create a Merge Request in cells/tissue. Get it reviewed and merged into main. Note the patch_id for tracking.

Checking Patch Status

View all patches and their statuses:

ringctl patch ls -e <cellsdev|cellsprod>

Note

Only one patch can be “In Progress” for a particular ring at any time. Patches run sequentially according to their priority. For urgent patches, consider setting priority to 3.

Each applied patch triggers the Instrumentor Stages pipeline. Find the pipeline on tissue by searching for your patch_id.

Deleting a Patch

Caution

A patch in the “in rollout” state cannot be stopped, as the Instrumentor stages pipeline is already running.

Delete a patch in the pending status

ringctl patch delete <patch_id> -e <cellsdev|cellsprod>

Delete a patch in the failed status

A failed patch might have been partially applied, so we need to revert the patch along with deleting the failed patch.

To revert a failed patch, make the following changes in a single MR:

• Delete the failed patch

  ringctl patch delete <patch_id> -e <cellsdev|cellsprod>

  This will create a branch on the remote and commit to it. Push the following changes to the same branch.

• Introduce a new patch reverting the attribute(s) that were changed in the failed patch to its previous value. You can determine the previous value of the attribute(s) by looking at the commit that applied the patch to the ring (example).

  Make sure you set the following attributes correctly:

  • Set the ring key in the new patch to the same ring where the previous patch failed.

  • Set the completed_after_ring key in the new patch to the same value as the ring key, so that the new patch is only applied to the ring where the previous patch failed.

  • Set a higher priority for the new patch, so that it is executed first, before any patches that are already in the queue. For example, if the first patch in the queue has a priority of 4, set the priority of the new patch to 3, so that it is placed at the front of the queue.

For an auto-deploy patch

For a patch that modifies only prerelease_version and has only_gitlab_upgrade: true, it is safe to just delete the failed patch file if the next patch in the queue is also an auto-deploy patch. If not, you can follow the above steps to revert the failed patch.

Retrying a failed patch

You can retry a patch that has failed after verifying that the failure will not repeat when retried. If the failure was due to something transient, it should be safe to retry.

If the failure was due to CI related flakiness, such as a “runner system failure” (example), you can retry the failed job itself.

If you would like to start a new pipeline to apply the patch, you can do the following. Note that this will add the patch to the back of the queue.

Create a new branch from the default branch:

git checkout main
git pull
git checkout -b retry-patch-<patch_id>

Update the patch file:

ringctl patch retry <patch_id> -e <cellsdev|cellsprod> --local

Push your branch, and create an MR. When the MR is merged, the patch will be re-queued.