Connecting to a Cell's Toolbox Pod

This guide covers how to connect to a cell’s toolbox pod for debugging and troubleshooting purposes.

Caution

Connecting to a toolbox pod and making manual changes should be a last resort. Always prefer making changes through Infrastructure as Code (IaC) or automated processes.

Prerequisites

For GCP Cells

Before connecting to a toolbox pod, you need:

1. PAM access to the cell’s GCP project - Follow the breakglass access guide to request the appropriate level of access:

   • Use project_read for read-only investigation
   • Use project_admin for write access
   • Use project_breakglass during incidents with low team availability

2. Cluster credentials - Once PAM access is granted, get the cluster credentials:

   TENANT_ID="<tenant_id>"
   
   gcloud container clusters get-credentials ${TENANT_ID} \
     --region us-east1 \
     --project cell-${TENANT_ID}

   This will update your kubeconfig and set the current context to the cell’s cluster.

For AWS Cells

1. Login to Hub Account - Run assume-env-local.sh in the AMP project

   AMP_ENVIRONMENT=<environment>;eval "$(./scripts/assume-env-local.sh "${AMP_ENVIRONMENT}")"

   This will open a prompt to sign-in using SSO, select Confirm and continue and then Allow access.

2. Assume role - Run assume-tenant-admin-operator.sh to assume dedicated/GitLabTenantAdminOperatorRole IAM role in the target tenant account

   eval $(./scripts/assume-tenant-admin-operator.sh <TENANT_ACCOUNT_ID>)

3. Cluster credentials - update kubeconfig with the credentials using the TENANT_ID

      aws eks update-kubeconfig  --region us-east-1 --name <TENANT_ID>

Note

Both the TENANT_ID and the TENANT_ACCOUNT_ID can be found in the cell’s tenant model in rings/<AMP_ENVIRONMENT>/<RING_NUMBER> folder in Tissue

Connecting to the Toolbox Pod

Find the Toolbox Pod

List the toolbox pods in the default namespace:

kubectl get pods -l app=toolbox

Example output:

NAME                              READY   STATUS                  RESTARTS      AGE
gitlab-toolbox-8c6bff946-8lt4v    1/1     Running                 0             5d

Execute a Shell in the Toolbox Pod

Execute a bash shell in the toolbox container:

kubectl exec -it <toolbox-pod-name> --container toolbox -- bash

Example:

kubectl exec -it gitlab-toolbox-8c6bff946-8lt4v --container toolbox -- bash

You should now have a shell inside the toolbox pod:

git@gitlab-toolbox-8c6bff946-8lt4v:/$

Common Operations

Check Migration Status

View the status of all database migrations:

gitlab-rails db:migrate:status

Filter for specific migrations:

gitlab-rails db:migrate:status | grep -E '<migration_version>'

Run Migrations Manually

Caution

Running migrations manually should only be done as a last resort when automated deployment migrations fail and you need to unblock deployments.

Use the db-migrate script to execute migrations in the correct order (regular migrations first, then post-deployment migrations):

./scripts/db-migrate

This script will:

• Check if migrations are up-to-date
• Run regular migrations (db:migrate)
• Run post-deployment migrations
• Perform custom instance setup

Note

For common kubectl operations, see the Kubernetes documentation.

Troubleshooting

Cluster Not Running Warning

If you see a warning that the cluster is not running:

WARNING: cluster c01j98xncnbxvaqayp is not RUNNING. The kubernetes API may or may not be available.

This may indicate the cluster is being upgraded or is in maintenance. Check the cluster status in the GCP/AWS console or wait for the operation to complete.

No Toolbox Pods Found

If no toolbox pods are found, check:

1. You’re in the correct namespace (default is default)
2. The toolbox deployment exists: kubectl get deployments -l app=toolbox
3. Check for any deployment issues: kubectl describe deployment <toolbox-deployment>