Kubernetes on-call setup

Complete these steps before your first on-call shift with cluster access.

Cluster config lives across a handful of repositories — see Where things live in the Kubernetes overview.

CI jobs on ops.gitlab.net apply changes. Deployer changes some cluster state outside git — the CI job that applied a given change is your audit trail.

Kubernetes API access

Our GKE clusters are private. Access requires either the corporate VPN or an SSH tunnel via bastion.

Prerequisites

1. Set up Yubikey SSH keys.
2. Set up bastion access.
3. Install gcloud: https://docs.cloud.google.com/sdk/docs/install-sdk.
4. Install the GKE auth plugin: gcloud components install gke-gcloud-auth-plugin. See Bind kubectl to GKE.
5. Install kubectl — macOS, Linux.
6. gcloud auth login.

If you see permission warnings under ~/.config/gcloud/, fix ownership: sudo chown -R "$(whoami)" ~/.config.

Option A: VPN

1. glsh kube setup --no-proxy — writes kubeconfig entries for all clusters without a proxy. See glsh.
2. Launch NordLayer and connect to one of the organization gateways — not to a shared gateway.
3. Pick a cluster: glsh kube use-cluster lists them; glsh kube use-cluster gstg --no-proxy selects one. kubectx also works.
4. Confirm: kubectl get nodes.

Option B: SOCKS5 proxy via SSH

1. glsh kube setup — writes kubeconfig entries with the SOCKS5 proxy configured.
2. glsh kube use-cluster gstg.
3. In a separate window, kubectl get nodes.

GUI and terminal UIs

Optional but handy:

• k9s — terminal UI over the local kubeconfig; fast for browsing pods, logs, events, and resource usage during an incident.
• Lens IDE — desktop GUI over the local kubeconfig.
• The GKE console shows most of the same information in a browser.

Metrics and dashboards: https://dashboards.gitlab.net/dashboards/f/kubernetes/kubernetes.

Shell access to nodes and pods

Node access via IAP

kubectl get pods -o wide   # get the node name
gcloud compute --project gitlab-production ssh <node> --tunnel-through-iap

You may receive a SIRTBot notification. GKE COS nodes run containerd; use crictl for container operations:

crictl ps
crictl exec -u root -it <container> /bin/bash

Do not install packages directly on the node. Use toolbox (see k8s-operations.md) or an ephemeral debug container instead:

kubectl debug -it <pod> --image=busybox --target=<pod>

See Debug Running Pods.

Pod access

kubectl exec -n <namespace> -it <pod> -- sh

Some images lack a shell — use kubectl debug instead.

Local k-ctl for k8s-workloads/gitlab-com

k-ctl is a shell wrapper over helmfile used only by k8s-workloads/gitlab-com. You need it if you must render or apply the GitLab chart locally — typically only during an incident when CI is unavailable. Follow the setup instructions in that repository’s README.

All other platform workloads (Vault, Prometheus, Mimir, KEDA, cert-manager, and so on) are managed by ArgoCD via argocd/apps. Local helmfile invocations are not the operational path for those.

HAProxy state for zonal clusters

Draining a zonal cluster from HAProxy is destructive to that zone’s traffic. Use only in extreme circumstances (for example, evacuating traffic before a rebuild). Scripts live in chef-repo:

./bin/get-server-state gprd gke-us-east1-b
./bin/set-server-state -z b -s 60 gprd maint gke-us-east1-b

See k8s-cluster-rebuild.md for the full HAProxy drain procedure used during cluster rebuilds.

Troubleshooting

kubectl get nodes fails with connection refused

Bastion SSH is not set up. Follow bastions and yubikey, then re-run glsh kube setup.

kubectl fails with an auth error

Refresh credentials: gcloud auth login then glsh kube setup.

glsh is missing or out of date

Follow running helper scripts from runbook in the top-level README.