Kubernetes runner managers

Part of the runner fleet’s managers run as Kubernetes Deployments on GKE, managed by ArgoCD. Nothing on this page involves Chef, knife, or SSH to a manager: the manager is a pod, its configuration is Helm values in git, and all changes go through merge requests.

The ephemeral job VMs are unchanged: the manager pod runs the same docker+machine executor and provisions the same COS VMs as the Chef-based managers. Everything in the debugging guide about job VMs and their connectivity still applies once you are inside the manager pod.

Background and rationale live in the design document and the central cluster epic.

Layout

One manager cluster per environment. All manager pods for an environment run on it; the ephemeral VMs stay in their per-project networks and never move.

Cluster	Project	Environment label
runner-managers-gprd-1	gitlab-ci-155816	gprd-ci

Regional in us-east1, on the ci network, defined as explicit resources in config-mgmt/environments/ci/gke-runner-managers.tf. Its ArgoCD cluster secret carries gitlab.com/runner-cluster: "true", which services use to exclude runner clusters without listing them.

A shard is a directory under services/gitlab-runner/env/gprd-ci/shards/, and each shard spans one or more ephemeral VM projects. One manager Deployment exists per shard-project pair:

Shard	Runner	VM projects
p-s	k8s.private...gitlab.com/gitlab-org	gitlab-r-saas-l-p-amd64-{1..3}
p-m	k8s.private...gitlab.com/gitlab-org/medium	gitlab-r-saas-l-p-amd64-{1..3}
p-l	k8s.private...gitlab.com/gitlab-org/large	gitlab-r-saas-l-p-amd64-{1..3}
p-com	k8s.private...gitlab.com/gitlab-com	gitlab-r-saas-l-p-amd64-{1..3}
l-s-amd64	k8s.saas-linux-small-amd64...default	gitlab-r-saas-l-s-amd64-1

The private shards keep their deployed shorthand names. New shards are named after their ephemeral project family minus the saas- prefix (l-s-amd64, l-m-arm64, …).

What	Where
Namespace	gitlab-runner
Deployments	one per shard-project: gitlab-runner-<shard>-<project>
ArgoCD service	services/gitlab-runner/ in argocd/apps
Shared objects	services/gitlab-runner-commons/
Values	services/gitlab-runner/env/gprd-ci/shards/<shard>/ and .../clusters/<cluster>/
ArgoCD UI	https://argocd.gitlab.net/ (applications gitlab-runner--<shard>--<project>--<cluster hash>)
Chart	gitlab-runner Helm chart, version pinned per shard-cluster in projects.yaml

Unlike most services, gitlab-runner is not rendered by the generic-services ApplicationSet. The argocd-runner-managers chart (rendered by applications/gitlab-runner.yaml) creates three ApplicationSets:

• per project: shard-cluster directories with a projects.yaml get one runner Application per project, 1 through projectCount. The TOML derives all GCP fields from the project number (google-project={{ .Values._projectPrefix }}{{ .Values._project }}), so a shard’s projects share one values file and adding a project is a one-line count bump.
• per shard: one secrets Application per shard-cluster (gitlab-runner--<shard>--secrets--<hash>) renders the shard’s ExternalSecrets, independent of the project count.
• per cluster: directories with a legacy app.yaml instead of a projects.yaml get the old one-Application-per-cluster shape. The deployer and gvisor-* shards in env/ci still use it.

Cluster-shared objects are owned by the gitlab-runner-commons service (a normal generic service): the runner-manager ServiceAccount all manager pods run as, the gitlab-runner-secrets SecretStore, the external-secrets ServiceAccount and RBAC, and the registry pull secret. No shard’s Application owns anything another shard needs.

Each shard-project deployment runs several replicas that share one runner token, so they appear as a single runner in GitLab with multiple system_ids. Capacity settings (concurrent, limit, IdleCount) are per replica; deployment capacity is the per-replica value times replicas. For what the settings mean and how to size them, see the sizing guide in the epic description.

Mapping a job to a manager pod

The job log names the runner:

Running with gitlab-runner 18.5.0~pre... on
k8s.private.runners-manager.gitlab.com/gitlab.com/gitlab-org/medium Jn8J2S38P, system ID: r_...

The runner name maps to a shard via the table above (this one is p-m), and the VM name in the job log carries the project number. To find the specific pod for a system_id, check the system_id label on the runner metrics or grep the pods’ logs.

Access and logs

glsh kube use-cluster runner-managers-gprd-1

kubectl get pods -n gitlab-runner
kubectl logs -n gitlab-runner deploy/gitlab-runner-p-s-1 --since=1h

Logs are JSON and also ship to Elasticsearch through the same pipeline as the VM fleet (pubsub-runner-inf-gprd-*, filter on json.shard).

Debugging ephemeral VM creation

The equivalent of running docker-machine on a manager VM is exec-ing into the manager pod. Machine state and certificates live in the pod filesystem:

POD=$(kubectl get pods -n gitlab-runner -l app=gitlab-runner-p-s-1 -o name | head -1)
kubectl exec -it -n gitlab-runner ${POD#pod/} -- sh

# inside the pod
docker-machine ls
docker-machine ssh runner-xxxxxxxx-private-...

From inside a job VM the connectivity checks from the debugging guide apply unchanged.

Every pod runs a boot_verify canary at startup: it creates one VM end-to-end before the readiness endpoint (/health/ready on port 9252) reports healthy, and the Deployment’s startupProbe gates on it. A pod stuck 0/1 Running with restarts means VM provisioning is broken; check its logs for the failing create step before anything else.

Ready does not mean polling

boot_verify proves the manager can provision a VM and run a job on it. Job polling is a different endpoint (ci-gateway) through a different network path, and it is not covered: a manager with a broken path to ci-gateway reports Ready indefinitely while every job request times out. If a Ready manager runs no jobs, check its logs for job request errors before suspecting capacity or queueing.

Auth failures look like missing projects

The pods authenticate to GCP through Workload Identity: they all run as the runner-manager ServiceAccount (owned by gitlab-runner-commons), and the GCP-side binding lives in config-mgmt/environments/ci/runner-managers.tf. A missing binding does not produce a permission error: GCP returns Project with ID "..." not found on the pre-create check. If a new or renamed deployment fails like this, check the binding before anything else.

Configuration changes, deploys, rollbacks

All configuration lives under services/gitlab-runner/env/gprd-ci/shards/<shard>/:

• values.yaml (shard level): everything shared across clusters
• clusters/<cluster>/values.yaml: replicas, resources, and the full config.toml as runners.configOverride
• clusters/<cluster>/projects.yaml: chart version pin, projectCount, projectPrefix
• clusters/<cluster>/values-vault-secrets.yaml: the shard’s ExternalSecrets

runners.configOverride is templated by the runner chart, which is how the per-project GCP fields work. envVars values are not templated; a template expression there ships to the pod literally.

Change process is a merge request. After merging, the ApplicationSet’s progressive RollingSync rolls the change out automatically; expect minutes of delay, or sync the application in the ArgoCD UI to expedite. Rollback is a revert MR.

Do not edit resources with kubectl: ArgoCD reverts drift on the next sync, and git stops matching what actually runs.

Deploys rotate pods with maxSurge: 100% / maxUnavailable: 0 and long termination grace periods sized to each shard’s max job timeout. A deploy therefore keeps the old generation draining for hours; both generations polling at once is normal, as are 409 conflicts on jobs/request (replicas racing on the shared token).

The manager image is pinned by digest-tagged version in values. Never use the floating bleeding tag: it derives a helper image tag that is never published, which breaks the boot_verify canary.

Secrets

Runner tokens and the Sentry DSN come from Vault via External Secrets:

• Tokens are per shard and shared by all of the shard’s deployments. The Vault paths still carry the legacy environment names (k8s/env/r-saas-l-p-amd64/gitlab-runner/* for private shards, k8s/env/r-saas-l-s-amd64/... for saas-small); check the shard’s values-vault-secrets.yaml for the path and property.
• The shard’s secrets Application renders them into gitlab-runner-<shard>-secret, using the SecretStore and external-secrets plumbing owned by gitlab-runner-commons.

Seeding a new shard’s token is a manual step before its first deploy.

ExternalSecrets with refreshInterval: 0 never re-fetch: a changed remoteRef (new path, new property) needs a manual delete of the target secret before it picks up the new value. The runner secrets use 15m, but the gotcha applies to anything copied from older definitions.

Health signals

• Pod ready = boot_verify passed. kubectl get pods -n gitlab-runner. But see “Ready does not mean polling” above.
• gitlab_runner_jobs vs gitlab_runner_limit per instance: pinned at limit means the deployment is saturated (for saas-small, spilling to the remaining Chef fleet is harmless while it exists; the shared queue routes around it).
• gitlab_runner_worker_processing_failures_total{failure_type="no_free_executor"}: poll attempts that found no warm machine. Routine at low rates; sustained high rates during steady state mean the idle pool cannot keep up.
• gitlab_runner_autoscaling_machine_states: warm pool depth per state.
• Shard-level saturation: gitlab_component_saturation:ratio{component="private_runners"} (jobs over limits summed across the shard, VM and k8s managers together).

Metrics flow through the monitoring stack on the cluster to the gitlab-gprd Mimir tenant, stamped with the same labels as the VM fleet (environment=gprd, shard, worker, plus cluster). k8s-monitoring-stack rolls out through RollingSync like the runner service; sync it in the UI to expedite a scrape config change.

Adding a project to a shard

The cheap operation, once the VM project itself exists (quota, CIDRs, network, env-projects module: the project half of provisioning a new shard is unchanged).

1. Bump projectCount in the shard’s projects.yaml.
2. For shards still sharing load with Chef managers (saas-small), pair the bump with the corresponding Chef drain; project totals stay level.
3. Verify the new deployment before relying on it: a real job completed, Mimir series carry shard and worker labels, logs in Elasticsearch.

Adding a shard

1. Seed the shard’s runner token in Vault.
2. New directory under services/gitlab-runner/env/gprd-ci/shards/, named after the ephemeral project family minus saas-. Copy the closest existing shard and diff every setting against its current source of truth (the Chef shard config, or the donor shard), justifying each difference. Stale copies have reintroduced fixed bugs on every migration wave.
3. projects.yaml with the chart version, projectPrefix, and projectCount: 1, plus the token ExternalSecret in values-vault-secrets.yaml.
4. Start with concurrent = 1 and limit low, verify (real job, metrics labels, logs, vector and wiz clean), then raise to sized capacity.

No infrastructure work: the ServiceAccount, SecretStore, Workload Identity, vector, and monitoring all exist at cluster level.

Adding a cluster

The rare operation: a new environment (gstg, ops, dev) or a second gprd cluster.

1. config-mgmt (environments/ci): subnet with secondary ranges, cluster and node pools (copy gke-runner-managers.tf), the pods range in the docker-machine firewalls and in the environment’s ci-gateway firewall, a vector account for the environment’s log topic. Keep node pool names short: GKE truncates cluster+pool in node names, which breaks node-family selectors.
2. config-mgmt again, after the cluster applies: the vault auth entry in environments/vault-production (it reads the cluster-info secret, which exists only once the cluster does). Clusters in a new GCP project also need AppProject destinations in argocd/config (the existing ones wildcard on the project) and a runner-manager Workload Identity member (the binding is per workload pool, so same-project clusters are already covered).
3. runbooks: the new ranges (nodes and services /20s, a pods range) in the subnet allocations registry, and a kubernetes/clusters.json entry for glsh kube use-cluster.
4. argocd/apps: an app.yaml for gitlab-runner-commons under the new cluster, then per shard a projects.yaml (and values file if anything differs from the shard level).
5. Verify before ramping: a real job completed, job polling works, Mimir series carry cluster, shard, and worker labels, logs queryable in Elasticsearch, vector and wiz error-free. Each of these has silently failed at least once. Vector drops what it cannot publish, so log gaps do not backfill.

Removing a deployment or cluster

Order matters; every step below has been learned the hard way.

1. Scale to zero first (replicas: 0), do not delete: pods drain their jobs and delete their machines on shutdown, and rollback is a revert.
2. Delete the Applications before deleting their files. The ApplicationSets run applicationsSync: create-update: removing a values file orphans the Application, which then renders chart defaults (3 replicas, no config) and crashloops. Cascade-delete the apps, then merge the file removal.
3. Cluster teardown in config-mgmt: keep the kubernetes provider alive on a data google_container_cluster source for the destroy apply (removed blocks cannot forget resources inside an implicitly-destroyed module), and remove it in a follow-up. On Shared VPC, delete clusters before removing their host-project IAM: the GKE robots need container.hostServiceAgent.use on the host project to delete, not just to create. Subnets delete only after nodes and the control plane’s PSC endpoints are gone; a failed subnet delete right after cluster deletion is usually a race against GKE’s async cleanup, retry.
4. Sweep orphaned VMs after the cluster is gone: filter on the runner_manager_name label, which carries the cluster name. Do not sweep earlier; see the zombie pitfall below.

Pitfalls

• Never force-kill a runner pod. The termination grace period is the job drain, and the pod deletes its machines on shutdown. Force-killing orphans the VMs and kills the jobs on them.
• A pod stuck Terminating past its grace period with an unreachable kubelet is not a stuck deletion. The shutdown signal may never have been delivered: manager pods in that state have kept polling, running jobs, and creating VMs for a day while Kubernetes showed them Terminating. Check the VM project for machines with recent creation timestamps and the pod’s manager label before treating it as dead weight.
• RollingSync-triggered syncs prune with syncPolicy.automated.prune even when automated sync is disabled. The runner service sets prune: false explicitly for this reason; do not remove it, and do not rely on “automated sync is off” to protect shared resources anywhere else.
• One ownership conflict freezes the whole ApplicationSet. If an Application already exists with a different owner, the ApplicationSet stops reconciling all of its Applications, not just that one. Delete the conflicting Application and kick the ApplicationSet with an annotation.
• A deletion stuck on an in-flight sync can be cleared by removing the operation: kubectl -n argocd patch application <name> --type json -p '[{"op":"remove","path":"/operation"}]'.
• Services that must not run on runner clusters exclude the gitlab.com/runner-cluster label (DoesNotExist) instead of maintaining name lists; the label exists from the moment the cluster registers, so there is no window for stray Applications.
• Manager node pools use autoscaled blue-green upgrades with safe-to-evict: "false" pods: node drains wait for pod rotation instead of evicting mid-job. Do not force-drain manager nodes; jobs up to the full grace period are running on the pods.
• google-flex-selection requires google-bulk-insert. The error surfaces only on real VM creation, so without boot_verify a manager with this mismatch reports Ready and then fails every job it takes.
• Monitoring failures are silent by default: an empty relabel capture group drops labels without errors, and vector publishes to nonexistent pubsub topics with only pod-log errors to show for it. After any change to the monitoring values, confirm the series and labels actually arrive.
• ApplicationSet generator behaviour differs across ArgoCD versions (selectors especially). Verify chart changes against the deployed controller version, not the docs; the diff preview is the integration test.