GitLab.com on Kubernetes

The following is a high level guide on what it takes to build out the necessary bits for adding GKE and bringing over components of GitLab into Kubernetes.

Our current application configuration components:

For GitLab.com there is a regional cluster and multiple zonal clusters to service traffic for each environment. This document covers how to build a new cluster, note that currently this procedure is not automated and may take hours to complete.

Provision the cluster in Terraform

Three modules create the IP reservations needed for monitoring and ingress, the cluster and node pools, and external DNS, see this example for one of the zonal clusters or the full regional cluster terraform configuration.
Set IAM user permissions on cluster
- This is manual, documented here: https://gitlab.com/gitlab-com/gl-infra/k8s-workloads/gitlab-com/blob/master/README.md#service-account
- The service account will have an authentication key file in json format created, we’ll need this for the next step.
Set the appropriate environment variables in the application configuration repositories:
- Repositories:
- ENV Vars:
  - SERVICE_KEY
  - This key is gathered from following the documentation in the previous step and must be added to each repo since environment scoped group level variables are not a feature of GitLab
  - Environment names use wildcards to cover the zonal clusters, which are prefixed with gprd or gstg.

Configure gitlab-helmfile

See bootstrapping new clusters for how to apply the initial set of helm charts on the cluster.

Prometheus rules

New Clusters

Inside of our runbooks repo, we need to add a configuration inside of .gitlab-ci.yaml to deploy to our new cluster.
Ensure the appropriate variables are added to the ops instance Utilize this MR as a guideline: https://gitlab.com/gitlab-com/runbooks/merge_requests/1200

Replacement Clusters

Go to the runbooks CI jobs, find the latest green pipeline on our default branch, then find the job associated with the existing cluster and retry it
Check the CI output, we should see the addition of our custom resources required by our various Prometheus components
- This includes various service monitors, alert rules, and prometheus rules

Thanos configuration

Thanos query needs to know about the prometheus endpoints, these are set in the ops-base.json chef role

Configure gitlab-com

See bootstrapping new clusters for how to apply the gitlab helm chart on the cluster.