Vault Administration

Adding a GitLab instance to Vault

In order to enable authentication to Vault from CI for a GitLab instance, add it to the jwt_auth_backends map in environments/vault-production/vault_config.tf:

module "vault-config" {
  [...]

  jwt_auth_backends = {
    [...]

    ops-gitlab-net = {
      description  = "GitLab CI JWT for ops.gitlab.net"
      jwks_url     = "https://ops.gitlab.net/-/jwks"
      bound_issuer = "https://ops.gitlab.net"
    }
  }

  [...]
}

Terraform will then configure Vault with the JWT authentication method and some default policies for this GitLab instance.

Adding a Kubernetes cluster for authentication and the External Secrets Operator

Master access IP allowlisting

The ops-gitlab-gke cluster (which is hosting the Vault service) has to be allowed to connect to the target cluster to be able to do the Service Account verification. This can be done by adding the named IP addresses gitlab-gke-01 and gitlab-gke-02 from the gitlab-ops project to the authorized_master_access parameter of the GKE cluster module, see this merge request for an example.

Kubernetes Authentication secrets

To be able to authenticate into a Kubernetes Cluster with Vault, a service account with the necessary permissions needs to be installed to enable Vault to dynamically create temporary service accounts and roles in the cluster. This can be done by installing the Helm chart vault-k8s-secrets:

environments:
  pre:
    values:
      - [...]
        vault-k8s-secrets:
          installed: true

This will also create a secret vault-k8s-secrets-token which will be used when configuring Vault below.

See the Vault documentation about the Kubernetes secrets engine for more information.

External Secrets operator

The External Secrets operator can be installed in a Kubernetes cluster by enabling it in bases/environments.yaml in the gitlab-helmfiles repository:

environments:
  pre:
    values:
      - [...]
        external_secrets:
          installed: true
          # renovate: datasource=helm depName=external-secrets registryUrl=https://charts.external-secrets.io versioning=helm depType=pre
          chart_version: 0.7.0

See the Vault documentation about the Kubernetes authentication method for more information.

Vault configuration

The cluster information must be saved in a Vault secret that will be used by Terraform to configure the Kubernetes authentication method and/or the Kubernetes secrets engine for this cluster.

If the cluster is provisioned by config-mgmt then use the following module along with it:

module "gitlab-gke-vault-cluster-info" {
  source = "../../modules/vault-gke-cluster-info"

  environment                = var.environment
  gke_cluster_name           = module.gitlab-gke.cluster_name
  gke_cluster_ca_certificate = module.gitlab-gke.cluster_ca_certificate
  gke_cluster_endpoint       = module.gitlab-gke.cluster_endpoint
}

If not, you will need to save the cluster information manually instead:

KUBERNETES_ENDPOINT="$(kubectl config view -o jsonpath='{.clusters[?(@.name == "gke_gitlab-pre_us-east1_pre-gitlab-gke")].cluster.server}')"
CA_CERT="$(kubectl config view --raw -o jsonpath='{.clusters[?(@.name == "gke_gitlab-pre_us-east1_pre-gitlab-gke")].cluster.certificate-authority-data}' | base64 -d)"

vault kv put shared/kubernetes/clusters/pre/pre-gitlab-gke endpoint="${KUBERNETES_ENDPOINT}" ca_cert="${CA_CERT}"

If the vault-k8s-secrets chart has been installed to enable authentication into the cluster via Vault, you will also need to save its JWT under a different path:

JWT_TOKEN="$(kubectl --namespace vault-k8s-secrets get secret vault-k8s-secrets-token -o jsonpath='{.data.token}' | base64 -d)"

vault kv put ci/ops-gitlab-net/gitlab-com/gl-infra/config-mgmt/vault-production/kubernetes/clusters/pre/pre-gitlab-gke service_account_jwt="${JWT_TOKEN}"

Finally, add the cluster in environments/vault-production/kubernetes.tf:

locals {
  [...]

  kubernetes_clusters = {
    [...]

    pre-gitlab-gke = {
      environment               = "pre"
      enable_kubernetes_secrets = true

      auth_roles    = {}
      secrets_roles = {}
    }

    [...]
  }
}

Terraform will then configure Vault with the Kubernetes/JWT authentication method and some default policies for this cluster.