Vault Secrets Management

Summary

Vault is an identity-based secret and encryption management system. It can be used as a central store to manage access and secrets across applications, systems, and infrastructure.

Vault Environments

We currently have two specific Vault instances setup to provide secrets to our infrastructure.

• https://vault.gitlab.net for all environments using secrets
• https://vault.pre.gitlab.net for testing configuration/deployments

Architecture

High Level Overview

graph TB

https://vault.$env.gitlab.net

subgraph GCP

  https://vault.$env.gke.gitlab.net
  IAP-Load-Balancer
  Internal-Clients
  Internal-Load-Balancer
  Vault-Raft-Snapshots-Bucket

  subgraph GKE-Vault-Deployment
    Vault
    Raft-PVC
    K8s-CronJob
  end

end

https://vault.$env.gitlab.net --> IAP-Load-Balancer --> Vault
Internal-Clients --> https://vault.$env.gke.gitlab.net --> Internal-Load-Balancer --> Vault
Vault --> Raft-PVC
K8s-CronJob --> Vault
K8s-CronJob --> Vault-Raft-Snapshots-Bucket

The application is deployed in Kubernetes using the official Vault Helm chart from Hashicorp.

• Helm chart deployment
• Internal and external ingress (see ingress section below)

Availability

We run Vault in High Availability mode. This consists of one active Vault server and several standby servers. Since we’re using the community version, standby instances are not unsealed and are only replicating the data but are not able to read it, and they will forward all requests to the leader instance. The standby instances only unseal when being promoted to leader.

We have enabled automatic unseal with GKMS. The unsealing process is delegated to Google KMS in the event of a failure. The Vault cluster will coordinate leader elections and failovers internally.

We have configured 5 replicas with spread constraints for pods to be in different hosts and be distributed in different zones, which gives us multi-zone failure tolerance across 3 different zones.

Raft storage is configured with regional SSD persistent disks which provide durable storage and replication of data between three zones in the same region.

Additionally, we have multi-region backups we can restore as disaster recovery (in case of full region failure) (see storage section below).

In short, we have redundancy across zones in the us-east1 region, but not across different regions. However we are able to easily and rapidly restore the service in a different region when needed.

Raft autopilot

Vault’s Raft autopilot is configured in environments/vault-production/vault_config.tf:

resource "vault_raft_autopilot" "autopilot" {
  cleanup_dead_servers               = true
  dead_server_last_contact_threshold = "12h0m0s"
  last_contact_threshold             = "10s"
  max_trailing_logs                  = 1000
  min_quorum                         = 3
  server_stabilization_time          = "10s"
}

Autopilot automatically prunes dead peers 12 hours after last contact and requires a stabilization period before a new server is promoted to voter. Combined with the vault-pvc-rotater CronJob this ensures nodes are recycled cleanly on a regular basis.

Ingress

We have one internal and one external endpoint, keeping the service from being directly exposed to the internet.

• External ingress for web user access is accessible through a GCP HTTPS load balancer that uses Google Identity-Aware Proxy (IAP). It can be accessed through https://vault.gitlab.net (production) and https://vault.pre.gitlab.net (preprod).
  • Note: Vault CLI does not work through this load balancer at the time of this writing, see this feature request.
• Internal ingress for CI/Kubernetes/Terraform/Chef/Ansible/etc (API) is exposed through a Kubernetes service with a zonal network endpoint group (NEG) as a backend. The Vault service uses the default port 8200 which is then exposed on port 443 by the ingress. It can be access through https://vault.ops.gke.gitlab.net (production) and https://vault.pre.gke.gitlab.net (preprod).
  • Prometheus metrics are gathered from the endpoint /v1/sys/metrics on that same port.

Storage

Vault’s Integrated Storage engine Raft is used for storage. This allows all the nodes in a Vault cluster to have a replicated copy of Vault’s data locally. It is also used as the HA backend.

Source

We are not using the enterprise version for Vault which come with automatic backups. Instead we are running a Kubernetes CronJob to create and save raft snapshots to a GCS bucket every hour. Manual testing was done in the pre environment for validation, snapshot backup, and restoration.

Snapshot backups are saved across multiple regions in the United States and can be used for region failure disaster recovery.

Resources and Configuration

Vault is configured across several Terraform modules and environments, each with a well-defined scope. The table below summarises what each piece owns.

Layer	Location	Owns
GCP scaffold	config-mgmt/modules/vault-project	KMS keyring and unseal key (auto-unseal), the Raft snapshots GCS bucket and its IAM bindings. Deployed in the dedicated gitlab-vault-<env> GCP projects.
Cluster-side GCP resources	config-mgmt/modules/vault	IAM bindings and service accounts used by the Vault Kubernetes workload in the gitlab-ops and gitlab-pre projects.
Cluster-wide Vault configuration	terraform-modules/vault/vault-configuration	All authentication methods (Okta OIDC, GCP, Kubernetes, JWT per GitLab instance, AppRole for the single vault-provisioning role used to bootstrap the environment), all KV/PKI/Transit/GCP/Kubernetes secrets engines, admin and user OIDC roles, shared policies, audit sink, identity-entities-cleaner role.
Environment wiring	config-mgmt/environments/vault-production / vault-staging	Instantiates the vault-configuration module and owns Okta group membership (groups.tf), secret policies (secrets_policies.tf), GCP projects and rolesets (gcp.tf, gcp_projects.tf), Kubernetes cluster registration (kubernetes.tf), Chef environments (chef.tf), and the Raft autopilot resource (vault_config.tf).
Cluster-side wiring	config-mgmt/environments/pre/vault.tf / ops/vault.tf	Instantiates the modules/vault module in the pre and ops environments (i.e. the GCP projects hosting the Vault Kubernetes workload).
Per-project Vault resources	terraform-modules/gitlab/project — vault.tf, vault_policies.tf, vault_transit_key.tf	Per-project JWT roles (readonly, protected read-write, per-environment ro/rw), matching policies, transit key, and the per-project CI variables (VAULT_AUTH_ROLE, VAULT_AUTH_ROLE_SUFFIX, VAULT_SECRETS_PATH, VAULT_TRANSIT_KEY_NAME, plus per-environment variants).
Group-level CI variables	terraform-modules/gitlab/group//modules/vault-variables	Group-level GitLab CI variables: VAULT_ADDR, VAULT_SERVER_URL, VAULT_AUTH_PATH, VAULT_SECRETS_SHARED_PATH, VAULT_TRANSIT_PATH.

Maintenance automation

Some Kubernetes cron jobs have been setup to help with regular maintenance of the Vault cluster:

vault-identity-entities-cleaner

• Cronjob definition: https://gitlab.com/gitlab-com/gl-infra/argocd/apps/-/blob/b25b54707ca3431dd697eaf810ed52a2cbc9cb6b/services/vault/values-extras.yaml#L223
• Container image: https://gitlab.com/gitlab-com/gl-infra/ci-images/-/tree/master/vault-identity-entities-cleaner
• Script: https://gitlab.com/gitlab-com/gl-infra/ci-images/-/blob/master/vault-identity-entities-cleaner/scripts/cleaner.rb

Vault creates an identity entities for each user authenticating into it, which includes humans via OIDC, Kubernetes service accounts and GitLab CI jobs authenticating via JWT. In particular, each CI job has its own unique identity based on their job ID claim, that are then never used again (unless the job is retried later). Vault currently doesn’t have any mechanism to automatically cleanup old unused identities so they keep accumulating indefinitely, filling the storage, causing high memory usage and degrading performances (see this issue).

This job remediates this problem by deleting in small batches all identity entities that are older than 7 days (max TTL for OIDC logins) every hour, keeping the storage size under control.

vault-pvc-rotater

• Cronjob definition: https://gitlab.com/gitlab-com/gl-infra/argocd/apps/-/blob/b25b54707ca3431dd697eaf810ed52a2cbc9cb6b/services/vault/values-extras.yaml#L156

Vault uses a BoltDB database as its Raft logs storage, which cannot shrink to recover space when data is deleted. Although it tries to reuse old pages when possible, the database file inevitably keeps growing indefinitely (see BoltDB Raft logs). This eventually leads to memory pressure and/or OOM events as the database file is loaded into memory as page cache which increases memory usage as the file keeps growing.

This job remediates this problem by deleting the oldest Persistent Volume Claim and rotating its attached pod once a week, so that the node bootstraps itself with a fresh Raft snapshot. This also serves as a chaos monkey ensuring that automatic Vault node recovery keeps working properly.

Upgrades

Vault upgrades come as two independent Renovate MR streams:

• Vault container image bumps via the pinned image.tag in services/vault/values.yaml. This is the one that actually changes the Vault server version.
• Helm chart version bumps on the Vault Helm chart in services/vault. Because image.tag is pinned in our values, chart bumps never change the Vault version by themselves; they only bring template and default changes.

Before merging, check the relevant changelog for breaking changes:

• Image bumps → Vault changelog
• Chart bumps → Helm chart changelog

Breaking changes may require additional work in:

• the Helm chart values at services/vault/values.yaml (new config knobs, deprecated fields)
• the terraform-modules/vault/vault-configuration module (auth methods, secrets engines, policies)

Once merged, ArgoCD applies the manifest and the Vault StatefulSet rolls pods one at a time. Watch the rollout with kubectl -n vault rollout status sts/vault and confirm each pod rejoins Raft as a follower (vault operator raft list-peers) before the next pod is replaced. Also monitor the cluster status row at the top of the Vault service dashboard and watch for any VaultLowFailureTolerance alerts firing in #f_fleet_alerts during the rollout.

Authentication

User authentication

We’re using Google Identity-Aware Proxy for the external load balancer and Okta OIDC is required to log into the Vault web interface.

API authentication

• CI runners can be authenticated with JWT using JWKS
• GCP servers (chef client, ansible, etc) can be authenticated with Google Cloud auth (service accounts or instance service accounts)
• Kubernetes can be authenticated with Kubernetes Service Account Tokens

Identities/Roles and Policies are used to enforce RBAC and limit scope of access to necessary secrets. See the Secret Access section for more information.

Security Considerations

Data Encryption and Unsealing

Vault data is encrypted at all times. When Vault is started, it is always started in a sealed state and will not be able to decrypt data until it is unsealed.

To unseal Vault, a root key is needed to decrypt the Vault data encryption key.

To ensure that the root key is never known or leaked, we have configured auto-unseal using GCP KMS which lets us leverage GCP KMS to encrypt and decrypt the root key. Only the KMS key is able to decrypt the root key in our configuration. There is no other method possible to decrypt the root key. The Terraform configuration can be found here.

graph TD
    A[Encryption Key] -->|Encrypted by | B(Root Key)
    B --> D[GCP Cloud KMS key]

For better security access control, the KMS key is hosted in a separated GCP project:

• Production: gitlab-vault-production
• Staging: gitlab-vault-staging

Recovery keys

Vault uses an algorithm known as Shamir’s Secret Sharing to split the recovery key into shards. It is important to know that the recovery key can only used to generate a root token but cannot be used to unseal Vault or decrypt any data.

Additionally, we’re also using end-to-end TLS encryption for Vault.

Secret Access

Role-based Access Control (RBAC) Policies

Vault uses policies to govern the behavior of clients and instrument Role-Based Access Control (RBAC). A policy defines a list of paths. Each path declares the capabilities (e.g. “create”, “read”, “update”, “delete”, “list”, etc) that are allowed. Vault’s denies capabilities by default unless explicitly stated other wise.

There are some built in policies generated by Vault and we’ve currently configured the following policies (all in terraform-modules/vault/vault-configuration):

• General usage policies
• Terraform policies
• GCP secrets engine policies
• Kubernetes secrets engine policies
• Per-KV-mount policies:
  • ci: policies_kv_gitlab.tf (shared) and policies_kv_gitlab_ci_identity.tf (per-identity)
  • k8s: policies_kv_kubernetes.tf and policies_kv_kubernetes_identity.tf
  • chef: policies_kv_chef.tf and policies_kv_chef_identity.tf
  • runway: policies_kv_runway.tf and policies_kv_runway_identity.tf
  • shared: policies_kv_shared.tf and policies_kv_shared_identity.tf
  • observability: policies_kv_observability.tf and policies_kv_observability_identity.tf

For each KV mount, the paired _identity.tf file generates per-identity (per-Okta-group, per-project, per-cluster) policies, while the base file defines the cluster-wide policies shared across all consumers of the mount.

JWT Authentication and Bound Claims

The JSON Web Token (JWT) method can be used to authenticate with Vault by a JWT authentication method or an OIDC. These JWTs can contain claims or a key/value pair. These can be used by Vault to validate that any configured “bound” parameters match which provide more granularity to authentication permissions.

For an example, see the bound claims configured for OIDC users based on Okta group memberships.

More details and specifications can be found in the Vault documentation.

Observability

Prometheus / Mimir

Vault is monitored via Prometheus in GKE. Configuration is done through a ServiceMonitor which scrapes the /v1/sys/metrics endpoint. Our GKE prometheus metrics are also accessible in our Mimir cluster (example metric).

A service overview dashboard can be found in Grafana.

Logs

We’ve configured audit logging to output to STDOUT which forwards logs to Kibana. The Vault logs can be viewed here.

Administration

See Vault Administration.

Usage

See How to use Vault for Secrets Management in Infrastructure.

Troubleshooting

See Troubleshooting Hashicorp Vault.