Troubleshooting Hashicorp Vault

Determining Pod status and logs

Connect to the appropriate GKE cluster, then list/look at all pods in the vault namespace:

kubectl -n vault get pods
kubectl -n vault logs vault-0

The active node can be found with the label vault-active=true:

kubectl -n vault get pods -l app.kubernetes.io/name=vault -l vault-active=true

Determining status of Vault from Vault itself

You can use a SOCKS5 proxy or forward the port 8200 and run Vault locally (see CLI):

# In a separate shell session
ssh -D 18200 bastion-01-inf-ops.c.gitlab-ops.internal
# In your first session
export VAULT_ADDR=https://vault.ops.gke.gitlab.net
export VAULT_PROXY_ADDR=socks5://localhost:18200
vault login -method oidc role=admin
vault operator peers
vault operator raft list-peers
# Ensure all Vault pods are listed and their `State` is either `leader` or `follower`

# In a separate shell session
kubectl -n vault port-forward svc/vault-active 8200
# In your first session
export VAULT_ADDR=https://localhost:8200
export VAULT_TLS_SERVER_NAME=vault.ops.gke.gitlab.net
vault login -method oidc role=admin
vault operator peers
vault operator raft list-peers
# Ensure all Vault pods are listed and their `State` is either `leader` or `follower`

Alternatively, you can connect to one of the Vault pods and run:

kubectl -n vault exec -it vault-0 sh
$ vault status
$ export VAULT_SKIP_VERIFY=true
# Ensure `Initialized` is `true` and `Sealed` is `false`
$ vault login
# enter root token
$ vault operator peers
$ vault operator raft list-peers
# Ensure all Vault pods are listed and their `State` is either `leader` or `follower`

No Active Vault Instance / Vault Sealed / Vault Low Failure Tolerance

The Vault pods are failing to start, have lost quorum or are unable to auto-unseal.

Vault is deployed in a cluster of 5 nodes, so it needs at least 3 healthy nodes to have a quorum and be operational.

Check the status of the Vault deployment and investigate any failing pod for errors:

kubectl --namespace vault get pods
kubectl --namespace vault logs vault-X

You can also check the logs in Elasticsearch instead.

In case of unseal errors:

Verify that the Kubernetes Service Account is still associated to its Google Service Account [email protected]:
Terminal window
```
kubectl --namespace vault describe serviceaccount vault
```
Verify that this Service Account has permission to use the unseal KMS key for encryption/decryption.

Vault Audit Log Request Failure

Vault is unable to send its audit log and thus has stopped all operations until it is able again.

At the time of this writing, the Vault audit logs are written directly to stdout, so they can be collected by Fluentd and shipped to Elasticsearch, which makes failure extremely unlikely.

If Vault fails to write its audit logs it could mean:

a bug introduced in Vault: has it been upgraded recently? Search the issues on GitHub.
containerd not able to handle the container’s output, possibly affecting other workloads, check the health of node running the active Vault pod.

Debugging CI Vault permission errors

Vault access from CI has several layers of configuration (infra-mgmt, config-mgmt/environments/vault-production, the vault-configuration Terraform module, and Vault itself) and it can be difficult to reason about which one is failing when a permission error is reported. Follow the steps below in order to narrow it down.

Verify the secret actually exists at the expected path, and fix on the writer side (Terraform outputs, manual vault kv put, or External Secrets Operator sync) if it doesn’t:
Terminal window
```
vault kv metadata get <mount>/<full-path>
```
Verify the project role in infra-mgmt. The project must have vault = { enabled = true, auth_path = local.vault_auth_path } in its module definition. If the secret is outside the project’s default paths (ci/<instance>/<full-path>/*), the project must additionally list it in one of readonly_secret_paths, readwrite_secret_paths, protected_secret_paths, extra_readonly_policies, or extra_protected_policies. See terraform-modules/gitlab/project for the full input schema.
Verify the role config and policies in Vault directly.
Terminal window
```
vault read auth/<gitlab-instance>/role/<project-path-underscores>
vault read auth/<gitlab-instance>/role/<project-path-underscores>-rw
vault policy read <gitlab-instance>-project-<project-path-underscores>
vault policy read <gitlab-instance>-project-<project-path-underscores>-rw
```
Confirm that:
- bound_claims contains the correct project_id (and ref_protected = "true" for the -rw role);
- token_policies lists the expected policies;
- each policy grants the required capabilities on the target path.
If the role or policies don’t match what infra-mgmt should have created, run the Atlantis plan/apply on the infra-mgmt MR — the drift may simply not have been applied.
Confirm the CI job’s ID token audience and JWT claims. The job must define:
```
id_tokens:
  VAULT_ID_TOKEN:
    aud: https://vault.gitlab.net
```
and use the expected VAULT_AUTH_ROLE / VAULT_AUTH_PATH values (set on the project or group). See Using Vault secrets in CI.
For writes from Atlantis (Terraform-triggered vault_kv_secret_v2 resources), verify that Atlantis itself has the right to write to the target path. Atlantis’s own policy is defined in environments/vault-production/secrets_policies.tf; if the Atlantis project doesn’t have write access to the path, the apply will fail with a permission error even though the user running the pipeline has access via OIDC.

Cross-references: access.md (Okta groups), administration.md (adding a GitLab instance or Kubernetes cluster), usage.md#authorizing-a-gitlab-project.

Debugging External Secrets Operator (ESO) / Kubernetes secret delivery breakage

When an ExternalSecret in a Kubernetes namespace stops syncing (or never populates), work through the following in order:

Confirm the secret exists at the configured path in Vault:

vault kv metadata get <mount>/<full-path>
# e.g. vault kv metadata get k8s/<cluster>/<namespace>/<secret>
# or   vault kv metadata get shared/...

Fix on either side if the path doesn’t match.

Check the ESO role in environments/vault-production/kubernetes.tf, under kubernetes_clusters.<cluster>.auth_roles (used by the External Secrets Operator to authenticate) and/or secrets_roles (for dynamic Kubernetes credentials). Add or fix it if missing.
Read the role directly from Vault to confirm what actually got applied:
Terminal window
```
vault read auth/kubernetes/<cluster>/role/<role>
```
Fix drift in Terraform if the role config doesn’t match expectations.
Inspect the policies attached to the role:
Terminal window
```
vault policy read <policy-name>
# e.g. vault policy read k8s_<cluster>_<role>
```
The default policy attached to an ESO auth-role is k8s_<cluster>_<role> (single underscore separator). This is distinct from Kubernetes secrets-engine role policies, which follow kubernetes_<cluster>--<role> (see usage.md).
Recovering from a misconfigured secret value:
- If the user wrote the wrong value: vault kv patch with the correct value, or roll back by pinning version in the ExternalSecret spec to a previous version.
- If the secret was destroyed entirely (versioning history lost): the user must recreate it as a new secret.

Generating a root token

Official documentation: https://learn.hashicorp.com/tutorials/vault/generate-root

When admin access is not sufficient or broken, a root token can be generated using the recovery keys. For this you will need access to the entry Vault Production in the Production vault in 1password.

:warning: Root tokens are dangerous as they allow to do anything in the cluster and don’t expire. For this reason, please remember to revoke it using vault token revoke -self once you are done using it! :warning:

First, setup CLI access by following the steps described here
Verify that Vault is unsealed:
Terminal window
```
vault status
```
If not, troubleshoot the issue

Grab 3 recovery keys of the 5, and run the following:

vault operator generate-root -init
# Note the OTP in the output
vault operator generate-root
# Enter recovery key #1
vault operator generate-root
# Enter recovery key #2
vault operator generate-root
# Enter recovery key #3
# The output should now show "Complete true"
# Grab the Encoded Token from the output and the OTP from the first step and run:
vault operator generate-root -decode $ENCODED_TOKEN -otp $OTP
# The output is your root token, now you can export it:
export VAULT_TOKEN=hvs.thetokenfromabove
vault token lookup
# This should show something like this:
# Key                 Value
# ---                 -----
# accessor            XYZ
# creation_time       1658469449
# creation_ttl        0s
# display_name        root
# entity_id           n/a
# expire_time         <nil>
# explicit_max_ttl    0s
# id                  hvs.thetokenfromabove
# meta                <nil>
# num_uses            0
# orphan              true
# path                auth/token/root
# policies            [root]
# ttl                 0s
# type                service

When you are done troubleshooting, please revoke the token:
Terminal window
```
vault token revoke -self
```

Backing up and restoring Vault

Vault Raft snapshots are taken hourly. They are created by a Kubernetes CronJob that runs inside the GKE cluster that runs Vault. It uses vault operator raft snapshot save to create an encrypted copy of all Vault data, and then uploads to it to a GCS bucket inside a separate GCP project:

Production: gitlab-vault-production-vault-raft-snapshots in gitlab-vault-production
Staging: gitlab-vault-staging-vault-raft-snapshots in gitlab-vault-staging

Restoring Vault from a snapshot into an existing functional installation

Note that doing this restores everything entirely back to the state it was at the snapshot, removing all new data since the snapshot was taken.

First identify the Vault snapshot you wish to restore and download it locally:

gsutil ls gs://gitlab-vault-production-vault-raft-snapshots/
gsutil cp gs://gitlab-vault-production-vault-raft-snapshots/2022/07/22/raft-20220722-120000+0000.snap .

Then, setup CLI access by following the steps described here
Then generate a root token

Restore the snapshot:

vault operator raft snapshot restore raft-20220722-120000+0000.snap

:warning: Finally, don’t forget to revoke your root token!
Terminal window
```
vault token revoke -self
```

Restoring Vault from a snapshot into an empty installation

:warning: As we use GKMS auto-unseal for our Vault, you need to make sure the GKMS key that was originally used with the Vault snapshot is still available. As this is controlled by Google, they are responsible for maintaining its snapshot and availability. :warning:

First identify the Vault snapshot you wish to restore and download it locally:

gsutil ls gs://gitlab-vault-production-vault-raft-snapshots/
gsutil cp gs://gitlab-vault-production-vault-raft-snapshots/2022/07/22/raft-20220722-120000+0000.snap .

Make sure the Vault application in ArgoCD is synced.
You should see the Vault pods running and crashing in a loop (because they are uninitialized and don’t have a cluster to join):
Terminal window
```
kubectl -n vault get pods
```
Scale down the cluster to a single pod so that replication doesn’t start until the snapshot has been restored:
Terminal window
```
kubectl -n vault scale sts/vault --replicas=1
```
To make the following steps easier, edit failureThreshold of the healthchecks to some high number in the StatefulSet to prevent the unhealthy pod from restarting before you initialize it:
Terminal window
```
kubectl -n vault edit sts vault
# livenessProbe:
#   failureThreshold: 1000
# readinessProbe:
#   failureThreshold: 1000
```
Note the previous values so you can restore them later; at the time of writing they are livenessProbe.failureThreshold: 30 and readinessProbe.failureThreshold: 3 (see services/vault/values.yaml).
In a separate shell session, establish port-forwarding to the pod:
Terminal window
```
# In a separate shell session
kubectl -n vault port-forward pod/vault-0 8200
```

Then initialize Vault, take note of the root token and export it (no need to save it elsewhere, it will be gone after the snapshot restoration):

# In your first session
export VAULT_ADDR=https://localhost:8200
export VAULT_TLS_SERVER_NAME=vault.ops.gke.gitlab.net
vault operator init
export VAULT_TOKEN=hvs.something

Now you can force-restore the snapshot:

vault operator raft snapshot restore -force raft-20220722-120000+0000.snap

Check in the pod logs that Vault was able to restore and unseal:
Terminal window
```
kubectl -n vault logs vault-0
```
You can now restore failureThreshold to its original values in the StatefulSet (see the note above for the current defaults):
Terminal window
```
kubectl -n vault edit sts vault
# livenessProbe:
#   failureThreshold: 30
# readinessProbe:
#   failureThreshold: 3
```

And scale back to 5 replicas:

kubectl -n vault scale sts/vault --replicas=5

You should now see the other pods spin up and become healthy, as they will join the Vault cluster and start replicating automatically:
Terminal window
```
kubectl -n vault get pods
```

To verify that the cluster is healthy, login as admin (or generate a root token) and then:

vault login -method=oidc role=admin
vault operator peers
vault operator raft list-peers
vault operator raft autopilot state