gkms

GKMS and Vault

At the time of this writing, the GKMS secrets are in the process of being migrated into Vault under the chef/ KV mount. Both share the same process for certificate management.

Automation

This process should be automated via the Certificates Updater. This tool pulls all Chef secrets from GKMS and Vault (under the chef/ KV mount), checks the validity of every certificate found with a valid associated private key, and updates them with a new certificate from SSLMate when possible. It runs twice a week in a scheduled CI pipeline on GitLab.com.

In cases where this automation fails to update a certificate (or skips it), you can follow the instructions below.

Replacement

Make sure you know the item (e.g. frontend-loadbalancer gprd) and fields (if they differ from ssl_certificate and ssl_key). Refer to the certificate table for that information.

Obtain the new certificate from SSLMate.
(GKMS only) Create a local backup of the gkms-vault:
Terminal window
```
./bin/gkms-vault-show ${item} > ${item}_bak.json
```

(Vault only) Note the current version of the Vault secret:

vault kv metadata get -mount=chef  env/${env}/${item} | grep current_version

Format the new certificate (and/or key) to fit into JSON properly and copy the output to the clipboard. (The following command is executed with GNU sed)
Terminal window
```
sed -E ':a;N;$!ba;s/\r{0,1}\n/\\n/g' ${new_certificate}.pem
```
Update the values in the gkms-vault. Make sure to only edit the fields that were specified. Some data bags will contain multiple certificates!
Terminal window
```
# GKMS
./bin/gkms-vault-edit ${item}
# Vault
glsh vault edit-secret chef env/${env}/${item}
```
See [../vault/usage.md#interact-with-vault-secrets] for more information on how to access and edit Chef secrets in Vault.
(GKMS only) This should give you an error if the new gkms-vault is not proper JSON. Still you should validate that by running ./bin/gkms-vault-show ${item} | jq .. If that runs successfully, you have successfully replaced the certificate! Congratulations!
Finally trigger a chef-run on the affected node(s). This should happen automatically after a few minutes, but it is recommended to observe one chef-run manually.

Rollback of a replacement

Sometimes stuff goes wrong. Good thing we made a backup! :)

GKMS

Copy the contents of ${item}_bak.json into your clipboard
Update the values in the gkms-vault. Clear out the whole write-buffer and paste the JSON you just copied.
Terminal window
```
./bin/gkms-vault-edit ${item}
```
Done!

Vault

Rollback to the previous version of the secret:

vault kv rollback -mount=chef -version=${version} env/${env}/${item}

Done!