Skip to content
Runbooks

OpenBao for GitLab Secrets Manager Service

Logging

Section titled “Logging”

Audit Logging

Section titled “Audit Logging”

We suggest the following filters to focus on relevant project audit logs:

resource.labels.service_name="secrets-manager"
labels.container_name = "ingress"
logName = "projects/gitlab-runway-production/logs/run.googleapis.com%2Fstdout"
jsonPayload.type = "response"

Filters

Section titled “Filters”
  • jsonPayload.request.namespace.path="<ns_type>_<ns_id>/<obj_type>_<obj_id>/" can be used to filter audit logs to a particular tenant’s project or group.
    • ns_type and ns_id are the type and ID of the owning tenant; e.g., user_12345 or group_54321.
    • obj_type and obj_id are the type and ID of the final project or group depending on the scope of the secrets manager instance in question.
    • For instance, a working example filter on production is jsonPayload.request.namespace.path = "group_108099358/project_70952462/".
  • jsonPayload.request.path =~ "secrets/kv/data/explicit/.*" can be used to filter to just secret value read operations.
    • An explicit secret name can also be given with jsonPayload.request.path = "secrets/kv/data/explicit/<SECRET-NAME>".
    • This is best used in conjunction with the above.

Service Logging

Section titled “Service Logging”

We suggest the following filters to focus on relevant project service logs:

resource.labels.service_name="secrets-manager"
labels.container_name = "ingress"
logName = "projects/gitlab-runway-production/logs/run.googleapis.com%2Fstderr"

Summary

Section titled “Summary”

GitLab Secrets Manager is a built-in secrets management solution for CI pipelines. Secrets are created and managed using GitLab UI, and consumed by CI jobs.

GitLab Secrets Manager relies on the secrets-manager Runway service. The service is configured and deployed using the gitlab-secrets-manager-container project.

secrets-manager runs OpenBao, which is a fork of HashiCorp Vault. The source code of OpenBao lives in openbao-internal, a build project that is intended to modify the upstream OpenBao releases.

Architecture

Section titled “Architecture”

The Rails backend and runners connect to the secrets-manager service (running OpenBao) through the CloudFlare WAF and Runway.

OpenBao stores data on the Cloud SQL instance provided by Runway, and gets the unseal key from Google KMS.

OpenBao posts audit logs to the Rails backend.

The GitLab Secrets Manager design docs provides request flow diagrams.

flowchart TB
    CloudFlare(CloudFlare: secrets.gitlab.com)
    KMS[GCP KMS]
    PostgreSQL[GCP CloudSQL from Runway]

    Rails-- Manage OpenBao -->CloudFlare
    CloudFlare-- https://secrets-manager.production.runway.gitlab.net -->Runway
    Runway-->OpenBao
    Runner-- Fetch Pipeline Secrets -->CloudFlare
    OpenBao-- Decrypt Unseal Key -->KMS
    OpenBao-- Storage -->PostgreSQL
    OpenBao-->Runway

The service runs multiple OpenBao nodes:

  • a single active node
  • multiple standby nodes

Nodes connect to the PostgreSQL backend to store data and to acquire a lock.

flowchart TD
    Ingress


        Service_OB([HTTP API])

    subgraph OpenBao
        OB_1[Primary]
        OB_2[Standby A]
        OB_3[Standby B]

        Service_Primary([Primary gRPC])
        end

    Ingress --> Service_OB
    Service_OB --> OB_1
    Service_OB --> OB_2
    Service_OB --> OB_3

    OB_2 -. forward .-> Service_Primary
    OB_3 -. forward .-> Service_Primary

    Service_Primary --> OB_1

    OB_1 -->Service_DB
    OB_1 -. lock maintenance .->Service_DB
    OB_2 -. lock monitor .->Service_DB
    OB_3 -. lock monitor .->Service_DB

    Service_DB([PostgreSQL]) -->    DB[(PostgreSQL)]

    OB_1 -- auto-unseal --> KMS
    OB_2 -- auto-unseal --> KMS
    OB_3 -- auto-unseal --> KMS

Performance

Section titled “Performance”

Benchmarking and sizing recommendations are covered by gitlab#568356.

Scalability

Section titled “Scalability”

The service is deployed using Runway and its scaling is handled by Cloud Run.

Scalability is configured in runway.yml.

Availability

Section titled “Availability”

GitLab Secrets Manager is limited to the Ultimate tier. The feature needs to be enabled in a project.

The service is configured to be deployed to the us-east1 region.

Durability

Section titled “Durability”

Runway performs backup and backup restore validation as configured for the secrets-manager service.

On Runway backups are always on.

Backup procedure:

  1. Back up Cloud SQL PostgreSQL database.
  2. Back up the unseal key material stored on Google Cloud KMS. See runbooks for our internal Vault service, which similarly relies on Google Cloud KMS.

For restore, we suggest the following steps:

  1. Stop OpenBao.
  2. Perform the PostgreSQL restore.
  3. Start OpenBao.

Security/Compliance

Section titled “Security/Compliance”

The Cloud SQL PostgreSQL database only contains encrypted data, and the unseal is stored on Google KMS.

Monitoring/Alerting

Section titled “Monitoring/Alerting”

The service comes built-in Runway observability:

Direct OpenBao Access

Section titled “Direct OpenBao Access”

Unlike Vault, we do not intend for OpenBao to be directly modified or accessed by SREs to protect customer secrets. Only in the event of OIDC issuer reconfiguration of the global auth/gitlab_rails_jwt authentication mount would this be necessary. To do so, use Terraform in config-mgmt to use recovery keys to create a highly privileged root token and remediate problems via Terraform for auditability.

Reset

Section titled “Reset”

To reset OpenBao’s data:

  1. Verify staging Secrets Manager is online and operational (metrics from dashboard and/or logs).
  2. Scale OpenBao down to zero instances.
  3. Perform backup of database (runway-db-secrets-manager). (Via Google Cloud WebUI; requires the cloudsql.backupRuns.create permission).
  4. Delete database contents (TRUNCATE openbao_kv_store; + TRUNCATE openbao_ha_locks; per defaults at OpenBao PostgreSQL Storage Config and configuration in the project). (Via Google Cloud WebUI; requires the cloudsql.instances.executeSql permission).
  5. Scale OpenBao up to one instance.
  6. Monitor logs until new instance comes up: log console link.
  7. Perform additional backup so most-recent backup is on the new instance.
  8. Scale back to regular OpenBao instances.
  9. Verify connection with GitLab Staging instance works as expected; enable on new project/group.
    • Recovery procedure is same as before; revert to earlier backup and reconsider failure mode.
Section titled “Links to further documentation”