Breakglass

We follow the Principle of Least Privilege whereby SREs don’t have default read/write access to Cells infrastructure in GCP. This helps limit blast radius in case of security incidents or misconfigurations of running scripts locally to production. We use Privileged Access Manager (PAM) to provide short lived access to GCP via the console or gcloud.

When to escalate

---
title: PAM access entitlements
---
flowchart TD
    need_access[I need access to Cells]

    is_incident{Are you in
    an incident?}
    is_change_incident{Need changes
    to fix an incident?}
    is_change_normal{Need to
    make changes?}
    is_specific_cell{Do you know which Cell
    you need to look at?}
    is_team_members_available{Are team members
    available?}
    is_change_operational{Do you need to
    make temporary changes
    as part of a CR?}

    iac_change[Make changes through IaC]

    project_read
    project_admin
    project_breakglass
    org_read

    need_access-->is_incident

    is_incident-->|Yes| is_change_incident
    is_change_incident-->|No|project_read
    is_change_incident-->|Yes, need write access to quickly solve incident
    I will sync IaC after|is_team_members_available
    is_team_members_available-->|Yes|project_admin
    is_team_members_available-->|No|project_breakglass

    is_incident-->|No| is_change_normal
    is_change_normal-->|No|is_specific_cell
    is_change_normal-->|Yes|is_change_operational
    is_specific_cell-->|Yes|project_read
    is_specific_cell-->|No|org_read
    is_change_operational-->|Yes|project_admin
    is_change_operational-->|No
    OR
    Yes, but I can make changes through IaC|iac_change

How to escalate

Google provides guides to request access through the console and through the gcloud CLI.

Specific Examples:

• org_read
• project_read
• project_admin
• project_breakglass
• amp

Levels of access

Default Access

By default we give enough permissions to SREs to view the organization, view projects, and use PAM. For any further action through GCP changes must be made through IaC or using PAM escalation.

Cells Dev	Cells Prod

Org-wide Access

org_read

• Get access with gcloud:

  # $ORG_ID for dev: `425712905585`
  # $ORG_ID for prod: `1073525295030`
  # $CELL_ENV for dev: `dev`
  # $CELL_ENV for prod: `prod`
  gcloud beta pam grants create \
            --entitlement="org-readonly-entitlement-${CELL_ENV}" \
            --requested-duration="1800s" \
            --justification="$ISSUE_LINK" \
            --location=global \
            --organization="$ORG_ID"

• Get access with console: dev | prod

This entitlement provides the ability to list resources at an org-wide level, but does not provide the ability to get details for resources, other than IAM bindings. This is useful for initial discovery and exploration of the resources available across the org.

Project Access

project_read

• Get access with gcloud:

  # TENANT_ID: Can be found in the "tenant_id" field in the tenant model in `cells/tissue`.
  # example TENANT_ID="c01j2gdw0zfdafxr6"
  gcloud beta pam grants create \
            --entitlement="gldt-readonly-entitlement-${TENANT_ID}" \
            --requested-duration="1800s" \
            --justification="$ISSUE_LINK" \
            --location=global \
            --project="cell-${TENANT_ID}"

• Get access with console: dev | prod

This entitlement provides the ability to view resources within a project scope. This allows you to list and get resources to discover and describe specific resource configurations, e.g. Viewing the details for a GKE cluster and its nodes.

project_admin

• Get access with gcloud:

  # TENANT_ID: Can be found in the "tenant_id" field in the tenant model in `cells/tissue`.
  # example TENANT_ID="c01j2gdw0zfdafxr6"
  gcloud beta pam grants create \
            --entitlement="gldt-admin-entitlement-${TENANT_ID}" \
            --requested-duration="1800s" \
            --justification="$ISSUE_LINK" \
            --location=global \
            --project="cell-${TENANT_ID}"

• Get access with console: dev | prod

Admin access provides typical operational readwrite abilities for managing cell infrastructure.

This level of access requires an approver, and typically will also link to an issue detailing why the access is required. We should prefer making changes through IaC wherever possible.

[!note] This entitlement requires approval from a member of the [email protected] Google group. Who can approve: Any SRE or Infrastructure Manager in this group. Next steps: Please ask a team member to approve your PAM request.

project_breakglass

• Get access with gcloud access:

  # TENANT_ID: Can be found in the "tenant_id" field in the tenant model in `cells/tissue`.
  # example TENANT_ID="c01j2gdw0zfdafxr6"
  gcloud beta pam grants create \
            --entitlement="gldt-breakglass-entitlement-${TENANT_ID}" \
            --requested-duration="1800s" \
            --justification="$ISSUE_LINK" \
            --location=global \
            --project="cell-${TENANT_ID}"

• Get access with console: dev | prod

Breakglass provides the same level of access as the project_admin entitlement.

This access does not require an approver. The entitlement is intended to be used during an incident response when there is low team member availability.

We should use breakglass as a last resort when we need to make an emergency change to speed up incident response.

Amp

• Get access with gcloud

  # AMP_PROJECT_ID for dev: `amp-b6f1`
  # AMP_PROJECT_ID for prod: `amp-3c0d`
  gcloud beta pam grants create \
                  --entitlement="gld-readonly-entitlement" \
                  --requested-duration="1800s" \
                  --justification="$ISSUE_LINK" \
                  --location=global \
                  --project="amp-3c0d"

• Get access with console: dev | prod

The AMP GCP project has the same entitlements as a Cell, meaning project_read, proejct_admin, project_breakglass