Skip to content
Runbooks

GATE - iam-data-access — Runbook

iam-data-access is the composite Lookup + Update gRPC service in the GATE/IAM monorepo (gitlab-org/auth/iam): it serves identity/access relationship reads (Lookup API) and writes (Update API) on one shared gRPC port, backed by a Yugabyte data store (see the YugabyteDB Aeon runbook for the DB side). It is deployed via Runway under the service ID iam-data-gke-grpc.

Use this when something is wrong or a consumer reports errors calling into IAM. Sections are ordered for an incident: triage first, then observability, alerts, troubleshooting, and escalation. The Reference section at the bottom covers environments, Vault paths, and config file locations.

1. Quick triage (start here)

Section titled “1. Quick triage (start here)”
  • Scope: staging or production? (Runway service ID either way: iam-data-gke-grpc)
  • Health: kubelet already probes Health continuously (liveness/readiness), so a one-off manual call mostly just re-checks what Kubernetes already knows. Instead, watch the probe heartbeat itself: same production/staging A steady stream of Health calls means probes are succeeding; gaps or errors mean they’re not. (A manual external call is still useful for testing the full client→gateway→pod path specifically — see 4.1.)
  • Deployment status: check the Runway deployment pipeline for the latest run. A green “Upload Resource Manifests” job is not a successful deploy — it only publishes a Flux OCI artifact; Flux reconciles it async on the GKE cluster. To confirm the actual pod state — see 4.1: kubectl get pod -n iam-data-gke-grpc.
  • Client vs. service: is only one caller failing (suspect auth, not this service) or is everyone failing (suspect deployment or the backing DB)?
  • Logs: production · staging (Grafana Explore, health checks excluded; see Observability to adjust the query).
  • Recent change: check the security-mirror main pipeline for a recent Runway trigger, and recent MRs on gitlab-org/auth/iam — did a deploy or config change trigger this? If so, assess a rollback (4.5).
  • Capture: exact error text (gRPC status + message), affected RPC, timestamps, env — you’ll need these for escalation.

Then jump to the matching section: Observability · Alerts · Troubleshooting · Incident response · Reference.

2. Observability

Section titled “2. Observability”

Logs

Section titled “Logs”

Start here: production logs · staging logs — Grafana Explore, health checks excluded. These are also the service catalog’s logging links for iam-data-gke-grpc.

To adjust the query: datasource ClickHouse - Runway Production → table observability → query type logs.

  • Primary filter: ServiceName = 'iam-iam-data-gke-grpc-iam-data-gke-grpc'
  • Other useful fields: SeverityText (info/warn/error), Attributes['env'] (staging/production), Attributes['cloud.runtime'] (gke), k8s.pod.name, k8s.deployment.name, k8s.namespace.name, k8s.cluster.name.
  • Example filter: WHERE ServiceName = 'iam-data-gke-grpc' AND SeverityText = 'error' AND Attributes['env'] = 'production'.

Metrics

Section titled “Metrics”

iam-data-gke-grpc overview.

  • Shows the default runway-k8s-archetype SLIs: apdex/error-rate/RPS from runway_lb_* (frontend/backend load-balancer metrics), plus saturation (CPU/memory/HPA).
  • No custom application metrics (RPC-level latency/error-rate, DB-query timing) are emitted today — only the generic Runway LB/saturation metrics above. Separate iam-update/iam-read SLIs need that instrumentation to exist first.

Health / readiness

Section titled “Health / readiness”

gRPC Health RPC per sub-service (gitlab.iam.lookup.v1.LookupService/Health, gitlab.iam.update.v1.UpdateService/Health), served on the same port as regular gRPC traffic (:5005) — there is no separate metrics/health port for this service.

3. Alerts

Section titled “3. Alerts”

The Runway metrics-catalog entry for iam-data-gke-grpc provides the default runway-k8s-archetype alerts via Alertmanager/Slack/incident.io:

  • SLO violation alerts: apdex, error rate, traffic cessation — based on the generic runway_lb_* SLI (no RPC-method-level breakdown yet, see Metrics).
  • Saturation alerts: CPU, memory, HPA utilization.
  • Default severity is S4 (Slack #feed_alerts-general); S1/S2 pages on-call. See alert routing conventions.

These alert on service-level symptoms only — they won’t catch an iam-update-only or iam-read-only regression while the other half of the composite service looks healthy. For that, also watch consumer-side symptoms: GitLab Rails-side errors on calls into IAM, or reports from teams integrating with the Lookup/Update APIs.

4. Troubleshooting

Section titled “4. Troubleshooting”

4.1 Service unreachable / deployment unhealthy

Section titled “4.1 Service unreachable / deployment unhealthy”

Symptoms: connection refused/timeouts on all RPCs (including Health), or a deploy that “succeeded” in CI but nothing changed.

  • Check the Runway deployment pipeline — a green “Upload Resource Manifests” job only publishes a Flux OCI artifact; Flux reconciles it asynchronously on the target GKE cluster.

  • Check deployment health; two options:

    1. Via kubectl:
      • Use kubectl via the KAS proxy on ops.gitlab.net — see Connecting with kubectl for setup (glab cluster agent update-kubeconfig, pointed at the right agent — find it on the kas-clusters agents page).
      • Then kubectl get pod -n iam-data-gke-grpc to see actual pod status/restarts/events.
    2. Via GCP console: Go to deployment details

  • Check logs for crash/startup errors (bad CONFIG_TOML, missing Vault secret — see 4.4).

  • To test the full external path (client → gateway → pod) rather than just pod-level health, call the Health RPC directly — exempt from both auth layers, so no token needed:

    Terminal window
    # production: iam-data-gke-grpc.svc.gitlab.net
    # staging:    staging.iam-data-gke-grpc.svc.gitlab.net
    buf build -o - | grpcurl -protoset /dev/stdin \
      -d '{}' <host>:443 gitlab.iam.lookup.v1.LookupService/Health

    Useful when the health-check heartbeat in logs looks fine (pod is healthy) but callers still can’t reach the service — points at routing/gateway rather than the pod itself.

4.2 Authentication failures (Unauthenticated / 401)

Section titled “4.2 Authentication failures (Unauthenticated / 401)”

Symptoms: calls fail with opaque Unauthenticated (gRPC) or 401 (HTTP); Health still works (it’s exempt from both auth layers).

Two independent layers, both required on every protected RPC:

LayerChecksFailure mode
Service token (pkg/serviceauth)Shared secret sent in the gitlab-iam-data-access-token header — wrong header name is rejected the same as no tokenOpaque Unauthorized, never says missing vs. wrong
End-user JWT (pkg/userauth)authorization: Bearer <jwt>; aud claim must match expected_audiencesStructured VerifyError/FailReason (audience mismatch, expired, key not found)
  • Confirm the caller is using the gitlab-iam-data-access-token header specifically (per-service header names are intentional — sending it under another service’s header name gets the same opaque rejection).
  • Confirm the JWT’s aud claim matches this service’s configured expected_audiences.
  • Service token value: Vault-managed (see Vault paths).
  • Full details: docs/service-access.md in the iam repo.

4.3 Backing database issues

Section titled “4.3 Backing database issues”

iam-data-access is backed by a YugabyteDB Aeon cluster. Connection failures, latency, connection-pool exhaustion, node/cluster health — all covered in the YugabyteDB Aeon runbook, not duplicated here.

4.4 Config/secrets issues

Section titled “4.4 Config/secrets issues”

Symptoms: service fails to start, or starts with wrong config (wrong DB host, CORS rejecting a valid origin, wrong OAuth issuer).

  • Runway can’t mount configMap volumes, so non-secret config is passed inline as a CONFIG_TOML env var (.runway/values-{staging,production}.yaml) — the container entrypoint writes it to disk and points CONFIG_FILE_PATH at it.
  • Secrets are injected as individual env vars from Vault via Runway’s secretEnvFrom (workloadSecrets: [yugabyte, service-token] in .runway/deployment.yaml). A missing secret shows as its config-file placeholder value (VAULT_MANAGED) leaking through unchanged — check the Vault paths below.
  • Confirm expected values against Vault paths and the environment’s CONFIG_TOML block in .runway/values-<env>.yaml.

4.5 Bad deploy / need rollback

Section titled “4.5 Bad deploy / need rollback”
  • Production deploys are manual-gated (runway_manual_production_deploy: true) — staging auto-deploys on every release tag.
  • Chart version is pinned per deploy (oci://registry.gitlab.com/gitlab-org/auth/iam/chart/iam:<ChartVersion> in .runway/deployment.yaml) — a rollback is re-triggering the Runway deployment pipeline against the previous known-good tag.
  • See docs/release.md in the iam repo for the full release/deploy flow.

5. Incident response & escalation

Section titled “5. Incident response & escalation”

5.1 Severity

Section titled “5.1 Severity”
SeverityExampleAction
S1 / CriticalProduction unreachable for all callers; data corruption via Update APIPage on-call; declare incident
S2 / HighPartial outage (e.g. one RPC failing); elevated error rate reported by a consumer teamPage on-call
S3 / ModerateSingle-pod issue self-healing; staging-only issue blocking a releaseInvestigate same business day
S4 / LowSmoke-test flakiness, minor config drift, capacity planningRoutine MR / follow-up

Staging issues are normally ≤ S3 unless they block a release.

5.2 First responder checklist

Section titled “5.2 First responder checklist”

See Quick triage. Mitigate what’s in our control (rollback, config/secret fix); escalate deployment/cluster-level issues to Runway.

5.3 Internal escalation

Section titled “5.3 Internal escalation”

5.4 Runway / Infra escalation

Section titled “5.4 Runway / Infra escalation”
  • Ask in #f_runway for deployment/cluster-level issues (Flux reconciliation, pod crash diagnosis, PSC/network issues) — the Runway team owns the platform.
  • Read-only kubectl access for non-Infra engineers is now available for both staging and production — see Connecting with kubectl.

5.5 What we handle vs. what Runway/Infra handles

Section titled “5.5 What we handle vs. what Runway/Infra handles”
We handleRunway/Infra handles
App config, Vault secret values, service-token/JWT configCluster health, Flux reconciliation, autoscaling
Deploy trigger (release tags, manual production approval)GKE node/pod infrastructure, PSC networking
App-level bugs, DB query issues (see Aeon runbook)Platform-level metrics/logs pipeline (ClickHouse, Mimir)

6. Reference: environments, access & config

Section titled “6. Reference: environments, access & config”

Environments

Section titled “Environments”

There are two distinct entrypoints into this service — which one a caller uses depends on whether it runs in the same GKE cluster as this service:

EnvRunway service IDenv labelPublic entrypointInternal entrypointOAuth issuer
Staging (gstg)iam-data-gke-grpcstagingstaging.iam-data-gke-grpc.svc.gitlab.net:443direct pod IPs (designed, not yet exercised — see below)https://staging.gitlab.com
Production (gprd)iam-data-gke-grpcproductioniam-data-gke-grpc.svc.gitlab.net:443direct pod IPs (designed, not yet exercised — see below)https://gitlab.com

Public entrypoint (Cloudflare zone): .runway/deployment.yaml sets loadBalancing.externalLoadBalancer.enabled and .cloudflareSetting.enabled to true, which provisions a dedicated Cloudflare zone per environment — production at iam-data-gke-grpc.svc.gitlab.net, staging at staging.iam-data-gke-grpc.svc.gitlab.net. These front the same GKE external load balancer Runway provisions from the GCP origin (<workload-name>[.staging].runway.gitlab.net). This path is for callers outside this service’s GKE cluster — e.g. the main GitLab.com GKE clusters, which run separately from Runway’s. No auth header is required to reach the host itself — the service’s own two-layer auth still applies once connected.

Internal entrypoint (direct pod IPs via Headless Service): the intent is that other services co-located in the same GKE cluster — e.g. Artifact Registry, which does run in the same staging cluster (runway-gke-stg-us-east1) — skip the public internet entirely and reach this service’s pods directly via in-cluster Service discovery (plain Kubernetes DNS, not a Private Service Connect setup), rather than the external load balancer/Cloudflare path above.

Vault paths

Section titled “Vault paths”

Secrets are injected via Runway’s secretEnvFrom; deep links to the output secrets:

  • Yugabyte creds: staging · production
  • Service token: same Runway KV mount, env/<env>/service/iam-data-gke-grpc/service-token (see workloadSecrets in .runway/deployment.yaml).

Config file locations (in gitlab-org/auth/iam)

Section titled “Config file locations (in gitlab-org/auth/iam)”

Making changes / deploy pipeline

Section titled “Making changes / deploy pipeline”
  • Release flow, mirrors, and manual-production-gate details: docs/release.md.
  • Actual Runway staging/production deploy jobs run in the separate Infra-owned project — deployments pipeline list, not in iam’s own CI. Staging auto-deploys on release tag; production requires manual approval.
Section titled “Related links”