Skip to content
Runbooks

Project authorization refresh fan-out

Owned by the Authorization Group.

Covers AuthorizedProjectsWorker and the AuthorizedProjectUpdate::* workers.

1. Triage

Section titled “1. Triage”

1.1 Confirm authorization refresh is the source

Section titled “1.1 Confirm authorization refresh is the source”

Open the Sidekiq worker detail dashboard for all four workers and check the enqueue rate. Check if the enqueue rate for at least one of these workers is well above its trailing 24h baseline.

1.2 Confirm safety net is the root cause

Section titled “1.2 Confirm safety net is the root cause”

The safety net path tags every job it enqueues with meta.authorized_projects_refresh_purpose: safety_net, so you can separate the two paths directly in the logs. Run this in Kibana (data view pubsub-sidekiq-inf-gprd-*):

json.meta.authorized_projects_refresh_purpose : "safety_net"

Open this query in Kibana

Select json.meta.root_namespace field on the left sidebar to group the results by namespace. A fan-out is almost always concentrated in a single root_namespace with one dominant root_caller_id.

2. Incident response & escalation

Section titled “2. Incident response & escalation”

2.1 Mitigate the fan out

Section titled “2.1 Mitigate the fan out”

Enable the do_not_run_safety_net_auth_refresh_jobs feature flag. This gates the LOW_PRIORITY branch of UserProjectAccessChangedService, so no new safety-net jobs are scheduled.

The primary workers keeps running by design, only the safety net workers are disabled. The flag only affects the enqueue site, so jobs already in the Sidekiq scheduled set still fire and run to completion. Expect the fan-out to taper over the next hour.

2.2 Rollback

Section titled “2.2 Rollback”

Once the incident has been resolved, disable the feature flag. The safety net jobs repair project_authorizations rows that drifted from the source of truth. While the flag is enabled that repair is not running, so any drift the primary path misses goes uncorrected and affected users can keep seeing stale access.

Treat this as a temporary incident mitigation, not a steady state.

Record in the incident issue when the flag was enabled and disabled, and notify #g_authorization if it needs to stay on beyond the incident so the group can assess the correctness risk.

3. Background

Section titled “3. Background”

3.1 Why it fans out

Section titled “3.1 Why it fans out”

Project authorizations are cache for a user’s access levels in projects. Any change to the user’s access level / membership triggers a re-computation of this cache.

Every request schedules two workers - a primary worker (AuthorizedProjectsWorker or ProjectRecalculateWorker or ProjectRecalculatePerUserWorker) which is enqueued instantly. And a safety worker which is enqueued an hour later.

The safety nets read from the replica, however they usually cause the fan out becuase they trigger an access level recomputation for ever member in the group in which the membership change occured.

All the different workers write to the same project_authorizations table which results in a high number of writes and WAL accumulation on the table.

3.2 Common triggers

Section titled “3.2 Common triggers”

Not exhaustive, but these are what we have seen cause bursts:

  • SAML/SCIM group sync — calls the per-member callback once per member, so a sync of a large group enqueues one job per member.
  • Group sharing and unsharing, and project/group link create, update, or destroy.
  • Bulk member add, remove, activation, or await (seat management).
  • Project creation under a group, project transfer, and namespace transfer or path change.
Section titled “Related”