Duo dependencies on GitLab.com authentication
This page maps which GitLab Duo request paths depend on GitLab.com (Rails and Postgres) for authentication, how tokens and JWKS are cached along each path, and what fails when GitLab.com is unavailable. Use it during an incident to predict which Duo features break, and to check that Duo has actually recovered once the main site has.
Background: INC-15105, corrective action production-engineering#30190. During that incident, Postgres I/O stalls caused HTTP 503s on GitLab.com. Duo also returned 503s on token fetch and on the Duo Workflow WebSocket.
Related runbooks:
- GitLab Duo triage
- AI gateway
- Code Suggestions
- Duo Agent Platform
- Duo Workflow Service
- Cloud Connector
Summary
Section titled “Summary”- Every token-issuing endpoint needs Postgres on every request. Authentication (PAT or OAuth lookup), policy checks, seat or add-on checks, and the Cloud Connector token mint all read the database. While Postgres is unavailable, no client can get a new Duo token.
- The only path that keeps working is direct Code Suggestions with a token already issued. The
IDE talks to the AI gateway at
cloud.gitlab.comwith a user JWT, and the AI gateway checks it against keys it already has in memory. This works until the 1 h token expires. - Duo Agent Platform (flows, agentic chat) depends on GitLab.com throughout. The WebSocket goes through Workhorse and Rails pre-authorization on every connection. Every tool call and checkpoint goes back through Rails. Clients no longer receive a Duo Workflow Service (DWS) token that would let them bypass Rails.
- GitLab.com Duo traffic is served by Cloud Run. EKS serves only the
cloud.gitlab.com/aws/*routes. See Runtimes. - The AI gateway
/monitoring/readycheck fetches GitLab.com OIDC discovery and JWKS live. On Cloud Run it is only the startup probe. Running instances keep serving direct Code Suggestions from cached keys, but new instances cannot start while GitLab.com is down, so scale-up and deploys stall. On EKS it is also the readiness probe, so pods are expected to go NotReady in about 60 s. See AI gateway readiness. - New DWS instances cannot start while GitLab.com JWKS is unavailable, on either runtime, because DWS requires a complete key prefetch at startup. Running instances keep serving from cache.
- Main-site recovery does not mean Duo has recovered. Some client-side circuit breakers keep direct Code Suggestions off for up to 10 minutes after GitLab.com recovers. Failed agent flows are not resumed automatically. See Recovery checks.
Runtimes
Section titled “Runtimes”The AI gateway and DWS are deployed by Runway to both Cloud Run and EKS. Cloudflare on
cloud.gitlab.com routes by path prefix (config-mgmt environments/cloud-connect-prd/rules.tf):
cloud.gitlab.com prefix | Origin | Runtime |
|---|---|---|
/ai, /auth | ai-gateway.runway.gitlab.net | Cloud Run |
/DuoWorkflow | duo-workflow-svc.runway.gitlab.net | Cloud Run |
/aws/ai, /aws/auth | ai-gateway-eks.cloud.gitlab.com | EKS |
/aws/DuoWorkflow | duo-workflow-svc-eks.us-east-1.aws.runway.gitlab.net | EKS |
GitLab.com Rails builds AI gateway URLs as CloudConnector::Config.base_url plus /ai or /auth
(ee/lib/gitlab/ai_gateway.rb), so GitLab.com traffic goes to Cloud Run. The EKS deployments
serve only the /aws/* prefixes.
Probes that matter for this page:
| Service | Runtime | Startup probe | Readiness probe | Liveness probe |
|---|---|---|---|---|
| AI gateway | Cloud Run (.runway/runway.yml) | /monitoring/ready: 20 s initial delay, 17 s period, 10 s timeout, failure_threshold: 24 (about 7 min) | None | /monitoring/healthz |
| AI gateway | EKS (.runway/ai-gateway-eks/default-values.yaml) | /monitoring/healthz | /monitoring/ready: 30 s period, 5 s timeout, failure_threshold: 2 (about 60 s) | /monitoring/healthz |
| DWS | Cloud Run (.runway/duo-workflow-svc/runway-production.yml) | Runway default | None | Runway default |
| DWS | EKS (.runway/duo-workflow-svc-eks/default-values.yaml) | 15 s period, failure_threshold: 20 (about 5 min) | 10 s period, failure_threshold: 3 | 15 s period, failure_threshold: 4 |
Request paths
Section titled “Request paths”Direct to AI gateway (token from GitLab.com, request bypasses it)
Section titled “Direct to AI gateway (token from GitLab.com, request bypasses it)”| Feature | Token fetch (needs GitLab.com) | Request path | Survives a GitLab.com outage? |
|---|---|---|---|
| Code completion (IDE) | POST /api/v4/code_suggestions/direct_access | IDE → cloud.gitlab.com (AI gateway) | Yes, until the token expires (≤ 1 h), on AI gateway instances that were already running when the outage started |
| Third-party agents | POST /api/v4/ai/third_party_agents/direct_access | Client → AI gateway model-provider proxy | Same as above |
Proxied through the monolith
Section titled “Proxied through the monolith”| Feature | Path | Survives a GitLab.com outage? |
|---|---|---|
| Code generation and streaming (IDE), and completion fallback | POST /api/v4/code_suggestions/completions: Rails runs the checks, then Workhorse SendURL forwards to the AI gateway | No. It fails on every request |
| Duo Chat (classic) and other AI actions | GraphQL aiAction → Llm::CompletionWorker (Sidekiq, retry: 3) → AI gateway | No |
| Chat completions API | POST /api/v4/chat/completions (synchronous in Puma) | No |
| Duo Code Review | Llm::ReviewMergeRequestService → Llm::CompletionWorker | No |
| Duo Agent Platform: IDE flows and agentic chat | GET /api/v4/ai/duo_workflows/ws: Workhorse pre-authorizes with Rails, upgrades the connection, then opens a gRPC stream to DWS | No. New connections fail at pre-authorization. Existing streams stay open, but each tool call and checkpoint fails (see below) |
| Duo Agent Platform: web and remote flows | POST /api/v4/ai/duo_workflows/workflows/:id/execute (Workhorse BuildHTTP → Rails pre-authorization) | No |
| DWS tool calls and MCP | DWS sends an action over the stream, and the executor (Workhorse or the IDE) sends it to the GitLab API with the workflow OAuth token | No. Every call is a full Rails API request |
| DWS callbacks: status, checkpoints, audit events | /api/v4/ai/duo_workflows/workflows/:id/{,checkpoints,audit_events} via the executor | No. Status and checkpoint failures abort the flow. Audit events are dropped after 3 retries |
| GitLab MCP server | /api/v4/mcp | No |
DWS has no HTTP client of its own for GitLab. Every call it makes to GitLab goes through the
executor (ExecutorGitLabHttpClient). On GitLab.com the executor is Workhorse or the user’s IDE,
so a DWS session depends on GitLab.com however it validates tokens.
Tokens
Section titled “Tokens”| Token | Issued by | TTL | Database at issuance | Database at validation |
|---|---|---|---|---|
| Code Suggestions user JWT | AI gateway POST /v1/code/user_access_token, called by Rails during direct_access (3 s timeout) | 1 h (TokenAuthority, fixed) | Yes: many policy and namespace checks in Rails | No. The AI gateway validates against its JWKS cache |
| Cloud Connector self-signed JWT (Rails → AI gateway and DWS) | Rails CloudConnector::Tokens.issue_token, RS256 | 1 h | Yes: an AddOnPurchase query on every mint (not cached). The signing key is loaded once per process from cloud_connector_keys | No. AI gateway and DWS use JWKS |
| Duo Workflow OAuth token (human user) | Ai::DuoWorkflows::CreateOauthAccessTokenService | 2 h | INSERT into oauth_access_tokens on the primary, a new token on every call | Yes: OauthAccessToken.by_token |
| Duo Workflow composite-identity and autonomous tokens | CreateCompositeOauthAccessTokenService, CreateAutonomousOauthAccessTokenService | 1 h | INSERT plus audit event | Yes |
Stateless access token (glsat-) | Authn::Tokens::StatelessAccessToken, behind the duo_workflow_use_token_issuer feature flag (default off) | 2 h for user tokens, 1 h for composite | No token row written | Still yes: cloud_connector_keys and users reads. Revocation is checked against a Redis denylist |
| DWS token | DWS GenerateToken, minted locally | 1 h | Rails mints a Cloud Connector JWT to call it. DWS checks usage quota with CustomersDot | DWS JWKS cache. The token is no longer given to clients: direct_access returns token: '' and only the expiry |
| IDE OAuth token | Doorkeeper /oauth/token | 2 h (oauth_access_token_expires_in, default 7200 s) | Refresh writes to Postgres | Yes |
Rails-side caching
Section titled “Rails-side caching”- Rails can cache the direct-access user JWT in
Rails.cachewith TTLexpires_at - 5 min. This is behind the feature flagcache_code_suggestions_direct_access_token, which defaults to off. It does not help during a Postgres outage, because the cache is read only after the database checks have run. - A failed paid-namespace check is cached for 10 min, behind
cache_code_suggestions_paid_namespace_check(default off). Only negative results are cached. - No token caching was found for Duo Workflow
direct_accessor for/wspre-authorization.
Endpoints served by GitLab.com
Section titled “Endpoints served by GitLab.com”| Endpoint | Cache headers | Key source | Needs Postgres? |
|---|---|---|---|
/oauth/discovery/keys (alias /-/jwks) | Cache-Control: max-age=86400, public, must-revalidate | OpenID signing key (credentials), CI JWT signing key (application settings, cached 60 s in process), and EE: CloudConnector::Keys.all_as_pem | Yes, an uncached SELECT on cloud_connector_keys per request |
/-/cloud_connector/keys | max-age=3600, public, must-revalidate | CloudConnector::Keys.valid | Yes, a SELECT per request |
The AI gateway and DWS do not hard-code a JWKS URL. They read
<gitlab_url>/.well-known/openid-configuration and follow its jwks_uri. CustomersDot is
fetched the same way. To confirm: which URL jwks_uri resolves to in production, and whether
anything in front of GitLab.com (Cloudflare) serves these responses from cache during an outage.
AI gateway and DWS key cache
Section titled “AI gateway and DWS key cache”Both services use gitlab-cloud-connector (Python, 3.28.1) with providers LocalAuthProvider
(self-signed keys), GitLabOidcProvider("Gitlab") (AIGW_GITLAB_URL, default
https://gitlab.com), and GitLabOidcProvider("CustomersDot") (AIGW_CUSTOMER_PORTAL_URL).
| Behaviour | Value |
|---|---|
| Cache | In-process dict, one per worker process. Not shared, not persisted |
| TTL | 24 h (CompositeProvider default expiry_seconds=86400, not overridden) |
| Refresh trigger | Lazy, on the first request after TTL expiry. An unknown kid does not trigger a refresh |
| Fetch timeout and retries | 10 s per HTTP call, one retry per endpoint. The call is synchronous on the request path, so worst case is about 40 s per provider |
| Refresh fails, old cache exists | The whole old key set is cached for another 24 h (fail-safe, keeps validating). Fresh keys from the providers that did succeed are discarded. Log: Old JWKS re-cached: some key providers failed |
| Refresh fails, no old cache (cold worker) | The partial key set is cached for 24 h. The worker then rejects GitLab.com-signed tokens until its cache expires or a successful prefetch replaces it. Log: Incomplete JWKS cached: some key providers failed, no old cache to fall back to. The startup and readiness probes normally prevent this, because an instance gets no traffic until a full prefetch succeeds. It can still happen if a container runs several workers (WEB_CONCURRENCY > 1): only the worker that answered the probe is warmed. To confirm: the production WEB_CONCURRENCY, which is not set in the repository’s Runway config |
| No keys at all | CriticalAuthError("No keys founds in JWKS; are OIDC providers up?"). Requests get 401 |
| One provider down | Tolerated. Tokens signed by the other providers still validate |
| Validation | RS256 with aud checked. exp, nbf and iat are checked with 0 s leeway. iss is not checked. Realm and instance ID claims must match the request headers. Nothing is called per request |
Existing alerting: the Elastic watcher rule cloud_connector_ai_gw_jwks_fetch_fail matches the
two log messages above.
AI gateway readiness is coupled to GitLab.com
Section titled “AI gateway readiness is coupled to GitLab.com”/monitoring/ready runs validate_cloud_connector_ready, which calls prefetch_jwks(). That
fetches from all providers live every time it runs, and fails if any of them fails. It does
not fall back to the cache. /monitoring/healthz has no dependencies. How much this matters
depends on the runtime (see Runtimes).
Cloud Run (GitLab.com traffic): /monitoring/ready is only the startup probe, and liveness
uses /monitoring/healthz.
- Instances that were already running keep serving direct Code Suggestions from cached keys.
- New instances cannot pass the startup probe while GitLab.com cannot serve OIDC discovery or
JWKS, so scale-up and deploys stall. Each new instance gives up after about 7 minutes and Cloud
Run retries it. Capacity is limited to the instances that were running when the outage began
(
min_instances: 8per region). - To confirm against Cloud Run instance counts, startup probe failures and
cloud.gitlab.com5xx metrics from INC-15105.
EKS (cloud.gitlab.com/aws/* only): /monitoring/ready is also the readiness probe, with a
30 s period, 5 s timeout and failure_threshold: 2.
- Pods are expected to go NotReady about 60 s after GitLab.com (or CustomersDot) stops serving OIDC discovery or JWKS, even though their cached keys are still valid.
- Liveness uses
/monitoring/healthz, so pods are taken out of rotation but not restarted. - The library’s 10 s fetch timeout is longer than the 5 s probe timeout.
On both runtimes the fetch is synchronous and blocks the event loop while it runs.
DWS startup is coupled to GitLab.com
Section titled “DWS startup is coupled to GitLab.com”The DWS gRPC authentication interceptor raises AuthenticationError("Could not prefetch keys") at
startup unless all providers return keys. The interceptor is built when the server starts, so
the process fails to start. On both Cloud Run and EKS, a new or restarted DWS instance cannot start
while GitLab.com JWKS is unavailable. Already-running instances keep serving from cache. The DWS
gRPC health check does not check GitLab.com or JWKS.
CustomersDot usage quota (AI gateway and DWS)
Section titled “CustomersDot usage quota (AI gateway and DWS)”When usage quota checks are enabled, the AI gateway and DWS call CustomersDot
HEAD /api/v1/consumers/resolve with a 1 s timeout, and cache successful answers for 1 h
(± 10% jitter). Errors fail closed: the AI gateway returns 403 USAGE_QUOTA_NOT_ENTITLED
and DWS returns PERMISSION_DENIED. This does not depend on GitLab.com, but it is a second
per-request dependency to rule out when Duo returns 403s.
Client behaviour (GitLab Language Server and VS Code)
Section titled “Client behaviour (GitLab Language Server and VS Code)”JetBrains and other clients have not been checked yet.
Code Suggestions direct-access token
Section titled “Code Suggestions direct-access token”| Behaviour | Value |
|---|---|
| Cache | In memory in the Language Server. Lost on restart |
| Refresh | Lazy, on the next completion request after expires_at - 40 s |
| Refresh retries | 3 attempts, backoff 1 s × 1.5, 10% jitter, honours Retry-After. 5xx is retried. 429, timeouts and network errors are not |
| Refresh fails | The error is logged, and the old, expired token is kept and still sent |
| Fallback | If the direct connection is unavailable, the request falls back to the monolith (/api/v4/code_suggestions/completions) |
| Direct-connection circuit breaker | Exponential, 1 s → 5 s → 25 s → 125 s → 600 s (cap), 5 s request timeout. It is reset only when the API is reconfigured, not after a successful request, so after a handful of failures each new failure opens it for 10 min |
| Overall suggestions circuit breaker | Opens after 4 errors for 10 s. Resets on the first success |
| Token cache cleared | On any API reconfiguration, including the OAuth token refresh below |
A user whose token is still valid when GitLab.com goes down keeps getting completions until the token expires, as long as an AI gateway instance that was already running serves the request. After that, refresh fails, and the fallback to the monolith also fails.
OAuth access token
Section titled “OAuth access token”- Refresh runs in the VS Code extension, not the Language Server. A timer fires at token expiry (2 h TTL). On failure it retries indefinitely with exponential backoff, 1 s up to 60 s, and only logs the failure.
- While the Language Server cannot validate the token (5xx, rate limit or network error on
/oauth/token/info), it marks the API state invalid. That disables suggestions, the direct connection and Duo Agent Platform. It revalidates without limit (5 s → 60 s), so it recovers automatically when GitLab.com is back. - Only a definitive 401
invalid_tokenorinvalid_grantprompts the user to re-authenticate. - To confirm: whether a refresh that the database commits but whose response is lost during
saturation leaves the client with
invalid_grant, which would force a re-authentication.
Duo Agent Platform
Section titled “Duo Agent Platform”- The Language Server connects through the WebSocket at
/api/v4/ai/duo_workflows/wswith the IDE’s OAuth token, not over gRPC to DWS. - Workflow tokens come from
direct_access. They are cached per workflow untilmin(DWS expiry, Rails expiry) - 30 s. On GitLab 19.2 and later, server capabilities come from metadata anddirect_accessis skipped. - Errors: a 5xx, 429 or network error on token fetch or upgrade →
SERVICE_CONNECTION_FAILED(retryable). 401 or 403 →AUTH_TOKEN_ERROR(not retried). 1006 →SERVICE_CONNECTION_DROPPED. - Retry budget: 5 attempts, backoff 0, 1, 2, 4, 8 s with up to 50% jitter, about 15 s in total. A longer outage fails the flow or chat turn. The user has to resend the message or restart the flow. Nothing resumes automatically.
- User-facing message: “Workflow could not connect to the Workflow service.”
DWS tool-call retries
Section titled “DWS tool-call retries”Executor HTTP and GraphQL calls from DWS retry 4 times (3 s, 9 s, 27 s, about 39 s in total) on
5xx, 401, 429, timeouts and network errors. They honour Retry-After up to 30 s. After that, REST
calls return the last response to the tool, and GraphQL calls raise.
What fails when GitLab.com Postgres is unavailable
Section titled “What fails when GitLab.com Postgres is unavailable”| Feature | During the outage | After GitLab.com recovers |
|---|---|---|
| Code completion, valid token held | Works until the token expires (≤ 1 h), on AI gateway instances that were already running. Capacity cannot grow (Cloud Run) | — |
| Code completion, token expired or LS restarted | Fails: refresh 503, and the monolith fallback 503 | The monolith fallback recovers within about 10 s. Direct AI gateway traffic can stay off for up to 10 min because of the circuit breaker |
| Code generation and streaming | Fails | Recovers on the next request |
| Duo Chat (classic), Code Review | Fails. Sidekiq retries jobs 3 times | Recovers. Jobs that ran out of retries are lost |
| Duo Agent Platform, new session | Fails at WebSocket pre-authorization (503) | Recovers. The user starts again |
| Duo Agent Platform, running session | The stream stays open, but tool calls and checkpoint saves fail and the flow aborts | Not resumed automatically |
| IDE OAuth refresh | Fails, and is retried without limit | Recovers automatically, unless the client ends up with invalid_grant (to confirm) |
| JWKS endpoints | Fail when not served from a cache | — |
| AI gateway instances | Running instances keep serving, but new instances cannot start (Cloud Run). On EKS, pods are expected to go NotReady | New instances start and EKS pods become Ready on the next successful probe |
| New DWS instances | Cannot start (Cloud Run and EKS) | Start normally |
Recovery checks
Section titled “Recovery checks”Main-site HTTP recovery is not enough to confirm that Duo has recovered. After GitLab.com recovers, check each of these separately:
- AI gateway capacity: new Cloud Run instances pass the startup probe, the instance count
is back to normal, and
cloud.gitlab.com5xx is back to baseline (AI gateway overview). For the/aws/*routes, also check that EKS pods are Ready. - JWKS: the
cloud_connector_ai_gw_jwks_fetch_failwatcher has stopped firing, and there are no newIncomplete JWKS cachedlog lines from AI gateway or DWS instances. If there are, restart those instances. Otherwise they keep rejecting GitLab.com-signed tokens for up to 24 h. - Token issuance: the 5xx rate on Rails
POST /api/v4/code_suggestions/direct_accessandPOST /api/v4/ai/duo_workflows/direct_accessis back to baseline. These endpoints have no SLI of their own, so check them in Rails API logs, filtered by route. - Duo Agent Platform: WebSocket upgrades on
/api/v4/ai/duo_workflows/wssucceed (Workhorse orai-assistedfleet logs). Theworkhorse_dws_sessionsSLI covers the gRPC stream, so it probably does not count rejected upgrades. - DWS: the DWS instance count is back to normal and no instance is failing startup with
Could not prefetch keys. - Code Suggestions traffic: the
server_code_completionsrequest rate on the AI gateway is back to its normal level. Allow about 10 min for the client circuit breakers to close before treating a low rate as a continuing problem.
Known gaps and follow-ups
Section titled “Known gaps and follow-ups”- AI gateway
/monitoring/readyre-fetches JWKS live instead of passing when the cache holds valid keys. On Cloud Run this blocks scale-up during a GitLab.com outage. On EKS it takes pods out of rotation. - DWS refuses to start without a complete key prefetch.
- A JWKS refresh discards the fresh keys from providers that succeeded instead of merging them.
- A cold worker caches a partial key set for the full 24 h.
- An unknown
kidnever triggers a refresh, so a rotated key is rejected until the TTL expires. - The JWKS fetch is synchronous inside an async server.
- Both JWKS endpoints query Postgres on every uncached request.
- The Language Server direct-connection circuit breaker is never reset by a successful request.
Sources
Section titled “Sources”Code read at these commits (2026-11-02, AI gateway and routing updated 2026-11-03):
| Repository | Commit | Key files |
|---|---|---|
| gitlab-org/gitlab | 02a640a2 | ee/lib/api/code_suggestions.rb, ee/lib/api/ai/duo_workflows/workflows.rb, ee/lib/gitlab/llm/ai_gateway/code_suggestions_client.rb, ee/lib/cloud_connector/tokens.rb, ee/app/services/ai/duo_workflows/create_oauth_access_token_service.rb, app/controllers/jwks_controller.rb, ee/app/controllers/cloud_connector/keys_controller.rb, workhorse/internal/ai_assist/duoworkflow/ |
| gitlab-org/modelops/applied-ml/code-suggestions/ai-assist (AI gateway, DWS) | 77bd76ef | ai_gateway/api/server.py, ai_gateway/api/monitoring.py, duo_workflow_service/interceptors/authentication_interceptor.py, duo_workflow_service/gitlab/executor_http_client.py, scripts/run.sh, .runway/runway.yml, .runway/ai-gateway-eks/default-values.yaml, .runway/duo-workflow-svc/runway-production.yml, .runway/duo-workflow-svc-eks/default-values.yaml |
| gitlab-org/cloud-connector/gitlab-cloud-connector | python-v3.28.1 | src/python/gitlab_cloud_connector/providers.py, probes.py, token_authority.py |
| gitlab-com/gl-infra/config-mgmt | main (2026-11-03) | environments/cloud-connect-prd/rules.tf |
| gitlab-org/editor-extensions/gitlab-lsp | 9d43884f | packages/ls/legacy/common/suggestion/direct_connection_{client,details_service}.ts, packages/ls/legacy/common/api.ts, packages/core/workflow_executor/src/executors/node/ |
| gitlab-org/gitlab-vscode-extension | 0cd38f5f | src/desktop/accounts/token_refresh_service.ts, src/desktop/gitlab/token_exchange_service.ts |