Skip to content

Duo dependencies on GitLab.com authentication

This page maps which GitLab Duo request paths depend on GitLab.com (Rails and Postgres) for authentication, how tokens and JWKS are cached along each path, and what fails when GitLab.com is unavailable. Use it during an incident to predict which Duo features break, and to check that Duo has actually recovered once the main site has.

Background: INC-15105, corrective action production-engineering#30190. During that incident, Postgres I/O stalls caused HTTP 503s on GitLab.com. Duo also returned 503s on token fetch and on the Duo Workflow WebSocket.

Related runbooks:

  • Every token-issuing endpoint needs Postgres on every request. Authentication (PAT or OAuth lookup), policy checks, seat or add-on checks, and the Cloud Connector token mint all read the database. While Postgres is unavailable, no client can get a new Duo token.
  • The only path that keeps working is direct Code Suggestions with a token already issued. The IDE talks to the AI gateway at cloud.gitlab.com with a user JWT, and the AI gateway checks it against keys it already has in memory. This works until the 1 h token expires.
  • Duo Agent Platform (flows, agentic chat) depends on GitLab.com throughout. The WebSocket goes through Workhorse and Rails pre-authorization on every connection. Every tool call and checkpoint goes back through Rails. Clients no longer receive a Duo Workflow Service (DWS) token that would let them bypass Rails.
  • GitLab.com Duo traffic is served by Cloud Run. EKS serves only the cloud.gitlab.com/aws/* routes. See Runtimes.
  • The AI gateway /monitoring/ready check fetches GitLab.com OIDC discovery and JWKS live. On Cloud Run it is only the startup probe. Running instances keep serving direct Code Suggestions from cached keys, but new instances cannot start while GitLab.com is down, so scale-up and deploys stall. On EKS it is also the readiness probe, so pods are expected to go NotReady in about 60 s. See AI gateway readiness.
  • New DWS instances cannot start while GitLab.com JWKS is unavailable, on either runtime, because DWS requires a complete key prefetch at startup. Running instances keep serving from cache.
  • Main-site recovery does not mean Duo has recovered. Some client-side circuit breakers keep direct Code Suggestions off for up to 10 minutes after GitLab.com recovers. Failed agent flows are not resumed automatically. See Recovery checks.

The AI gateway and DWS are deployed by Runway to both Cloud Run and EKS. Cloudflare on cloud.gitlab.com routes by path prefix (config-mgmt environments/cloud-connect-prd/rules.tf):

cloud.gitlab.com prefixOriginRuntime
/ai, /authai-gateway.runway.gitlab.netCloud Run
/DuoWorkflowduo-workflow-svc.runway.gitlab.netCloud Run
/aws/ai, /aws/authai-gateway-eks.cloud.gitlab.comEKS
/aws/DuoWorkflowduo-workflow-svc-eks.us-east-1.aws.runway.gitlab.netEKS

GitLab.com Rails builds AI gateway URLs as CloudConnector::Config.base_url plus /ai or /auth (ee/lib/gitlab/ai_gateway.rb), so GitLab.com traffic goes to Cloud Run. The EKS deployments serve only the /aws/* prefixes.

Probes that matter for this page:

ServiceRuntimeStartup probeReadiness probeLiveness probe
AI gatewayCloud Run (.runway/runway.yml)/monitoring/ready: 20 s initial delay, 17 s period, 10 s timeout, failure_threshold: 24 (about 7 min)None/monitoring/healthz
AI gatewayEKS (.runway/ai-gateway-eks/default-values.yaml)/monitoring/healthz/monitoring/ready: 30 s period, 5 s timeout, failure_threshold: 2 (about 60 s)/monitoring/healthz
DWSCloud Run (.runway/duo-workflow-svc/runway-production.yml)Runway defaultNoneRunway default
DWSEKS (.runway/duo-workflow-svc-eks/default-values.yaml)15 s period, failure_threshold: 20 (about 5 min)10 s period, failure_threshold: 315 s period, failure_threshold: 4

Direct to AI gateway (token from GitLab.com, request bypasses it)

Section titled “Direct to AI gateway (token from GitLab.com, request bypasses it)”
FeatureToken fetch (needs GitLab.com)Request pathSurvives a GitLab.com outage?
Code completion (IDE)POST /api/v4/code_suggestions/direct_accessIDE → cloud.gitlab.com (AI gateway)Yes, until the token expires (≤ 1 h), on AI gateway instances that were already running when the outage started
Third-party agentsPOST /api/v4/ai/third_party_agents/direct_accessClient → AI gateway model-provider proxySame as above
FeaturePathSurvives a GitLab.com outage?
Code generation and streaming (IDE), and completion fallbackPOST /api/v4/code_suggestions/completions: Rails runs the checks, then Workhorse SendURL forwards to the AI gatewayNo. It fails on every request
Duo Chat (classic) and other AI actionsGraphQL aiAction → Llm::CompletionWorker (Sidekiq, retry: 3) → AI gatewayNo
Chat completions APIPOST /api/v4/chat/completions (synchronous in Puma)No
Duo Code ReviewLlm::ReviewMergeRequestService → Llm::CompletionWorkerNo
Duo Agent Platform: IDE flows and agentic chatGET /api/v4/ai/duo_workflows/ws: Workhorse pre-authorizes with Rails, upgrades the connection, then opens a gRPC stream to DWSNo. New connections fail at pre-authorization. Existing streams stay open, but each tool call and checkpoint fails (see below)
Duo Agent Platform: web and remote flowsPOST /api/v4/ai/duo_workflows/workflows/:id/execute (Workhorse BuildHTTP → Rails pre-authorization)No
DWS tool calls and MCPDWS sends an action over the stream, and the executor (Workhorse or the IDE) sends it to the GitLab API with the workflow OAuth tokenNo. Every call is a full Rails API request
DWS callbacks: status, checkpoints, audit events/api/v4/ai/duo_workflows/workflows/:id/{,checkpoints,audit_events} via the executorNo. Status and checkpoint failures abort the flow. Audit events are dropped after 3 retries
GitLab MCP server/api/v4/mcpNo

DWS has no HTTP client of its own for GitLab. Every call it makes to GitLab goes through the executor (ExecutorGitLabHttpClient). On GitLab.com the executor is Workhorse or the user’s IDE, so a DWS session depends on GitLab.com however it validates tokens.

TokenIssued byTTLDatabase at issuanceDatabase at validation
Code Suggestions user JWTAI gateway POST /v1/code/user_access_token, called by Rails during direct_access (3 s timeout)1 h (TokenAuthority, fixed)Yes: many policy and namespace checks in RailsNo. The AI gateway validates against its JWKS cache
Cloud Connector self-signed JWT (Rails → AI gateway and DWS)Rails CloudConnector::Tokens.issue_token, RS2561 hYes: an AddOnPurchase query on every mint (not cached). The signing key is loaded once per process from cloud_connector_keysNo. AI gateway and DWS use JWKS
Duo Workflow OAuth token (human user)Ai::DuoWorkflows::CreateOauthAccessTokenService2 hINSERT into oauth_access_tokens on the primary, a new token on every callYes: OauthAccessToken.by_token
Duo Workflow composite-identity and autonomous tokensCreateCompositeOauthAccessTokenService, CreateAutonomousOauthAccessTokenService1 hINSERT plus audit eventYes
Stateless access token (glsat-)Authn::Tokens::StatelessAccessToken, behind the duo_workflow_use_token_issuer feature flag (default off)2 h for user tokens, 1 h for compositeNo token row writtenStill yes: cloud_connector_keys and users reads. Revocation is checked against a Redis denylist
DWS tokenDWS GenerateToken, minted locally1 hRails mints a Cloud Connector JWT to call it. DWS checks usage quota with CustomersDotDWS JWKS cache. The token is no longer given to clients: direct_access returns token: '' and only the expiry
IDE OAuth tokenDoorkeeper /oauth/token2 h (oauth_access_token_expires_in, default 7200 s)Refresh writes to PostgresYes
  • Rails can cache the direct-access user JWT in Rails.cache with TTL expires_at - 5 min. This is behind the feature flag cache_code_suggestions_direct_access_token, which defaults to off. It does not help during a Postgres outage, because the cache is read only after the database checks have run.
  • A failed paid-namespace check is cached for 10 min, behind cache_code_suggestions_paid_namespace_check (default off). Only negative results are cached.
  • No token caching was found for Duo Workflow direct_access or for /ws pre-authorization.
EndpointCache headersKey sourceNeeds Postgres?
/oauth/discovery/keys (alias /-/jwks)Cache-Control: max-age=86400, public, must-revalidateOpenID signing key (credentials), CI JWT signing key (application settings, cached 60 s in process), and EE: CloudConnector::Keys.all_as_pemYes, an uncached SELECT on cloud_connector_keys per request
/-/cloud_connector/keysmax-age=3600, public, must-revalidateCloudConnector::Keys.validYes, a SELECT per request

The AI gateway and DWS do not hard-code a JWKS URL. They read <gitlab_url>/.well-known/openid-configuration and follow its jwks_uri. CustomersDot is fetched the same way. To confirm: which URL jwks_uri resolves to in production, and whether anything in front of GitLab.com (Cloudflare) serves these responses from cache during an outage.

Both services use gitlab-cloud-connector (Python, 3.28.1) with providers LocalAuthProvider (self-signed keys), GitLabOidcProvider("Gitlab") (AIGW_GITLAB_URL, default https://gitlab.com), and GitLabOidcProvider("CustomersDot") (AIGW_CUSTOMER_PORTAL_URL).

BehaviourValue
CacheIn-process dict, one per worker process. Not shared, not persisted
TTL24 h (CompositeProvider default expiry_seconds=86400, not overridden)
Refresh triggerLazy, on the first request after TTL expiry. An unknown kid does not trigger a refresh
Fetch timeout and retries10 s per HTTP call, one retry per endpoint. The call is synchronous on the request path, so worst case is about 40 s per provider
Refresh fails, old cache existsThe whole old key set is cached for another 24 h (fail-safe, keeps validating). Fresh keys from the providers that did succeed are discarded. Log: Old JWKS re-cached: some key providers failed
Refresh fails, no old cache (cold worker)The partial key set is cached for 24 h. The worker then rejects GitLab.com-signed tokens until its cache expires or a successful prefetch replaces it. Log: Incomplete JWKS cached: some key providers failed, no old cache to fall back to. The startup and readiness probes normally prevent this, because an instance gets no traffic until a full prefetch succeeds. It can still happen if a container runs several workers (WEB_CONCURRENCY > 1): only the worker that answered the probe is warmed. To confirm: the production WEB_CONCURRENCY, which is not set in the repository’s Runway config
No keys at allCriticalAuthError("No keys founds in JWKS; are OIDC providers up?"). Requests get 401
One provider downTolerated. Tokens signed by the other providers still validate
ValidationRS256 with aud checked. exp, nbf and iat are checked with 0 s leeway. iss is not checked. Realm and instance ID claims must match the request headers. Nothing is called per request

Existing alerting: the Elastic watcher rule cloud_connector_ai_gw_jwks_fetch_fail matches the two log messages above.

AI gateway readiness is coupled to GitLab.com

Section titled “AI gateway readiness is coupled to GitLab.com”

/monitoring/ready runs validate_cloud_connector_ready, which calls prefetch_jwks(). That fetches from all providers live every time it runs, and fails if any of them fails. It does not fall back to the cache. /monitoring/healthz has no dependencies. How much this matters depends on the runtime (see Runtimes).

Cloud Run (GitLab.com traffic): /monitoring/ready is only the startup probe, and liveness uses /monitoring/healthz.

  • Instances that were already running keep serving direct Code Suggestions from cached keys.
  • New instances cannot pass the startup probe while GitLab.com cannot serve OIDC discovery or JWKS, so scale-up and deploys stall. Each new instance gives up after about 7 minutes and Cloud Run retries it. Capacity is limited to the instances that were running when the outage began (min_instances: 8 per region).
  • To confirm against Cloud Run instance counts, startup probe failures and cloud.gitlab.com 5xx metrics from INC-15105.

EKS (cloud.gitlab.com/aws/* only): /monitoring/ready is also the readiness probe, with a 30 s period, 5 s timeout and failure_threshold: 2.

  • Pods are expected to go NotReady about 60 s after GitLab.com (or CustomersDot) stops serving OIDC discovery or JWKS, even though their cached keys are still valid.
  • Liveness uses /monitoring/healthz, so pods are taken out of rotation but not restarted.
  • The library’s 10 s fetch timeout is longer than the 5 s probe timeout.

On both runtimes the fetch is synchronous and blocks the event loop while it runs.

The DWS gRPC authentication interceptor raises AuthenticationError("Could not prefetch keys") at startup unless all providers return keys. The interceptor is built when the server starts, so the process fails to start. On both Cloud Run and EKS, a new or restarted DWS instance cannot start while GitLab.com JWKS is unavailable. Already-running instances keep serving from cache. The DWS gRPC health check does not check GitLab.com or JWKS.

CustomersDot usage quota (AI gateway and DWS)

Section titled “CustomersDot usage quota (AI gateway and DWS)”

When usage quota checks are enabled, the AI gateway and DWS call CustomersDot HEAD /api/v1/consumers/resolve with a 1 s timeout, and cache successful answers for 1 h (± 10% jitter). Errors fail closed: the AI gateway returns 403 USAGE_QUOTA_NOT_ENTITLED and DWS returns PERMISSION_DENIED. This does not depend on GitLab.com, but it is a second per-request dependency to rule out when Duo returns 403s.

Client behaviour (GitLab Language Server and VS Code)

Section titled “Client behaviour (GitLab Language Server and VS Code)”

JetBrains and other clients have not been checked yet.

BehaviourValue
CacheIn memory in the Language Server. Lost on restart
RefreshLazy, on the next completion request after expires_at - 40 s
Refresh retries3 attempts, backoff 1 s × 1.5, 10% jitter, honours Retry-After. 5xx is retried. 429, timeouts and network errors are not
Refresh failsThe error is logged, and the old, expired token is kept and still sent
FallbackIf the direct connection is unavailable, the request falls back to the monolith (/api/v4/code_suggestions/completions)
Direct-connection circuit breakerExponential, 1 s → 5 s → 25 s → 125 s → 600 s (cap), 5 s request timeout. It is reset only when the API is reconfigured, not after a successful request, so after a handful of failures each new failure opens it for 10 min
Overall suggestions circuit breakerOpens after 4 errors for 10 s. Resets on the first success
Token cache clearedOn any API reconfiguration, including the OAuth token refresh below

A user whose token is still valid when GitLab.com goes down keeps getting completions until the token expires, as long as an AI gateway instance that was already running serves the request. After that, refresh fails, and the fallback to the monolith also fails.

  • Refresh runs in the VS Code extension, not the Language Server. A timer fires at token expiry (2 h TTL). On failure it retries indefinitely with exponential backoff, 1 s up to 60 s, and only logs the failure.
  • While the Language Server cannot validate the token (5xx, rate limit or network error on /oauth/token/info), it marks the API state invalid. That disables suggestions, the direct connection and Duo Agent Platform. It revalidates without limit (5 s → 60 s), so it recovers automatically when GitLab.com is back.
  • Only a definitive 401 invalid_token or invalid_grant prompts the user to re-authenticate.
  • To confirm: whether a refresh that the database commits but whose response is lost during saturation leaves the client with invalid_grant, which would force a re-authentication.
  • The Language Server connects through the WebSocket at /api/v4/ai/duo_workflows/ws with the IDE’s OAuth token, not over gRPC to DWS.
  • Workflow tokens come from direct_access. They are cached per workflow until min(DWS expiry, Rails expiry) - 30 s. On GitLab 19.2 and later, server capabilities come from metadata and direct_access is skipped.
  • Errors: a 5xx, 429 or network error on token fetch or upgrade → SERVICE_CONNECTION_FAILED (retryable). 401 or 403 → AUTH_TOKEN_ERROR (not retried). 1006 → SERVICE_CONNECTION_DROPPED.
  • Retry budget: 5 attempts, backoff 0, 1, 2, 4, 8 s with up to 50% jitter, about 15 s in total. A longer outage fails the flow or chat turn. The user has to resend the message or restart the flow. Nothing resumes automatically.
  • User-facing message: “Workflow could not connect to the Workflow service.”

Executor HTTP and GraphQL calls from DWS retry 4 times (3 s, 9 s, 27 s, about 39 s in total) on 5xx, 401, 429, timeouts and network errors. They honour Retry-After up to 30 s. After that, REST calls return the last response to the tool, and GraphQL calls raise.

What fails when GitLab.com Postgres is unavailable

Section titled “What fails when GitLab.com Postgres is unavailable”
FeatureDuring the outageAfter GitLab.com recovers
Code completion, valid token heldWorks until the token expires (≤ 1 h), on AI gateway instances that were already running. Capacity cannot grow (Cloud Run)—
Code completion, token expired or LS restartedFails: refresh 503, and the monolith fallback 503The monolith fallback recovers within about 10 s. Direct AI gateway traffic can stay off for up to 10 min because of the circuit breaker
Code generation and streamingFailsRecovers on the next request
Duo Chat (classic), Code ReviewFails. Sidekiq retries jobs 3 timesRecovers. Jobs that ran out of retries are lost
Duo Agent Platform, new sessionFails at WebSocket pre-authorization (503)Recovers. The user starts again
Duo Agent Platform, running sessionThe stream stays open, but tool calls and checkpoint saves fail and the flow abortsNot resumed automatically
IDE OAuth refreshFails, and is retried without limitRecovers automatically, unless the client ends up with invalid_grant (to confirm)
JWKS endpointsFail when not served from a cache—
AI gateway instancesRunning instances keep serving, but new instances cannot start (Cloud Run). On EKS, pods are expected to go NotReadyNew instances start and EKS pods become Ready on the next successful probe
New DWS instancesCannot start (Cloud Run and EKS)Start normally

Main-site HTTP recovery is not enough to confirm that Duo has recovered. After GitLab.com recovers, check each of these separately:

  1. AI gateway capacity: new Cloud Run instances pass the startup probe, the instance count is back to normal, and cloud.gitlab.com 5xx is back to baseline (AI gateway overview). For the /aws/* routes, also check that EKS pods are Ready.
  2. JWKS: the cloud_connector_ai_gw_jwks_fetch_fail watcher has stopped firing, and there are no new Incomplete JWKS cached log lines from AI gateway or DWS instances. If there are, restart those instances. Otherwise they keep rejecting GitLab.com-signed tokens for up to 24 h.
  3. Token issuance: the 5xx rate on Rails POST /api/v4/code_suggestions/direct_access and POST /api/v4/ai/duo_workflows/direct_access is back to baseline. These endpoints have no SLI of their own, so check them in Rails API logs, filtered by route.
  4. Duo Agent Platform: WebSocket upgrades on /api/v4/ai/duo_workflows/ws succeed (Workhorse or ai-assisted fleet logs). The workhorse_dws_sessions SLI covers the gRPC stream, so it probably does not count rejected upgrades.
  5. DWS: the DWS instance count is back to normal and no instance is failing startup with Could not prefetch keys.
  6. Code Suggestions traffic: the server_code_completions request rate on the AI gateway is back to its normal level. Allow about 10 min for the client circuit breakers to close before treating a low rate as a continuing problem.
  • AI gateway /monitoring/ready re-fetches JWKS live instead of passing when the cache holds valid keys. On Cloud Run this blocks scale-up during a GitLab.com outage. On EKS it takes pods out of rotation.
  • DWS refuses to start without a complete key prefetch.
  • A JWKS refresh discards the fresh keys from providers that succeeded instead of merging them.
  • A cold worker caches a partial key set for the full 24 h.
  • An unknown kid never triggers a refresh, so a rotated key is rejected until the TTL expires.
  • The JWKS fetch is synchronous inside an async server.
  • Both JWKS endpoints query Postgres on every uncached request.
  • The Language Server direct-connection circuit breaker is never reset by a successful request.

Code read at these commits (2026-11-02, AI gateway and routing updated 2026-11-03):

RepositoryCommitKey files
gitlab-org/gitlab02a640a2ee/lib/api/code_suggestions.rb, ee/lib/api/ai/duo_workflows/workflows.rb, ee/lib/gitlab/llm/ai_gateway/code_suggestions_client.rb, ee/lib/cloud_connector/tokens.rb, ee/app/services/ai/duo_workflows/create_oauth_access_token_service.rb, app/controllers/jwks_controller.rb, ee/app/controllers/cloud_connector/keys_controller.rb, workhorse/internal/ai_assist/duoworkflow/
gitlab-org/modelops/applied-ml/code-suggestions/ai-assist (AI gateway, DWS)77bd76efai_gateway/api/server.py, ai_gateway/api/monitoring.py, duo_workflow_service/interceptors/authentication_interceptor.py, duo_workflow_service/gitlab/executor_http_client.py, scripts/run.sh, .runway/runway.yml, .runway/ai-gateway-eks/default-values.yaml, .runway/duo-workflow-svc/runway-production.yml, .runway/duo-workflow-svc-eks/default-values.yaml
gitlab-org/cloud-connector/gitlab-cloud-connectorpython-v3.28.1src/python/gitlab_cloud_connector/providers.py, probes.py, token_authority.py
gitlab-com/gl-infra/config-mgmtmain (2026-11-03)environments/cloud-connect-prd/rules.tf
gitlab-org/editor-extensions/gitlab-lsp9d43884fpackages/ls/legacy/common/suggestion/direct_connection_{client,details_service}.ts, packages/ls/legacy/common/api.ts, packages/core/workflow_executor/src/executors/node/
gitlab-org/gitlab-vscode-extension0cd38f5fsrc/desktop/accounts/token_refresh_service.ts, src/desktop/gitlab/token_exchange_service.ts