GitLab Duo in Slack
[[TOC]]
About GitLab Duo in Slack
Section titled “About GitLab Duo in Slack”Users mention @GitLab in a Slack channel and GitLab Duo answers in the thread.
Each mention starts or continues a Duo Agent Platform session.
The session runs in one top-level group chosen for the Slack workspace, called the Duo group.
It is part of the GitLab Rails application, not a separate service. A mention passes through these parts:
- Slack sends an event to the GitLab for Slack app endpoint.
- Rails checks the user, the channel, and the workspace’s Duo group.
- A Sidekiq job (
Ai::Messaging::ServerSideTurnWorker) starts the turn. - Workhorse and the Duo Workflow Service run it.
- Rails posts progress and the answer back to the Slack thread.
An outage means users get no reply, a late reply, or an error in Slack. It does not affect the rest of GitLab.
Contact Information
Section titled “Contact Information”- Team: External Agents (
group::ai catalog,category:external agents) - Slack: team channel
- Escalation: the
tier2 - AIon-call rotation. Use/inc escalatein Slack and selecttier2 - AIfrom the “Oncall team” dropdown (see How to Escalate). - Rollout issue: https://gitlab.com/gitlab-org/gitlab/-/work_items/638482
Requirements
Section titled “Requirements”- GitLab.com: Limited Availability. Self-Managed: Experiment, off by default.
- The Slack workspace has a Duo group, set in the group’s GitLab for Slack app settings.
- The user has linked their Slack and GitLab accounts.
- The user meets the Duo group’s seat and role rules.
- Channels shared with another organization (Slack Connect) are always refused.
Feature flags
Section titled “Feature flags”The “off” behavior below comes from reading the code. TODO confirm each row on staging.
| Flag | Scope | Effect of turning it off |
|---|---|---|
slack_duo_group | top-level group or instance | Mentions fall back to the old per-user experiment path. Does not stop Duo. |
slack_duo_api_flow | top-level group | In a Duo group: new mentions are refused, running sessions can no longer save progress, and approval buttons stop working. On the old per-user path: mentions run the Duo Developer flow instead. |
slack_duo_mcp_server | user | Duo loses Slack search and channel history tools. Mentions still work. |
duo_workflow_detached_server_side_execution | user | On for everyone on GitLab.com since 2026-11-13. When off, each turn holds a Sidekiq thread for its whole run (limit 50 at once). |
slack_app_workspace_rate_limit | instance | The per-workspace limit is logged but not enforced. |
slack_duo_agent_app | instance | New installs stop asking for the DM and agent view scopes. Not a rollback once the Slack app is published. |
To turn a flag off for everyone:
/chatops gitlab run feature set <flag_name> falseTurning off slack_duo_group does not stop Duo replies in Slack.
To stop one workspace, see Turn off one workspace.
To stop all of them, see Stop everything.
Initial Triage
Section titled “Initial Triage”Collect this before you start:
- Slack workspace ID (starts with
T) and channel - GitLab username of the affected user
- The Duo group (top-level group) of the workspace
- When it happened, in UTC
- A link to the Slack message or the Duo session, if there is one
- Whether it affects one user, one workspace, or everyone
NOTE: Do not put customer message content in public issues.
Is it a refusal or a failure?
Section titled “Is it a refusal or a failure?”A lock reaction on the message plus a short reply is a refusal. The feature is working and the user or group does not meet a requirement. The reply names the reason. Common ones:
| Reason | Meaning |
|---|---|
user_not_linked | The user has not linked Slack and GitLab accounts. |
externally_shared_channel | The channel is shared with another organization. |
duo_group_not_set_up | The workspace has no Duo group. |
duo_group_experiments_off | The Duo group has experiment and beta features off. |
duo_features_off, agent_platform_off | A Duo setting is off in the Duo group. |
not_licensed, seat, role | The group or user lacks the subscription, seat, or role. |
Refusals are counted in the block_slack_duo_mention event, with the reason as its property.
Where to look
Section titled “Where to look”- Duo in Slack health dashboard.
- Outcomes: sessions, completed, waiting for approval, broken, killed as stuck.
- Early warnings: turns waiting for a slot, deferrals.
- Rails and Sidekiq logs. Search for messages that start with
Duo Messaging:.Duo Messaging: flow triggeredandDuo Messaging: flow trigger failed(withfailure_reason)Duo Messaging: retries exhausted; giving up- TODO saved log searches
- Rate limit hits:
Application_Rate_Limiter_Requestwith aslack_workspacefield. - Duo Workflow Service dashboard
- TODO Sentry link
Alerting
Section titled “Alerting”TODO. Alerts are tracked in https://gitlab.com/gitlab-org/gitlab/-/work_items/634044. Add one section per alert here when they exist.
Common Resolution Steps
Section titled “Common Resolution Steps”Duo does not reply, or replies late
Section titled “Duo does not reply, or replies late”On GitLab.com, turns run detached: the Sidekiq job only starts the turn and Workhorse keeps it going.
Each Workhorse process holds at most 100 detached runs, and a run ends after 55 minutes at most.
When a process is full, it refuses new turns with a 503.
- Check the Duo Workflow Service dashboard. If it is unhealthy, follow its runbook. No Slack flag fixes that.
- Check for refused turns. Look for
too many detached workflow runsin the Workhorse logs, or a rise ingitlab_workhorse_duo_workflow_detached_runs_rejected_totalon the Workhorse Runs dashboard. Refusals mean the per-process limit is reached. Escalate to the External Agents team. - Check the “Early warnings” row of the health dashboard for turns waiting for a slot.
- If detached mode itself is failing, turning off
duo_workflow_detached_server_side_executionis the fallback. Turns then run inside the Sidekiq job, 50 at once, so expect queueing under load. The flag belongs togroup::agent foundations: see its rollout issue. - Confirm: no new refusals, turns waiting for a slot returns to zero, and a test mention gets a reply.
Duo replies with errors for everyone
Section titled “Duo replies with errors for everyone”- Check whether other Duo Agent Platform surfaces also fail. If yes, it is not a Slack problem. Follow the Duo Workflow Service and AI Gateway runbooks.
- If only Slack fails, look for
Duo Messaging: flow trigger failedand readfailure_reason. - Check for a recent deploy or flag change (
/chatops gitlab run feature get <flag_name>). - Escalate to the External Agents team. If the errors do harm and cannot wait, see Stop everything.
One workspace floods us with requests
Section titled “One workspace floods us with requests”- Find the workspace in the
Application_Rate_Limiter_Requestlogs (slack_workspacefield). - Confirm
slack_app_workspace_rate_limitis on. If it is off, turn it on. Requests over the limit are then dropped. - If the workspace still causes load, turn it off.
Dropped requests get no reply in Slack. That is by design. GitLab answers a dropped request with a success status, because Slack counts any other answer as a failed delivery. Too many failed deliveries make Slack turn off events for every workspace: see the next section.
Slack has disabled event subscriptions for the app
Section titled “Slack has disabled event subscriptions for the app”Slack turns off event delivery for an app when more than 95% of deliveries fail within 60 minutes. See the Events API documentation. This affects every workspace. No mention reaches GitLab.
- Confirm: the mention endpoint receives no requests at all, for all workspaces.
- Fix the cause of the failed deliveries first (an endpoint outage or a bad deploy).
- Turn event subscriptions back on in the Slack app configuration. TODO who has access.
- Confirm with a test mention.
Slack context (search, channel history) misbehaves
Section titled “Slack context (search, channel history) misbehaves”- Turn off
slack_duo_mcp_server. - Confirm: a mention still gets a reply, without Slack search or history.
Installs fail with invalid_scope
Section titled “Installs fail with invalid_scope”This happens when slack_duo_agent_app is on before the updated Slack app is published.
- If the app is not published yet, turn off
slack_duo_agent_app. - If the app is published, do not turn the flag off. Escalate to the External Agents team.
Turn off one workspace
Section titled “Turn off one workspace”An Owner of the Duo group opens the group’s GitLab for Slack app settings and selects Turn off GitLab Duo.
Only an Owner can do this. The action is recorded as a duo_in_slack_disabled audit event.
New mentions in that workspace are then refused with duo_group_not_set_up. Sessions already running are not stopped.
Stop everything
Section titled “Stop everything”There is no dedicated stop switch. The closest is the slack_duo_api_flow flag.
- Turn off
slack_duo_api_flow. - In every workspace that runs in a Duo group, new mentions are refused with “This flow is not available” and logged as
Duo Messaging: flow trigger failed. Running sessions can no longer save progress. - A workspace that is not yet on
slack_duo_groupis not stopped. Its mentions run the Duo Developer flow instead.
Do not turn off slack_duo_group to stop replies. It sends mentions to the old experiment path.
Self-Managed
Section titled “Self-Managed”TODO. Admins cannot use ChatOps. To be added later.