Skip to content

GitLab Duo in Slack

[[TOC]]

Users mention @GitLab in a Slack channel and GitLab Duo answers in the thread. Each mention starts or continues a Duo Agent Platform session. The session runs in one top-level group chosen for the Slack workspace, called the Duo group.

It is part of the GitLab Rails application, not a separate service. A mention passes through these parts:

  1. Slack sends an event to the GitLab for Slack app endpoint.
  2. Rails checks the user, the channel, and the workspace’s Duo group.
  3. A Sidekiq job (Ai::Messaging::ServerSideTurnWorker) starts the turn.
  4. Workhorse and the Duo Workflow Service run it.
  5. Rails posts progress and the answer back to the Slack thread.

An outage means users get no reply, a late reply, or an error in Slack. It does not affect the rest of GitLab.

  • GitLab.com: Limited Availability. Self-Managed: Experiment, off by default.
  • The Slack workspace has a Duo group, set in the group’s GitLab for Slack app settings.
  • The user has linked their Slack and GitLab accounts.
  • The user meets the Duo group’s seat and role rules.
  • Channels shared with another organization (Slack Connect) are always refused.

The “off” behavior below comes from reading the code. TODO confirm each row on staging.

FlagScopeEffect of turning it off
slack_duo_grouptop-level group or instanceMentions fall back to the old per-user experiment path. Does not stop Duo.
slack_duo_api_flowtop-level groupIn a Duo group: new mentions are refused, running sessions can no longer save progress, and approval buttons stop working. On the old per-user path: mentions run the Duo Developer flow instead.
slack_duo_mcp_serveruserDuo loses Slack search and channel history tools. Mentions still work.
duo_workflow_detached_server_side_executionuserOn for everyone on GitLab.com since 2026-11-13. When off, each turn holds a Sidekiq thread for its whole run (limit 50 at once).
slack_app_workspace_rate_limitinstanceThe per-workspace limit is logged but not enforced.
slack_duo_agent_appinstanceNew installs stop asking for the DM and agent view scopes. Not a rollback once the Slack app is published.

To turn a flag off for everyone:

/chatops gitlab run feature set <flag_name> false

Turning off slack_duo_group does not stop Duo replies in Slack. To stop one workspace, see Turn off one workspace. To stop all of them, see Stop everything.

Collect this before you start:

  • Slack workspace ID (starts with T) and channel
  • GitLab username of the affected user
  • The Duo group (top-level group) of the workspace
  • When it happened, in UTC
  • A link to the Slack message or the Duo session, if there is one
  • Whether it affects one user, one workspace, or everyone

NOTE: Do not put customer message content in public issues.

A lock reaction on the message plus a short reply is a refusal. The feature is working and the user or group does not meet a requirement. The reply names the reason. Common ones:

ReasonMeaning
user_not_linkedThe user has not linked Slack and GitLab accounts.
externally_shared_channelThe channel is shared with another organization.
duo_group_not_set_upThe workspace has no Duo group.
duo_group_experiments_offThe Duo group has experiment and beta features off.
duo_features_off, agent_platform_offA Duo setting is off in the Duo group.
not_licensed, seat, roleThe group or user lacks the subscription, seat, or role.

Refusals are counted in the block_slack_duo_mention event, with the reason as its property.

  • Duo in Slack health dashboard.
    • Outcomes: sessions, completed, waiting for approval, broken, killed as stuck.
    • Early warnings: turns waiting for a slot, deferrals.
  • Rails and Sidekiq logs. Search for messages that start with Duo Messaging:.
    • Duo Messaging: flow triggered and Duo Messaging: flow trigger failed (with failure_reason)
    • Duo Messaging: retries exhausted; giving up
    • TODO saved log searches
  • Rate limit hits: Application_Rate_Limiter_Request with a slack_workspace field.
  • Duo Workflow Service dashboard
  • TODO Sentry link

TODO. Alerts are tracked in https://gitlab.com/gitlab-org/gitlab/-/work_items/634044. Add one section per alert here when they exist.

On GitLab.com, turns run detached: the Sidekiq job only starts the turn and Workhorse keeps it going. Each Workhorse process holds at most 100 detached runs, and a run ends after 55 minutes at most. When a process is full, it refuses new turns with a 503.

  1. Check the Duo Workflow Service dashboard. If it is unhealthy, follow its runbook. No Slack flag fixes that.
  2. Check for refused turns. Look for too many detached workflow runs in the Workhorse logs, or a rise in gitlab_workhorse_duo_workflow_detached_runs_rejected_total on the Workhorse Runs dashboard. Refusals mean the per-process limit is reached. Escalate to the External Agents team.
  3. Check the “Early warnings” row of the health dashboard for turns waiting for a slot.
  4. If detached mode itself is failing, turning off duo_workflow_detached_server_side_execution is the fallback. Turns then run inside the Sidekiq job, 50 at once, so expect queueing under load. The flag belongs to group::agent foundations: see its rollout issue.
  5. Confirm: no new refusals, turns waiting for a slot returns to zero, and a test mention gets a reply.
  1. Check whether other Duo Agent Platform surfaces also fail. If yes, it is not a Slack problem. Follow the Duo Workflow Service and AI Gateway runbooks.
  2. If only Slack fails, look for Duo Messaging: flow trigger failed and read failure_reason.
  3. Check for a recent deploy or flag change (/chatops gitlab run feature get <flag_name>).
  4. Escalate to the External Agents team. If the errors do harm and cannot wait, see Stop everything.
  1. Find the workspace in the Application_Rate_Limiter_Request logs (slack_workspace field).
  2. Confirm slack_app_workspace_rate_limit is on. If it is off, turn it on. Requests over the limit are then dropped.
  3. If the workspace still causes load, turn it off.

Dropped requests get no reply in Slack. That is by design. GitLab answers a dropped request with a success status, because Slack counts any other answer as a failed delivery. Too many failed deliveries make Slack turn off events for every workspace: see the next section.

Slack has disabled event subscriptions for the app

Section titled “Slack has disabled event subscriptions for the app”

Slack turns off event delivery for an app when more than 95% of deliveries fail within 60 minutes. See the Events API documentation. This affects every workspace. No mention reaches GitLab.

  1. Confirm: the mention endpoint receives no requests at all, for all workspaces.
  2. Fix the cause of the failed deliveries first (an endpoint outage or a bad deploy).
  3. Turn event subscriptions back on in the Slack app configuration. TODO who has access.
  4. Confirm with a test mention.

Slack context (search, channel history) misbehaves

Section titled “Slack context (search, channel history) misbehaves”
  1. Turn off slack_duo_mcp_server.
  2. Confirm: a mention still gets a reply, without Slack search or history.

This happens when slack_duo_agent_app is on before the updated Slack app is published.

  1. If the app is not published yet, turn off slack_duo_agent_app.
  2. If the app is published, do not turn the flag off. Escalate to the External Agents team.

An Owner of the Duo group opens the group’s GitLab for Slack app settings and selects Turn off GitLab Duo. Only an Owner can do this. The action is recorded as a duo_in_slack_disabled audit event. New mentions in that workspace are then refused with duo_group_not_set_up. Sessions already running are not stopped.

There is no dedicated stop switch. The closest is the slack_duo_api_flow flag.

  1. Turn off slack_duo_api_flow.
  2. In every workspace that runs in a Duo group, new mentions are refused with “This flow is not available” and logged as Duo Messaging: flow trigger failed. Running sessions can no longer save progress.
  3. A workspace that is not yet on slack_duo_group is not stopped. Its mentions run the Duo Developer flow instead.

Do not turn off slack_duo_group to stop replies. It sends mentions to the old experiment path.

TODO. Admins cannot use ChatOps. To be added later.