Skip to content
Runbooks

Duo Agent Platform Service

Summary

Section titled “Summary”

duo-agent-platform serves two purposes:

  1. Incident reporting umbrella. It is used in incident.io so that all incidents affecting Duo Agent Platform in any way are properly tagged and recorded, regardless of which underlying component failed.
  2. Monitored virtual service. It carries the SLIs for the GitLab monolith side of the Duo Agent Platform — signals emitted by Workhorse, and in future Rails and Sidekiq. It has no infrastructure of its own.

Note the asymmetry: the incident.io umbrella is broader than the error budget. An incident tagged Service::DuoAgentPlatform may originate in gitlab-lsp or duo-workflow-svc, which are not covered by this service’s SLIs.

Duo Agent Platform spans multiple components, each with its own observability surface:

Relationship to duo-workflow-svc

Section titled “Relationship to duo-workflow-svc”
ServiceCoversEmitted byMimir tenant
duo-workflow-svcThe Duo Workflow Service itself (gRPC server, LLM calls, tool use, checkpoints)Cloud Run / Runwayrunway
duo-agent-platformThe GitLab monolith side (Workhorse WebSocket handler; Rails/Sidekiq signals in future)ai-assisted fleet, sidekiq, webgitlab-gprd, gitlab-gstg

The split is not cosmetic. The two halves are scraped into different, isolated Mimir tenants; a recording rule evaluated in the runway tenant cannot read monolith series, and vice versa — see mimir-rules-jsonnet/README.md. Tenant selection is per-service (tenants: in the service definition), not per-SLI, so monolith-emitted Duo Agent Platform SLIs need their own service definition.

Monitoring/Alerting

Section titled “Monitoring/Alerting”

SLIs

Section titled “SLIs”
SLIRequest rateError rateEmitted by
workhorse_dws_connectionsgitlab_workhorse_duo_workflow_connections_totalgitlab_workhorse_duo_workflow_connection_errors_totalai-assisted
workhorse_dws_sessionsgitlab_workhorse_duo_workflow_sessions_totalgitlab_workhorse_duo_workflow_session_errors_totalai-assisted

Workhorse metrics

Section titled “Workhorse metrics”

Workhorse exposes six Prometheus counters and one histogram for the duoworkflow package, declared in workhorse/internal/ai_assist/duoworkflow/metrics.go. See the duoworkflow package README for full documentation including example PromQL queries. The four connection/session counters back the SLIs above; the three http_action metrics are not yet covered by an SLI.

They are emitted by the Workhorse process on the ai-assisted fleet. The Duo Agent Platform WebSocket handler is registered on the /api/v4/ai/duo_workflows/ws route (duo_workflow_ws in workhorse/internal/upstream/routes.go), which is an API-pattern route in Workhorse’s own routing table — but HAProxy routes the request to ai-assisted, so that is the fleet whose counters move. Hence the type="ai-assisted" selector and emittedBy: ['ai-assisted'] in the service definition. On that fleet the scrape job is plain gitlab-workhorse, not a fleet-suffixed variant like gitlab-workhorse-api.

Do not infer the fleet from the route table. These counters are package-level promauto counters registered at process init, so every Workhorse fleet exports all four of them with a permanent value of 0. Selecting the wrong type therefore produces a recorded series that reads 0 rather than returning no data, which is indistinguishable from a genuinely idle feature. Confirm the fleet empirically:

sum by (type) (increase(gitlab_workhorse_duo_workflow_sessions_total{env="gprd"}[24h]))

Alerts

Section titled “Alerts”
AlertSLIType
DuoAgentPlatformServiceWorkhorseDwsConnectionsErrorSLOViolationworkhorse_dws_connectionsError
DuoAgentPlatformServiceWorkhorseDwsSessionsErrorSLOViolationworkhorse_dws_sessionsError

Traffic cessation and traffic absent alerts are intentionally disabled (trafficCessationAlertConfig: false) on both SLIs. Duo Agent Platform WebSocket traffic can legitimately fall to zero when the feature is idle, and these SLIs have no established baseline ops rate yet — see Known gaps. Enable them once real volume is visible.

Troubleshooting

Section titled “Troubleshooting”

Because this is a virtual service, triage usually means following the signal to the component that emits it:

  1. Check the service overview dashboard to see which SLI is degraded.
  2. workhorse_dws_connections errors are Workhorse-local — WebSocket upgrade failures, runner initialisation failures, or runner execution failures. The error_type label distinguishes quota_exceeded and locked (both expected under load / contention) from other (needs investigation). Follow the ai-assisted service runbook if the whole ai-assisted fleet is unhealthy.
  3. workhorse_dws_sessions errors are failures on the gRPC ExecuteWorkflow stream between Workhorse and the Duo Workflow Service. The grpc_code label points at the cause: ResourceExhausted and DeadlineExceeded usually indicate DWS-side pressure, Unavailable a connectivity or deployment issue. Cross-check duo-workflow-svc before suspecting the monolith — duo-workflow-svc is a declared dependency, so its alerts may already explain the failure.

Known gaps

Section titled “Known gaps”
  • No observed baseline. These SLIs have never recorded real data. They first lived on duo-workflow-svc, selecting type="duo-workflow-svc" in the runway tenant, where neither the tenant nor the type label ever matched. Moving them here fixed the tenant but selected type="api", which recorded a flat zero. Only the correction to type="ai-assisted" produces real numbers, and recording rules do not backfill — expect an empty history before that change was deployed. The errorRatio: 0.95 thresholds have been carried through all three revisions unchanged and have never been checked against observed data.
  • gitlab_duo_workflow_stuck_workflows_cleaned_total (emitted by Ai::DuoWorkflows::FailStuckWorkflowsWorker on Sidekiq) belongs in this service but has no denominator metric today — there is no sessions-created counter. Tracked as a follow-up.
  • The broader Rails-side instrumentation gap is catalogued in GitLab Duo Agent Platform observability (Rails), which lists ~14 planned gitlab_duo_agent_platform_* metrics. Those should land here as they ship.

Service Changes

Section titled “Service Changes”
Section titled “Links to further Documentation”