Skip to content
Runbooks

DuoWorkflowSvcServiceServerTrafficCessation

Overview

Section titled “Overview”
  • This alert fires when the Duo Workflow Service server component stops receiving traffic after previously receiving traffic.
  • The alert detects an abnormal cessation of traffic: the service was receiving traffic 1 hour ago but has received zero traffic over the last 30 minutes.
  • This could indicate a service outage, deployment issue, or upstream routing problem.
  • Possible user impacts
    • This alert may indicate Agent Platform is completely unusable.

Services

Section titled “Services”

Metrics

Section titled “Metrics”
  • The metric used is gitlab_component_ops:rate_30m for the server component of duo-workflow-svc.
  • This metric measures the rate of GRPC requests, expressed as average requests per second over 30 minutes.
  • The alert fires when the rate drops to zero after previously being non-zero.
  • Link to metric catalogue

Alert Behavior

Section titled “Alert Behavior”
  • To silence the alert, please visit Alert Manager Dashboard
  • This alert is expected to be rare under normal conditions. It should only fire during actual outages or maintenance windows.

Severities

Section titled “Severities”
  • This alert creates S2 incidents (High severity, pages on-call).
  • All gitlab.com, self-managed and dedicated customers (other than those using self-hosted DAP) using Duo Workflow features are potentially impacted.
  • Review Incident Severity Handbook page to identify the required Severity Level.

Verification

Section titled “Verification”

Recent changes

Section titled “Recent changes”

Troubleshooting

Section titled “Troubleshooting”
  1. Check service health:

Possible Resolutions

Section titled “Possible Resolutions”
  • N.A. We don’t have historical data on this alert’s resolutions.

Dependencies

Section titled “Dependencies”
  • GitLab Rails + Postgres DB
  • Workhorse
  • AI Gateway / Duo Workflow Service

Escalation

Section titled “Escalation”
  • For investigation and resolution assistance, reach out to #g_agent_foundations on Slack.

Definitions

Section titled “Definitions”
Section titled “Related Links”