Skip to content
Runbooks

DuoWorkflowSvcServiceServerTrafficAbsent

Overview

Section titled “Overview”
  • This alert fires when the Duo Workflow Service server component stops reporting traffic metrics after previously reporting traffic.
  • Unlike traffic cessation, this alert indicates the signal itself is absent (not just zero), which could indicate a metrics collection issue or a complete service failure.
  • This could be caused by a change to the metrics used in the SLI, or by the service not receiving traffic.
  • Possible user impacts
    • This alert may indicate Agent Platform is completely unusable.

Services

Section titled “Services”

Metrics

Section titled “Metrics”
  • The metric used is gitlab_component_ops:rate_5m for the server component of duo-workflow-svc.
  • This metric measures the rate of GRPC requests, expressed as average requests per second over 5 minutes.
  • The alert fires when the metric is absent (no data) after previously being present.
  • Link to metric catalogue

Alert Behavior

Section titled “Alert Behavior”
  • To silence the alert, please visit Alert Manager Dashboard
  • This alert is expected to be rare under normal conditions. It should only fire during actual outages or monitoring issues.

Severities

Section titled “Severities”
  • This alert creates S2 incidents (High severity, pages on-call).
  • All gitlab.com, self-managed and dedicated customers (other than those using self-hosted DAP) using Duo Workflow features are potentially impacted.
  • Review Incident Severity Handbook page to identify the required Severity Level.

Verification

Section titled “Verification”

Recent changes

Section titled “Recent changes”

Troubleshooting

Section titled “Troubleshooting”
  1. Check service health:

Possible Resolutions

Section titled “Possible Resolutions”
  • N.A. We don’t have historical data on this alert’s resolutions.

Dependencies

Section titled “Dependencies”
  • GitLab Rails + Postgres DB
  • Workhorse
  • AI Gateway / Duo Workflow Service

Escalation

Section titled “Escalation”
  • For investigation and resolution assistance, reach out to #g_agent_foundations on Slack.

Definitions

Section titled “Definitions”
Section titled “Related Links”