Skip to content
Runbooks

DuoWorkflowSvcServiceServerErrorSLOViolation

Overview

Section titled “Overview”
  • This alert fires when the error rate of GRPC requests to the Duo Workflow Service server component exceeds the SLO threshold.
  • The alert indicates that the service is experiencing a higher-than-acceptable error rate, which impacts users relying on the Duo Workflow Service.
  • This is a user-impacting alert that requires immediate investigation and remediation.
  • Possible user impacts
    • Users will see Duo Agent Platform sessions failing.

Services

Section titled “Services”

Metrics

Section titled “Metrics”
  • The metric used is gitlab_component_errors:confidence:ratio_1h and gitlab_component_errors:confidence:ratio_6h for the server component of duo-workflow-svc.
  • This metric measures the error rate of GRPC requests, expressed as a percentage (0-100%).
  • The SLO threshold is 5% error rate, meaning the alert fires when errors exceed this threshold.
  • Link to metric catalogue

Alert Behavior

Section titled “Alert Behavior”
  • To silence the alert, please visit Alert Manager Dashboard
  • This alert is expected to be rare under normal conditions. High frequency indicates systemic issues.

Severities

Section titled “Severities”
  • This alert creates S2 incidents (High severity, pages on-call).
  • All gitlab.com, self-managed and dedicated customers (other than those using self-hosted DAP) using Duo Workflow features are potentially impacted.
  • Review Incident Severity Handbook page to identify the required Severity Level.

Verification

Section titled “Verification”

Recent changes

Section titled “Recent changes”

Troubleshooting

Section titled “Troubleshooting”

  1. Identify the scope of errors:

    • Visit Error Breakdown Dashboard.
    • Use flow_type, gitlab_version, client_type, gitlab_realm variable dropdowns on the dashboard to narrow down the problem

  2. Check service health:

  3. Check for recent changes:

    • Review recent changes mentioned under Recent changes section.
    • If a recent change caused the issue, consider rolling back.

Possible Resolutions

Section titled “Possible Resolutions”
  • N.A. We don’t have historical data on this alert’s resolutions.

Dependencies

Section titled “Dependencies”
  • GitLab Rails + Postgres DB
  • Workhorse
  • AI Gateway / Duo Workflow Service

Escalation

Section titled “Escalation”
  • For investigation and resolution assistance, reach out to #g_agent_foundations on Slack.

Definitions

Section titled “Definitions”
Section titled “Related Links”