Skip to content
Runbooks

DuoWorkflowSvcServiceToolUseErrorSLOViolation

Overview

Section titled “Overview”
  • This alert fires when the error rate of tool calls within agent platform sessions exceeds the SLO threshold.
  • Tool failures are non-fatal events that don’t immediately terminate sessions since agents can retry or use alternative approaches.
  • However, increasing failure rates serve as early warning indicators of potential system issues.
  • This alert indicates that tools are failing more frequently than expected. These tools can be invoked within Duo Workflow Service or in the clients (eg. users machine in editor extension)
  • Possible user impacts
    • Some tools are not usable (eg gitlab search, run command).
    • Overall quality of the platform would be low, as some tools cannot be used.

Services

Section titled “Services”

Metrics

Section titled “Metrics”
  • The metric used is gitlab_component_errors:confidence:ratio_1h and gitlab_component_errors:confidence:ratio_6h for the tool_use component of duo-workflow-svc.
  • This metric measures the error rate of tool calls, expressed as a percentage (0-100%).
  • The SLO threshold is 5% error rate, meaning the alert fires when errors exceed this threshold.
  • Link to metric catalogue

Alert Behavior

Section titled “Alert Behavior”
  • To silence the alert, please visit Alert Manager Dashboard
  • This alert is expected to be rare under normal conditions. High frequency indicates tool integration issues.

Severities

Section titled “Severities”
  • This alert creates S2 incidents (High severity, pages on-call).
  • All gitlab.com users using Duo Workflow features with tool integrations are potentially impacted.
  • Review Incident Severity Handbook page to identify the required Severity Level.

Verification

Section titled “Verification”

Recent changes

Section titled “Recent changes”

Troubleshooting

Section titled “Troubleshooting”

  1. Check which tools are failing

    • See tool errors graph for a breakdown of errors per tool.
    • Check whether the issue is limited for a specific tool

  2. Check duo workflow service logs:

  3. Check for recent changes:

    • Review recent changes mentioned under Recent changes section.
    • Check if a recent deployment affected tool errors.
    • If a recent change caused the issue, consider rolling back.

Possible Resolutions

Section titled “Possible Resolutions”
  • N.A. We don’t have historical data on this alert’s resolutions.

Dependencies

Section titled “Dependencies”
  • AI Gateway / Duo Workflow Service

Escalation

Section titled “Escalation”
  • For investigation and resolution assistance, reach out to #g_agent_foundations on Slack.

Definitions

Section titled “Definitions”
Section titled “Related Links”