Skip to content
Runbooks

AI-Assisted Service

Summary

Section titled “Summary”

AI Assisted is a dedicated Rails fleet that provides an AI Abstraction Layer in front of AI Gateway. It handles AI-specific requests that are often long running and depend on external services (such as Vertex AI and Anthropic). To prevent these requests from occupying Puma workers on the main fleet, potentially impacting performance, AI Assisted runs on an isolated fleet to safely serve these endpoints without affecting core GitLab traffic.

Architecture

Section titled “Architecture”

For diagram, refer to architecture blueprint.

Operational Roles and Responsibilities

Section titled “Operational Roles and Responsibilities”

Most traffic served by the AI Assisted service supports Code Suggestions features, such as Code Generation and Completion, which are owned by the Code Creation Group. If additional feature teams begin contributing endpoints in the future, service ownership may need to be reassessed.

Note that the fleet is not limited to /api/v4/ai_assisted*. HAProxy also routes the Duo Agent Platform WebSocket endpoint /api/v4/ai/duo_workflows/ws here, so the gitlab_workhorse_duo_workflow_* counters are emitted under type="ai-assisted" — see duo-agent-platform.

The service is implemented across the following projects:

Deployment

Section titled “Deployment”

AI-Assisted Service is deployed in:

  • Staging
  • Production

Regions

Section titled “Regions”

AI-Assisted Service is currently deployed across 3 pods in the following regions:

  1. us-east1-b
  2. us-east1-c
  3. us-east1-d

Performance

Section titled “Performance”

AI-Assisted Service includes the following SLIs/SLOs:

Section titled “Links to further Documentation”