Skip to content
Runbooks

Is a missing billing event billable?

When to use this runbook

Section titled “When to use this runbook”

Use this runbook when AI Gateway or Duo Workflow Service (DWS) logs No LLM operations available for billing tracking.

This error means the service sent no billing event for a request. That is lost revenue only if CustomersDot (CDot) would have charged for the event. CDot decides what is billable, not the service.

Step 1: Find the log entry

Section titled “Step 1: Find the log entry”

In GCP Logging, project gitlab-runway-production, use the filter for the service.

AI Gateway (code suggestions):

resource.labels.service_name="ai-gateway"
jsonPayload.error="No LLM operations available for billing tracking"

DWS (Duo Agent Platform):

resource.labels.service_name="duo-workflow-svc"
jsonPayload.event="No LLM operations available for billing tracking"

Step 2: Check if CDot would bill the event

Section titled “Step 2: Check if CDot would bill the event”

Read the log entry

Section titled “Read the log entry”

AI Gateway logs the error text in error:

{
  "message": "Failed to track billing event for code generations",
  "error": "No LLM operations available for billing tracking",
  "level": "error",
  "logger": "codesuggestions",
  "billing_event": "code_generations",
  "execution_env": "code_generations",
  "gitlab_feature_enablement_type": "duo_core",
  "gitlab_feature_enabled_by_namespace_ids": "<root namespace id>",
  "gitlab_saas_duo_pro_namespace_ids": "<root namespace id>",
  "gitlab_root_namespace_id": "<root namespace id>",
  "gitlab_realm": "saas",
  "gitlab_version": "<gitlab version>",
  "is_gitlab_team_member": "false",
  "correlation_id": "<correlation id>",
  "timestamp": "<timestamp>"
}

DWS logs the error text in event, through log_exception. It has no billing_event field because the event type is always duo_agent_platform_workflow_completion. This example leaves out the exception and stack fields.

{
  "event": "No LLM operations available for billing tracking",
  "level": "error",
  "logger": "exceptions",
  "exception_class": "ValueError",
  "additional_details": {
    "context": "Error sending billing event for workflow"
  },
  "workflow_id": "<workflow id>",
  "workflow_definition": "chat",
  "gitlab_feature_enablement_type": "duo_core",
  "gitlab_feature_enabled_by_namespace_ids": "<root namespace id>",
  "gitlab_root_namespace_id": "<root namespace id>",
  "gitlab_realm": "saas",
  "gitlab_version": "<gitlab version>",
  "is_gitlab_team_member": "false",
  "correlation_id": "<correlation id>",
  "timestamp": "<timestamp>"
}
FieldUse it to
billing_event (AI Gateway)Get the event type
workflow_definition (DWS)Get the feature qualified name
workflow_id (DWS)Find other logs for the same session
correlation_idFind other logs for the same request
gitlab_feature_enablement_typeSee how the user got access, for example duo_core, duo_amazon_q, tier or an add-on name
gitlab_realmFind the customer: gitlab_root_namespace_id for saas, gitlab_instance_id for self-managed
gitlab_saas_duo_pro_namespace_idsSee if the namespace has Duo Pro
gitlab_versionCheck the minimum version in ProductMinGitlabVersionCheck. Code suggestions events skip this check.
is_gitlab_team_memberUse as a hint only. CDot uses a value from the signed token, not this header, to skip team members. To confirm, check skip_reason in ClickHouse (step 3).

Apply the billing rule

Section titled “Apply the billing rule”

The rule depends on the event type.

code_completions or code_generations: billable when one of these is true:

  • gitlab_feature_enablement_type is duo_core.
  • The subscription has no Duo Enterprise, Duo Pro or Ultimate with Amazon Q plan.

See bill_code_suggestions? in subscription_service.rb.

All other event types: billable when all of these are true:

  • The feature qualified name, without its version suffix, is in BILLABLE_WORKFLOWS. For DWS, use workflow_definition. For AI Gateway, use the name set at the billing call site (static names are in FeatureQualifiedNameStatic). A name that is only in FLOW_TYPE_LIST is not billable.
  • The event is on or after the billable date for the deployment, for example saas_billable_date.
  • If the entry sets feature_flag, the flag is on. Ask Fulfillment to confirm.
  • If the entry sets feature_ai_catalog_item_required (for example chat), the catalog item check passes. DWS error logs do not include feature_ai_catalog_item.

All event types also need a namespace or instance with credits to consume:

  • A Premium or Ultimate subscription, a free subscription with the GitLab Credits add-on, or a trial.
  • For self-managed, an activated instance.

Some event types have extra skip rules. See skip_conditions/.

Step 3: Confirm with data

Section titled “Step 3: Confirm with data”

  1. Find why the LLM operations list was empty. Search for other logs with the same correlation_id, then compare with the known causes.

    Flat rate features, for example code_suggestions, code_review and sast_fp_detection, are charged per request, not per LLM call. For these features, an empty list is still a missed charge. See FLAT_RATE_FEATURE_CATEGORIES in usage_credit_calculator.rb.

    CauseHow to verifyBilling event expected?
    The model does not report token usage, for example Amazon QCheck model_name.Usually no. Amazon Q plans are not billed for code suggestions.
    A DAP workflow ran deterministic steps only (no LLM call)Find the flow ID and version. Check the flow definition for a path where every step is deterministic.No
    A retry path ran (ai-assist#2188)No

  2. Compare with past successful events. Search logs over a longer time range.

    • AI Gateway: find billing_events_client “Tracking billing event” logs whose event_type equals the error’s billing_event, with the same enablement type.
    • DWS: find successful billing events for the same workflow_definition and enablement type.

    If you find successful events, these requests are normally billed, so this is a real gap. If you find none, the path is probably never billed.

  3. Check ClickHouse. To get read-only access, request “ClickHouse Cloud Fulfillment” in Lumos (Okta tile).

    Always filter on IngestionTimestamp as well as Timestamp, or the query scans the whole table.

    QuestionQueryQuery results
    Does CDot charge this feature? 
    SELECT SkipConsumption, flow_type, skip_reason, count() AS events, round(sum(credits), 2) AS credits
    FROM (
      SELECT EventId,
             argMax(SkipConsumption, IngestionTimestamp) AS SkipConsumption,
             argMax(splitByChar('/', JSONExtractString(Metadata, 'feature_qualified_name'))[1], IngestionTimestamp) AS flow_type,
             argMax(JSONExtractString(Metadata, 'skip_reason'), IngestionTimestamp) AS skip_reason,
             argMax(GitlabCredits, IngestionTimestamp) AS credits
      FROM usage_billing_enriched
      WHERE RootNamespaceId = <root namespace id>   -- self-managed: InstanceId = '<instance id>'
        AND Timestamp          >= toDateTime64('<start>', 9) AND Timestamp < toDateTime64('<end>', 9)
        AND IngestionTimestamp >= toDateTime64('<start>', 9) AND IngestionTimestamp < now()
        AND EventType = '<event type>'   -- AI Gateway: `billing_event`, DWS: 'duo_agent_platform_workflow_completion'
        AND splitByChar('/', JSONExtractString(Metadata, 'feature_qualified_name'))[1] = '<feature>'   -- AI Gateway: 'code_suggestions', DWS: `workflow_definition`
      GROUP BY EventId
    )
    GROUP BY SkipConsumption, flow_type, skip_reason
    ORDER BY SkipConsumption, credits DESC

    Rows with SkipConsumption = false: CDot charges events like this one, so the missing event is a billing gap. Escalate (step 4, “Billable”).

    Only SkipConsumption = true: CDot does not charge this feature for this namespace (step 4, “Not billable”).

    No rows: see next row.
    Query 1 returned no rows. Is CDot failing to enrich this feature's events? 
    WITH
      <root namespace id> AS ns,
      now64(9) - INTERVAL 7 DAY AS since
    SELECT Reason, count() AS events
    FROM enrichment_failures
    WHERE IngestionTimestamp >= since
      AND EventId IN (
        SELECT EventId FROM raw_billing_usage_events
        WHERE ns IN (RootNamespaceId, NamespaceId)   -- self-managed: InstanceId = '<instance id>'
          AND IngestionTimestamp >= since
      )
      AND EventId NOT IN (
        SELECT EventId FROM usage_billing_enriched WHERE IngestionTimestamp >= since
      )
    GROUP BY Reason
    ORDER BY events DESC

    Rows: CDot has a separate problem. Reason shows why. Page Fulfillment (step 4).

    No rows: CDot has no events like this one in the time range. Use the billing rule in step 2 to decide.

Step 4: Decide and escalate

Section titled “Step 4: Decide and escalate”
FindingAction
Not in BILLABLE_WORKFLOWSConfirm with owning feature team whether this feature should be billed.
CDot fails to enrich feature’s eventsPage Fulfillment tier 2 on-call through Slack.
BillableThe event never reached CDot. Escalate to AI Model Services group, Gateway Services category.
Not billableAdd your findings to the incident or issue, including why the event is not billable (for example, Amazon Q plan). Close the investigation.

When you escalate, include:

  • The error log entry and related logs (same correlation_id)
  • The billing rule you applied and its result
  • The ClickHouse queries and their results
  • Other useful context, such as the number of affected requests

References

Section titled “References”