Skip to content
Runbooks

GitLab Query Language (GLQL)

Overview

Section titled “Overview”

GitLab Query Language (GLQL) is a tool that allows you to interact with data by creating queries to pull specific information, like issues (work items) or merge requests, directly from GitLab’s API. You can embed a GLQL query into text fields, such as comments, descriptions, and wikis, and dynamically display the queried data right within those fields.

Architecture

Section titled “Architecture”

The logic for GLQL is located in GitLab’s main repository and it plugs into GraphQL API, connecting to either PostgreSQL or Elasticsearch, depending on what data needs to be retrieved. GLQL also uses a Rust-based compiler to process queries. For more information you can refer to the following design document.

Monitoring/Alerting

Section titled “Monitoring/Alerting”

There is an existing dashboard for monitoring and alerting API performance metrics where we additionally added GLQL related definition. It is available on the API: Overview Grafana dashboard.

Alerting has been configured through Runbooks to alert into Slack channel #g_knowledge.

For any questions please reach out to the team in Slack via #f_glql or tag @knowledge-group or use GitLab’s group handle @gitlab-org/plan-stage/knowledge.

Logging

Section titled “Logging”

You can search for json.controller.keyword: "Glql::BaseController" in Kibana to find all GLQL-related logs. Additionally, you can search for json.graphql.glql_referer, which will provide the URL where GLQL was embedded, and json.graphql.glql_query_sha to filter logs for the same GLQL query used across different groups and projects.

Database

Section titled “Database”

Because GLQL queries are intended to be read-only, we wrap them in use_replicas_for_read_queries to ensure all reads hit Postgres replicas. This change improves resilience against long-running queries and avoids unnecessary load on the primary (for example, when a user’s recent write hasn’t yet been propagated to the replicas).

Rate Limiting

Section titled “Rate Limiting”

We rate limit all GLQL queries to ensure that if two requests for the same query fail within a 15-minute window with an ActiveRecord::QueryAborted error, that query (identified by its unique SHA) is blocked across all pages during that period. A dashboard for this specific rate limiting cluster is available for more details if needed.

Additionally, since GLQL is served under /api, it also inherits the standard GraphQL throttling settings:

Debug locally

Section titled “Debug locally”

To use the feature locally, you need to have the glql_integration feature flag enabled. Additionally, since GLQL relies on crypto.subtle, which is unavailable in insecure contexts (except for localhost), it won’t work if you’re using a custom URL like http://gdk.test:3000 locally. To resolve this, you might want to enable HTTPS in GDK by following this link.

Additional feature flags

Section titled “Additional feature flags”
  • glql_integration - the main feature that controls whether GLQL is enabled or disabled across the entire system.
  • glql_work_items - connect to WorkItems API instead of the legacy Issues API.
  • glql_es_integration - connect to Elasticsearch instead of PostgreSQL to query for data.
  • glql_load_on_click - when a GLQL block is embedded in a text field, it loads every time you scroll to it. With this feature flag enabled, GLQL blocks load on demand, and you’ll need to click a button to load the block if there are more than 20 GLQL views per page.

Incident Handling Steps

Section titled “Incident Handling Steps”

When an incident occurs where someone is overloading the database - either intentionally or unintentionally - by embedding multiple long-running queries in a popular issue that receives high traffic, you can follow these steps to mitigate the impact:

Monitor the GLQL Health Dashboard

Section titled “Monitor the GLQL Health Dashboard”

This GLQL Health Dashboard shows GLQL query performance over the last 14 days to help find slow or broken queries. It should help with finding issues with high GLQL traffic, seeing which queries are used most, and spotting queries that keep timing out.

Enable the glql_load_on_click feature flag

Section titled “Enable the glql_load_on_click feature flag”

When you have identified the specific issues or queries that are causing the problem, the first step is to enable the glql_load_on_click feature flag.

Since GLQL blocks automatically load every time a user scrolls to them, enabling glql_load_on_click changes GLQL blocks to load on-demand, which should help mitigate the risk.

The flag can be enabled at either the group or project level:

/chatops run feature set --project=gitlab-org/gitlab glql_load_on_click true
/chatops run feature set --group=gitlab-org glql_load_on_click true

Important to keep in mind:

  • Enabling this flag for a top-level group applies it to all descendant projects and sub-groups in the hierarchy.
  • Enabling this flag for a sub-group applies it only to that specific sub-group, not to its descendants.
  • Individual issues cannot be blocked, but this is unnecessary since the feature flag provides protection at the group/project level.

Manually remove the query

Section titled “Manually remove the query”

Additional mitigation option for a user with maintainer permission can be to edit the comment to remove problematic queries or delete the comment entirely as a last resort.

Post-incident cleanup

Section titled “Post-incident cleanup”

Once the incident is resolved, remember to disable the glql_load_on_click feature flag to restore normal functionality. Leaving this flag enabled will have a negative impact on user experience.