Rate Limiting Service

• Service Overview
• Alerts: https://alerts.gitlab.net/#/alerts?filter=%7Btype%3D%22rate-limiting%22%2C%20tier%3D%22sv%22%7D
• Label: gitlab-com/gl-infra/production~“Service::RateLimiting”

Overview of Rate Limits for https://gitlab.com

The handbook is the source of truth for Rate Limiting information.

If you are looking for information about requesting a rate limit bypass for GitLab.com, please see the Rate Limit bypass policy.

This section of documentation is targeted at SREs working in the production environment.

How-Tos

Even with rate limiting in place, it’s possible you need to explicitly block an IP or User sending exceptionally high volumes of traffic.

Alternatively, customers may request a bypass and if this is approved, you will need to follow the steps in this doc to temporarily implement an approved bypass.

Block an IP or Path

This can be done using Cloudflare WAF for HTTP traffic. Cloudflare is the preferred first option for blocking HTTP/S traffic — see the Cloudflare blocking and managing traffic runbook for the decision matrix, scenarios, and the canonical Terraform workflow.

In case of an incident that requires immediate intervention, this can be done in the Cloudflare dashboard and backported in Terraform. If a rule is not backported then it will be deleted on the next Terraform Apply.

• To block an IP using a Custom Rule, add it to this file.
• To throttle an IP using a Rate Limit rule, add it to this file

Please leave the appropriate incident or issue number, and information about when the rule can be deleted, in the rule description.

Work with an IMOC or peer to validate the change is reasonable and correct.

These will typically be temporary; anything permanent needs more careful discussion.

Block a User or Project

GitLab team members with Administrator access can log in using their admin credentials, find the user’s profile and block them. Follow steps for Contacting users about GitLab incidents or changes.

Blocking a user also cancels the imports they started. To verify, or to cancel imports the block does not cover, see Canceling running imports.

You can also follow the steps to block a project causing high load.

Allow an IP or User

See the Implementing Bypasses section below.

What Layer is a Rate Limit Happening?

Detailed instructions for this can be found in the Rate Limiting Troubleshooting handbook page.

Is a Specific Endpoint or Feature Being Rate Limited?

Is my endpoint being rate limited? answers this from the detail dashboard, for a team that owns an endpoint or a feature and wants to know whether rate limiting is rejecting its traffic.

Modifying a Rate Limit

If one endpoint (or a collection of endpoints) are being unduly rate-limited, we can consider increasing the limit for them. How you proceed will depend on the urgency of the request:

Is this urgent?

Yes: Implement the increased rate-limit and set the bypass header in Cloudflare where appropriate.

No: Consider other options:

• Can the limit be modified in the Application (either RackAttack or ApplicationRateLimiter)?
• Creating a limit that takes into account user-identify (rather than just IP address) is preferable.

Considerations

• Gather evidence using logs and metrics to determine what is going on and why the limit might need adjusted.
• Document your findings in an issue, getting input from @gitlab-org/production-engineering/managers
• Verify whether our infrastructure can handle the proposed increase.
  • Consider the Database, Gitaly, and Redis, as well as frontend compute.
• Would setting the Bypass Header for specific requests (URL patterns or other identifiers) be sufficient?
• If it is agreed to proceed, raise a production change issue, linked to the earlier discussion issue, to execute the change.
• Ensure GitLab.com Specific Rate Limits is updated to match the new values.

Bypasses and Special Cases

Published rate limits apply to all customers and users with no exceptions. Rate limiting bypasses are only allowed for specific cases.

We need special handling for various partners and other scenarios (e.g. excluding GitLab’s internal services). To permit this we have lists of IP addresses, termed allowlist that are permitted to bypass the rate limits.

Trusted IPs from customers/partners can be added to the allowlists, however we’d prefer to whittle this list down, not add to it. The Rate Limit bypass policy must be followed when considering adding to this list.

• Cloudflare
  • Custom rule bypass: example (confidential)

User-based bypasses are preferred over IP based, as IP addresses are a poor proxy for actual identity. User IDs are much less fungible, and carry implications of paid groups/users and permanent identities of customers, whereas there could be multiple users behind a single IP address and these can rot if they are no longer used by the original user.

Steps to follow before implementing a bypass

• Engage with the customer (via their TAM) and endeavour to find a way to achieve their goals without bypasses.
• May require development to enhance the API or webhooks (add more information so it can be pushed to the customer, rather than polled).
• In some cases, adding a couple of fields to a webhook can eliminate the need for many API calls.
• If implementing a bypass is unavoidable due to incident or temporary urgent customer need then follow the steps listed in the bypass policy

Customers with IPs present in the allow list can be assumed to have legacy grant and may have IPs added as necessary, as long as the ask is reasonable (e.g. adding a few more where there are already many; questions should be asked if they ask to add 100 when they currently have 2).

Bypass headers

The X-GitLab-RateLimit-Bypass header is set to 0 by default. Any value set for this by the client request is overwritten by Cloudflare.

Requests from IPs with a bypass configured will have the X-GitLab-RateLimit-Bypass header set to 1, which RackAttack interprets to mean these requests bypass the rate limits. Ideally we will remove this eventually, once the bypass list is smaller (or gone), or we’ve ensured that our known users are below the new limits.

There are a few other special cases that also set X-GitLab-RateLimit-Bypass - these include certain paths, internal infrastructure addresses such as runner managers, or 3rd party vendors who have integrations with us.

All current bypasses for customers are implemented here.

Tracking Bypasses

Link any bypasses created to Customers Bypassing Rate-Limiting Epic so that we can track it to completion.

These are never permanent, they are only stepping stones to making the API better or otherwise enhancing the product to eliminate the excessive traffic. In practice what we have found so far is issues like webhooks payloads lacking trivial details that must then be scraped/polled from the API instead, and so on.

Anytime an IP is added to the allowlist, an issue for removing the IP should be opened in the production engineering tracker cross-linking the original issue or incident where the IP was added and setting a due date for the IPs to be removed. In the case of allow-list requests, this is at most 2 weeks after the IP was added.

If a rate limit was disabled or relaxed as part of an incident mitigation, also follow the general process for mitigations that disable a safety limit to ensure it has an explicit DRI, a re-evaluation deadline, and an ~"infradev"-tracked follow-up.

Implementing Bypasses

Cloudflare (IP-based)

Cloudflare is responsible for IP-based rate limiting and bypasses on GitLab.com. Do not put IP addresses into HAProxy or RackAttack for allowlisting!

To add a new entry to the allowlist:

• Create a new variable containing the customer’s IP addresses in this file
  • Put a link to the rate limiting request issue in the comments so that we can easily attribute the IPs later.
• Bypass Cloudflare Rate Limits
  • Add a new custom rule using the variable in this file
  • The custom rule will tell Cloudflare WAF to skip all rate limiting rules for the listed IPs, bypassing them.
• Bypass RackAttack Rate Limits (configured in Cloudflare)
  • Add a new transform rule using the variable in this file
  • The transform rule will tell Cloudflare to apply the X-GitLab-RateLimit-Bypass: 1 header for all IPs in the allowlist, bypassing the RackAttack rate limits.

Rails (RackAttack)

User-based

Per the docs, we can designate specific user IDs as being able to bypass authenticated rate limits.

1. Update the Vault secret here by appending the user ID to the list.
2. Bump the version of the secret here.
3. Finally, use the new version of the secret in the Helm chart.

IP-based

We previously used to put a list of IPs to be allowlisted into the /var/opt/gitlab/rack_attack_ip_whitelist/ip_whitelist file where it would be read by the application. This method is no longer used; you should put the IPs into Cloudflare instead.

Rack Attack Rate Limits Configuration Summary

For where each limit is set in the Admin area, and its LabKit cohort, see Rate limits in the Admin area.

Name	Actor	Path Pattern	HTTP Method	Default Rate	GitLab.com Rate	Staging Rate	Pre Rate
throttle_unauthenticated_web	IP Address	Non-API paths	ALL	100 requests per 60 seconds	500 requests per 60 seconds	500 requests per 60 seconds	2000 requests per 10 seconds
throttle_authenticated_web	User	Non-API paths	ALL	100 requests per 60 seconds	1000 requests per 60 seconds	2000 requests per 60 seconds	100000 requests per 60 seconds
throttle_product_analytics_collector	Application	Product analytics endpoints	ALL	100 requests per 60 seconds	100 requests per 60 seconds	100 requests per 60 seconds	100 requests per 60 seconds
throttle_unauthenticated_protected_paths	IP Address	/users/password, /users/sign_in, /api/session, etc.	ALL	10 requests per 60 seconds	10 requests per 60 seconds	10 requests per 60 seconds	10 requests per 60 seconds
throttle_authenticated_protected_paths_api	User	/users/password, /users/sign_in, /api/session, etc.	ALL	10 requests per 60 seconds	10 requests per 60 seconds	10 requests per 60 seconds	10 requests per 60 seconds
throttle_authenticated_protected_paths_web	User	/users/password, /users/sign_in, /api/session, etc.	ALL	10 requests per 60 seconds	10 requests per 60 seconds	10 requests per 60 seconds	10 requests per 60 seconds
throttle_unauthenticated_get_protected_paths	IP Address	/users/password, /users/sign_in, /api/session, etc.	GET	10 requests per 60 seconds	10 requests per 60 seconds	10 requests per 60 seconds	10 requests per 60 seconds
throttle_authenticated_get_protected_paths_api	User	/users/password, /users/sign_in, /api/session, etc.	GET	10 requests per 60 seconds	10 requests per 60 seconds	10 requests per 60 seconds	10 requests per 60 seconds
throttle_authenticated_get_protected_paths_web	User	/users/password, /users/sign_in, /api/session, etc.	GET	10 requests per 60 seconds	10 requests per 60 seconds	10 requests per 60 seconds	10 requests per 60 seconds
throttle_authenticated_git_lfs	User	/info/lfs and Git LFS API paths	ALL	1000 requests per 60 seconds	1000 requests per 60 seconds	1000 requests per 60 seconds	1000 requests per 60 seconds
throttle_unauthenticated_git_http	IP Address	Git HTTP endpoints	ALL	3600 requests per 3600 seconds	500 requests per 60 seconds	500 requests per 60 seconds	2000 requests per 10 seconds
throttle_authenticated_api	User	/api/* paths	ALL	7200 requests per 3600 seconds	6000 requests per 60 seconds	2000 requests per 60 seconds	20000 requests per 10 seconds
throttle_unauthenticated_api	IP Address	/api/* paths	ALL	3600 requests per 3600 seconds	500 requests per 60 seconds	800 requests per 60 seconds	2000 requests per 10 seconds
throttle_raw_endpoint	Project/File Path	/*/raw/* paths	GET	300 requests per minute	300 requests per minute	-	-
throttle_authenticated_git_ssh	User	Git SSH protocols	ALL	600 requests per minute	600 per minute	-	-
git_basic_auth	IP Address	git, jwt/auth/	ALL	10 requests per 60 seconds	300 requests per minute	300 requests per minute	disabled
virtual_registries_endpoints_api_limit	User	/api/v4/virtual_registries/packages/maven/*	ALL	1000 requests per 15 seconds	1000 requests per 15 seconds	1000 per 15 seconds	1000 requests per 15 seconds
throttle_unauthenticated_packages_api	IP Address	/api/v4/packages/*	ALL	800 requests per 15 seconds	800 requests per 15 seconds	800 requests per 15 seconds	800 requests per 15 seconds
throttle_authenticated_packages_api	User	/api/v4/packages/*	ALL	1000 requests per 15 seconds	1000 requests per 15 seconds	1000 requests per 15 seconds	1000 requests per 15 seconds
throttle_unauthenticated_files_api	IP Address	/api/v4/projects/*/repository/files/*	ALL	125 requests per 15 seconds	125 requests per 15 seconds	125 requests per 15 seconds	125 requests per 15 seconds
throttle_authenticated_files_api	User	/api/v4/projects/*/repository/files/*	ALL	500 requests per 15 seconds	500 requests per 15 seconds	500 requests per 15 seconds	500 requests per 15 seconds
throttle_unauthenticated_deprecated_api	IP Address	Deprecated API endpoints	ALL	1800 requests per 3600 seconds	3600 requests per 3600 seconds	1800 requests per 3600 seconds	3600 requests per 3600 seconds
throttle_authenticated_deprecated_api	User	Deprecated API endpoints	ALL	3600 requests per 3600 seconds	3600 requests per 1800 seconds	3600 requests per 1800 seconds	3600 requests per 1800 seconds
throttle_incident_management_notification_web	Project	/projects/*/alert_management_alerts/notify.json	POST	3600 requests per 3600 seconds	3600 requests per 3600 seconds	3600 requests per 3600 seconds	3600 requests per 3600 seconds

Application (RackAttack)

Enable an Application Rate Limit in “Dry Run” mode

It is possible to enable RackAttack rate limiting rules in “Dry Run” mode which can be utilised when introducing new rate limits by setting the GITLAB_THROTTLE_DRY_RUN environment variable [source].

For GitLab.com these environment variables are managed in k8s-workloads, and set in the extraEnv.

Once the GITLAB_THROTTLE_DRY_RUN environment variable is configured in production, you can then turn the specified throttle on, for example throttle_authenticated_web. If the new limit that is being introduced is hit, you should see event_type="track" in the RackAttack metrics and logs.

After validating the rate limit threshold is behaving as expected, you should remove the event name from the GITLAB_THROTTLE_DRY_RUN environment variable which will allow the rate limit to start throttling requests.

• Metrics: RackAttack events by event name and type

LabKit cohort flags

The LabKit rack rate limiter runs next to RackAttack, behind six feature flags, until GitLab 19.7. To find which limiter returned a 429, and which flags to turn off for each symptom, see LabKit cohort flags: levers by symptom. That page applies to GitLab.com and GitLab Dedicated.

Tier-aware throttles (labkit)

GitLab.com applies request limits by subscription plan (Free, Premium, Ultimate), a plan-independent limit for unauthenticated traffic, 60 an hour per IP, and a per-plan burst limit on the MCP Server endpoint. They are labkit rules on the rack_request limiter, not RackAttack throttles, so nothing above about GITLAB_THROTTLE_DRY_RUN applies to them. Design: Tier-aware throttles.

Every rule name contains _traffic_per_, and each limit exists twice: a _log rule that only counts, and a :limit rule that blocks. A feature flag decides which one matches a request, so moving between observe and enforce is a flag flip, effective on the next request, with no deploy.

Flag	On its own
rate_limiter_plan_limits_<free|premium|ultimate>_info	that plan’s _log rules match and count
rate_limiter_plan_limits_<plan>_enforce	that plan’s :limit rules match and block, only while _info is also on
rate_limiter_unauthenticated_limits_info	the unauthenticated _log rule counts
rate_limiter_unauthenticated_limits_enforce	the unauthenticated :limit rule blocks, only while _info is also on
rate_limiter_mcp_limits_<plan>_info	that plan’s MCP _log rule counts requests to POST /api/v4/mcp
rate_limiter_mcp_limits_<plan>_enforce	that plan’s MCP :limit rule blocks, only while _info is also on

Every _enforce flag needs its _info flag on too. _enforce alone matches nothing, so to enforce a limit turn both on, and never delete _info first.

All 14 flags (Gitlab::RateLimit::FlagPolicy::FLAGS) are gitlab_com_derisk flags, so they exist only on GitLab.com. Any one of them starts the LabKit middleware, even when every cohort flag is off; see LabKit cohort flags.

Is a 429 tier-aware?

On the detail dashboard, the “Tier-aware” panels in the Labkit row break the rules out by name. result="log" is a would-be rejection during observe mode; result="block" is an enforcing rule deciding to reject. A tier-aware 429 carries the rule name in its RateLimit-Name header, which is how you tell one from any other 429. The MCP rules are named burst_mcp_traffic_per_user_<plan>_plan, and an MCP request is counted by both its MCP rule and the general per-plan rules.

No separate per-throttle counter for those 429s exists, and none is planned. result="block" on these panels is that count.

Rolling back

Turn off the _enforce flag for the affected plan. Its :limit rules stop matching and the _log twins take over, so the limit keeps counting but stops blocking. Turning off _info stops both rules at once.

/chatops run feature set rate_limiter_plan_limits_free_enforce false
/chatops run feature set rate_limiter_mcp_limits_free_enforce false

Two things to know when flipping:

• Each rule has its own Redis counter, keyed by rule name. Switching from the _log rule to the :limit rule (or back) starts from zero, so every client gets a fresh allowance at the flip.
• A _log rule never blocks, but a whole-check fail-open (see the Alerts section) skips every rule after the failure, so the tier-aware counters can under-report. A count_distinct rule that skipped a missing identifier key skips only that rule, the check still enforces the rest, and it needs its own triage. The check-error panel next to the tier-aware panels counts both and cannot tell them apart.

Tier resolution cache

Resolving a plan reads CachedPlanTier, which is backed by Rails cache with a ten minute expiry on every entry. Every request that resolves a tier reads it, so the hit rate is the difference between a cheap lookup and a database query on the request path.

gitlab_plan_tier_lookups_total{subject, result} counts them. subject is user or namespace, result is hit or miss. Two panels on the Detail dashboard show the hit rate and the lookup rate.

If the hit rate drops:

• Check the lookup rate panel first. A poor ratio on a few lookups a second is noise. Judge the two together.
• A drop across both subjects at once points at the cache itself rather than at traffic. Check Redis for the cache store, not redis-cluster-ratelimiting, which holds the rate-limit counters and not this.
• A drop on namespace alone usually means traffic has shifted towards paths that resolve a new root namespace each time, so the keys turn over faster than the ten minute expiry.
• Misses are not errors. The limiter still works, it just costs a database query per miss. There is no alert on this, so it is a triage aid rather than a page.

What a classification pass costs

The limiter resolves the requester before any endpoint authenticates it, using its own Gitlab::Auth::RequestAuthenticator. That is not a read-only lookup.

The pass only runs while a cohort shadow flag or a plan-tier or unauthenticated flag is on. Turning all of them off removes every cost below.

When it authenticates a request with a personal access token, it updates the token’s last-used state:

• last_used_at, at most once per ten minutes per token
• an Authn::PersonalAccessTokenLastUsedIp row, only for an IP not already stored, then a trim to the five newest distinct IPs
• an audit event, personal_access_token_used_from_unseen_ip, when a token used before appears from a new IP

The service holds a 60 second Redis lease per token and does not release it, so the whole write pass runs at most once a minute per token. The last_used_at column does not bound it. An IP write does not refresh that column, and it only moves every ten minutes.

Two writes run inside without_sticky_writes: the last_used_at update, and the IP row insert with its trim. Both still run on the primary. The block only stops the rest of the request from sticking there afterwards. The writes still mark the session as written, so at the end of an API request the user is stuck: their next requests read from a caught-up replica or from the primary until replication catches up. The audit event sits outside the block. It is defined as stream-only, so no audit_events row is written for it.

There is also a read, and it runs twice per request. The service decides whether it needs to write before it takes its Redis lease, and that decision reads personal_access_token_last_used_ips. The limiter and the endpoint each build their own LastUsedService, so the memoized lookup in one is invisible to the other. The read only runs when the request carries an IP and the token’s last_used_at is more than a minute old. On a token’s first use the service skips the lookup and goes straight to the write. That is an ordinary read, so the load balancer sends it to a replica unless the session already sticks to the primary. Both reads run before the API endpoint looks up the user’s sticking, so an earlier write that stuck the user does not send them to the primary. So expect this read on the replicas.

Over the 24 hours to 2026-10-14 00:00 UTC the table took 13,998 index scans a second against 19.2 inserts on the main cluster, so the cost of this path is reads rather than writes. That figure covers every reader of the table, not only the limiter. To measure it again:

avg_over_time((sum(rate(pg_stat_user_tables_idx_scan{env="gprd", fqdn=~"patroni-main.*", relname="personal_access_token_last_used_ips"}[5m])))[24h:5m])
avg_over_time((sum(rate(pg_stat_user_tables_n_tup_ins{env="gprd", fqdn=~"patroni-main.*", relname="personal_access_token_last_used_ips"}[5m])))[24h:5m])

Keep the fqdn filter. The same series exists on the ci and sec clusters, and on the archive, delayed and PITR hosts, and without the filter the sum spans all of them. The filter works on the host naming: the main cluster’s Patroni nodes are patroni-main-v17-*, while the archive, delayed and PITR hosts are named postgres-* and so fall outside it.

The read itself is tracked in production-engineering#30077, which covers it running before the Redis lease and twice per request.

If personal_access_token_last_used_ips appears in database load, the limiter is one contributor and this is why.

Two further costs per classified request, both expected:

• one Redis counter write per counting rule that matches
• the plan-tier reads described above

The audit event only reaches Sidekiq when an instance audit-event streaming destination is configured. When one is, it runs as AuditEvents::AuditEventStreamingWorker on the audit_events_audit_event_streaming queue, which you can watch on the Sidekiq worker detail dashboard.

The middleware order matters. Two middlewares are pinned above Gitlab::Middleware::LabkitRackRateLimit, and both use insert_before rather than use, so neither position depends on load order.

Gitlab::Middleware::IpAddress comes first, so the IP is already set when the limiter resolves the requester. Below the limiter, Gitlab::IpAddressState would be nil, the IP row and the unseen-IP audit event would both be skipped, and only last_used_at would still update.

Gitlab::Database::LoadBalancing::RackMiddleware is pinned there for the same kind of reason. Below the limiter, sticking would apply only after the limiter had loaded the session user, or the CI build on a job-token request, so those lookups could read from a lagging replica.

Neither failure raises, so both would be silent.

Alerts

The rate-limiting service in the metrics catalog generates SLO alerts for the application-level rate limiter driven by the Labkit::RateLimit library (the migration target for RackAttack and ApplicationRateLimiter call sites). The library fails open: a rate-limiter error never blocks a request, so these alerts signal lost rate-limiting coverage, not user-facing request failures.

Metrics behind the alerts

• gitlab_labkit_rate_limiter_checks_total{rate_limiter, action, matched, error} — one increment per check call, including checks that failed open. action is the caller-facing decision (allow or block); error="true" marks a check that encountered an error (Redis unavailable, or a count_distinct rule skipped over a missing identifier key).
• gitlab_labkit_rate_limiter_rule_evaluations_total{rate_limiter, rule, action, result} — one increment per evaluated rule; powers the per-rule breakdowns on the detail dashboard.

Two more SLIs on the service record rules but do not alert:

• rate_limiter_rule_evaluations: the error ratio is the share of rule evaluations that decided block. With the cohort flags on, which is the default, a block returns a 429; see LabKit cohort flags.
• rate_limited_responses: the error ratio is the share of Workhorse requests that got a 429, from any application-level limiter.

A limiter blocks on purpose, so a rising ratio can be expected, for example during an attack. For that reason, both SLIs are experimental and s3, and the team’s ignored_components in services/teams.yml keeps them out of the stage-group error budget. The plan is to set an SLO for each from the SLO analysis dashboard, and to use anomaly detection when observability#4457 is done.

The GitLab Dedicated version of the service is in reference-architectures/get-hybrid/src/services/rate-limiting.jsonnet. It has the same SLIs, but no traffic-cessation alert, because turning off the cohort flags would set it off.

What the alerts mean

• RateLimitingServiceRateLimiterChecksErrorSLOViolation — the fraction of checks encountering an error (checks_total{error="true"} / checks_total) burned through the error budget. Two distinct causes set error="true":
  • Whole-check fail-open — usually the redis-cluster-ratelimiting service being unhealthy. Rate limits are effectively not enforced for the affected traffic. Check that service’s alerts and dashboards first; the SLI declares the dependency, so a firing Redis alert is the upstream cause.
  • A count_distinct rule skipped over a missing identifier key — a caller configuration bug that fires with healthy Redis (no inhibiting Redis alert; the check still completes and enforces other rules). Search the gitlab-rails logs for rate_limit_missing_count_distinct — the log line carries the rule name.
• RateLimitingServiceRateLimiterChecksTrafficCessation / ...TrafficAbsent — the check rate dropped to zero or the metric disappeared. Rate-limit middleware is likely not running or not emitting metrics: check for recent gitlab-rails deployments or configuration changes that could have disabled the labkit middleware or its metrics.

Triage

1. Open the rate-limiting detail dashboard — the checks panel shows per-limiter request/block rates, the rule evaluations panel shows which rule is deciding.
2. Correlate with the redis-cluster-ratelimiting service overview if the alert is error-ratio based.
3. Rate-limit log lines are emitted in the calling service’s structured logs (gitlab-rails application logs), with name (limiter), rule, and error_type: rate_limit_error fields for fail-open events.
4. For a request that LabKit blocked, search auth_json.log (json.subcomponent: auth_json) for message: Rate_Limit_Block. The rate_limiter_rule_evaluations SLI links to the query. See Find the source of a 429 for the fields.