CloudflareCloudConnectorRateLimitExhaustion

Overview

See rate limiting for a general description of how we enforce rate limits in Cloudflare.

This alert fires when Cloudflare enforced rate limits that resulted in clients being blocked for an extended period of time.

This could be for good or bad reasons:

• Good: malicious traffic from 3rd parties we have no business relationship with
• Bad: traffic from paying customers who have outgrown their rate limit budget

The “bad” case can happen when actual usage of Cloud Connector powered features outgrows rate limits we allocate to this customer based on the number of user seats they purchased from us.

Metrics

The alert fires when the cloudflare_zone_firewall_events_count rate exceeds a given threshold for a given time window.

• Current alerts
• alert definition

Alert Behavior

The alert does not invoke pagers but is posted to the #cloud-connector-events Slack channel instead.

Troubleshooting

Cross-check in Cloudflare who was affected by these events:

• Cloudflare: Security analytics
• Cloudflare: Security events

If you suspect it was a paying customer, refer to possible resolutions for next steps. Otherwise it can be ignored.

Possible resolutions

This is most likely only relevant in case one of the Duo-specific rate limits was exhausted.

If you established that this is affecting a paying customer whose Cloud Connector features may now be degraded, escalate the issue with #g_cloud_connector. Depending on who the customer is and which features they use, further escalation to stage groups may be necessary.

If rate limits need to be adjusted, they can only be so for the entire rate limit bucket the customer falls into. This means increasing rate limits will affect all other customers falling into the same rate limit bucket too and means we need to consider increasing any limits upstream of cloud.gitlab.com such as AI vendor quotas. Refer to the AI gateway runbook in this case.

Cloudflare rate limit buckets are defined here.