Gitaly unusual activity alert

Symptoms

• Alert on Slack: Unusual Gitaly activity for a project has been detected. Review the runbook at https://gitlab.com/gitlab-com/runbooks/tree/master/docs/gitaly/gitaly-unusual-activity.md for more details

1. Review the suspicious activity

• Check out the abuse dashboard: https://dashboards.gitlab.net/d/9T-wXWbik/abuse-dashboard?orgId=1&panelId=2&from=now-1h&to=now
• Review the abuse reporting data in Kibana: https://log.gprd.gitlab.net/goto/6636a49add992f6326862df0afc6ae54
• Review the abuse dashboard: https://log.gprd.gitlab.net/app/kibana#/dashboard/AWSIfVZhTIzC7JP6Xxn1
• Keep in mind that this is an open-ended alert, so it alerts to suspicious activity, rather than pin-pointing an issue.
• Use this as an informational alert, combine it with other signals

2. Evaluate impact

• If the affected Gitaly server is under load due to the activity this project is generating, consider disabling the project:

  1. Archive the project - this is especially useful if the project name or description itself contains links or reference to scams/spam/malware, as it de-lists the project from search as well.

  2. Delete the project - as far as we can tell, this is the only way to shut down a project that’s publishing to gitlab pages.

• If the traffic is being generated by anonymous users accessing a public project, consider making the project private.

  1. We tend to do for people using GitLAb as a CDN, highly trafficked repos, etc. This doesn’t always help, at least one project has included authentication to access the private repos.

  2. Go through the UI: Settings -> General -> Permissions -> Project Visibility

• If the affected Gitaly server is under load due to multiple instances of git upload-pack --stateless-rpc processes corresponding to git clone requests, it’s possible that there is a packfile bitmap missing in the repo.

  1. Check if the repo has a pack/objects/.bitmap file.

  2. If it doesn’t, click the housekeeping button on the repo in the web UI: Project Settings -> General -> Advanced -> [Run housekeeping]