Gitaly profiling

Gitaly process

Gitally is written in Go which has built in profiling.

A convenient way is to use ssh port forwarding to download the profiles directly to your own workstation. The actual Gitaly port (9236) can be found here. We are forwarding this to our local port 6060 as this is the standard port for go pprof endpoints.

ssh -N -L 6060:localhost:9236 file-03-stor-gprd.c.gitlab-production.internal

curl -o cpu.bin 'http://localhost:6060/debug/pprof/profile'
go tool pprof -http :8080 cpu.bin


curl -o trace.bin 'http://localhost:6060/debug/pprof/trace?seconds=5'
go tool trace trace.bin


curl -o heap.bin 'http://localhost:6060/debug/pprof/heap'
go tool pprof -http :8080 heap.bin


curl -o goroutines.txt 'http://localhost:6060/debug/pprof/goroutine?debug=2'

GCP Profiling

GCP profiles are also available service wide.

System CPU Profile

Since Gitaly shells out to git the Go profile is not enough information. We can use Linux built in perf tool

[email protected]:~$ perf_flamegraph_for_all_running_processes.sh


[email protected]:~$ perf_flamegraph_for_pid.sh 12


[email protected]:~$ perf_flamegraph_for_user.sh git

Periodic System CPU Profiles

We run CPU profiles at a periodic internal and upload these to gs://gitlab-gprd-periodic-host-profile where you can choose a hostname and a specific time.