HAProxy on Kubernetes

The HAProxy fleet is moving from the Chef VMs to pods on GKE, one release per flavor and traffic kind. The design and the decisions behind it are in the HAProxy on Kubernetes design document. The migration is tracked in production-engineering#29644. This section contains knowledge that is useful for engineers on-call. Everything below works the same on pre, gstg and gprd.

Where things are

Environment	Cluster	glsh kube use-cluster
pre	pre-gitlab-gke	pre-gitlab-gke
gstg	gitlab-36dv2	gstg-gitlab-36dv2
gprd	gitlab-3okls	gprd-gitlab-3okls

Namespace haproxy. One release per shard, named haproxy-<shard>: main-http, main-ssh, main-kas, main-internal-http, main-internal-kas, ci, registry, pages (ADR 003). Each release has one Deployment per zone, haproxy-<shard>-<zone>, pinned to that zone, with its own ConfigMap and HPA (1 to 3 pods), all behind one LoadBalancer Service on a reserved address (ADR 004). Backends in the pod’s own zone are active, the other zones are backup, as on the VMs. A PodDisruptionBudget per Deployment allows one pod out at a time.

kubectl -n haproxy get pods -l app.kubernetes.io/instance=haproxy-main-http -o wide
kubectl -n haproxy get svc

Each pod runs the haproxy container and the agent sidecar (gitlab-haproxy-agent), the same agent as on the VMs. It reads the same Consul KV. The chatops canary commands and set-server-state apply to the pods as well. The stats socket is /run/haproxy/admin.sock in the haproxy container and socat is in the image:

kubectl -n haproxy exec deploy/haproxy-main-http-us-east1-b -c haproxy -- \
  sh -c 'echo "show servers state" | socat stdio /run/haproxy/admin.sock'

Stats and Prometheus metrics are on port 7331 of every pod (kubectl port-forward, then / for the stats page, /metrics). Metrics have type="haproxy" and a shard label, one shard per release, next to the VM fleet’s type="frontend".

Configuration

Everything is in argocd/apps, rendered by the haproxy chart (ADR 002). The haproxy.cfg of a shard is the config: template in services/haproxy/env/<env>/shards/<shard>/values.yaml. The backend servers are the pools in services/haproxy/env/<env>/values.yaml. A config change is a merge request, and the merge rolls the pods. Rollback is a revert.

The haproxy ApplicationSet syncs on its own in two steps: pre and gstg first, half of the Applications at a time, then gprd one Application at a time, each step waiting for the previous one to be Healthy. A broken config fails the startup probe of the first new pod and the rollout stops there. If the pods have not rolled a few minutes after a merge, look at the ApplicationSet first:

kubectl -n argocd get applicationset service-haproxy -o json | jq '.status.applicationStatus'

An Application that is still OutOfSync after its sync (for example a resource flagged requiresPruning after a rename) stops the step from completing, and with it every later step. A sync with prune on that Application clears it.

Certificates and the stats password come from Vault through External Secrets, the same paths as the Chef cookbook. The origin certificate for the router hostname is issued by cert-manager (ADR 005).

Deny lists and other lists

The lists under /etc/haproxy/lists/ on the pods (deny-403-ips.lst, deny-403-paths-regex.lst, deny-403-user-agents.lst, deny-403-pages-domains.lst, blocklist-uris.lst, require-captcha.lst) come from the front-end-security repository: a pipeline on master publishes them to Vault, External Secrets picks them up within 5 minutes, and Reloader restarts the pods that mount them. A merge in front-end-security is on the pods in about 10 minutes without any further action. See Blocking individual IPs and Net Blocks.

Taking a release out of traffic

Each release receives traffic through a different mechanism (ADR 001), and each mechanism has its own way out:

Release	Traffic comes through	To remove it
main-http	Cloudflare Spectrum on 80/443, and the http-router GITLAB_ORIGIN_OVERRIDES flag while the VMs still serve	Move the Spectrum origin. For the flag, set the rollout to 0% or remove the zone’s entry
main-ssh, main-kas	Spectrum origin lists (tcp/22, tcp/8150, altssh tcp/443)	Move the Spectrum origin
main-internal-http, main-internal-kas, ci	internal DNS records with the VM and the GKE address	config-mgmt, drop the GKE address from the record
registry, pages	the VM load balancer’s NEG backends, or the Service on the published address after the handover	config-mgmt, remove the NEG backends. After the handover the VMs are decommissioned and the Service is the only path

The http-router flag is a Flagship flag in the http-router Worker. A flag that reads DISABLED still serves its default variant. The override is off only when the default variant is {} or the host entry is removed. Every path routed by a Cloudflare origin rule on the zone must be in the flag’s excludePathPrefixes, or those requests fail with a 526.

Draining

A pod drains itself when it is deleted: the preStop hook disables the release’s check frontend (check_https, check_ssh or check_kas). The load balancer health check and the readiness probe fail and new connections stop within 30 seconds. Then HAProxy gets a soft stop and established connections finish, up to hard-stop-after (30 minutes). Node drains and rolling updates go through this path.

To drain one pod without deleting it, do what the hook does. enable frontend undoes it:

kubectl -n haproxy exec <pod> -c haproxy -- \
  sh -c 'echo "disable frontend check_https" | socat stdio /run/haproxy/admin.sock'

To take a zone out of a release, remove it from zonal.zones in the shard’s values and merge. The Deployment for that zone is deleted and its pods drain as above. There is no quicker zone switch: the HPA scales a Deployment back up if you scale it to 0 by hand.

Rolling updates need room for a surge pod in every zone. The haproxy-spare capacity buffer keeps one pod’s worth of node capacity free per zone. If pods sit in Pending during a roll, check the node pools (RESOURCE_POOL_EXHAUSTED and QUOTA_EXCEEDED show in the cluster-autoscaler status ConfigMap in kube-system).

Logs

Request logs are JSON on stdout. They are in BigQuery, dataset haproxy_logs, table stdout, in the environment’s project (gitlab-production, gitlab-staging-1), next to the VM tables, partitioned by day. See HAProxy Logging. The same lines are in Cloud Logging as resource.type="k8s_container", namespace haproxy, container haproxy, and kubectl logs -c haproxy on a pod for the last minutes.