Skip to content
Runbooks

gitlab-haproxy-agent: draining HAProxy servers via consul KV

gitlab-haproxy-agent runs on every HAProxy node and answers haproxy agent-check probes with the desired server state held in consul KV. Writing one KV key changes a server’s state on every enrolled HAProxy within seconds, with no reload, no chef converge, and no SSH to load balancers.

Design and history: https://gitlab.com/gitlab-com/gl-infra/production-engineering/-/work_items/29645

The KV contract

Section titled “The KV contract”
gitlab-haproxy-agent/state/<backend>/<server> = ready | drain | maint | <weight>% (0-256)

An absent key means ready. Deleting the key is how a state is cleared.

Use maint for take-out-and-return work and drain for instant removal from new-connection balancing: leaving maintenance triggers the configured slowstart ramp, leaving drain does not.

Only enrolled backends react. Enrollment is per backend via node['gitlab-haproxy']['agent']['check_backends'] in chef, and shows up as agent-check ... agent-send "<backend>/<server>\n" on the server lines in haproxy.cfg. Keys for backends that are not enrolled have no effect.

Currently enrolled: gstg canary_web.

Draining a server

Section titled “Draining a server”

From a chef-repo checkout, -k writes KV in addition to the classic socket commands:

Terminal window
./bin/set-server-state -k gstg drain gke-cny-web
./bin/set-server-state -k gstg ready gke-cny-web

Or directly on any node in the environment (every node runs a consul agent):

Terminal window
consul kv put gitlab-haproxy-agent/state/canary_web/gke-cny-web drain
consul kv delete gitlab-haproxy-agent/state/canary_web/gke-cny-web

Verifying

Section titled “Verifying”

An agent-driven state shows as DRAIN (agent) or MAINT (agent) in show stat, distinct from socket or config driven states:

Terminal window
echo "show stat" | sudo socat stdio /run/haproxy/admin.sock \
  | awk -F, '$1 == "canary_web" {print $1"/"$2": "$18}'

To see the full desired state:

Terminal window
consul kv get -recurse gitlab-haproxy-agent/state/

To check what a specific agent answers, on the HAProxy node:

Terminal window
printf 'canary_web/gke-cny-web\n' | nc 127.0.0.1 9777

When something is off

Section titled “When something is off”
  • The agent serves /-/liveness and /-/metrics on port 9778. Sync freshness, response latency, and error counters are all there.
  • If KV writes are not taking effect on a node, its consul watch is likely broken: see HAProxyAgentConsulSyncStale.
  • The agent never guesses. When it is stale or down, haproxy keeps each server’s last received state. A drain issued during that window needs to be confirmed or re-issued after recovery.
  • Logs are in the pubsub-system-inf-<env> index (data stream, query as .ds-pubsub-system-inf-<env>-*) with json.ident: gitlab-haproxy-agent.

Integrations

Section titled “Integrations”