Skip to content
Runbooks

HAProxyIPOptionsDropsHigh

Overview

Section titled “Overview”

The HAProxy VMs run an nftables rule, managed by the gitlab-server::nft_cve_2026_98432 Chef recipe, that drops IPv4 TCP packets carrying IP options. It mitigates CVE-2026-98432. Its drop counter is exported every 5 minutes through the node_exporter textfile collector as nftables_cve_2026_98432_dropped_packets_total.

This alert fires when a host drops more than 100 such packets in 15 minutes, for at least 5 minutes. That is either an exploitation attempt, which is already blocked, or a real client that uses IP options and can’t connect.

Most internet routers drop or strip IP options, so normal traffic should carry almost none. gstg only gets health-check traffic, so there the alert mainly tests the alert path.

Services

Section titled “Services”

Diagnosis

Section titled “Diagnosis”

In Grafana Explore, pick the datasource for the alert’s env label (Mimir - Gitlab Gstg for gstg, Mimir - Gitlab Gprd for gprd) and check which hosts are dropping and since when:

sum by (fqdn) (increase(nftables_cve_2026_98432_dropped_packets_total{env="gstg"}[15m]))

For gprd, change env="gstg" to env="gprd".

One host suggests traffic aimed at it, many hosts at once suggest a scan across the fleet. If the series is missing for a host, check the exporter on that host (below) before trusting the alert.

On the affected node (the fqdn label), confirm the counter is rising:

Terminal window
sudo nft list table inet cve_2026_98432
cat /opt/prometheus/node_exporter/metrics/nft_cve_2026_98432.prom

Find the sources with temporary, rate-limited logging:

Terminal window
sudo nft insert rule inet cve_2026_98432 input ip protocol tcp ip hdrlength '>' 5 limit rate 10/minute log prefix '"CVE-2026-98432 drop: "'
sudo journalctl -k --since "10 min ago" | grep "CVE-2026-98432 drop" | grep -o 'SRC=[^ ]*' | sort | uniq -c | sort -rn | head

Chef does not remove the log rule: it only reloads the table when the rules file changes. Always remove it yourself when you’re done, by reloading the table from its file, and confirm only the drop rule is left:

Terminal window
sudo systemctl reload-or-restart nft-cve_2026_98432
sudo nft list table inet cve_2026_98432

The reload resets the drop counter to 0. The alert handles this, because it uses increase().

Remediation

Section titled “Remediation”
  • Unknown or scanner sources: exploitation attempts, already blocked. Share the source IPs with SIRT.
  • A known partner or internal range: a real client is impacted. Check HAProxy errors for that source, and decide with SIRT whether to keep the mitigation.

If the nftables rule needs to be removed

Section titled “If the nftables rule needs to be removed”
  • Rollback, only if agreed with SIRT: set gitlab-server.nft_cve_2026_98432.enable to false in the HAProxy base role in chef-repo. This removes the rule from every HAProxy host in that environment.