Skip to content
Runbooks

UbuntuLivepatch

Overview

Section titled “Overview”

Ubuntu Livepatch is a service that provides online security fixes for High/Critical level severity security vulnerabilities in the Linux kernel. We use this service across our fleet in order to apply security fixes to systems that are difficult to reboot quickly.

Services

Section titled “Services”
  • Some Ubuntu VM instances that are managed by Chef should be enrolled in Ubuntu Advantage and have Livepatch enabled. This alert checks if a node is enrolled into Ubuntu Pro subscription and does not have Livepatch enabled.

Metrics

Section titled “Metrics”
  • The metric used for this alert is generated by this shell script, which is installed on all Chef managed VM instances.
  • The script writes a file with Prometheus formatted metrics, where the node_exporter process then exports them to Prometheus.
  • The metric canonical_livepatch_enabled is generated based on the output of the ua status command, where the statuses of “warning” or “enabled” are considered enabled, anything else is interpreted as not enabled.
  • A Systemd timer is installed to execute this script every 5 minutes.

Alert Behavior

Section titled “Alert Behavior”
  • The alert looks for any instances enrolled into Ubuntu Pro in GPRD and report Livepatch as being disabled, while also having run chef-client recently.

Severities

Section titled “Severities”
  • This alert is low severity, and intended to inform us when there are nodes that may not be receiving security patches as we intend.

Verification

Section titled “Verification”
  • Grafana Explore
  • You can verify the state of Livepatch on the machine using the command sudo ua status

Troubleshooting

Section titled “Troubleshooting”
  • Ensure chef-client runs are completing without error.
  • If Chef is running, verify that the attribute that enables the Livepatch feature is set to true:
    • knife search node 'name:<nodename>' -a gitlab-server.ubuntu-advantage.livepatch

Possible Resolutions

Section titled “Possible Resolutions”
  • If the attribute mentioned above is false, find the roles associated with the node, and check to see if there is a misconfiguration that is resulting in the attribute being overridden.
  • From the ua status command, ensure that the node is associated with our Ubuntu Advantage account.

Dependencies

Section titled “Dependencies”

  • Chef needs to be running on the instance for this alert to function.

  • If this alert fires, it is likely a good idea to first check with the service owner for the VM instance, and see if this isn’t intentional.

  • For generic Livepatch related help, you can reach out in #g_production-engineering_ops in Slack.

  • Alert definition

  • Update the template used to format this playbook