Redis
Overview
Section titled “Overview”Our VM based Redis deployments are managed by three different Chef cookbooks.
System patching may be performed on instances deployed by any of these, however, limitations may apply to instances using the omnibus-gitlab cookbook. It is not possible to update the gitlab-ee packages deployed on these systems without also updating Redis itself, so any vulnerabilities relating to this may require a migration to a new deployment mechanism using one of the other options.
Lead Time
Section titled “Lead Time”Our Redis infrastructure is deployed using highly available deployment topologies that should enable security patching with minimal notice.
Process
Section titled “Process”See Linux Patching Overview for generic processes applied to all Linux systems.
- Iterate over all nodes in the shard or deployment.
- If the node is the primary:
- Ensure we are able to failover.
- Perform a failover to promote another node.
- Validate that failover was successful.
- Reboot the node.
- Ensure the node booted, and redis started. Start the redis service if needed.
- Wait for the redis to configure itself as a replica and catch up with the primary.
Most of the required logic for doing so safely exists in the set of redis reconfigure scripts:
These could be adapted to perform a full reboot instead of just a service restart.
Additional Automation Tooling
Section titled “Additional Automation Tooling”No automation exists for the fleets deployed on virtual machines. The scripts mentioned above could be updated to help remove repetitive tasks that apply to individual servers, but today, each instance must be touched by a SRE.