macOS runner fleet deployments

Generally, we follow the standard runner blue/green deployment process.

However, given the nature of dedicated hosts and potential capacity issues we try to do so in periods of lower utilization. Peak utilization seems to be EMEA day-time.

View the current job and historic utlization on the CI runners dashboard.

Further, given we maintain a number of components for macOS (host image, job images, nesting etc), it is sensible to test any changes in our staging shard saas-macos-staging first. Once this is deployed to, test pipelines can be run in the saas-macos-staging test project.

macOS shard differences

saas-macos-medium-m1

Like saas-macos-staging, this shard uses mac2.metal dedicated hosts which are spread across all the availability zones that the AWS account has access to.

While these machines certainly don’t have the high quota availability of linux VMs, they are generally easily acquired. Therefore we can perform a blue/green switch without worrying too much about capacity, assuming we’re not at peak demand.

saas-macos-large-m2pro

The mac2-m2pro.metal dedicated hosts have a scarce supply in us-east-1.

Therefore, instead of releasing unused dedicated hosts (after 24h), we hold onto the hosts without instances running on them.

In this configuration the deployments (blue/green) share two availability zones which in turn allow sharing of the free dedicated hosts.

Before starting a deployment it’s recommended to log into the m2pro AWS account and verify that each availability zone has a few (2 at least) hosts free to be used immediately.

More context is available in the m2pro capacity limitations epic if necessary.