MacOS resources in AWS

This document outlines where most of the resources live in AWS, this can help you know where to look to debug issues.

Go to access.md for information on how to access the resources described in this document.

MacOS on AWS

Instances

All the MacOS instances are in the ‘us-east-1’ region.
All the job VMs are considered ephemeral VMs.
In the case of MacOS, hosts live for at least 24h.
- The 24h rule is due to licensing limitations, see licensing.md for details.
There are firewall rules between AWS and GCP (gitlab-ci-155816 project) to allow ssh and other traffic from these VMs.
See architecture.md for more details about the connections established between AWS and GCP.

Dedicated Hosts

EC2 Dedicated Host Lifecycle
Upgrading MacOS Dedicated Host
Perhaps the most important column in the dedicated hosts view is the State of each of the Hosts.
When a host is missing vCPU utilization info, it could indicate the instance is deleted, but not yet deleted from the account’s pool.
Released state means the instance is no longer connected to our AWS account, it’s not clear how long it takes for these entries to be deleted.
Pending indicates the instance is currently being reprovisioned.

AMIs

The images appearing in the AMI view are images that are used for provisioning the EC2 instances.

The AMIs generated here are stored in S3. See also image building.

NOTE: To understand the difference between an EC2 AMI and user-facing AMI, you should have a basic understanding of the architecture of these runners. In summary, each EC2 VM you see in the console, spins up two nested VMs within itself. These nested VMs use the user facing jobs images, while the parent instance, uses the EC2 instance images. For more details on the architecture of these runners, have a look at architecture.md.

Volumes and Snapshots

For performance reasons, EBS Volumes are used to store job VM disk images and provide persistent storage for MacOS hosts. MacOS’s SIP (System Integrity Protection) prevents programmatic access to volumes without user authorization. For example, the nesting daemon cannot access EBS volumes attached to the MacOS host without said user authorization. No API exists to bypass SIP, so we hack around it using automated VNC keyboard commands to click through permission dialogs.

More details about SIP and how we use EBS volumes in the image building doc.

Security Groups

Network Interfaces

Auto Scaling Groups

Service Quotas

console.aws.amazon.com/servicequotas

Quota limits for how many dedicated MacOS instances we can run at a time. To view these limits:

Go to Amazon Elastic Compute Cloud (Amazon EC2).
Filter for mac2.
Click Running Dedicated mac2 Hosts.

S3

When job images are built in the job-images project, they are uploaded to S3.

These job images are then pulled into MacOS hosts when they are first provisioned.

More details on the method for chunking and downloading the job images can be found in the s3pipe README.

console.aws.amazon.com/s3

VPC

Runner managers in GCP access MacOS hosts via a redundant, 4-tunnel VPN connection that allows secure communication between GCP and AWS networks with automatic failover and dynamic routing via BGP.