Debugging MacOS Runners

This document provides a comprehensive guide for debugging issues with AWS MacOS runner instances. It covers monitoring instance health, accessing instances, debugging nested VMs, and handling common problems.

Determining autoscaling group (ASG) health

Health Metrics and Monitoring

AWS ASG dashboard

Access via AWS Console
- Medium Macs are in account 215928322474
- Large Macs are in account 730335264460
Check for instances in unhealthy states
Check scaling activity history for unexpected terminations or failed operations

Mac host metrics

Check the EC2 Dedicated Hosts dashboard for abnormal states
Monitor for hosts in “pending” or “released” states that might indicate provisioning issues
Verify vCPU utilization is present for all active hosts

Runner manager metrics

Grafana dashboard

Logs Location

Nesting logs: /Users/ec2-user/nesting.log
MacOS init logs (from user-data script on Mac host): /var/log/amazon/ec2/ec2-macos-init.log

Access to MacOS VMs

Refer to access.md for information on how to access MacOS VMs.

Debugging connection between runner manager and host Mac

Confirm established connection with host Mac
Terminal window
```
# On runner manager
ss -tn | grep HOST_MAC_PRIVATE_IP4
```
Confirm established connections from host Mac to runner manager
Terminal window
```
# On the host Mac
sudo lsof -i@RUNNER_MANAGER_PRIVATE_IP4
```
List network connections between host Mac and nesting VMs. For every VM in nesting list there should be a connection.
Terminal window
```
# On the host Mac
sudo lsof [email protected] | grep nesting
```

Debugging Nesting Client/Server

Logs Locations

Nesting settings

# On the host Mac
cat /Users/ec2-user/nesting.json

List available nesting images

# On the host Mac
ls /Volumes/VMData/images

Managing Nested VMs with Nesting client

Confirm nesting service is running

launchctl list | grep nesting

Get nesting help

nesting

Get nesting version

nesting version

List running VMs

nesting list

Output will provide the ID, image, and localhost:port of the running VM. For example:

bb9wtbcq macos-14-xcode-15 127.0.0.1:60835

SSH into VM

The SSH userid and password for VMs can be found in the associated runner manager instance in the [runners.autoscaler.vm_isolation.connector_config] section of /etc/gitlab-runner/config.toml.

ssh -p PORT [email protected]

Possible Nesting Issues

Service not starting

# Restart the nesting server
sudo launchctl unload /Library/LaunchDaemons/nesting.plist
sudo launchctl load /Library/LaunchDaemons/nesting.plist

Connection issues

# Check network connectivity
sudo tcpdump -i any port 22 | grep RUNNER_MANAGER_PRIVATE_IP4