Hosted Runners Debugging Guide
Debugging a hosted runner involves two main steps:
- Verifying a runner-manager’s ability to spin up ephemeral VMs.
- Ensuring the ephemeral VMs can connect to GitLab.com or the CI Gateway.
Quick Overview
Section titled “Quick Overview”For a visual walkthrough, check out this video: Hosted Runners Testing.
Part 1: Testing Ephemeral VM Creation
Section titled “Part 1: Testing Ephemeral VM Creation”The most challenging aspect of testing runner-managers is composing the docker-machine command with all the required custom options. These options vary by manager, so we’ve created handy scripts to automate this process.
Using generate-create-machine.sh
Section titled “Using generate-create-machine.sh”This script is typically located in the /tmp folder of runner-manager VMs. It generates another script based on the configurations in the /etc/gitlab-runner/config.toml file of each runner-manager.
Steps to Run
Section titled “Steps to Run”$ sudo suExample Output of a Successful Run
Section titled “Example Output of a Successful Run”tmp# ./create-machine.shRunning pre-create checks...(test1) Check that the project exists(test1) Check if the instance already existsCreating machine...(test1) Generating SSH Key(test1) Creating host...(test1) Opening firewall ports(test1) Creating instance(test1) Waiting for Instance(test1) Uploading SSH KeyWaiting for machine to be running, this may take a few minutes...Detecting operating system of created instance...Waiting for SSH to be available...Detecting the provisioner...Provisioning with cos...Copying certs to the local machine directory...Copying certs to the remote machine...Setting Docker configuration on the remote daemon...Checking connection to Docker...Docker is up and running!
To connect your Docker Client to the Docker Engine running on this VM, run: docker-machine env test1Part 2: Testing Ephemeral VM Connectivity
Section titled “Part 2: Testing Ephemeral VM Connectivity”Once the ephemeral VM is created successfully, you can verify its connectivity.
Steps to Test Connectivity
Section titled “Steps to Test Connectivity”cos@test1 ~ $ curl -IL https://us-east1-c.ci-gateway.int.gprd.gitlab.net:8989cos@test1 ~ $ curl -IL https://gitlab.comExpected Outcome
Section titled “Expected Outcome”-
A successful call will return a
200status code. -
If any command times out, it may indicate a network misconfiguration.
Troubleshooting Tips
Section titled “Troubleshooting Tips”Common Issue: Network Misconfiguration
Section titled “Common Issue: Network Misconfiguration”One frequent issue is a missing network configuration for the CI Gateway. Ensure that the network is allowed in the CI Gateway configuration.
If problems persist, verify the VM’s network settings and access permissions.