Troubleshooting AWS runners

Troubleshoot AWS runner issues.

Network misconfigurations are the most common cause of issues. See access requirements first.

Inspect live runner logs

Open your runner’s details and select View live logs when available. The viewer shows timestamps in UTC, log levels, and messages. Select Fields to inspect an entry’s structured data, or use the copy button to copy the original log entry.

Search messages and fields, then select a minimum log level: Debug, Info, Warn, or Error. Each level includes entries at that level and above. The default is Info. For example, Warn shows warnings and errors. Entries without a known level are excluded from the filtered view. Search covers the buffered logs, with the latest 1,000 entries retained. Scrolling up, searching, or inspecting an entry pauses the view so incoming logs do not move what you are reading. A Tailing paused notice explains when the view is frozen and indicates when new logs are available. Select Resume tailing to jump to the latest matching logs and follow new entries. Changing the log level keeps the current tailing or paused state.

Use Runner Logs to open CloudWatch for older logs.

CloudFormation stack fails

Symptoms: ROLLBACK_COMPLETE or ROLLBACK_IN_PROGRESS with errors like Parameter validation failed: parameter value for EC2RunnerInstancesSubnet does not exist.

Fix: Ensure you select a VPC, at least one availability zone, and subnets across multiple AZs.

Runner task fails

Symptoms:

• CREATE_FAILED with ECS Deployment Circuit Breaker was triggered
• ResourceInitializationError in task logs
• Cannot pull images or access AWS services

Fix:

• Verify VPC has Internet Gateway or NAT Gateway
• Update route tables (public → IGW, private → NAT)
• For private subnets, add VPC endpoints for Secrets Manager, S3, ECR
• Check security groups allow outbound HTTPS

Instance type not available

Symptoms: Error like “m6i.xlarge is not available in us-east-1e”

Fix:

• Use multiple AZs (avoid us-east-1d and us-east-1e exclusively)
• Try a different region or instance type
• Update stack parameters or create new environment class
• Retry later (availability is transient)

Unexpected costs

Symptoms: Unexpected AWS charges, or continued billing after deleting a runner.

Fix:

• See managing costs to identify resources
• After deleting a runner, verify the CloudFormation stack is fully deleted
• Check for residual EC2 instances or EBS volumes and delete manually

SSM access blocked

Symptoms:

• Environments fail with AWS account policy blocks ssm:SendCommand
• Runner marked as degraded
• Slow startup (cache credentials can’t refresh)

Cause: Service Control Policies (SCPs) or an outdated runner IAM policy blocking SSM access. The runner needs ssm:SendCommand and ssm:GetCommandInvocation permissions.

Fix: Update the runner’s CloudFormation stack to apply its current IAM policy. Ask your AWS administrator to check SCP restrictions as well; an IAM allow cannot override an explicit SCP deny.

ssm:GetCommandInvocation requires "Resource": "*" because it does not support resource-level permissions. Scoping it to an SSM command ARN denies access. ssm:SendCommand can remain scoped to the environment instances and the AWS-RunShellScript document:

{
  "Version": "2012-10-17",
  "Statement": [{
    "Effect": "Allow",
    "Action": "ssm:GetCommandInvocation",
    "Resource": "*"
  }, {
    "Effect": "Allow",
    "Action": "ssm:SendCommand",
    "Resource": ["arn:aws:ec2:*:*:instance/*", "arn:aws:ssm:*::document/AWS-RunShellScript"]
  }]
}

See AWS Systems Manager permissions for the supported resource types. Retain the runner policy’s account and tag conditions when adjusting permissions.

Prebuilds fail due to policy restrictions

Symptoms:

• Prebuilds fail with AWS Service Control Policy blocks ec2:CreateSnapshot
• Prebuilds fail with AWS IAM policy does not allow ec2:CreateSnapshot
• Similar errors for ec2:RegisterImage, ec2:DescribeSnapshots, or ec2:DescribeImages

Cause: Prebuilds require creating EBS snapshots and AMIs. These operations can be blocked by:

1. Service Control Policies (SCPs) - Organization-level policies that deny EC2 snapshot/AMI actions
2. IAM policies - The runner’s IAM role is missing required permissions (outdated CloudFormation stack)

Fix for SCP restrictions

Request your AWS administrator to allow these actions for the runner’s IAM role in the SCP:

{
  "Version": "2012-10-17",
  "Statement": [{
    "Effect": "Allow",
    "Action": [
      "ec2:CreateSnapshot",
      "ec2:RegisterImage",
      "ec2:DescribeSnapshots",
      "ec2:DescribeImages",
      "ec2:DeleteSnapshot",
      "ec2:DeregisterImage"
    ],
    "Resource": "*"
  }]
}

Fix for IAM policy restrictions

Update your CloudFormation stack to the latest version. The latest stack template includes all required IAM permissions for prebuilds.

Custom CA certificate issues

The ca-trust-init container in the Runner’s ECS task logs can help diagnose CA issues. Check its Status (it should show STOPPED with exit code 0) and its Logs for errors and warnings.

Environment stops shortly after starting

Symptoms: Environment starts but stops within seconds. No error is visible in the Ona dashboard.

Cause: The environment instance failed to download or parse the CA bundle. The instance shuts down when this fails.

Fix:

• Verify the CA bundle source (S3 bucket or HTTPS URL) is accessible from the runner’s VPC
• For S3 URLs, ensure the bucket name starts with gitpod- — the runner’s IAM role only has access to buckets matching gitpod-*
• Confirm the S3 object exists and the path in the CustomCATrustBundle parameter is correct
• Verify the CA bundle is valid PEM — the -----BEGIN CERTIFICATE----- and -----END CERTIFICATE----- delimiters must each be on their own line
• Verify the CA bundle does not contain invalid content, such as displaying metadata before -----BEGIN CERTIFICATE-----

User data is limited to 16384 bytes

Symptoms: Environment creation fails with "User data is limited to 16384 bytes"

Cause: The CustomCATrustBundle CloudFormation parameter uses an SSM dynamic reference ({{resolve:ssm:...}}), which embeds the full PEM certificate content into EC2 user data. CA bundles with multiple certificates exceed the 16 KB AWS limit.

Fix: Switch to an S3 URL for the trust bundle:

1. Create an S3 bucket with a name starting with gitpod- (e.g. gitpod-myorg)
2. Upload your CA bundle to s3://gitpod-myorg/shared/ca-bundle.pem
3. Update the CustomCATrustBundle CloudFormation parameter to s3://gitpod-myorg/shared/ca-bundle.pem
4. Update the CloudFormation stack

See Custom CA Certificate for details on all supported formats.

CA not trusted in devcontainer builds

Symptoms: Devcontainer image builds or feature installs fail with TLS certificate errors, even though the custom CA works for other operations.

Cause: Custom CA certificates are applied to the runner and environment host but not injected into devcontainer build phases. Docker builds run in an isolated context that does not inherit the host’s CA trust store.

Fix: Add your CA certificates directly to your devcontainer image:

COPY my-ca-bundle.crt /usr/local/share/ca-certificates/
RUN update-ca-certificates

Network connectivity issues

Checklist:

• Security groups: port 29222 (SSH), outbound HTTPS, port 22999 (internal)
• Route tables: public subnets → IGW, private subnets → NAT
• Network ACLs: not blocking required traffic
• DNS: VPC DNS resolution enabled, can resolve app.gitpod.io

Test connectivity:

# Health endpoint (should return 200)
curl -v https://<your-domain>/_health

# Required endpoints
curl -I https://app.gitpod.io
curl -I https://public.ecr.aws

Restart runner after network changes

After changing security groups, route tables, or VPC endpoints, restart the runner:

Console: ECS console → Clusters → your cluster → Services → Update → check Force new deployment

CLI:

aws ecs update-service --cluster YOUR_CLUSTER_NAME --service YOUR_SERVICE_NAME --force-new-deployment

Verify: Check runner shows “Connected” in Settings → Runners, then test creating an environment.

Getting help

Use Support from the organization menu. Include:

• Runner ID and version (from Settings → Runners → ... menu)
• CloudFormation stack name and region
• Runner logs from CloudWatch (ECS task logs)