> ## Documentation Index
> Fetch the complete documentation index at: https://ona.com/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Alerts and Dashboards

> [!NOTE]
>
> Available on the Enterprise plan. [Contact sales](https://ona.com/contact/sales) to learn more.

Ona provides pre-built Grafana alerts and a dashboard for GCP Runners. These live in the [`terraform-google-ona-runner`](https://github.com/gitpod-io/terraform-google-ona-runner) repository under the [`monitoring/`](https://github.com/gitpod-io/terraform-google-ona-runner/tree/main/monitoring) directory and are designed to work with the Prometheus metrics your runner already exposes.

Before using these alerts and dashboards, you need to configure metrics collection on your runner. See [Custom metrics pipeline](https://ona.com/docs/ona/runners/monitoring-and-metrics.md) for setup instructions.

## Prerequisites

* A deployed GCP Runner with [metrics collection enabled](https://ona.com/docs/ona/runners/monitoring-and-metrics.md)
* A Grafana instance (or compatible alerting system) connected to your Prometheus data source

## Dashboard

The repository includes a Grafana dashboard at [`monitoring/dashboards/gitpod-runner-overview.json`](https://github.com/gitpod-io/terraform-google-ona-runner/blob/main/monitoring/dashboards/gitpod-runner-overview.json).

### What it covers

| Section | What it shows |
| - | - |
| **Version & Replicas** | Runner version tracking and replica count |
| **Health Status** | Health checks and active instance states by lifecycle stage |
| **GCP Runner Kit Interface** | Environment operation durations, function calls, error rates |
| **GCP API Operations** | API request metrics, success rates, error rates, latency heatmaps |
| **KV Store Operations** | Redis/key-value store operation rates and durations |
| **PubSub Operations** | Message processing, acknowledgments, connection health |
| **Environment Operations** | Compute environment operation rates and durations |
| **System Metrics** | Host-level CPU, memory, disk usage, and disk I/O |
| **WRI** | Workspace Runtime Interface performance metrics |

The dashboard uses template variables (`$project_id`, `$region`, `$runner_name`, `$instance`) so you can filter by deployment.

### Import the dashboard

1. In Grafana, go to **Dashboards → Import**
2. Upload `gitpod-runner-overview.json` from the repository
3. Select your Prometheus data source
4. Configure the template variables to match your deployment

## Alerts

The repository includes 19 alert definitions at [`monitoring/alerts/`](https://github.com/gitpod-io/terraform-google-ona-runner/tree/main/monitoring/alerts), each in its own folder with an `alert.yaml` (Grafana-compatible alert rule) and a `runbook.md` (troubleshooting steps).

### Alert overview

#### Critical — immediate response required

These indicate a service outage or severe degradation.

| Alert | Condition | Impact |
| - | - | - |
| [Service Down](https://github.com/gitpod-io/terraform-google-ona-runner/tree/main/monitoring/alerts/service-down) | Runner or auth proxy `up` metric is 0 for >1 min | Complete outage — users cannot create or manage environments |
| [High Error Rate](https://github.com/gitpod-io/terraform-google-ona-runner/tree/main/monitoring/alerts/high-error-rate) | >10% of environment operations failing over 5 min | Users experiencing environment creation failures |
| [High Latency](https://github.com/gitpod-io/terraform-google-ona-runner/tree/main/monitoring/alerts/high-latency) | 95th percentile operation time >5 min | Slow environment operations |
| [Goroutine Panics](https://github.com/gitpod-io/terraform-google-ona-runner/tree/main/monitoring/alerts/goroutine-panics) | Application panics detected | Potential service instability |

#### High — prompt attention required

These indicate degraded performance or functionality.

| Alert | Condition |
| - | - |
| [API Rate Limiting](https://github.com/gitpod-io/terraform-google-ona-runner/tree/main/monitoring/alerts/api-rate-limiting) | Hitting GCP API rate limits |
| [PubSub Backlog](https://github.com/gitpod-io/terraform-google-ona-runner/tree/main/monitoring/alerts/pubsub-backlog) | >1000 unprocessed messages |
| [PubSub Connection Health](https://github.com/gitpod-io/terraform-google-ona-runner/tree/main/monitoring/alerts/pubsub-connection-health) | PubSub connectivity issues |
| [Circuit Breaker Open](https://github.com/gitpod-io/terraform-google-ona-runner/tree/main/monitoring/alerts/circuit-breaker-open) | Circuit breaker protecting system from cascading failures |
| [Redis Connection Issues](https://github.com/gitpod-io/terraform-google-ona-runner/tree/main/monitoring/alerts/redis-connection-issues) | Redis connectivity problems |

#### Medium — monitor and track

These indicate reduced capacity or resource constraints.

| Alert | Condition |
| - | - |
| [High CPU Usage](https://github.com/gitpod-io/terraform-google-ona-runner/tree/main/monitoring/alerts/high-cpu-usage) | CPU usage >80% for extended period |
| [High Memory Usage](https://github.com/gitpod-io/terraform-google-ona-runner/tree/main/monitoring/alerts/high-memory-usage) | Memory usage >80% for extended period |
| [High Disk Usage](https://github.com/gitpod-io/terraform-google-ona-runner/tree/main/monitoring/alerts/high-disk-usage) | Disk usage >85% for extended period |
| [Network Connection Health](https://github.com/gitpod-io/terraform-google-ona-runner/tree/main/monitoring/alerts/network-connection-health) | Network connectivity issues |
| [Network Errors](https://github.com/gitpod-io/terraform-google-ona-runner/tree/main/monitoring/alerts/network-errors) | High rate of network errors |
| [Registry Health](https://github.com/gitpod-io/terraform-google-ona-runner/tree/main/monitoring/alerts/registry-health) | Container registry connectivity issues |
| [Zone Capacity Issues](https://github.com/gitpod-io/terraform-google-ona-runner/tree/main/monitoring/alerts/zone-capacity-issues) | GCP zone unavailable or at capacity |
| [Quota Exceeded](https://github.com/gitpod-io/terraform-google-ona-runner/tree/main/monitoring/alerts/quota-exceeded) | GCP resource quotas hit limits |

#### Info — optimization opportunities

| Alert | Condition |
| - | - |
| [High Process Memory Usage](https://github.com/gitpod-io/terraform-google-ona-runner/tree/main/monitoring/alerts/high-process-memory-usage) | Process memory usage >1GB for extended period |
| [High Goroutine Count](https://github.com/gitpod-io/terraform-google-ona-runner/tree/main/monitoring/alerts/high-goroutine-count) | >1000 active goroutines |

### Import alerts into Grafana

Each alert folder contains an `alert.yaml` file:

1. In Grafana, go to **Alerting → Alert Rules**
2. Click **Import**
3. Upload the `alert.yaml` from the alert folder you want (e.g., `service-down/alert.yaml`)
4. Configure notification channels for the alert's severity level

### Customize thresholds

The default thresholds work for most deployments. Adjust them based on your scale:

* **Smaller deployments** may need lower thresholds to catch issues earlier
* **Larger deployments** may need higher thresholds to reduce noise
* **Development environments** may want less sensitive alerts

### Runbooks

Each alert folder also contains a `runbook.md` with step-by-step troubleshooting instructions. Before using a runbook, set up these environment variables:

```bash theme={null}
export PROJECT_ID="your-gcp-project-id"
export REGION="your-region"          # e.g., us-central1
export RUNNER_ID="your-runner-id"    # from your Terraform configuration
```

The runbooks use `gcloud compute ssh` commands to inspect the runner instance and include resolution steps and escalation procedures.

## Notification channels

Configure notification channels in Grafana based on alert severity:

| Severity | Suggested channels |
| - | - |
| Critical | PagerDuty, SMS, phone |
| High | Slack, email |
| Medium | Email, ticket creation |
| Info | Email, dashboard review |

## Next steps

* [Custom metrics pipeline](https://ona.com/docs/ona/runners/monitoring-and-metrics.md) — Enable metrics collection and see all available metrics
* [Troubleshooting GCP Runners](https://ona.com/docs/ona/runners/gcp/troubleshooting-runners.md) — Diagnose common runner issues
* [`monitoring/` on GitHub](https://github.com/gitpod-io/terraform-google-ona-runner/tree/main/monitoring) — Browse alert definitions and dashboard source


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.