Alerts and routing
Evaluate resource conditions and deliver operational notifications.
Alert rules evaluate metric thresholds, health conditions, and missing data. Routes send matching alert events to configured notification destinations.
Create a rule
Open Observe → Alerts → Create rule. Select resources by IDs or labels, choose the condition, set its evaluation window, and assign severity and ownership. Preview the selected resources before enabling the rule.
Choose a threshold, named health condition, or no-data condition. Thresholds use an explicit metric unit and aggregation. Pending duration defaults to 5 minutes and must be a non-negative integer duration; severity defaults to warning.
Missing data defaults to unknown. It is not a zero or an automatic recovery. A dedicated no-data condition reports stalled collection, while a threshold rule reports an observed measurement. Select a route or inherit the project's default, then test the destination before enabling the rule.
The Alert rule API separates rule configuration, evaluated alert instances, and delivery attempts so automation can inspect each outcome.
A negative pending duration or a threshold with an incompatible unit is rejected. Use a dedicated no-data rule to report unreachable agents and stalled collection.
Alert lifecycle
A matching condition enters Pending and then Firing after the configured duration. It returns to Resolved when a valid evaluation no longer meets the condition. Missing samples leave the evaluation Unknown under the default policy; they do not prove recovery.
Acknowledgement records operator ownership and can suppress repeated reminders. It does not clear the underlying metric, change resource health, or stop the rule from evaluating.
Routes and delivery
Routes match project, labels, severity, and condition type. A destination can send email or an authenticated webhook to an approved endpoint. Test each route with a synthetic event before relying on it for production response.
Delivery attempts and destination failures are visible separately from alert state. A firing alert can have failed delivery. Use an independent path to monitor notification health when Infra is responsible for critical infrastructure.
Grouping and noise control
Group related resource alerts by project, region, or parent condition. A node failure can cause guest-agent and network symptoms for many droplets. Inspect the parent infrastructure condition before restarting individual guests.
Maintenance windows can suppress notifications for the reviewed scope. Evaluation continues and the alert remains visible. When the window ends, conditions still firing are eligible for notification under the route policy.
Suggested operational coverage
Create rules for node reachability, storage redundancy, pool free space, failed backup runs, recovery-point age, exhausted project quota, and control-plane operation backlog. Add application checks through your service monitoring system.
For Status integration, send selected events to an external-alert endpoint using an authenticated destination. See Status external alerts and webhook delivery for receiver and retry behavior.
Changes and deletion
Changing a selector can add or remove evaluated resources. The rule revision records the selection change. Deleting a rule stops future evaluation but retains its alert history under configured retention. Review active alerts before removing the rule so unresolved infrastructure conditions remain owned.
Slack and PagerDuty destinations document correlation, provider acknowledgement, and failed delivery. Anomaly detection uses learned history rather than an explicit fixed threshold.