Setup health and observability
Check configuration requirements for detection, response, and delivery.
Open Setup health for the operator-facing readiness checklist. It follows an alert through detection, response, delivery, and public communication and turns configuration gaps into links to the page where they can be fixed.
The checks
| Check | Severity | What it finds |
|---|---|---|
| Monitors without alert coverage | Critical | Enabled monitors not routed to any enabled notification channel. |
| Stale agents | Critical | Active monitoring agents that have not checked in for two minutes. |
| Failed notification channels | Critical | Enabled channels whose latest persisted delivery failed or is retrying. |
| Unowned monitors | Warning | Monitors whose owning team is no longer assigned. |
| Status components without monitors | Warning | Public-facing components with no monitor supplying health. |
| Overdue follow-ups | Warning | Open post-incident work whose due date has passed. |
| Missing public-page subscriptions | Warning | Public status pages with no confirmed email or webhook subscribers. |
Critical findings mean an alert raised now will not reach anyone; warnings mean the alert may arrive late or go to the wrong people. Re-run checks reloads the checklist from current state. A 100% checklist is configuration evidence, not a synthetic end-to-end probe, so still test channels and rehearse recovery.
Request correlation and logs
Every request receives X-Request-ID and a W3C traceparent. Use the request
ID to join proxy, web, migration, runner, and worker records. Worker logs also
report the durable outbound result-queue depth so operators can distinguish
target failures from a disconnected application.
The delivery runner
Outbound delivery—alert channels, subscriber notices, scheduled reports, and notification emails—is drained by a separate runner managed by the deployment administrator. The runner must execute every minute.
One pass evaluates state and time signals, drains organization and personal deliveries, sends subscriber notices and reports, and updates the durable runner heartbeat. A failed delivery is retried with backoff up to eight attempts; a stopped runner is reported by the readiness endpoint below.
Health endpoints
Health endpoints answer different questions:
GET /api/health/livereports only that the web process can answer.GET /api/health/readinessprobes the database, compares applied migrations with the shipped set, and independently checks the durable notification runner heartbeat. It returns 503 unless all required checks are healthy. SetSTATUS_BACKGROUND_RUNNER_REQUIRED=falseonly when the external runner is intentionally not part of a deployment.
The authenticated Prometheus endpoint adds status_queue_items for alert,
subscriber, and workbench queues by state, plus
status_worker_result_queue_items for each agent's durable local delivery
queue. Existing monitor and worker-liveness series remain unchanged.