Control-plane health

Monitor management readiness, dependency health, backlog, and observation freshness.

Control-plane health reports whether management services can authorize requests, record durable state, execute operations, and receive observations. It does not replace application health checks.

Health surfacesCopy link

Open Installation settings → Health for dependency conditions and Operations → Queue for pending work. The probe endpoints use the control-plane origin; restrict their detailed responses to the management path.

Use /health/live for process liveness and /health/ready for whether the service can accept normal traffic through its required dependencies. Restart policy and load-balancer admission therefore use different checks.

The installation health view explains named dependency conditions. Metrics quantify backlog, errors, storage pressure, and observation age. A green process check does not establish that a worker can commit a new operation or decrypt an integration credential.

Keep detailed dependency responses and metrics on a restricted management path. Public liveness should not expose database addresses, credential state, or internal topology.

Monitor dependenciesCopy link

Track database availability, connection usage, replication health, free disk, key availability, observation storage writes, object destination access, and worker capacity. A service can be live while unable to commit an API mutation.

For HA deployments, monitor shared dependencies independently. Multiple healthy API processes do not compensate for an unavailable database or inaccessible active encryption key.

Operation healthCopy link

Observe queued operation count, oldest queued age, reconciling count, worker assignment, and failure rates by operation type. A long migration and a stuck create operation require different thresholds.

Investigate whether the backlog comes from insufficient workers, an unavailable region, database latency, quota cleanup, or a component that stopped reporting. Adding workers cannot repair an unreachable node or a blocked storage lease.

Observation freshnessCopy link

Monitor the age of the latest node, storage, and network observations. A flat chart with fresh samples differs from a chart displaying the last value before a disconnected source. Use explicit no-data rules rather than interpreting missing data as idle infrastructure.

Journal ingestion can lag behind operation state. Compare event time and receive time when investigating delayed records.

External checksCopy link

Run an authenticated API read and a controlled resource-operation check from an independent monitoring system. Keep its credential limited to the test scope. Use a separate notification path so an Infra alerting outage does not hide the control-plane failure.

Status monitoring can check the console origin and API health endpoint. Restrict access to any detailed health path through the monitoring network or authenticated proxy.

Incident responseCopy link

Open Installation settings → Health and use the failed dependency to select its recovery procedure:

• Database writes unavailable: inspect the database connection error and restore access to the intended database. If its primary has failed, follow database failover. Resume mutations only after writes succeed and readiness recovers.
• Node disconnected: follow node connectivity checks. Keep the original operation IDs while connectivity is repaired so a retry does not create duplicate work.
• Writer ownership uncertain: complete node fencing before starting a replacement guest from shared storage.

Capture the affected time range, release versions, dependency conditions, request IDs, and operation IDs. Use Troubleshooting for resource-specific checks and backup recovery when management state must be restored.

Probe endpointsCopy link