Database failover and management reconciliation
Recover one authoritative PostgreSQL writer and reconcile interrupted management work.
Database failover restores the control plane's access to one authoritative writable PostgreSQL database. It does not transfer guest execution or recover application databases inside droplets.
Prepare the database path
Configure PostgreSQL in the deployment environment on the control-plane hosts. Inspect its reported condition at Installation settings → Health → Database. Primary promotion is performed in the database provider's console or supported administration tools, not by changing an Infra droplet label.
Use a supported PostgreSQL version and a stable connection endpoint that follows the database deployment's failover procedure. Configure TLS trust, credentials, connection limits, backup retention, and monitoring before adding API replicas.
A read replica is not a writable management endpoint. Pointing some API replicas at a writable primary and others at a read-only replica can create inconsistent request behavior even if every connection passes a network health check.
During loss of the primary
API mutations fail or become unavailable when they cannot record durable state. Workers stop admitting steps that require new coordination. Existing node actions can continue locally, which means their acknowledgement can arrive after the database path returns.
Do not resolve the incident by starting two independent primaries. The database system's fencing and promotion procedure must establish the authoritative writer. Infra's worker reconciliation cannot merge divergent management databases safely.
Promote and reconnect
- Follow the database provider's primary-loss procedure and fence the previous writer.
- Confirm the promoted endpoint accepts durable writes and required transactions.
- Verify the intended database identity and recovery point.
- Restore API and worker connectivity through their normal connection path.
- Inspect readiness, connection pressure, queued operations, and reconciliation.
- Resume scheduled maintenance only after uncertain operations are understood.
A transient reconnect does not require reenrolling nodes or generating new installation encryption material. Those changes create additional identity problems unrelated to database availability.
Lost management transactions
A promoted replica can lack recently acknowledged transactions according to the database deployment's recovery properties. Record the actual promotion point. Nodes or hosted services can still contain resources created after that point.
Inventory reconciliation must compare those components with the recovered database. Do not delete an unfamiliar guest because its create operation is missing from restored history. Preserve evidence and use the installation's recovery procedure to classify it.
Validate and test
Check organization reads, conditional updates, operation claims, secret decryption,
audit insertion, and agent observation writes. A successful SELECT 1 checks
less than the complete management path.
Run periodic controlled failover tests with representative in-flight operations. Measure recovery through authoritative operation outcomes, not only through the first successful console page response. See high availability and operation recovery.