Recover ambiguous operations
Resolve interrupted execution without submitting competing infrastructure changes.
Operation recovery establishes the authoritative result of work whose execution or acknowledgement was interrupted. It preserves the original operation and resource ownership records.
Identify the uncertainty
Open Operations → Queue for active work or Operations → History for a terminal result. Find the operation by ID or resource and select its Recovery tab. The details identify its blocker, last authoritative phase, and available resume or override action.
An operation enters reconciling when Infra cannot safely classify the last
step as applied or unapplied. Common cases include an agent disconnect after a
hypervisor action, a worker restart during migration commit, and a lost response
from a hosted service.
Read the operation's last confirmed phase, target IDs, ownership generation, and component observations. A client timeout alone does not imply a reconciling state; the server can still be running or already finished.
Restore the observation path
Use node connectivity checks for agent identity and network failures, or database failover for an unavailable management database. Restore the component that owns the affected state. When that component reconnects, the worker compares its result with the recorded intent before continuing.
Do not clear the resource lock by deleting operation rows or reenrolling the node under a new identity. That removes evidence used to prevent duplicate execution and can leave an active guest or volume writer unmanaged.
Reconcile common cases
For a droplet create, inspect whether the allocated ID exists at the hypervisor and whether its disks and interfaces match the operation. A guest that exists under that ID is adopted into the original result rather than created again.
For migration commit, Infra checks execution ownership and volume writer state. It cannot declare the source stopped solely because the source agent disappeared. Complete fencing if the previous writer cannot otherwise be excluded.
For external hosted actions, reconnect the Connector and read the hosted operation reference. A locally pending request can correspond to a completed hosted resource that continues incurring usage.
Retry versus resume
After correcting the blocker, return to the same operation's Recovery tab. If it offers Resume, use that control to continue the original execution record. If it instead requires a new action, follow the recovery instruction shown for the failed phase. A new action creates a separate request against the resource's current state; it is not a replacement for an unresolved resume.
When an API response was lost before the client learned its operation ID, replay the identical request with its original idempotency key. If the key retention has expired, search by resource and request identifiers before creating again.
Manual recovery record
A Recovery Override action requires a reason, evidence, and the expected resource generation. If the generation changed since review, the action is rejected. The record cannot be reused to authorize a later unrelated recovery.
After recovery, open the operation result and check its resource and cleanup IDs. For each retained resource, inspect its health and whether the result calls for reuse or deletion. Confirm the corresponding quota is released only after the cleanup operation succeeds. A terminal recovery result can still have unfinished cleanup.
Use Operation API for polling and cancellation. If the management database itself was restored, follow control-plane recovery before resuming normal automation.