Backend readiness and connection draining

Backend readiness and connection draining

Remove application instances from service without confusing probe state with session completion.

Connection draining stops new backend selection while allowing existing connections to finish within a configured timeout. Readiness determines whether a backend can receive new traffic; draining controls its removal.

Prepare the application

Expose a readiness check that fails when the instance should stop receiving new requests. Keep liveness separate so a deliberately draining instance is not restarted before it finishes active work.

For HTTP services, distinguish a completed response from a long-running stream. For TCP services, the balancer observes connections rather than application transactions. It cannot infer whether a database transaction committed safely.

Start draining

Open Load balancer → Backends, select a member, and choose Drain. Set drain_timeout_seconds, an integer from 0 through 3600, inclusive. The default is 300. Zero stops selection and closes remaining connections after the configuration applies; it does not provide a grace interval.

The member enters draining after the requested revision is applied. Existing flows can continue until they complete or reach the timeout. Pending configuration is not evidence that new selection already stopped on every service instance.

Coordinate guest maintenance

  1. Start backend drain and wait for applied membership state.
  2. Stop new application jobs and observe active requests or connections.
  3. Wait for completion or the approved timeout.
  4. Remove the member or stop the application.
  5. Perform the guest or node maintenance.
  6. Verify readiness before registering it for new traffic again.

Do not shut down the guest immediately after submitting drain. That removes the backend before the intended grace period can do useful work.

Unhealthy and empty groups

Health failure removes a backend from new selection after the configured unhealthy threshold. It does not necessarily close every existing connection immediately. An administrator can use explicit removal when the service must stop using a compromised member.

If no backend is eligible, HTTP listeners return service unavailable and TCP listeners cannot establish a usable backend connection. The balancer does not ignore the health result merely because the group is empty.

Cancel a drain

A member can return to active selection only after the new membership revision applies and its healthy threshold is met. Canceling a drain cannot reconstruct connections already closed at timeout. Client retry behavior remains an application concern.

The Load balancer API exposes membership state and active connection counts. Use maintenance approvals when the removal affects a protected production service.