Backup verification and restore tests

Backup verification and restore tests

Distinguish object checks, disk restoration, and application recovery validation.

Backup verification records whether a recovery point can be read and restored. Verification levels test different properties and remain visible independently of the original backup run result.

Verify the stored copy

Open Storage → Backups → Recovery points, select a completed backup, and choose Verify → Objects and manifest. Infra reads the manifest, checks required objects, and validates their stored checksums. The operation records the destination, manifest revision, and verification time.

A successful object check establishes that the selected stored copy is readable through that path. It does not boot a guest, run database recovery, or prove that an encryption key will remain available during a later incident.

Restore-test policy

In Storage → Backups → Verification profiles, create a restore-test profile with an isolated project, allowed region, storage policy, and resource limits. Select that profile under the recovery point's Verify → Restore test action. The test account needs enough quota to create temporary volumes and guests. Destination residency policy applies to test data as it does to production data.

A test creates new volumes and can boot a recovery guest when the verification profile requires it. The profile identifies the image, required member volumes, and checks to run. It must disable production traffic and external side effects.

Use separate credentials for a test database. A restored job scheduler or mail service can send real work if it receives production secrets and outbound routing.

Application checks

Define checks around application data rather than process startup alone. A database can start while reporting a missing relation or an unrecoverable segment. A web server can return 200 from a static page without opening its database.

Record the expected dataset or application invariant with the test profile. Keep its version in the result so a later change to the check does not alter what an earlier verification established.

Results and cleanup

A result records the backup capture time, verification level, restored member IDs, checks, duration, and cleanup status. Verified applies to that backup and profile revision; it is not inherited by the next scheduled backup.

A failed test leaves the original backup unchanged. Temporary resources are removed according to the test's cleanup policy. Cleanup failure retains their IDs and continues consuming quota until an operator resolves it.

Do not delete a failed test environment automatically if an approved incident procedure needs its disks for diagnosis. Restrict access and set an explicit retention deadline instead.

Recovery objectives

Measure recoverable age from capture time, not verification time. A backup verified today can still contain data from last week. Measure restoration through application readiness and traffic promotion when validating a recovery time objective.

Use alerts for unverified backup age and failed verification runs, Backup API for result inspection, and disaster recovery for production promotion.