Backups that can be restored, not backups that exist

01 — Context

The system had a manual dump and the conviction of being backed up. No automatic job, no verification, and not one restore ever attempted.

A file that was never restored is not a backup. It is a hope with a timestamp. Most backup incidents are not disks failing: they are restores that were assumed to work and then did not. This case is about turning that assumption into a tested procedure without stopping the service it protects.

02 — Constraints

One shared vCPU, 3.9 GB of RAM, a disk already at 67%, a database with PostgreSQL wire compatibility, and no maintenance window: the backup had to live alongside the workload, not pause it.

03 — Approach

One artefact, three parts, verified before it counts

1. A single archive. A compressed tarball holding the database dump, the deployment configuration that does not live in version control (compose file, environment, reverse proxy, monitoring, certificates) and a manifest with versions, counts and a SHA-256 per part. Stored outside the repository, mode 600.
2. Dump from an auxiliary client. The database container ships an interactive client but not the dump tool, and adding binaries to a production image is more risk than value. A throwaway client container talks to the local port instead.
3. Three gates before the file is accepted. gzip integrity, tar listing, and the checksum compared against its control file. A backup that cannot be read is the same as no backup.
4. Retention with a number. Fourteen snapshots, newest first, oldest deleted. Retention without a stated number becomes a disk-full incident.
5. Automation that stays out of the way. A systemd timer runs daily at 03:40 UTC with low CPU priority and idle I/O class, so it never competes with the service.

04 — Measured

The dump compresses to 10.6 KB and the deployment configuration to 25 KB. The restore was proven by loading the archive into a throwaway database: fourteen tables, identical row counts in users, requests, responses and migration history, zero errors. The test database is then dropped, so the next proof also starts clean.

05 — The part that mattered

The first restore failed

The archive passed its integrity checks, contained every table, and still would not load. The dump carried an internal object of the database engine that already exists at cluster level, so restoring inside the same cluster aborted with already exists.

Restoring is now part of the definition of done, not an optional extra.

06 — Operation

systemctl list-timers <backup>.timer     # when it runs next
journalctl -u <backup>.service           # what happened last time
backup.sh --keep 30 --dir /mnt/external  # ad-hoc run, other retention, other target

# restoring, on purpose, before it is needed
tar xzf backup.tar.gz ./db/dump.sql.gz -O | gunzip > /tmp/restore.sql
createdb restore_test && psql -d restore_test -v ON_ERROR_STOP=1 -f /tmp/restore.sql

07 — What is still missing

The copy lives on the same disk as the database. That protects against human error, accidental deletion and a failed migration — not against losing the host. The next step is replicating the archive off-host with encryption at rest, and that limitation is written down rather than glossed over.

08 — What it demonstrates

• Backups verified by restore, instead of backups declared in a runbook.
• Scheduling and resource control with systemd on a single-CPU host.
• Diagnosing a real restore failure instead of trusting a green check.
• Retention, checksums and manifests as part of the design, not afterthoughts.
• The habit of not accepting anything as working until it has been exercised.