compose-backup: back up, encrypt and version Docker stacks

If you’ve ever moved house, you know the moment. The box is taped shut, and a small voice asks whether the screws for the bookshelf are in there or still in the kitchen drawer. In September I was standing in exactly that spot, except the flat was a computer. Paperless had to move off my old workstation onto a VM of its own: six containers, a Postgres database, 11 GB of documents.

With Docker, copying a folder doesn’t cut it. The data sits in volumes that Docker manages somewhere on its own. The database files belong to a user that doesn’t even exist on the new machine. And three months from now, latest will point to a different image than today. So what has to go in the box for a stack to really run again on the other side? And how does a one-off move become a backup that happens by itself every night?

My answer is a small tool I wrote for the job: compose-backup, a single Python file that needs nothing but the standard library and the Docker command.

What goes in the box

A Compose stack is a group of containers described by one compose.yaml. The containers themselves are disposable; Docker rebuilds them from their images whenever you like. The valuable part is everything around them.

First, the data, which lives in two places. Volumes are storage areas Docker manages itself, such as Paperless’s database. Bind mounts are ordinary host folders or files mapped into a container. Then there’s the stack’s .env, which holds the passwords in plain text. Without it, nothing starts.

One run of compose-backup puts all of that into a folder that describes itself. It looks like this:

paperless-<copy>/
├── compose.yaml          original
├── compose.pinned.yaml   images pinned by digest
├── .env                  secrets (chmod 600)
├── manifest.json         volumes, paths, checksums
├── SHA256SUMS            for sha256sum -c
├── volumes/<name>.tar.gz
└── binds/<path>.tar.gz

The file with the odd name, compose.pinned.yaml, is my favourite. image: postgres:18 only tells you which label was stuck on the image. A tag like 18 or latest is a pointer the publisher can move at any time. A digest is the checksum of exactly one image. The pinned copy writes down both (digest shortened here):

image: docker.io/postgres:18@sha256:7157393f508f…

Where the original says latest, the tool fills in the version label many images carry, so the line still makes sense a year from now. Variables like ${DB_PASSWORD} stay variables, which means no password ever lands in the pinned file.

That leaves the user who doesn’t exist. Postgres runs as UID 999, and copying its files from the host either loses owners and permissions or fails outright. So the packing is done by a tiny Alpine container that gets the volume mounted read-only. Owners, permissions and timestamps arrive intact.

Stop, pack, seal

Now we know what goes in the box. Packing has one trap, though. A database that keeps writing while you pack it can end up in the archive in a state that never existed. That’s why --stop pauses the stack for the duration. Forget the option with a database in the stack, and you get a warning.

The left side shows the backup, and it runs the same way every time: stop, pin the images, pack, write a SHA-256 checksum for each archive, start again. The start sits in a finally block, so it happens even if packing goes wrong. A backup that leaves the service broken would be a poor trade.

For Paperless the call looks like this:

compose-backup backup -f /opt/paperless/compose.yaml --stop \
  --exclude consume --exclude export

The inbox is an NFS share and doesn’t belong here, and Paperless rebuilds its export every night anyway.

Unpacking, carefully

I care more about the right half of the picture than the left. A restore overwrites data, and restores rarely happen on a relaxed afternoon.

Before restore touches anything, it recomputes the checksum of every archive and compares it with the manifest. If a file is missing or a sum is off, it stops right there, with not a single byte written. Next it checks whether the target volumes are empty. If one already holds data, it wants an explicit --force, and it asks only after checking all of them. There’s no such thing as a half-restored stack.

Only then does it write. --pinned brings back the pinned compose file, and --up starts the stack with exactly the images from backup day:

compose-backup verify  /srv/backups/paperless/paperless-<copy>
compose-backup restore /srv/backups/paperless/paperless-<copy> \
  --force --pinned --up

verify on its own only checks and never writes. And I don’t take any of this on faith: the tests back up a real stack, delete it, restore it and compare content and ownership, and they make sure a tampered archive or a full volume gets refused.

Every night at half past three

So much for a single stack. In my homelab, though, every service runs in its own VM, and I don’t want to back up any of them by hand. Time to zoom out a little.

The Ansible role compose_backup sets up the same things on every Docker VM: rclone, the script as /usr/local/bin/compose-backup, a small wrapper called backup-host.sh, a config file with the credentials that only root can read, and a systemd timer. It fires at 03:30 plus up to 20 random minutes, so the VMs don’t all pull on the line at once.

The wrapper walks every stack under /opt/*/compose.yaml, backs it up with --stop, and keeps the last three copies locally under /srv/backups. If a disk is more than 85 % full, it sends me a note via ntfy.

Then the backup leaves the VM. rclone sync mirrors the folder into an S3 bucket at Hetzner, through a crypt remote that encrypts contents and file names before anything leaves. That part isn’t optional, since every copy contains a .env. I also keep the key in my password manager. Without it, every backup in that bucket is just expensive noise.

The mirror becomes a history

A mirror sounds good at first. It has one awkward trait, though: it’s loyal. Three copies live locally, so the bucket holds three as well. If a VM backs up rubbish one night, rclone sync dutifully uploads the rubbish and clears out the oldest good copy to make room. Three bad nights in a row, and nothing good is left.

So since 5 September, versioning has been switched on for the bucket. When rclone overwrites or deletes something now, it only creates a new version, and the old one stays retrievable. The Hetzner console had no switch for it at the time, so I turned it on through the S3 API with rclone backend versioning.

To keep the bucket from growing forever, a lifecycle rule tidies up. It’s a clean-up rule the storage runs by itself, and mine expires old versions after 30 days. The upshot is three current copies plus everything overwritten in the last 30 days. rclone can fetch the state from a given moment and decrypt it on the way:

rclone copy --s3-version-at "2026-09-05T18:00:00" \
  crypt:paperless/<copy> /restore/

To be honest, there’s a limit. Versioning protects me from my own mirror and from mistakes. It won’t stop someone who uses the same credentials to delete old versions on purpose.

Sundays at five

The second time compose-backup gets to work is the weekly update. Another Ansible role, compose_update, pulls new images on Sundays at 05:00. If a stack has none, nothing happens at all. If it has some, the script first backs the stack up locally, starts it with the new images, waits 30 seconds, and checks that every container is running and healthy.

Then a message lands on my phone, and it contains the exact command that brings back the pre-update state. It never runs on its own. A rollback also rolls back any data added since the update, and that’s a call I want to make myself. If a stack is sick after the update, the same command shows up as “rollback recommended” in a high-priority message.

From practice: One Sunday morning at the end of September, the update messages read as if everything on eight VMs had just been rolled back. Each one ended with “Rollback: … compose-backup restore … –force –pinned –up”. The journals showed that none of it had happened; the command was only the prepared way back. Since then the message spans several lines, says first what was updated, and states plainly above the command that it was NOT executed. At five in the morning, nobody reads a message the way it was meant.

And the move?

Back to the box from the start. The September move was really one big restore. I backed up Paperless on the old machine with --stop, 11 GB in five minutes. The folder went over to the new VM with rsync, and there restore recomputed every checksum before unpacking. Afterwards the database held 767 documents. Every service that moved the same way later was another dress rehearsal.

The mistakes along the way were mine. Twice I had one slash too many at the end of an rsync path, and the contents landed one level too high. And a restore under sudo left behind files owned by root.

The tool itself didn’t get away without a scratch either.

From practice: In mid-September I imported 50 GB of photos into Immich. The following night the backup dutifully packed the entire photo library into a 56 GB archive on the 120 GB data disk. On Sunday morning the update wanted to take one more backup first, the disk filled up, Postgres crashed, and Immich answered with nothing but 502. The same night the Authentik backup failed because the tool mistook the mounted Docker socket for a folder; at least that meant its update was skipped, and the old version simply kept running. Since then compose-backup skips sockets, backed by a test, and Immich uses --exclude to back up only its database, about 250 MB. The photos are covered by the nightly VM backup, which is better built for bulk data anyway.

What I take away

The question at the start was what goes in the box. Today I’d say: more than you’d think, plus a note on what stays out on purpose. Volumes alone aren’t enough. Only with bind mounts, the .env, the exact image version and checksums does an archive become something you can actually bring back to life three months later.

What helps me sleep most, though, is the caution when unpacking: a tool that checks first and won’t overwrite without --force at least doesn’t turn a bad day into a worse one. And versioning turned a loyal but forgetful mirror into a 30-day memory.

compose-backup is only one of three backup layers. How it fits together with backing up whole VMs and the alert path, and who actually wakes me when something goes wrong at night, is in part 6: backup and monitoring.