Part 6 of 6
What if something breaks tonight?
Sunday, just after five, and an update has broken a service. How my homelab gets through a night like that, and who tells me when it can't manage on its own.
Sunday, just after five in the morning. I’m still asleep, but my homelab isn’t. Every VM is checking whether its programs have new versions, and installing them. Now imagine Immich, my photo library, gets an update and won’t come back up afterwards. Or worse, a whole VM refuses to start.
Right at the start of the series, in part 1, I asked what happens if the server dies tonight. Since then we’ve seen how requests get home safely, how a letter finds its way into the archive, how a text file becomes a VM, and who runs that code. This last part deals with the bad night itself. That old question splits into two smaller ones: what can I get back, and who tells me I need to?
Which backup for which damage?
A backup is a copy you can bring back an earlier state from. Which copy you need depends on what broke. A VM that won’t boot needs something different from a database an update has mangled. So I plan backups from the damage backwards, and that gave me three layers that overlap on purpose.
Two words from part 4 help here. My services run in Docker containers, sealed boxes for programs, and all the boxes that make up one service form a stack. The table shows the three layers, when they run and what they protect against.
| Layer | When | Where to | Helps when … |
|---|---|---|---|
| 1 · whole VM | daily 01:00 | backup server at Hetzner | a VM won’t start |
| 2 · each stack on its own | daily 03:30 | on the VM, and encrypted at Hetzner | a stack’s data is wrong or gone |
| 3 · update with a way back | Sundays 05:00 | on the VM | an update breaks a stack |
Hetzner is a hosting company where I rent two small servers and some online storage; it’s the “outside the house” of this series. In the picture my server at home sits at the top, Hetzner at the bottom, and the numbers 1 to 3 mark the three layers.
Let’s go from coarse to fine.
Layer 1: whole VMs
The coarsest layer backs up every VM in one piece, every night at 01:00. The target is a Proxmox Backup Server, PBS for short, the backup software from the makers of Proxmox. Mine runs on one of the two servers at Hetzner.
The PBS cuts a VM’s disks into blocks called chunks and stores each one only once, even if it shows up in a hundred backups. Think of a storage unit that keeps one of ten identical moving boxes and just notes everywhere it belongs. Before a chunk leaves the house, my server encrypts it with a key only it has. Hetzner sees nothing but gibberish, and the connection is encrypted on top. The job is throttled to 80 MB/s and does the small VMs first, the big ones with photos, documents and pipeline data last. In the picture this layer also mentions fleecing; I’ll come to that further down.
A backup nobody ever reads is more of a hope than a backup. So every day at 15:30 the PBS reads chunks back and compares their checksums. A checksum is a kind of fingerprint for a file; change a single bit and it no longer matches. This run is called verify, and it gets round to every backup at least once every 30 days.
Layer 2: each stack on its own
For everyday mishaps, a whole VM is often too blunt. If all you need is yesterday’s database, you don’t want to roll back the entire VM. So at 03:30, with up to 20 minutes of random delay, every VM backs up each of its stacks separately. That’s the job of compose-backup, a small tool I wrote for it.
It pauses the stack briefly and packs up its data and its description. It also writes a list of checksums for every file and records which template each box was built from. Docker calls such a template an image, and the tool doesn’t note “the latest” but the exact fingerprint of that version, the digest. The last three backups stay on the VM.
Then rclone, a copy tool for online storage, sends the backups to Hetzner and encrypts both contents and file names on the way. That’s not optional, because every backup includes the stack’s .env file from part 4, with passwords in plain text. After each run the script reports in: quietly on success, loudly on failure, and with a warning when a disk is more than 85 % full.
Layer 3: back to Sunday morning
The finest layer is the update from the opening scene. On Sundays at 05:00 every VM fetches new images. If a stack has none, nothing happens. If it has some, the script first backs the stack up on the VM, starts it on the new images, waits 30 seconds and checks that every box is running and healthy.
Then it sends a message to my phone containing exactly the command that restores the state from before the update. My messages are in German; translated, with the backup name shortened, one for Immich reads:
Updated: immich
All containers running, nothing to do.
Only if there are problems: way back to the state before the update (NOT executed):
immich: compose-backup restore /srv/backups/immich/<snapshot>/ --force --pinned --up
The command never runs by itself. A rollback that fires automatically would also wipe out data that arrived after the update, so the decision stays with me. If a stack is unhealthy after the update, the same command arrives as “rollback recommended” with high priority.
Everything ends up at Hetzner
Two of the three layers send their copies to Hetzner’s object storage. That’s storage you don’t plug in like a disk but talk to over the internet, file by file, a bit like a warehouse full of lockers. The language you use with such a warehouse is called S3, and nearly every tool speaks it, in my case the PBS, rclone and OpenTofu among others.
A locker is called a bucket, and I keep one per purpose, for example for the PBS chunks, for the stack backups, and for OpenTofu’s logbook from part 4. Hetzner bills per account, not per bucket, so one bucket per purpose costs nothing extra, and each can get its own access rights and clean-up rules.
Turning a mirror into a history
rclone keeps the bucket in step with the VM like a mirror. Whatever disappears on the VM disappears from the bucket on the next run, and a broken backup gets uploaded just as faithfully as a good one. That’s why versioning is switched on for every bucket. Overwriting or deleting a file only creates a new version; the old one stays in the locker.
So that the locker doesn’t fill up forever, a clean-up rule that the storage runs on its own lets old versions of the stack backups expire after 30 days.
That leaves the key, because encryption only moves the problem there. The rclone key lives in my password manager. A key that exists only on the server it protects is gone with that server when you need it most.
Getting things back
Now for the emergency. Which path is right depends, once again, on the damage. The picture starts at the top with the question of what’s broken and then branches into the three cases.
After a bad update I copy the command from the message and run it on the VM. If a stack’s data is wrong or gone, I pick one of the three backups on the VM. If the VM itself is gone, rclone fetches the backup from the bucket and decrypts it on the way.
Either way, compose-backup restore does the rest, and does it carefully. It checks every checksum first and only writes once all of them match. It only overwrites existing data when I explicitly ask for it (--force), and it starts exactly the images that were running at backup time (--pinned), not the newest ones.
If the whole VM won’t start, Proxmox shows the PBS backups like just another disk. I pick one, and my server fetches the chunks and decrypts them with its key.
Has this ever actually worked?
Yes, and on a large scale, because my move to the new homelab in September was really one big restore. I backed up Paperless on the old machine with compose-backup, 11 GB in five minutes, and restored it on the new VM with the checksum check. Afterwards the database held 767 documents. Every service that moved this way was a dress rehearsal. The mistakes along the way were mine, not the tool’s: twice, one slash too many at the end of a copy path.
Who wakes me up
The best backup is useless if nobody notices it’s needed. Otherwise I’d only find out about the broken Immich from the opening the next time I wanted to look at photos.
Every warning reaches me through a single channel, ntfy. It’s a plain push service that works like a doorbell: a script sends a short message to ntfy, and my phone shows it. ntfy runs at Hetzner, not on my server at home, so if that server goes down, the doorbell still works.
The next picture shows who rings it. My server with the sources of warnings is at the top, ntfy at Hetzner below it, and the red lines are messages on their way to the phone.
You’ve already met the first source, the backup and update scripts on every VM.
The second is Uptime Kuma, the watchman from part 4. Every minute it checks 22 places, from the services and both DNS servers to the backup server and ntfy itself. Often it looks for a keyword in the answer rather than just any answer at all. It retries before raising the alarm, so a single hiccup wakes nobody.
The third source watches measurements. InfluxDB is a database for time series, a kind of tachograph that stores values with timestamps. Proxmox writes every VM’s load into it every few seconds, my server adds its disks’ self-reports (SMART) every minute, and a collector called Telegraf sends readings from the server at Hetzner. Small Python scripts look at all of this every five minutes and speak up when a runner from part 5 runs low on memory, when my server can’t be reached, or when a disk starts acting up. They only report changes, once when something tips over and once when it recovers.
Where the doorbell stays silent
It isn’t airtight, and I know of two gaps. The nightly VM backup puts load on the server, and early on Kuma reported short night-time blips that cleared up by themselves. Since then there’s a maintenance window from 01:00 to 03:00 in which the affected checks show “maintenance” instead of “down”. A real outage inside that window only shows up afterwards. I know the price and I pay it.
The second gap is bigger. Kuma and the Python watchers run in VMs on my server themselves. If the whole server hangs, they hang with it, and ntfy waits for a message nobody sends. A watchdog, a kind of dead man’s switch, deals with the hang itself by restarting the server on its own. A check from outside that raises the alarm when my server goes quiet would be the next step.
Two restless nights
The nights weren’t always this quiet, though.
From practice: One morning in mid-October my docs, the proxy’s admin page and the login service were all gone. On three VMs the file system had given up, each time in the middle of that very VM’s backup, on nights when the upload to the PBS was stalling. The VM keeps running during a backup, you see. If it wants to overwrite a block that hasn’t been backed up yet, that block has to reach the backup server first. If the upload stalls, the write stalls, the VM reports a timeout, and its file system gives up. The fix is called fleecing. Old blocks go into a scratch area on my server first, and the VM keeps writing at full speed. A check on reboot repaired the three file systems. Fleecing has been on since 16 October, and the nights have been quiet ever since.
From practice: In September verify kept failing, more than 1,700 individual chunk errors in 30 days. Part of it came from outside. Hetzner reported an object storage incident, and its front door, the gateway, answered with errors (503 and 504). The other part I had built myself. The backup job had grown and ran into the afternoon, verify started at 13:00 regardless, and both hammered the same bucket at once. Verify now runs at 15:30 and reads less at a time, and the next run had exactly one error left.
If you’ve read part 5, the pattern will look familiar. There, two jobs shared a runner; here, two jobs shared a bucket, and neither knew about the other.
Back on the train
That brings us to the end of the series, and I’ll take one more ride on the train from part 1. I type paperless.home.example into my phone, and a second later the letter is there. By now you know what happens in that second. NetBird and a single proxy bring the request home without an open port (part 2). The letter reached the archive through the scanner, Paperless and a local language model (part 3). The VM it lives on is a file in the repo (part 4) that a pipeline applied on my own runners (part 5).
And if my server dies tonight? Then the letter is sitting encrypted at Hetzner, twice in fact: once as part of the whole VM and once in the Paperless stack’s backup. In most cases my phone will tell me. Only if the whole server hangs without a sound does nobody outside notice yet, and that’s the next thing on the list.
If you look at the big picture in part 1 again now, you’ll probably read it with different eyes. Which of the four parts would you build first in your own homelab?