ntfy: who's knocking at five in the morning?

Sunday, just after five. The phone on the bedside table lights up and buzzes once. I reach for it, still half asleep, and read “Update”, followed by a list of services. Before the words really sink in, two questions go through my head, the same ones anybody asks when a strange text message turns up. Who is talking to me? And why should I believe it?

This week a third question joined them, and it’s a less comfortable one. What about the messages that never arrive at all?

This post is about all three. My homelab, the small collection of servers at home that this series is about, sends me every important message through the same channel: a little service called ntfy (say “notify”). In part 6 it got a single paragraph. It deserves more.

A letterbox with a doorbell

Start with the phone. A push notification is a message a server sends to your phone on its own initiative, so the app doesn’t have to keep asking every few minutes. For the big apps that almost always goes through Google (on Android) or Apple (on the iPhone). The app makers hand their message over, and Google or Apple deliver it.

ntfy is an open-source program that does this job itself. You put it on a server of your own, and it really only does two things: it accepts messages, and it passes them on to everyone who is listening. In between sits a topic, a named channel. Picture a letterbox with a name plate and a doorbell attached. Whoever drops something in rings the phone of everyone subscribed to that box.

Why not just use ntfy.sh, the public instance the developer runs? There, knowing a topic’s name is basically enough to read along. On my own server I decide who may write and who may read, and the messages stay with me. The Android app keeps its own connection open to my server for this; I haven’t set up any relay through ntfy.sh or Google.

Any script can post

What won me over is how you drop a message in. ntfy speaks HTTP, the same language your browser uses to fetch web pages. So any script that can call a web address can also send an alert. On Linux, curl is all it takes, a small command-line program made for exactly this kind of request:

curl -H "Authorization: Bearer <token>" \
     -H "Title: Test" -d "Hello from the server" \
     https://ntfy.example/<topic>

The first line says who is sending. A token is a long random password meant for programs. Then come the title and the text, and finally the topic’s address. That’s it: no library, no account with anyone. Keep an eye on the quotation marks in that command, by the way. They come back at the end.

Why it doesn’t live at home

That’s the idea. The next question is where the letterbox stands. Deliberately, not at my place.

At home, nearly everything runs on a single machine I call pve1. On it run lots of virtual machines (VMs), computers that exist only as software, one per service. If pve1 goes down, every VM goes with it. An alert service on pve1 would be like a smoke detector that has to phone the fire brigade from the burning room.

So ntfy runs on a small rented server at Hetzner, the same one my VM backups go to. In front of it sits a reverse proxy, a gatekeeper that takes every request from the internet, handles the encryption (HTTPS) and hands each request to the right program behind it. ntfy itself only accepts requests that come through that gatekeeper.

The next picture shows the whole journey of one message, from home to the phone.

The senders at the top are the ones we’ll meet in a moment. They all take the same route. ntfy also keeps every message for a while, so if the phone was out of signal, the app catches up.

To be fair, the location has its limits. The letterbox survives an outage at home, but most of the senders don’t. When pve1 hangs, they go quiet with it. That gap is still open; part 6 has more on it.

Who may write and who may only read

Now back to the bedroom question: why do I believe the message?

Because on my server nobody may do anything I haven’t explicitly allowed. The configuration’s default rule is “deny”: no permission, no reading and no writing. On top of that there are rules for exactly three users, and the picture shows who may do what.

The sender user is the one for the machines. Its token sits in every sender: on each VM in a file only the VM’s administrator can read, plus in Uptime Kuma and in the Paperless chain. Strictly speaking it may read as well, which it never needs.

The reader user belongs to the app on my phone. It can subscribe to my topics and nothing else. If I lost the phone, whoever found it could read along, but couldn’t send fake alarms. The third user, an admin, is for me when I change users or rules.

So the answer to “why do I believe it?” is fairly dry. Only someone holding the sender user’s token can write to the topic, and the whole way there is encrypted. That only tells me who wrote, though. Whether the script is right about what it says is another matter.

Five senders, five tones of voice

So who actually writes to me? Five senders, each with its own tone.

Uptime Kuma is a watchman that knocks on 22 doors every minute: services, DNS servers, the network storage and ntfy itself. If something doesn’t answer, it tries twice more before raising the alarm. Between one and three at night it stays quiet about most services, because the VM backup is keeping the machine busy and short hiccups are normal then.

The watches on InfluxDB are small scripts. InfluxDB is a database for measurements over time, such as free memory or temperatures. Every five minutes the watches look at it and only speak up when something tips over and again when it recovers: when one of the machines that run my software tests is short on memory, when pve1 stops answering, when a disk gets too warm.

The backup script backs up all services on a VM every night at half past three (how, is in the post about compose-backup). The update script fetches new versions on Sundays at five. That was my message from the beginning. And the Paperless chain, which reads scanned letters and files them, reports every finished letter with its title, sender, document type and a link.

So that the phone isn’t equally loud about everything, each sender adds a priority from 1 to 5. The app turns that into behaviour: 1 and 2 arrive quietly, 3 is normal, 4 and 5 get increasingly insistent. On top of that comes a tag, which the app shows as a small picture in front of the title, a floppy disk for the backup, for instance. On the lock screen I know who’s talking before I’ve read a word.

The Paperless chain was too chatty at first. I had Paperless itself report whenever a document changed. Trouble is, the chain changes every document four times, so each letter produced four messages, the first ones still without a sender or type. Now only the chain’s last step sends a single message, once everything is done.

What can go wrong

An alert channel has one nasty trait: when it breaks, it doesn’t tell you. I’ve seen that twice so far, once loudly and once very quietly.

From practice: In mid-September the app suddenly claimed my reader user had no permission. The user was fine, and so were the rules. The fault sat one level earlier, at the gatekeeper. The proxy on the Hetzner server forwarded requests to a host name rather than a fixed address. Sometimes it resolved that name to an IPv6 address it couldn’t reach from where it sat, sometimes to nothing at all. Then it answered with a 502, “the one behind me isn’t answering”, and the app turned the failed login into a missing permission. Since then all seven of the proxy’s entries point to a fixed internal address.

The second story is very fresh, from last night.

From practice: Overnight into 30 October, a review of the scripts turned up a single line. In the backup script, the quotation marks in the curl call, exactly the ones from the example above, had backslashes in front of them: \"…\" instead of "…". In a shell script, quotation marks hold together what belongs together, such as “Title: Backup ok” including its space. A backslash in front turns the quotation mark into an ordinary character. So curl received a pile of chopped-up pieces with quote marks stuck to them, including an address that wasn’t one, and failed. After it stood || true: if this goes wrong, pretend nothing happened. The idea was that a lost message should never make the backup itself fail. What it actually did was make sure that not one backup message had arrived since early September. No “backup ok”, no failure, and no warning about a full disk, even though that very warning had been added in September after a disk filled up. The update script, written separately, reported reliably every Sunday. The phone buzzed, the channel looked alive, and nobody missed the quiet messages from half past three. The fix went in the same night. In a test the old call got no answer at all and the new one a clean OK from the server, and then the script went out to all eleven VMs. Tellingly, that same night two system disks stood at 99 and 100 percent, exactly what the swallowed warning was there to warn about.

What annoys me most is that I’d done it right with the watches on InfluxDB. There I set the threshold to 100 % as a test and waited until the alarm actually showed up on my phone. With the backup script I skipped that step. A silent || true in an alert path swallows exactly the failure you want to hear about, which is why you have to test the alert path itself, and not just the thing it’s supposed to report.

Takeaways

Back to Sunday morning. Who is talking to me? One of five senders, and from the title and the little picture in front of it I can tell which, even half asleep. Why do I believe it? Because only the sender user’s token can write to my topics, and the whole way is encrypted. That was the easy part.

The third question will stay with me longer. Seven weeks of silence from the backup script felt like seven weeks of “all good”. I read silence differently now. It isn’t a message, just the absence of one. For every new sender I now want to hear a real test message buzz on the phone before I call it done.

How ntfy fits together with the three backup layers, Uptime Kuma and InfluxDB is told in part 6: Backup & monitoring. And if you’re wondering what actually gets backed up at half past three (and, since the fix, reported again), that’s the story of compose-backup.