Running NetBird yourself: the matchmaker that steps aside

Every good party has that one person who knows everyone. They take you by the elbow, say “This is Jana, she’s building a garden shed too”, and two minutes later you’re deep in a conversation about foundations while they’ve long moved on to the next group. You don’t need them for the conversation. You’d just never have found each other without them.

I run exactly that kind of person, as a small server at Hetzner. I realised it in September. I started an upgrade, and for about fifteen seconds everything on the server stood still: no management, no dashboard, no login. Nobody in the homelab noticed. The VMs kept talking to each other, and my phone kept its way home.

In part 2 I introduced NetBird as the network over which all my devices talk to each other, encrypted. This piece is about the part you normally don’t see: the matchmaker itself, and why it doesn’t mind being gone for a moment.

A server that only makes introductions

NetBird separates two things that are easy to lump together. The control plane knows who belongs and who may talk to whom. The data itself travels through WireGuard tunnels directly between the devices, which NetBird calls peers. My server only does the first job. Every peer creates its own key and registers just the public half, so the server can’t look into the tunnels.

That doesn’t make it harmless. It holds the list of every device, group and rule, plus the setup keys and the login data. NetBird is available as a cloud service, but the server is open source, and I wanted that list on a machine I manage myself, with no account at a provider. The flip side: updates, backups and debugging are on me.

What runs on the little server

The machine is a small Debian server at Hetzner, and everything on it is one Docker Compose stack under /srv/netbird. Up front sits Traefik, a reverse proxy that fetches certificates from Let’s Encrypt and routes each request on port 443 by its path. The real work happens in one container, netbird-server, which in recent versions bundles everything that used to run separately.

Next to them run NetBird’s own reverse proxy, which can publish internal services to the internet, and a few small nginx containers of mine that I’ll get to with the traps. All the state lives in SQLite files in a Docker volume: peers and rules, the logins of the built-in identity provider, and an event log.

This server is the only part of my network with open ports, 443 for almost everything plus two UDP ports. Even the relay runs over HTTPS, under /relay. So much for the building. Now let’s watch what happens inside when two peers meet for the first time.

How two peers find each other

Take my phone, which wants to reach proxy-01. Both sit behind a router doing NAT, network address translation, which turns away connections from outside unless someone inside started them. Just like at the party, someone has to make the introduction.

First, each peer collects where it might be reachable. Its address in its own network is the host candidate. Its public address, which it learns from the STUN service, is the srflx candidate. And the relay is the last resort. The two swap these lists through signal, encrypted with their own keys.

Then comes the trying. ICE, a technique borrowed from video calls, has both sides send test packets to every candidate at the same time. Because both send from the inside, each router opens a small hole for the reply, and the best pair that gets through wins. If nothing works, the relay steps in. WireGuard has already encrypted the packets by then, so all the relay sees is noise.

Once the tunnel is up, it no longer needs the server. That explains the quiet upgrade in September. During the pause, only things that need fresh negotiation have to wait: a new peer, a changed rule, a phone switching from Wi-Fi to mobile data. netbird status -d shows for each peer which path won. Between my VMs it says P2P with host/host, straight through the LAN. How often the relay jumps in when I’m out and about, I honestly have never measured.

Keeping order: roles, keys, DNS

A host has to know who goes with whom. Every peer in my network has exactly one role and exactly one DNS group, and those two axes keep the rule set small. The roles are mobile, lan-servers, lan-infra and remote-servers. On top of that there’s a group called lan-home with no peers at all. It only carries a resource, meaning a network whose devices don’t run NetBird themselves, in this case the home LAN.

Who may talk to whom is settled by four policies:

Each role has its own setup key, reusable and valid for a year. It drops a new peer into its role and its DNS group automatically, without me clicking anything in the dashboard. A new VM joins during its setup with netbird up --management-url https://netbird.example --setup-key …. I forgot the management URL at first. The client then registers with the cloud service, which has never seen my key, and replies with a curt setup key is invalid.

For DNS, dns-adguard is the normal case: AdGuard on dns-01, backed by the replica on the Raspberry Pi. The Pi itself asks Quad9, because an AdGuard host can’t use itself as its nameserver. The backup server at Hetzner gets no rule at all, since it can’t reach AdGuard. NetBird only writes its own resolver, with no fallback; how I learned that the hard way with a rule for the group All is in part 2.

The gateway

One peer has a special job. A small VM on pve1 gives my phone access to the home LAN and acts as exit node, which means that when I want, all my traffic on the road goes out to the internet via home. NetBird sets up forwarding and NAT by itself. If the gateway goes down, the Pi takes over. The same route lists both as routers with a metric where the lower number wins: 100 for the gateway, 200 for the Pi.

The exit route applies to mobile only, and it stays that way. Give a server with a public address a route like that and its default route moves into the tunnel, so it goes dark from the outside. Only the hoster’s console helps then, which is why I always keep a way in that doesn’t depend on NetBird.

Back to the upgrade

Which brings us back to September. The server was still on version 0.72.2, while 0.78 was current. Six version jumps at once, and the SQLite database holds every peer, every key and every rule. If a database migration fails when the new version starts, worst case I get to re-enrol every single device. I didn’t like the sound of that, so one script now does backup and upgrade, in that order.

The script first copies compose-backup to the server, the same small tool that backs up every Docker VM in part 6. It stops the stack for about 15 seconds, saves volumes and configuration along with checksums, and starts it again. Those were the 15 seconds nobody noticed. The backup is then verified, mirrored to my workstation, checked against the checksums there and taken along by the nightly S3 run.

Then comes the upgrade. The running images first get a pre-upgrade-<date> tag, because after the pull :latest points at the new version and the old one would otherwise only exist in the registry. After the start, the script searches the log for errors and failed migrations, calls the dashboard and the login over HTTP, and asks the API how many peers are connected.

I read the version from the log, from the line management server version. With :latest, the label on the image tells you nothing. On 7 September it said 0.78.1, with the dashboard at v2.92.0. I didn’t need the way back, but it’s ready:

compose-backup restore backups/<version> --force --pinned --up

Traps in disguise

That leaves the hours I spent debugging. Each of these traps dressed up as something else.

NetBird SSH and port 22

NetBird has its own SSH feature, where you log in with your NetBird account instead of a key. When it’s on, the peer redirects port 22 of its NetBird address via NAT to NetBird’s own SSH server. A perfectly ordinary ssh lands there, and the host’s real SSH daemon never even sees the attempt. Since September it’s switched off on every peer through the API.

Peer names exist twice

NetBird gives each peer a name like pve1.peers.netbird.example, and only the NetBird resolver knows it. My public zone, however, has a wildcard record. Look up the same name through the router, AdGuard or Quad9 and you end up at the NetBird server at Hetzner, which asks for an SSH password that will never fit. So I resolve peer names only through NetBird, or use the NetBird address directly.

From practice: One day I couldn’t SSH from my laptop into my workstation any more. “Permission denied”, even though key and password were right. I spent a long time digging on the target until I noticed its SSH log showed no attempt at all. I had been talking to someone else the whole time. These days the first thing I compare in a case like that is the host key fingerprint.

A legal notice in the dashboard

Being publicly reachable, the server needs an imprint under German law, a page naming who runs it. Patching the dashboard image would be undone by the next upgrade. So an nginx container of its own serves the page, and Traefik sends /impressum there with a higher priority than the catch-all route to the dashboard. The link inside the dashboard comes from a thin proxy in front of it that adds a small script before </body>. It took three attempts: injected HTML gets swept away by the React app, an inline script is blocked by the dashboard’s Content Security Policy, an external script file from the same server gets through.

From practice: The link showed up on the login page, and every check was green. In the dashboard after login it was missing anyway, unnoticed for ten days. The culprit was my Traefik rule PathPrefix(/impressum), a plain string match with no path boundary that also caught /impressum-link.js. Instead of the script, the browser got the legal notice as HTML and silently dropped it. I only spotted it when I looked at the actual response for that one file. The rule now reads Path(/impressum) || PathPrefix(/impressum/).

What I take away

What helped me most was thinking of control and data as two separate things; it explains nearly everything NetBird does, the quiet upgrade included. Beyond that: one setup key per role, a backup before every upgrade, and when something looks like a login failure, first check that I’m reaching the right service at all.

So the host’s job is done as soon as everyone has been introduced. What the guests get up to afterwards, how the phone travels through AdGuard, the proxy and a wildcard certificate all the way to Paperless, is told in part 2: Network & access.