A green padlock for services nobody can see

You’ve seen this page. You type in the address of your router, or of a little server in the basement, something like 192.168.x.x, and instead of a login you get a full-screen warning: “Your connection is not private.” You click “Advanced”, then “Proceed to 192.168.x.x (unsafe)”, and everything works. Next time you click faster. At some point you click without looking.

That’s the real problem. The warning exists for the one day when somebody actually is sitting between you and the device. If you dismiss it every day, you won’t read it on that day either.

When I built my homelab, the small collection of servers at home this series is about, I wanted to break that habit. Every service was supposed to get a real green padlock in the browser. The catch: none of these services can be reached from the internet, and that’s normally exactly what a padlock like that requires. Here’s how it works anyway, from the very beginning.

Names and house numbers

Every device on a network has an address made of numbers, its IP address. Think of it as a house number: that’s where data packets get delivered. Numbers starting with 192.168 only mean something inside your home network, a bit like a hotel room number that tells nobody out on the street anything.

Since nobody wants to remember numbers, we have names. paperless.home.example is one of them, a domain name. The last part belongs to a domain I rent, and in front of it I can make up as many names as I like. (home.example is a placeholder on this blog; my real domain is different.)

Turning names into numbers is the job of DNS, the Domain Name System. Picture a phone book: you ask for a name, a DNS server looks it up and gives you the number. What matters later is that there’s more than one phone book. The public one belongs to the internet, and the entries for my domain in it are kept by Hetzner, the hosting company where I manage it. On top of that, any network can keep a small phone book of its own that only applies inside.

An ID card for the server

When your browser loads a page over HTTPS, two things happen. The connection gets encrypted, so nobody along the way can read it. And the server has to identify itself, so you can be sure you’re really talking to paperless.home.example and not to somebody who slipped in between.

That ID is called a certificate. It says which names it covers, until when, and who signed it. Signing is done by a certificate authority, and your browser ships with a list of authorities it trusts. A server that signed its own certificate is like an ID card someone made at the kitchen table. The details may be correct, but nobody can check them.

That explains the warning. Your router shows either such a home-made ID, or one with a different name on it than the one you typed. And no public authority will issue an ID for a bare house number like 192.168.x.x, because that same number exists in millions of homes. A domain name, on the other hand, belongs to exactly one owner.

My certificate authority is Let’s Encrypt. It issues certificates for free and fully automatically, each valid for 90 days. Before signing, though, it wants to know whether the name really is yours.

An inspector who can’t ring the bell

The usual proof is called the HTTP-01 challenge, a challenge being a small test. Let’s Encrypt hands you a piece of text, you put it on your web server, and Let’s Encrypt fetches it from the outside. Whoever answers under that name owns it. Convenient, but your server has to be reachable from the internet.

Mine isn’t. My router has no door open to the inside, not a single port, as it’s called. All my devices talk to each other through NetBird, a private, encrypted network laid over the ordinary internet; part 2 explains how it works. For Let’s Encrypt this means the inspector is standing outside and there’s no bell to ring.

Luckily there’s a second way, the DNS-01 challenge. Here Let’s Encrypt doesn’t knock on the server at all. It looks in the public phone book. I’m asked to place a TXT record under the name _acme-challenge.home.example, a line of free text that contains the token. If you can change a domain’s entries, the domain is yours, that’s the reasoning. So my server only needs to be able to talk to Hetzner. That happens through their API, the interface programs use instead of people to change entries, and the key for it is an API token.

At my place the proxy does this, a program I’ll come to further down. The picture shows who asks whom along the way.

Every connection here goes out from my proxy; nothing has to come in. The orange step is the only one that needs more than internet access, namely the key to my DNS entries. The whole thing takes one to three minutes, mostly waiting: the new record has to reach all of Hetzner’s DNS servers before Let’s Encrypt looks.

One asterisk for everything

If I’m proving control over the whole domain anyway, I may as well ask for more than one name. A wildcard certificate covers *.home.example, and the asterisk stands for any name in that spot: paperless, nextcloud, immich, and whatever I set up tomorrow. Let’s Encrypt only issues wildcards after a check in DNS, by the way. So my detour also happens to be the only road there.

The asterisk covers exactly one level, though. *.home.example matches paperless.home.example, but not home.example itself, where my start page lives. That’s why my certificate carries a second wildcard for the domain one level up, which covers the start page. Putting the start page’s name into the same request as well gets rejected by Let’s Encrypt, because the wildcard already covers it.

Two phone books

Before the browser sees any ID, it has to find the server, and for that it needs an address for paperless.home.example.

That address comes from AdGuard Home, a DNS server on my network that also filters out ads and trackers. AdGuard holds exactly one entry for all my services, a so-called rewrite: every name ending in .home.example lives at the proxy’s address. NetBird makes sure my devices ask this very phone book. There’s one rule here I learned the hard way: the instruction “ask AdGuard” must never go out to all devices at once. That story is in part 2.

And the public phone book? To be honest, it holds the same wildcard with the same address. That sounds like a leak at first, but it’s harmless. It’s the proxy’s address inside the NetBird network, and the internet doesn’t route addresses like that anywhere. It’s a room number without a hotel. No individual service name appears there, just the asterisk.

The textbook term for this is split DNS, one phone book for inside and one for outside. In my case the two differ less in their answers than in who can do anything with them. The picture puts both sides next to each other.

Left is the path my own requests take. Right is what the world learns about my services, which is pleasantly little.

One doorman, one ID

All names lead to the same address, so somebody there has to direct the visitors. That’s Nginx Proxy Manager, a reverse proxy, from here on simply the proxy. Think of it as a doorman: it accepts every request, shows the wildcard certificate, reads which name the request is for and passes it on to the right service.

For a new service I add one entry in the proxy: name, target, port, plus the existing certificate. I don’t touch DNS, and I don’t need a new certificate. As the target I always use the service’s NetBird address, never a host name. That keeps the leg from the proxy to the service encrypted too, and it avoids a trap that once cost me sporadic 502 errors on a different proxy. The ntfy story has the details.

Which leaves the question of what happens after 90 days. Nothing I have to worry about. The proxy renews the certificate on its own, about 30 days before it expires, and runs through the whole TXT record routine again. You can count along in the public logbook I’ll get to in a moment: mid-February, mid-April, mid-June, mid-August, a new one every 60 days. The current one dates from early September and expires in early December. So the next renewal is due any day now, and I won’t notice a thing.

The logbook where everything is written down

Every certificate from a public authority ends up in public logbooks, the Certificate Transparency logs. They exist so that it gets noticed when an authority issues an ID for a name it shouldn’t, and the major browsers now insist that a certificate is logged there. Anyone can search them, for example on the website crt.sh.

For a homelab that would rather not be seen, that’s awkward. With one certificate per service, my complete list of services would sit there with dates. With the wildcard, only the asterisk shows up, one more reason to use one.

My record isn’t entirely clean, though. When I looked, I also found a certificate for one single service, with its full name. Once something is in the logbook, it stays there. That’s not a disaster, since a name alone opens no door. But it’s a good reminder that “internal” is not the same as “invisible”.

Where I tripped

Two stumbling blocks to finish with.

From practice: In September I wanted to create a DNS record for a new server with a script. But the old interface at dns.hetzner.com only redirected to Hetzner’s new console. DNS management has moved into the general Cloud API, to api.hetzner.cloud/v1/zones/<id>/rrsets, with the same Authorization: Bearer login as for servers. Changing records works differently too: you don’t overwrite an existing record with a PUT, you send a POST to …/rrsets/<name>/<type>/actions/set_records. For the certificate I was prepared, luckily: the proxy knows both APIs, and my helper script uses the new one by default.

The second one sits one floor up. The domain above home.example also has a wildcard record in the public phone book, and it points to an entirely different server on the internet. As long as a device asks AdGuard and I spell the name right, I never notice. But if I forget the home in a name, or a device asks a public DNS server for a name only my internal phone book knows, it ends up somewhere else. And there it is again, the warning from the beginning, or a password prompt that never accepts anything. How exactly that cost me a while with SSH is in the article about the NetBird server.

Only this time, the warning is right.

Takeaways

A server doesn’t have to be reachable to get a real certificate. It’s enough that it may change its domain’s entries. One wildcard, one entry in the internal phone book and a doorman in front turn every new service into a single entry in the proxy, with no DNS change and no new certificate.

The most important part is the habit from the beginning, though. At home there’s no longer any reason for a certificate warning. If one does show up, I don’t click through. I stop and look where I’ve ended up.

What the rest of the route looks like, from the phone on the road through the tunnel to the service, and why traffic stays encrypted even inside my own LAN (with one honest exception), is in part 2: Network & access.