Code: one VM per service with OpenTofu and Ansible

Remember the last time you set up a new laptop? Install the system, install your programs, hunt for that one setting. On the old machine everything had been just right, but you never wrote any of it down, so now you’re guessing what you changed where, years ago.

That’s exactly how I used to set up my servers. In my homelab a server is almost always a VM, a virtual machine: a computer that exists only as software, a bit like a flat in an apartment block. The building is my one real server, and Proxmox is the software on it that manages the flats. A new VM used to mean clicking it together in Proxmox, installing Linux, setting up the programs it needed, and hoping I’d remember what I had done. Five VMs from those days are still around.

The first three parts of this series covered what you can see from the outside: the big picture in part 1, the way from my phone into the home network in part 2, and Paperless, my digital archive for paperwork, in part 3. From here on we go backstage, into the area the series map calls Operations. First up is Code, or how a new VM comes to life without me clicking anything.

A blueprint instead of clicks

These days every new VM starts as a short text file. I read a plan, and a little later the service answers at <service>.home.example, backed up and monitored. The VM for Paperless was born that way too.

The idea is called infrastructure as code. Instead of setting a VM up by hand, I write down what it should look like, the way an architect draws a blueprint, and a program builds from it. All these files live in my repo, short for repository: a folder that keeps every change with a date attached, like a document with its full edit history.

Two programs share the work. OpenTofu is the builder who puts up the shell. It talks to Proxmox and creates the VM with its processor cores, memory and disks. Ansible does the interior. It logs into the finished VM over SSH, an encrypted remote control, and sets up whatever the service needs, changing only what isn’t right yet.

Every new service comes to life in the same four steps. The next picture shows them from top to bottom.

First I write the file and read the plan OpenTofu makes from it. Then Proxmox builds the VM, Ansible configures it, and finally the service gets its name, its watchman and its backup. Let’s walk through the stages in order, starting with the file.

One file per VM

A VM’s file is surprisingly short, because the real work lives in a module. In OpenTofu, a module is a reusable blueprint, a bit like a prefab house from a catalogue: every VM shares the same floor plan, and the file only says how big this particular house should be. Mine is called debian-vm, since every VM gets Debian, a widely used Linux.

I call the VM for Paperless paperless-01. Here is its file, trimmed a little. You don’t need to know the language; each line is just a value:

module "paperless_01" {
  source = "./modules/debian-vm"

  name         = "paperless-01"
  vm_id        = 215
  cores        = 6
  memory       = 8192
  disk         = 30
  data_disk_gb = 100
  ip           = "<ip>"
  # … user, password and SSH keys come from variables
}

There’s the name, a number, six processor cores, 8 GB of memory and two disks, 30 GB for the system and 100 GB for data. The number is the VMID, the ID Proxmox files the VM under. It also sets the VM’s address on my home network, its IP, whose last part is simply the VMID.

The next picture shows what the module does with those values.

It creates exactly two things: the VM itself, and a small note with instructions for its very first start, the cloud-init snippet. At the end it reports the VM’s IP. From that, a second file, inventory.tf, writes the inventory for Ansible, which is simply the list of all VMs and their addresses. So a new VM means one new file plus two lines in inventory.tf. Twelve of my VMs were built this way.

The first boot

Let’s stay with that note for a moment, because the first boot is where I learned the most. cloud-init is a program that ships inside the Debian image and works through the note once, on the VM’s very first start, like a caretaker handing over the keys on moving day. After that it does nothing.

At the very beginning it did nothing at all. Debian offers its ready-made image in several flavours, and I had picked nocloud first. After booting I found myself straight in a root shell, a console with full admin rights, because that flavour ignores cloud-init entirely. The right one is called generic.

The second problem went deeper. My first note was long: install Docker, the program my services will later run in, join NetBird, set keyboard and language. Every VM also runs a small helper called the guest agent, which among other things tells Proxmox the VM’s IP. OpenTofu reports “done” as soon as that IP shows up. It has no idea whether cloud-init is still busy inside, so right afterwards Docker sometimes wasn’t even installed yet.

Today the note says almost nothing. cloud-init creates the user with an SSH key and starts the guest agent, and that’s it. Everything else is Ansible’s job, where I can watch what happens and repeat it as often as I like. On the test VM, the machine was up after 28 seconds and cloud-init was done after 17.

The plan I read

Back to OpenTofu. Before it builds anything, there’s the step I care about most. For it, OpenTofu needs a memory, called the state. Think of it as a logbook recording which real VM belongs to which file. Mine sits in online storage at Hetzner that keeps old versions, so I can get a damaged logbook back. I’ll explain how that storage works in part 6.

From the files and the state, OpenTofu first produces a plan, much like a tradesperson writes a quote before starting the job. It lists what would be created, changed or replaced. Only when I agree does the apply carry it out. Replace sounds harmless, but for a VM it means delete and rebuild, system disk and all. If a running VM shows up as “must be replaced”, something is off, and nothing gets applied.

I’m strict about this for a reason, and to tell it I need to borrow a word from part 5. OpenTofu doesn’t only run on my machine. Most of the time it runs in a pipeline, an automated process on GitLab, the platform that hosts my repo. Both should write exactly the same cloud-init note for the same VM.

From practice: On the morning of 8 September I ran OpenTofu on my own machine, only to rewrite the Ansible inventory. The apply deleted and rebuilt three VMs, including the one for DNS, my network’s phone book. The cause was hiding in the VMs’ password. It contains a character that GitLab, inside pipeline variables, reads as a reference to another variable. So the pipeline had written a different note than my machine did, OpenTofu saw a changed note, and back then a changed note meant replacing the VM. DNS carried on through the spare DNS server on a Raspberry Pi, but I had to rebuild AdGuard, the DNS program. Since then the pipeline passes the variable through untouched, and the password goes into the note escaped so that no character can be reinterpreted.

The module has carried two safety rules ever since. One says that a changed cloud-init note is no reason to replace a VM; it’s only read on the first boot anyway. The other says that a changed setting is no reason to reboot (reboot_after_update = false). That one dates from the afternoon of the same day, which part 5 tells.

And the five old VMs from the beginning? OpenTofu added them to its logbook after the fact. For those it only manages name, cores, memory and start behaviour, and the important ones are locked against accidental deletion.

Ansible moves in

Once the VM is up, it’s still fairly bare: a fresh Debian with one user and one key. Now the interior work begins. Ansible reads the inventory and follows a set of instructions called a playbook, mine is site.yml. A playbook is made of roles. A role is a recipe for exactly one job, and the playbook decides which recipe gets cooked on which VM.

We need one more word first. My services run in Docker. Docker packs a program and everything it needs into a container, a sealed box that runs the same anywhere. A service often needs several boxes, say the program and its database. Together I call them a stack, and a file named compose.yaml lists the boxes that belong to it.

The next picture shows the order of the roles: basics for every VM first, then the actual service, and finally backup and updates.

The table says what each of the five roles does. Keep an eye on the last two; they come back in part 6.

Every role is idempotent, which is a long word for a simple promise. Once a run has worked, the next one changes nothing, like a light switch labelled “on” rather than “toggle”. On the test VM the first run reported 15 changes and the second none. The first service to go all the way through the pipeline was DNS. Its VM answered 30 seconds after the apply, and Ansible was finished after about two minutes.

Where do the passwords go?

With this much code in a repo, you might wonder where the passwords live. Never in the repo, that much is certain. Each service gets a .env file on its VM, a plain text file with its credentials that only the administrator can read. I send it over SSH before Ansible runs. If it’s missing, Ansible still creates the stack but won’t start it, and prints a hint instead. Credentials the pipeline needs are stored as protected variables in GitLab.

To make sure nothing slips through anyway, gitleaks, a scanner for accidentally saved secrets, checks every upload to the repo. That habit has a history. An older attempt in this repo once held an access token in plain text, since revoked.

One at a time

When I change a recipe, the change has to reach every VM. I never do that on all of them at once, only one or two at a time. The one time I let the playbook loose on every VM together, the smallest VM, with 1 GB of memory, ran out of it during a big system upgrade until it only answered ping, the most basic sign of life on a network. On top of that, the SSD in my server slows down when it gets hot under sustained load.

Since then even the smallest VM gets 2 GB. And a change only counts as done once it has reached every VM, because what’s in the repo isn’t automatically what’s running.

The finishing touches

With the stack running, the service is still only reachable on its own VM. To show up under its name, it needs an entry at the proxy, the doorman from part 2 who takes every request and passes it to the right service. A script adds <service>.home.example there. DNS needs no change at all, since it already sends every name under home.example to the proxy.

Then the service gets a watchman. Uptime Kuma checks every minute whether it answers and raises the alarm if it doesn’t; more on that in part 6. A script creates that entry too, and the shared login through Authentik from part 2 if the service supports it. Add the new VM to the nightly backup job, write a page in my docs, and that’s it.

Tearing a service down runs the same way in reverse. At the end, a full plan has to say “No changes”. Only then do code and reality agree again.

From clicking to reading

So the service is there: one file, one plan I actually read, a couple of minutes of Ansible, and it answers under its own name. Unlike that new laptop, this time everything is written down, and my job has shifted from clicking to reading. The reading is the part that matters. A tool that replaces or reboots VMs on its own needs guard rails before it touches a machine you depend on, and a human who reads the plan.

Up to now, though, I’ve pretended to be the one at the keyboard. Most of the time plan and apply run in the pipeline, on machines that are themselves VMs in this homelab. What happened to one of those machines on the afternoon of 8 September, and why a nightly test suddenly ran too slow, is the story of part 5.