Pipeline: who actually runs my code?

It’s a little after eleven at night, and a test suite fails, a batch of automated tests for one of my software projects. No red assertion, no bug in the code, just a timeout. The test had simply taken longer than it was allowed to.

The natural first move is to dig through the code. The cause wasn’t there, though. It sat one level further down, in a question I rarely ask: who actually runs this code, and who is it sharing the machine with?

That question fits this point in the series nicely. In part 4 I showed how a short text file turns into a finished VM, and I pretended to be the one typing the commands. Most of the time I don’t type them at all. A pipeline does, and that’s what this part is about. Then we’ll come back to that timeout.

An assembly line for code

My code lives on GitLab, a platform for repos, those folders that remember every change. Each time I upload a change, called a push, GitLab starts a pipeline. Picture an assembly line with several stations, called jobs. One job might check the formatting, another runs the tests. If one fails, the line stops.

GitLab itself only hands out the work. The work is done by a runner, a program on some computer that keeps asking GitLab for something to do, like a courier who drops by the depot to pick up the next round of deliveries. GitLab.com offers its own runners for this, the shared runners, used by many projects at once. For small projects they’re perfectly fine. I still had two reasons to run my own.

Why run my own runners?

The first reason is my home network. Jobs in the homelab repo, where the code from part 4 lives, talk to Proxmox and log into the VMs over SSH. A shared runner somewhere on the internet can’t reach any of that, and I wouldn’t want it to.

The second reason showed up on 14 October. My software project paper-reader had burned through its free minutes on the shared runners, and every pipeline stopped with ci_quota_exceeded. That project only takes a change into its main version, a step called a merge, once the change’s pipeline is green, meaning every job passed. No pipeline, no merges, so work came to a halt. I enabled my own runner for the project and switched the shared runners off, all of it through GitLab’s API, its interface for programs rather than people.

Side by side, the difference looks like this:

There’s one more detail that fits the picture from part 2. Because the runner asks GitLab for work over HTTPS on its own initiative, GitLab never has to connect to me. So the pipeline doesn’t need a single open port on my router either.

Two runners, one server

I run two of them, both Debian VMs on my server, pve1. Every job starts in a fresh Docker container, the sealed box from part 4, and leaves nothing behind for the next one. The first runner predates my move to the new homelab. I added the second on 16 September, built with the same OpenTofu module as every other VM, because pipelines kept queuing up behind a single runner.

Besides the homelab repo, three of my software projects use these runners: paper-reader, e-learning and telefix. Two knobs decide which runner gets which job. A tag is a label, like the address on a parcel. A job that asks for the tag homelab only lands on a runner carrying that label. Jobs with no tag at all go to any runner that accepts untagged jobs, and both of mine do. The second knob is concurrent, which sets how many jobs a runner works on at the same time.

In the next picture the projects sit on the left, the two runners on the right, and every arrow carries the tag a job uses to find its runner.

Here’s how the two are set up today. vCPU stands for virtual CPU, the slice of the real processor a VM gets to use.

Almost every row hides a correction. After the move, runner 1 had just 8 GB of memory, until a test job that measures code coverage hung for 57 minutes. It’s had 16 GB ever since, and a small script sends a message to my phone as soon as a runner drops below 15 % free memory. Then, in October, three jobs suddenly ran at once, because the config said concurrent = 3, a value no script had ever written.

These days I read concurrent = 2 as half a runner rather than a double one. Each job shares processor, memory and disk with a neighbour it knows nothing about. Remember that neighbour; it’s coming back.

From a push to a VM

The runners’ most important job, though, is close to home. Every job in the homelab repo carries the homelab tag, which sends it to runner 1. The pipeline runs exactly the steps from part 4: plan and apply with OpenTofu, then Ansible. The picture shows them from top to bottom.

At the top sits lint, a kind of spell checker for code that runs on every push. Then plan writes its plan, the quote from part 4, and saves it as a file. Apply never starts by itself. I sit in between and read the plan, and if a VM shows up as “must be replaced”, I leave the button alone. When I do press it, apply runs exactly the saved file I read, without planning again. Whatever happens between my reading and my clicking can’t change the outcome anymore.

That leaves one question: what if two pipelines want to touch the state, OpenTofu’s logbook, at the same time? For that, plan and apply share a resource group. It works like the key to the only bathroom in a shared flat: whoever has it is inside, everyone else waits. So GitLab never runs two of those jobs at once.

Passwords live as protected variables in GitLab, and gitleaks scans every push for anything saved by mistake. On 8 September the first service VM went all the way through this pipeline. The VM was up after 30 seconds, and Ansible had finished after about two minutes.

At the end of part 4 I left the afternoon of that day hanging. It has a lot to do with this pipeline.

From practice: On the afternoon of 8 September an apply was only supposed to change one setting of the Proxmox provider, the component OpenTofu uses to talk to Proxmox. By default the provider restarts a VM when its configuration changes, so it restarted three of them, including the runner the job was running on. The job died along with its runner, and two VMs stayed off. Every VM definition has said reboot_after_update = false, don’t restart after a change, ever since. A pipeline that manages infrastructure is part of that infrastructure, and whatever it does to “all VMs”, it does to itself too.

The other mistake from those weeks was a quieter one.

From practice: An early Ansible job came back green, and nothing had happened. The copy of the repo inside the runner is writable by everyone, and in a directory like that Ansible ignores its settings file, ansible.cfg, for safety reasons. So it no longer knew where the inventory was. What ran was a playbook against zero VMs, without a single error. Today the job tells Ansible explicitly where its settings are, and fails when the inventory is empty.

Back to that night

That’s the runners and the pipeline. With tags and concurrent in mind, the timeout from the beginning is quickly told.

paper-reader has a nightly, a pipeline that starts by itself every evening at 23:00. It runs a single job with two property-based tests. Tests like these invent random inputs on their own and check that a rule always holds, in this case 50,000 times each. On 28 October the pipeline after a merge was running on runner 1 at the same time, and the nightly got the second slot. There’s our neighbour. The two shared 8 vCPUs, the test took up to 62 % longer than usual, and it blew through its fixed limit of 120 seconds.

The obvious reflex would be to give runner 1 more vCPUs. It wouldn’t have helped much. pve1 runs on a laptop processor with 16 threads, 16 lanes of work that every VM has to share, and all VMs together have about 50 vCPUs assigned. A bigger runner would mostly have given the neighbouring job more elbow room. So I changed two other things.

First, the time limit. In a merge request, a proposed change still waiting to go into the main version, the test runs 500 cases, at night 50,000, and no fixed limit fits both. Measured on the runner, a case takes about 1.8 ms. The limit is now 60 seconds plus 3 ms per case, which works out to 210 seconds at night and just over 60 in a merge request.

Second, the runner. Runner 2 has concurrent = 1, so a job there has the VM to itself. It got the tag paper-nightly, and the nightly asks for it. There’s a trap here, because runner 2 also accepts untagged jobs. Had I tagged only the nightly, every other paper-reader job would have started landing on runner 2 as well. So every job got a tag. In the project’s pipeline file that takes two spots:

default:
  tags: [docker]          # only runner 1 carries it

frontend:property-nightly:
  tags: [paper-nightly]   # only runner 2 carries it

The first run on runner 2 passed. Yet the test took 119 seconds there, one second under the old limit, with no other job on its VM. Meanwhile, on runner 1, the pipeline for merging this very change was running. For comparison, the same test takes 24 seconds on my workstation.

The last picture sums up the night. The big frame is the server with its 16 threads. At the top, the nightly of 28 October shares runner 1 with its neighbour; below, it now runs alone on runner 2.

Those 119 seconds bothered me more than the timeout did. A VM of its own separates programs, memory limits and settings. It does nothing about the hardware underneath. That’s why it takes both changes: the dedicated runner removes the neighbouring job, and the growing limit soaks up the noise from all the other VMs.

Your own VM, shared hardware

The nightly still starts at 23:00 every evening, now alone on its runner and with 210 seconds of headroom. Honestly, that night taught me more about my homelab than about the test. A time limit in a test says as much about the machine as about the code, and once you pack lots of VMs onto one server, they end up sharing the same threads after all.

The pipeline now builds and changes my VMs reliably. But what if one gets lost anyway? Or a service quietly dies in the night and nobody notices? That’s the subject of the last part, part 6: how my homelab gets through a bad night, and who wakes me when it can’t manage on its own.