AI without the cloud: a language model on a laptop GPU

A letter from the doctor goes into the scanner. I press the button, and a little later my phone buzzes: “Paperless: verarbeitet” (processed), followed by a title, the sender, the kind of document and a link. I typed none of it. Something read that letter, worked out who sent it and filed it in my archive.

How the letter gets from the scanner into the archive is the story of part 3. The archive is called Paperless, a program that stores scanned mail and makes it searchable. What that part left open is who actually reads the letter. That’s what this post is about.

A language model, but at home

The letter was read by a language model. That is a program trained on huge amounts of text so that it can understand and write text, the same kind of technology behind the well-known AI chats. Hand it a letter and ask “Who sent this?”, and it answers with the sender.

The easy option would be to send the letter to one of the big models on the internet, in “the cloud”. But that is exactly where I don’t want a letter like this, let alone a bank statement or a tax assessment. A cloud model would probably be smarter. But only a model running at my place guarantees that nobody else reads my mail.

There is a second rule on top of that. The automated steps that talk to the model only log things like the document number, never the text of the letter.

My model has two users. One is the automatic chain that files new letters and, at the end, makes my phone buzz. The other is me, when I want to chat with it in a browser. Before I show how both reach the model, let’s look at the machine it runs on.

A graphics chip without memory of its own

A language model mostly does one thing: a huge number of simple multiplications at once. A graphics chip, or GPU, is far better at that than the normal processor, because it does exactly these sums in parallel for images. That is why language models like to live on graphics cards.

I don’t have a big graphics card. My desktop machine is a mini PC with a Ryzen 7 7840HS, a processor you would normally find in a laptop. A small graphics chip is built right into that processor, a Radeon 780M; this is called integrated graphics. It has no memory of its own and shares the machine’s 50 GB of RAM with everything else.

To run the model on that chip you need a program that loads it and answers questions. Mine is llama.cpp, a free program that runs language models on ordinary hardware. Think of it as a counter at the post office: other programs hand it a question over the network, and it passes back the model’s answer. Because it speaks the same language as OpenAI’s interface, many ready-made tools can talk to it unchanged.

How the chat and the chain get there

The other services in my homelab don’t run on the desktop machine but on a separate server. There, every service has its own virtual machine, or VM: a small computer of its own that exists only as software. Three of them matter here. paperless-01 is the VM with the archive. n8n-01 is the VM with n8n, a tool where you plug automated steps together from building blocks; that’s where the filing chain runs. And openwebui-01 is the VM with Open WebUI, a chat page for the browser.

All three talk to the model through NetBird, an encrypted private network between my devices, like a tunnel nobody outside can look into. Part 2 explains it in more detail. The picture shows both paths.

On the left, the chat: Open WebUI passes my question on to llama.cpp. Its VM is tiny, two cores and 2 GB of RAM, because the model doesn’t live there. On the right, the chain: Paperless reports a new letter to n8n, which puts its questions to the model one after the other. The grey arrow belongs to the things that didn’t work; more on that below.

The model isn’t on the server because the server’s older graphics chip is also its only display output and doesn’t hand over well to a VM. So is the small chip in my desktop machine up to the job?

Lots of experts, little work

To answer that, we need two terms. Parameters are the numbers in which a model stored what it learned during training, a gigantic collection of numbers. My model has 30 billion of them. And a token is the piece of text a model reads and writes in, usually part of a word. For my letters, 24,000 characters come to roughly 7,000 tokens.

A classic model goes through all of its parameters for every single token. On a graphics chip without memory of its own that is the bottleneck: the chip could compute faster, but it waits for the numbers to arrive from RAM.

My model, Qwen3-30B-A3B, is built differently. It is a “mixture of experts”. Picture a company with many departments whose front desk sends each question only to the two or three departments that know about it. In a similar way the model picks only a small part of its parameters for each token, about 3 billion, which is what “A3B” in the name means. All 30 billion still sit in memory, but each token only reads about a tenth of them.

I tried out how much that matters in August, with a classic model of similar size on the same chip. The figure for my current model comes from the llama.cpp log since September:

With the classic model, a 100-token answer took about 45 seconds, like watching a fax come in. My current model is more than ten times faster, even though it is bigger overall.

To fit 30 billion numbers into memory, they are stored more coarsely, a bit like a more heavily compressed photo. That brings the file down to 18.6 GB.

And in everyday use? Since the move in early September, the log shows 862 requests. A typical one takes 19 seconds, the longest took almost 80. For a letter nobody is impatiently waiting for, that’s fine.

Two seats at the table

Fast enough for a letter, then. But what happens if I chat while a letter is being filed?

One more term for that. The context is everything the model has in view for a request: my instructions, the letter’s text and its own answer. llama.cpp reserves room in RAM for it. On my machine that is 32,768 tokens, split into two equal seats, called “slots”, with 16,384 tokens each. Each slot handles exactly one request, like two chairs at a consultation desk. A third request waits until a chair is free.

It used to be four times as large, but memory is tight. Nothing has been missing; the longest request in the log was 6,626 tokens.

Two slots don’t double the speed, since both share one chip. Running alone, a request writes a median of 19 tokens per second; sharing, it gets 15. What you gain is that one long request no longer holds up everyone else.

On top of that there is a safety line. No answer may be longer than 1,024 tokens, which at my speed means it stops after about a minute. Why that matters so much to me is the first story further down.

Choose, don’t invent

In the chat, the model is allowed to ramble. In the chain it isn’t. When it names a sender there, it has to be one my archive already knows. Paperless calls senders “correspondents”, by the way. Without a rule, after a month I would have “City Utilities”, “City Utilities Ltd” and “Ctiy Utilities” as three different correspondents.

So in the chain, the model gets a form to fill in. Technically that is a JSON schema. JSON is a simple text format for data with field names and values, for example {"correspondent": "Unbekannt"}. The schema is the template for it: it says which fields exist and which values are allowed. For the correspondent question, the important part looks like this:

{"correspondent": {"type": "string",
                   "enum": ["<correspondent 1>", "<correspondent 2>", "…", "Unbekannt"]}}

enum is the list of allowed answers (“Unbekannt” is German for unknown). The chain fetches that list fresh from Paperless for every letter.

The nice part happens inside llama.cpp. While writing each token, it checks that the answer still fits the schema and simply doesn’t allow anything else. So the model can’t invent a correspondent that isn’t on the list, nor write a polite sentence in front of the answer. It may choose, but not invent. If nothing fits, “Unbekannt” stays, and I see it.

n8n checks the answer once more afterwards. If it is empty or unreadable, the step stops with an error instead of quietly filing the letter as “Unsortiert” (unsorted). An error in n8n is something I’ll find. A misfiled letter, maybe never.

What didn’t work

Of course it wasn’t this smooth from the start.

From practice: Paperless comes with its own AI feature that makes suggestions for every document. That was my first attempt: 64 timeouts, not a single success. Paperless sends the letter, all tags, all correspondents and similar documents, about 5,000 tokens, and expects an answer in a fixed format. The smaller model I had back then wrote free text instead, more than 4,000 tokens over five minutes. Paperless gave up after two minutes, but the slot stayed busy. With the current model and the answer limit, the suggestions came back in 56 seconds. In the browser even that was too slow: every document I opened asked the model, both slots stayed busy, and at some point even saving failed. The feature has been off ever since (the grey arrow in the first picture), and the AI only works in the background through n8n.

From practice: Before a language model can read a letter, the scanned image has to become text. That is called text recognition, or OCR. I wanted AI for that too, a model called PaddleOCR-VL that understands a page as an image. But it needs an Nvidia graphics chip and doesn’t run on my AMD one. An older variant running on the processor alone kept almost eight of its sixteen cores busy for every letter, which was too much. I went back to Tesseract, the text recognition that ships with Paperless. For printed letters, its text is all the language model needs.

One limit remains. The model runs on exactly one machine. When it’s off, there are no answers, and only one large model fits into memory at a time. A VM on the server running a smaller model on the processor alone is an option in the plan. I haven’t built it, and honestly I don’t know whether it would be fast enough.

Takeaways

• On a small graphics chip, memory is the brake. A model that needs only a few parameters per token is many times faster there.
• Fixed slots and a limit on answer length cost almost nothing and stop one runaway request from blocking all the others.
• A template, a list and a check afterwards turn a chatty model into a tool I can rely on.
• A slow model belongs in the background, not where someone is waiting for a click.

So the next time my phone buzzes, I know who read the letter: a laptop-class graphics chip, a few experts inside the model and a list that allows no excuses. How the letter becomes text in the first place, and what the chain then does with sender, tags and title, is in part 3.