Eight tokens a second. That is the number that ended my experiment with running AI locally, and it is the reason my personal AI stack now sits on someone else’s GPUs.
I did not want that outcome. I run a homelab specifically because I like owning the thing that does the work. But after a few months of trying, the honest answer is that local inference did not earn its place, and the reasons are more interesting than “it was slow”.
Here is what I actually run.
The stack on one diagram
Obsidian Trilium
(reference, RO) (working memory, RW)
│ │
└────────┬──────────┘
▼
┌─────────────┐
│ Hermes │ LXC on the Proxmox cluster
│ orchestrator│ web UI over Tailscale
└──────┬──────┘ cron, Telegram, MCP tools
│
▼
┌─────────────┐
│ OpenRouter │ one key, one endpoint
└──────┬──────┘
│
┌────────────┼────────────┐
▼ ▼ ▼
DeepSeek Claude whatever
(default) (heavy lifting) is nextFour moving parts. Two of them store memory, one of them decides what to do, and one of them is a switchboard. Nothing here is exotic, and that is deliberate.
Hermes does the deciding
Hermes is the orchestrator. It runs as an LXC container on my Proxmox cluster, currently version 0.20.1, on Python 3.11. It runs as its own unprivileged hermes user rather than as root, which is the sort of thing that feels pedantic right up until an agent with tool access does something you did not expect.
What makes it useful is not the chat interface. It is the two things bolted to the side of it.
The first is MCP. Hermes has two Model Context Protocol servers wired in, one for Proxmox and one for my Synology NAS. That is the difference between an assistant that can talk about my homelab and one that can query it. Asking “which containers are running and how much memory is free” gets a real answer read from the actual cluster, not a guess assembled from whatever I mentioned three messages ago.
The second is cron with somewhere to report. Hermes runs scheduled jobs and pushes the results to Telegram. My favourite is a weekly patch patrol every Friday morning that checks every node and container in the cluster for pending updates. When everything is current it says nothing at all. A monitoring job that only speaks when something needs attention is worth ten dashboards I have to remember to open.
All of that would be a great deal less useful if talking to it meant SSHing into a container. Hermes has a web UI, and because every device I own sits on the same Tailscale network, I can open it from my phone on the sofa, my laptop at a cafe, or any desktop in the house. Same address every time, no port forwarding, no reverse proxy, no VPN client to remember to switch on, and nothing published to the internet. It is behind a login, and the only machines that can route to it at all are ones I have deliberately added to the tailnet.
This is the part of the setup I would miss most. An assistant that can only be reached from one desk is a tool you use when you remember to sit at that desk. One that answers from a phone is one you actually use.
Its default model is DeepSeek V4 Flash 0731, which is a deliberate choice I will come back to.
OpenRouter is the switchboard
Everything goes through OpenRouter. One API key, one base URL, and every model I might want to use sitting behind it.
The obvious objection is that this adds a middleman between me and the model providers, and that is true. Here is why I do it anyway.
Switching models becomes a config change instead of a code change. Moving Hermes from DeepSeek to Claude is one line in a YAML file. No new SDK, no new auth flow, no second billing relationship, no rewriting the bits of my setup that assume a particular response shape. When a new model lands and everyone spends a week arguing about whether it is any good, I can find out for myself in about thirty seconds.
One key to manage rather than five. Every provider I talk to directly is another credential to store, rotate and worry about. I have enough of those.
Spend lives in one place. I can see what this whole stack costs by looking at a single number, which turns out to matter for the argument at the end of this post.
The cost is a small amount of latency and a dependency on a company that could change its terms tomorrow. Both are real. Neither has bitten me yet, and the abstraction means that if OpenRouter did something I disliked, I would be swapping a base URL rather than rebuilding.
Cheap by default, expensive on request
People are surprised that my default model is not the best one available. It is DeepSeek V4 Flash, and it is default precisely because most of what an orchestrator does is not hard.
Checking whether a container needs patching, formatting a status message, deciding which tool to call, summarising a log. None of that needs a frontier model. Routing it to one is paying a premium for capability that goes unused, on a task that runs every day.
Claude handles the heavy lifting. Writing, reasoning through an architecture decision, anything involving code, anything where being wrong is expensive. I do most of that in Claude Code, and Hermes can reach Claude through the same OpenRouter endpoint when a job actually warrants it.
The split is not about loyalty to a model. It is that a stack which reaches for its most expensive option by default is a stack nobody keeps running for long.
The part where I gave up on local inference
I wanted local inference to work. Privacy, no per-token cost, no dependency on anyone’s uptime but my own. I have written before about wanting the AI’s memory to stay on my disks, and running the model on my own hardware is the natural extension of that.
So I put Ollama on my M4 Mac Mini and pointed the stack at it. This bit is easy, and worth writing down because it catches people out: Ollama binds to localhost by default, so nothing else on your network can reach it. You need OLLAMA_HOST=0.0.0.0 before it will answer anything but the machine it is running on.
Then I loaded a 9B model and started using it properly.
Eight to ten tokens per second.
That is slower than reading speed. Every interaction became something you sit and wait through, and the thing about an assistant you have to wait for is that you stop asking it things. Within about two weeks I had quietly gone back to the API for everything, which is its own kind of answer.
It was not the chip, it was the memory
Here is the part I got wrong at the time, and it matters if you are thinking about trying this yourself.
An M4 is not a slow chip. A 9B model at that quantisation should run considerably faster than I was seeing. The bottleneck was that mine is the base M4 with 16GB of unified memory, and on Apple silicon that 16GB is shared between the operating system, everything else you have open, and the model. Load a 9B model into it and you are immediately in memory pressure, and throughput falls off a cliff.
The same model on a machine with 32GB or more would tell a completely different story. So the correct conclusion is not “local inference is too slow”. It is “local inference on 16GB is too slow”, which is a much more useful thing to know.
There is a small joke buried in the hardware here. The one specification I paid extra for on that Mac Mini was the 10 gigabit ethernet port. It is a genuinely great feature and it is completely irrelevant to this problem. Network throughput does nothing for you when the constraint is memory bandwidth on the other side of the wire.
What it actually costs
This is the argument that settled it, and I can put real numbers on it because everything runs through one account.
- Total OpenRouter spend to date: $16.78
- Mid-July: $1.52. Late July: $3.36. Today: $16.78
- Current run rate: about $0.42 a day, or roughly $13 a month
My daily rate has roughly doubled since July, which is what happens when a thing becomes useful enough to reach for without thinking about it. Even so, this entire stack costs me less per month than a couple of coffees.
Now price the alternative. The memory upgrade that would have made local inference viable is a few hundred dollars, and Apple does not let you add it later. At my current rate I would need to run this stack for well over a year before that hardware paid for itself.
And that is the argument that actually matters. It is not that the hardware is expensive. It is that hardware depreciates against a moving target. In the time it would take that upgrade to break even, the models will have moved on twice, and the machine I bought to run a 9B model will be running a 9B model while everyone else has moved to something better. The API has no such problem. When a better model appears, I change one line.
Buying hardware to run a specific generation of model is buying a depreciating asset to chase an appreciating one.
Where the memory lives
The one part of this I have kept entirely local is the memory, and I wrote about that split in detail in Two Brains for My AI. The short version: Obsidian holds curated reference knowledge and the agent reads it but never writes to it. Trilium holds the daily working log and the agent reads and writes freely, because it is a real database with an API and permissions.
That division has survived every other change to this stack, which is worth noting. The models moved, the inference moved off my hardware entirely, and the memory architecture did not need touching once. It turns out the durable decision was not which model to use. It was where the knowledge lives and who is allowed to write to it.
What happens next
The Mac Mini is running Plex for the moment, which is a slightly undignified role for a machine I bought with ambitions. I have other plans for it.
Local inference gets another try eventually, on hardware chosen for the job rather than hardware I happened to own. Until then the stack stays as it is: memory on my disks, orchestration on my cluster, and the actual thinking rented by the token from whoever is currently best at it.
Sixteen dollars and seventy-eight cents says that is the right call for now.