A blog about technology, football and all the other random stuff in my life

Latest Posts

  • The Layer the Cable Map Doesn’t Show

    I’ve already drawn the cables. This is where the packets actually go.

    Back in May I published the full topology of the house: every switch, every 10G run, every zone. That was the easy half. Cables tell you what’s physically possible. They tell you nothing about where a DNS query goes, which door a request came through, or why one machine’s traffic leaves the country and another’s doesn’t. If you go back far enough in this blog, you would have noticed that I’ve been using the Asus RT routers since the original N56U (in 2011!). While they can do most of the basic functions (and some of the advanced ones with the Merlin firmware), I’ve taken advantage of my homelab to add on a few more bells n whistles.

    its always DNS

    The Router Does Three Separate Jobs

    Everything starts at an Asus RT-BE88U running Merlin firmware. Stock firmware would route fine. I run Merlin for three features that turned out to be the backbone of everything downstream.

    DHCP with reservations. Boring, until it isn’t. My reverse proxy references backend IPs directly, so a container that gets a new lease silently breaks a service while the proxy keeps happily forwarding to an address nobody’s at. I once had four services sitting proxied-but-unreserved for four days without noticing. Reserve the address before you point anything at it.

    DNS Director. This force-redirects port 53 for most client on the LAN to my Pi-holes, so a device that hardcodes 8.8.8.8 gets intercepted and filtered anyway. It works completely, which is the problem. My Pi-holes are LAN clients too. For a while their own upstream queries were being redirected back at the router, so the recursive resolver behind them never reached the root servers at all. Recursion was an illusion. Any host running its own resolver needs to be set to “No Redirection” — that one exemption is the whole trick.

    VPN Director. Policy-based routing: named clients leave through a commercial VPN, everything else takes the native connection. My desktop is on the list; the Proxmox nodes aren’t. The catch nobody mentions is that it’s IPv4 only. Checked from my desktop while writing this: IPv4 exits at a CDN77 address, IPv6 exits on my actual ISP prefix. Anything that resolves AAAA first walks straight past the VPN with my real address attached. I spent a while treating that as a bug before accepting it as the design.

    DNS: Pi-hole in Front, Unbound Behind

    Two Pi-holes, both containers on the Proxmox cluster. One is the source of truth, the other a replica, and nebula-sync copies one to the other every fifteen minutes. Two of them because DNS failing takes the whole house with it.

    Behind each Pi-hole sits unbound, doing full recursion out to the root servers. No Cloudflare, no Quad9, nobody upstream keeping a log of what I look up. Pi-hole blocks and forwards; unbound resolves.

    Getting that right cost me about 3,975 SERVFAILs. Unbound validates DNSSEC itself. I had also ticked DNSSEC on in Pi-hole, so every signed lookup got validated twice and the second validator kept rejecting the first one’s work. The fix was one checkbox. Leave Pi-hole’s DNSSEC off when unbound is upstream — unbound is the validator, and it still rejects bogus signatures without any help.

    Caddy: One Certificate, Thirty-Three Names

    Everything internal answers on a single wildcard domain, served by a Caddy container on one of the cluster nodes. One Let’s Encrypt certificate, issued over DNS-01 against Cloudflare, which means nothing has to be exposed to the internet to get it. Renewal happens entirely through a DNS record.

    Thirty-three hostnames as of this week. Each is a Pi-hole record pointing at Caddy’s address plus a five-line block in one config file. Adding a service takes two minutes and no new certificate. I tried to get NGINX set up 3 times and failed miserably, so I got my LLM to set up and manage Caddy for me. It worked at the first try!

    The one I got wrong: for eleven days my router’s admin interface was proxied on that domain. The highest-value target on the network, protected by nothing but the router’s own login, reachable from anything that could resolve LAN DNS. I had added it, never written it down, and only found it during an audit. I deleted the entry rather than putting an authentication layer in front of it. Removing the exposure was cheaper than defending it.

    Tailscale: The Private Overlay

    Tailscale is how I get in from outside — laptop, phone, ipad, wherever. The interesting part is DNS, because MagicDNS and Pi-hole both want to be the resolver.

    The tailnet uses split DNS: queries for my domain go to both Pi-holes, everything else resolves normally. Those records answer with LAN addresses, not Tailscale addresses, so devices already on the LAN keep working unchanged. Remote devices reach the same addresses through a subnet router advertising the whole /24.

    That subnet router is currently one node. If it goes down, remote access to the LAN goes with it. I know. A second one is on the list.

    The trap here is subtle and cost me an afternoon. 100.100.100.100 is a forwarder, not a resolver. Tailscale answers tailnet names and hands everything else to whatever the system’s real nameservers were at the moment it took over. One node had its interfaces bounced during a NIC migration, tailscaled restarted while the config was momentarily empty, and it captured nothing as its upstream. Every public name on that host failed silently for eleven hours. All three nodes had byte-identical resolver config. The state that mattered lived in a backup file that none of the obvious commands show you.

    Cloudflare Tunnel: The Public Door

    External access runs on a different domain entirely, through a Cloudflare tunnel. No inbound ports open anywhere. Most hostnames sit behind Cloudflare Access; Plex is deliberately bare, because it is built to face the internet and enforces its own login (with 2FA, naturally).

    The two domains differ by a single hyphen. That is a genuinely bad idea I am now stuck with, and I have written down which is which in three separate places. (The one with the hypen is external!)

    The tunnel originally ran one connector, on the NAS, which was also the origin for one of the services it fronted. A DSM update took down the door and one of the rooms behind it, orphaning three perfectly healthy services that had nothing to do with the NAS. There are two connectors now, on separate boxes, serving the same tunnel. I tested it by stopping the first: twelve of twelve requests still served.

    Getting that wrong is easy. A second connector has to use the existing tunnel’s token. Clicking “create a tunnel” gives you two tunnels and zero redundancy.

    The Pattern

    Five systems, and most of the faults above were the same shape: two layers both trying to do one job. Two DNSSEC validators. A router intercepting the resolver it was pointing at. A door mounted on the room it opened onto. Draw the logical layer, not just the cables. You cannot spot a duplicate when you are only looking at one diagram.

    None of them announced themselves. They presented as “the internet is slow”, “flatpak is broken”, “apt can’t reach the repos” — never as the thing that was actually wrong. Each fix was a single setting. Each one took days to find. Just remember the wise advice of the interwebs: if you have a problem, its always DNS.

    Next up for me is figuring out VLANS. My new 10G Managed switch (DXS-F108T) just arrived today, so stay tuned!

  • I Built a Dashboard for the Things I Actually Check

    Five separate logins became one page. The interesting part was how often I turned out to be wrong.

    I check five things regularly: whether the homelab is healthy, my portfolio, my health trend, whatever trip is coming up, and my to-do list. Each one lived somewhere different. Each had its own login. None of them talked to each other. I have to go looking for information by opening a browser, going to my bookmarks page and finding the right tile to click:

    My Heimdall Dashboard of links to my self-hosted services.

    Information you have to go and fetch is information you check less often than you should.

    What It Looks Like Now

    One page. Five modules, each running in its own container, pulled together by a dashboard shell called Glance. The shell is configuration, not a product — it reads each service’s API and renders the numbers. That’s the whole trick (admittedly greatly helped by LLMs to design):

    The architecture for my data dashboard

    The health module was the biggest piece by some distance. I exported four years of Fitbit history and imported it alongside live Garmin data, so the trend line runs continuously from March 2022 to today instead of restarting when I changed watches. About 7 million data points. There’s a real gap of six days in January where neither device recorded anything, which is the week I switched. I’ve left it in. It’s honest.

    The finance module pulls positions from my broker and reconciles them against manually entered accounts for the things a broker doesn’t know about — Provident Fund, cash, the rest. Everything now sits behind a single login.

    Three of My Four API Guesses Were Wrong

    This is the part worth writing down.

    I planned the whole thing on paper first (with the help of AI, of course, including this post), including which endpoint each module would call and which fields it would read. When I came to build it, three of the four were wrong.

    The monitoring tool returned a flat structure where I’d assumed a nested one (I needed the LLM to explain this to me too!). The finance app had moved its performance endpoint to a new API version, and the old path returned a 404. The travel app used different field names to what I was expecting.

    None of these would have crashed anything. Each would have quietly rendered an error tile on a dashboard I’d then have trusted. The fix was boring and it worked: read a live response from every endpoint before writing a single line of template code.

    The Service I Decided Not to Build

    The last gap was authentication. Several services were reachable on my network with no login at all, and the obvious answer was to put single sign-on in front of everything.

    So I drafted the whole build. Container, secrets, rollout order, backups. Then I actually tested my assumptions and found that most of the services I’d listed as unprotected were, in fact, protected. The real list was less than half of what I’d written down (i.e. in my issue backlog in my Obsidian Vault). A lot of the services were things I wanted open so I could use them on the fly on an ipad or my phone.

    I closed the gap with tools already running instead. I stopped proxying my router’s admin interface entirely rather than putting a login in front of it, and turned off open registration on one app that had it. That took an afternoon and added nothing new to maintain.

    A new service is a permanent cost. Backups, updates, secrets to store, another thing that breaks at 2am. Since I already stayed up late last night to figure out why Jellyfin was transcoding when no one was using it, I figured “Simplex sigillum veri“.

    What I’d Tell Someone Starting

    Verify before you build. Every assumption I wrote in my .md vault was cheap to check and expensive to get wrong.

    Back up the databases properly. A snapshot of a running database is a snapshot of a database mid-write. Every module here dumps itself fifteen minutes before the nightly backup, and each dump refuses to complete unless it passes an integrity check. A backup you haven’t verified is a hope.

    And scope the problem before you scope the solution. I nearly built an identity provider to solve a problem that turned out to be four services and a router.

  • Two Brains for My AI: How Obsidian and Trilium Split the Work

    I've spent the last few months tinkering with a local AI stack on my desktop — Ollama for local inference, Hermes orchestrating the whole thing, and a Claude subscription for the heavy lifting. It works. But there was always one nagging gap I kept dancing around: where does the AI's memory actually live?

    For a while I went back and forth on whether to use Obsidian or Trilium as the knowledge store. I tried picking one. I tried picking the other. And then it finally clicked that I was asking the wrong question. They're not competitors. They do genuinely different jobs, and once I let them do those different jobs, the architecture got simpler, not more complicated.

    So here's where I landed.

    The Realisation: Two Kinds of Memory

    The thing nobody tells you about "AI memory" is that it's actually two separate problems wearing one coat:

    1. Reference knowledge — the curated, stable stuff. Things I've written down deliberately and want to keep. This is read-mostly. The AI pulls from it, but it shouldn't be scribbling all over it.
    2. Working memory — the daily log. Decisions, scratch notes, "remember I changed X today." This is high-frequency, messy, and constantly being written to by both me and the AI.

    The moment I framed it that way, the tool choice answered itself. I already use both apps for exactly these two purposes, so I'd been overthinking it.

    • Obsidian holds my .md files — my curated reference library. Plain markdown on disk, portable, mine forever.
    • Trilium is where I dump my daily notes. It's a proper database with an API, which turns out to matter a lot for the working-memory job.

    One is the library. The other is the journal. Don't make the library be the journal.

    Why The Split Plays To Each App's Strengths

    This isn't arbitrary. Each app is built for the role I'm giving it.

    Obsidian being plain markdown means anything can read it — Claude Code just cds into the folder, and an in-vault RAG plugin like Smart Connections does local semantic search through Ollama without phoning anywhere. Brilliant for retrieval. But writing back to markdown with an automated agent is fragile — no schema, no permissions, and with files syncing around my network, I'm one race condition away from a merge headache. So I don't let the AI write here. If it produces something worth keeping, I promote it by hand.

    Trilium is the opposite shape. It's a database behind an API (ETAPI), with native MCP support and proper permission control — read-only or read/write, my choice. That makes the agent-writes-back loop actually reliable. Dated entries, structured attributes, an interface Hermes and Claude can both hit cleanly. Exactly what you want for a journal the AI helps maintain.

    So: Obsidian gets read-only treatment because it's my canon. Trilium gets read/write because it's the working log and it's structurally built to handle it.

    The Architecture

    Here's the whole thing on one diagram:

       Obsidian Vault (.md)              Trilium (daily notes / DB)
   reference knowledge               working memory + journal
        │                                   │
        │ READ-mostly                       │ READ + WRITE
        ▼                                   ▼
   ┌─────────────┐                   ┌──────────────┐
   │ Smart Conn. │                   │ Built-in AI  │
   │ + Ollama    │                   │ + ETAPI/MCP  │
   │ Claude Code │                   │              │
   └──────┬──────┘                   └──────┬───────┘
          │                                 │
          └──────────► Hermes ◄─────────────┘
                  (orchestrator: routes
                   reads vs writes)
                          │
                  ┌───────┴───────┐
                  ▼               ▼
               Ollama          Claude
            (local worker)   (escalation)

    The key shift here is that Hermes stops being just an orchestrator and becomes a router. It knows where each operation belongs. The rule set is almost embarrassingly simple:

    • Reads hit both stores — reference from Obsidian, recent context from Trilium.
    • Writes go to Trilium only, scoped to the daily-notes subtree.
    • Obsidian writes are human-gated. Always.

    Underneath both, the same two-tier model I've been running all along: Ollama does the routine local work (free, private, fast), and Claude gets the escalation — the complex reasoning and long-context synthesis that the local models can't touch.

    The Flow That Makes It Worth The Effort

    Running two apps only pays off because of how they feed each other:

    1. Capture. Daily stuff — decisions, meeting notes, half-formed ideas — gets written to Trilium. By me, or by the agent.
    2. Retrieve. When I ask a question, Hermes pulls reference context from Obsidian and recent working memory from Trilium, then stitches them into the prompt. Recent Trilium entries first (what's top of mind), then Obsidian reference matches (the background), then my actual question. Keeps the prompt focused and stops it bloating past the context window.
    3. Promote. Every so often, the good stuff from the Trilium daily log graduates into a proper Obsidian .md note. The journal is ephemeral; the library is forever. This step is manual on purpose — it's the quality gate that keeps my curated vault clean.

    That promotion step is genuinely the whole point. Trilium absorbs all the messy, high-frequency writes — which protects my Obsidian vault from being cluttered up by an over-eager agent — and only vetted, distilled knowledge ever crosses the line into the library.

    A Couple Of Guardrails

    Because I don't trust anything with write access until it's earned it:

    • Trilium's AI features are still experimental, so I started the MCP connection and Hermes both at read-only, and only promoted to read/write once I'd watched them behave for a while.
    • The agent writes into a dedicated subtree — an AI Memory parent note — rather than scattering machine-written content through my real notes. Keeps it quarantined, easy to audit, trivial to roll back if it goes sideways.

    Where This Leaves Me

    No separate vector database, no extra infrastructure to babysit. Obsidian's RAG handles reference retrieval, Trilium's ETAPI handles dated working memory, and each app does the one thing it's actually good at. Hermes routes between them, Ollama does the grunt work, Claude handles the hard problems.

    The lesson, as usual, was that I'd been overcomplicating it. The answer wasn't a clever tool or a new piece of kit — it was just letting two tools I already used keep doing their jobs, and being disciplined about which one the AI is allowed to write to.

    If you've made it this far: yes, I really did spend an unreasonable amount of time deciding where a chatbot is allowed to save its notes. Worth it though.