{"id":497,"date":"2026-08-15T12:12:24","date_gmt":"2026-08-15T12:12:24","guid":{"rendered":"https:\/\/nazeem.me\/?p=497"},"modified":"2026-08-15T12:12:24","modified_gmt":"2026-08-15T12:12:24","slug":"the-layer-the-cable-map-doesnt-show","status":"publish","type":"post","link":"https:\/\/nazeem.me\/?p=497","title":{"rendered":"The Layer the Cable Map Doesn’t Show"},"content":{"rendered":"\n

I’ve already drawn the cables. This is where the packets actually go.<\/em><\/p>\n\n\n\n

Back in May I published the full topology of the house<\/a>: every switch, every 10G run, every zone. That was the easy half. Cables tell you what’s physically possible. They tell you nothing about where a DNS query goes, which door a request came through, or why one machine’s traffic leaves the country and another’s doesn’t. If you go back far enough in this blog, you would have noticed that I’ve been using the Asus RT routers since the original N56U (in 2011!). While they can do most of the basic functions (and some of the advanced ones with the Merlin firmware<\/a>), I’ve taken advantage of my homelab to add on a few more bells n whistles.<\/p>\n\n\n\n

\"its<\/figure>\n\n\n\n
\n\n\n\n

The Router Does Three Separate Jobs<\/h2>\n\n\n\n

Everything starts at an Asus RT-BE88U<\/strong> running Merlin<\/strong> firmware. Stock firmware would route fine. I run Merlin for three features that turned out to be the backbone of everything downstream.<\/p>\n\n\n\n

DHCP with reservations.<\/strong> Boring, until it isn’t. My reverse proxy references backend IPs directly, so a container that gets a new lease silently breaks a service while the proxy keeps happily forwarding to an address nobody’s at. I once had four services sitting proxied-but-unreserved for four days without noticing. Reserve the address before<\/em> you point anything at it.<\/p>\n\n\n\n

DNS Director.<\/strong> This force-redirects port 53 for most client on the LAN to my Pi-holes, so a device that hardcodes 8.8.8.8 gets intercepted and filtered anyway. It works completely, which is the problem. My Pi-holes are LAN clients too. For a while their own upstream queries were being redirected back at the router, so the recursive resolver behind them never reached the root servers at all. Recursion was an illusion. Any host running its own resolver needs to be set to “No Redirection” \u2014 that one exemption is the whole trick.<\/p>\n\n\n\n

VPN Director.<\/strong> Policy-based routing: named clients leave through a commercial VPN, everything else takes the native connection. My desktop is on the list; the Proxmox nodes aren’t. The catch nobody mentions is that it’s IPv4 only. Checked from my desktop while writing this: IPv4 exits at a CDN77 address, IPv6 exits on my actual ISP prefix. Anything that resolves AAAA first walks straight past the VPN with my real address attached. I spent a while treating that as a bug before accepting it as the design.<\/p>\n\n\n\n

\n\n\n\n

DNS: Pi-hole in Front, Unbound Behind<\/h2>\n\n\n\n

Two Pi-holes, both containers on the Proxmox cluster. One is the source of truth, the other a replica, and nebula-sync<\/strong> copies one to the other every fifteen minutes. Two of them because DNS failing takes the whole house with it.<\/p>\n\n\n\n

Behind each Pi-hole sits unbound<\/strong>, doing full recursion out to the root servers. No Cloudflare, no Quad9, nobody upstream keeping a log of what I look up. Pi-hole blocks and forwards; unbound resolves.<\/p>\n\n\n\n

Getting that right cost me about 3,975 SERVFAILs. Unbound validates DNSSEC itself. I had also ticked DNSSEC on in Pi-hole, so every signed lookup got validated twice and the second validator kept rejecting the first one’s work. The fix was one checkbox. Leave Pi-hole’s DNSSEC off when unbound is upstream \u2014 unbound is the validator, and it still rejects bogus signatures without any help.<\/p>\n\n\n\n

\n\n\n\n

Caddy: One Certificate, Thirty-Three Names<\/h2>\n\n\n\n

Everything internal answers on a single wildcard domain, served by a Caddy<\/strong> container on one of the cluster nodes. One Let’s Encrypt certificate, issued over DNS-01 against Cloudflare, which means nothing has to be exposed to the internet to get it<\/strong>. Renewal happens entirely through a DNS record.<\/p>\n\n\n\n

Thirty-three hostnames as of this week. Each is a Pi-hole record pointing at Caddy’s address plus a five-line block in one config file. Adding a service takes two minutes and no new certificate. I tried to get NGINX set up 3 times and failed miserably, so I got my LLM to set up and manage Caddy for me. It worked at the first try!<\/p>\n\n\n\n

The one I got wrong: for eleven days my router’s admin interface was proxied on that domain. The highest-value target on the network, protected by nothing but the router’s own login, reachable from anything that could resolve LAN DNS. I had added it, never written it down, and only found it during an audit. I deleted the entry rather than putting an authentication layer in front of it. Removing the exposure was cheaper than defending it.<\/p>\n\n\n\n

\n\n\n\n

Tailscale: The Private Overlay<\/h2>\n\n\n\n

Tailscale is how I get in from outside \u2014 laptop, phone, ipad, wherever. The interesting part is DNS, because MagicDNS and Pi-hole both want to be the resolver.<\/p>\n\n\n\n

The tailnet uses split DNS: queries for my domain go to both Pi-holes, everything else resolves normally. Those records answer with LAN addresses<\/strong>, not Tailscale addresses, so devices already on the LAN keep working unchanged. Remote devices reach the same addresses through a subnet router advertising the whole \/24.<\/p>\n\n\n\n

That subnet router is currently one node. If it goes down, remote access to the LAN goes with it. I know. A second one is on the list.<\/p>\n\n\n\n

The trap here is subtle and cost me an afternoon. 100.100.100.100<\/code> is a forwarder, not a resolver<\/strong>. Tailscale answers tailnet names and hands everything else to whatever the system’s real nameservers were at the moment it took over. One node had its interfaces bounced during a NIC migration, tailscaled restarted while the config was momentarily empty, and it captured nothing as its upstream. Every public name on that host failed silently for eleven hours. All three nodes had byte-identical resolver config. The state that mattered lived in a backup file that none of the obvious commands show you.<\/p>\n\n\n\n

\n\n\n\n

Cloudflare Tunnel: The Public Door<\/h2>\n\n\n\n

External access runs on a different domain entirely, through a Cloudflare tunnel<\/strong>. No inbound ports open anywhere. Most hostnames sit behind Cloudflare Access; Plex is deliberately bare, because it is built to face the internet and enforces its own login (with 2FA, naturally).<\/p>\n\n\n\n

The two domains differ by a single hyphen. That is a genuinely bad idea I am now stuck with, and I have written down which is which in three separate places. (The one with the hypen is external!)<\/p>\n\n\n\n

The tunnel originally ran one connector, on the NAS, which was also the origin for one of the services it fronted. A DSM update took down the door and one of the rooms behind it, orphaning three perfectly healthy services that had nothing to do with the NAS. There are two connectors now, on separate boxes, serving the same tunnel. I tested it by stopping the first: twelve of twelve requests still served.<\/p>\n\n\n\n

Getting that wrong is easy. A second connector has to use the existing<\/em> tunnel’s token. Clicking “create a tunnel” gives you two tunnels and zero redundancy.<\/p>\n\n\n\n

\n\n\n\n

The Pattern<\/h2>\n\n\n\n

Five systems, and most of the faults above were the same shape: two layers both trying to do one job. Two DNSSEC validators. A router intercepting the resolver it was pointing at. A door mounted on the room it opened onto. Draw the logical layer, not just the cables. You cannot spot a duplicate when you are only looking at one diagram.<\/p>\n\n\n\n

None of them announced themselves. They presented as “the internet is slow”, “flatpak is broken”, “apt can’t reach the repos” \u2014 never as the thing that was actually wrong. Each fix was a single setting. Each one took days to find. Just remember the wise advice of the interwebs: if you have a problem, its always DNS<\/em>.<\/p>\n\n\n\n

Next up for me is figuring out VLANS. My new 10G Managed switch (DXS-F108T<\/a>) just arrived today, so stay tuned!<\/p>\n","protected":false},"excerpt":{"rendered":"

I’ve already drawn the cables. This is where the packets actually go: Merlin, Pi-hole with unbound, Caddy, Tailscale and a Cloudflare tunnel.<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"ai_generated_summary":"","wpai_meta_description":"","footnotes":""},"categories":[5],"tags":[],"class_list":["post-497","post","type-post","status-publish","format-standard","hentry","category-homelab"],"_links":{"self":[{"href":"https:\/\/nazeem.me\/index.php?rest_route=\/wp\/v2\/posts\/497","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/nazeem.me\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/nazeem.me\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/nazeem.me\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/nazeem.me\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=497"}],"version-history":[{"count":6,"href":"https:\/\/nazeem.me\/index.php?rest_route=\/wp\/v2\/posts\/497\/revisions"}],"predecessor-version":[{"id":503,"href":"https:\/\/nazeem.me\/index.php?rest_route=\/wp\/v2\/posts\/497\/revisions\/503"}],"wp:attachment":[{"href":"https:\/\/nazeem.me\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=497"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/nazeem.me\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=497"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/nazeem.me\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=497"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}