Nazeem.Me

A blog about technology, football and all the other random stuff in my life

Tag: proxmox

  • Five Copies, Two and a Half Fault Domains

    3-2-1 is the rule everyone quotes. Three copies of your data, on two different media, with one offsite. It is a good floor and it is easy to think you have cleared it when you have not.

    I counted my copies earlier this year and got five. Primary NAS (NAS1), secondary NAS (NAS2), a cold box, a drive at the office, and OneDrive. Comfortably past the bar. Then I counted what could fail independently and got roughly two and a half.

    Copies are not fault domains

    The problem is that 3-2-1 counts copies, and copies are the wrong unit. What matters is how many distinct ways your data can die, and how many of those each copy actually survives.

    NAS2 replicates from NAS1 continuously. That protects me from a drive failing, or the primary unit dying. It protects me from nothing else. A deletion propagates in seconds. So does ransomware encryption, and so does silent corruption. Always-on replication is an uptime feature wearing a backup’s clothes.

    The cold box is a genuine second copy, but it is reachable during a sync, so a bad actor with the right access reaches it too. OneDrive covers documents and photos and nothing else. Which left the drive at the office as the only copy that survives losing the house, and I refresh it roughly every two months.

    That last number is the whole story. In a site-loss event, my recovery point is whatever date is on that drive. Worst case is eight weeks of loss. Average at a random moment is about a month. My stated goal was “never lose more than a month,” so I was meeting it by luck, with no margin.

    Split the dataset before you fix anything

    The instinct is to back up everything harder. That is expensive and it is the wrong move, because the data is not one thing.

    Roughly 15 TB of mine is media. It barely changes. A copy that is eight weeks old is not stale, because the source has not moved either. The exposure is concentrated in a much smaller slice that changes weekly: documents, photos, project files, configuration, VM state.

    Once you see it that way the strategy splits cleanly. Bulk static data wants cheap capacity and a slow cadence. The changing subset wants automation, frequency, and somewhere the house cannot take with it. Trying to apply one policy to both gets you an expensive plan you will not follow.

    Media: stale is fine

    Media lives on NAS1, replicates to NAS2, and has a third copy on a cold Unraid box. The office drive carries it too, and being two months behind on files that have not changed in two years costs nothing.

    What I will not do is put 15 TB in the cloud. I went looking for a one-time-fee option and there is exactly one permanent-payment provider in the market, Arweave, at somewhere between $294,000 and $455,000 for 15 TB, with data that is public and undeletable by design. Cloud storage is rent. The only genuine one-time payment for bulk storage is buying disks.

    Recurring cold storage is affordable in isolation, roughly $330 a year on OVHcloud Cold Archive with free egress, but retrieval is where archive tiers bite, and for replaceable media it is not worth it.

    The Proxmox cluster

    Proxmox Backup Server runs as a VM on NAS2, holding 30 restore points for every guest in the cluster. My guess before measuring was somewhere in the terabytes. The actual datastore is 242 GB.

    PBS deduplicates at the chunk level, and thirty restore points of slowly-changing Debian containers are almost entirely the same chunks. Capacity turned out to be a non-issue. Which was useful, because it meant the real risk was somewhere else entirely.

    PBS ran on the NAS2 and its datastore lived on that same NAS. One unit failure would have taken the backup server and every restore point for the whole estate in a single event. Not a site-loss scenario, just a one-box scenario, and the only thing standing behind it was an office drive two months stale.

    The second PBS, and why it pulls

    The fix was a second PBS instance on gollum, one of the other nodes, with a 1 TB USB SSD as its datastore. Different room, different UPS, different failure domain.

    Two decisions in there matter more than the hardware.

    It pulls, it does not push. The secondary reaches out to the primary on a schedule and takes what it needs. Nothing on the primary has credentials for the secondary. If the primary is compromised, the attacker gets no path to the copy. A push architecture hands them both.

    Vanished snapshots are not removed. If something deletes backups on the primary, whether that is ransomware or me at 1am, the secondary keeps its own copies until its own retention decides otherwise. A sync that faithfully mirrors deletions is a very efficient way to lose everything twice.

    It syncs at 22:30, prunes at 02:30, and runs garbage collection at 03:30 — deliberately after the prune, so GC actually reclaims what the prune released. Verification runs Sunday mornings. I restore-tested it before trusting it, which is the only reason I am willing to describe it as working.

    A third PBS that goes with me

    I run a third PBS on my laptop (as a docker container), which I usually take with me everywhere. It follows the schedule of the second PBS, but with a 1 hour difference. It backs up to an external USB SSD that I also keep in the office.

    Workstations, and the 465 GB that was actually 19 GB

    The desktops back up nightly to the NAS with restic over NFS, encrypted, with restore tests and a proper systemd timer.

    The useful discovery was the size. My root filesystem showed 465 GB used, which sounded like an expensive backup. The actual backup set is about 19 GB. Steam accounted for 245 GB, caches for 51 GB, and downloaded model weights for another 29 GB. All of it re-downloadable, none of it worth a single byte of backup storage.

    What is left compresses to about 44%. A cold run takes 69 seconds on 10 GbE. An incremental takes under ten. Once you have excluded the things that are large and replaceable, workstation backup stops being a capacity problem and becomes a scheduling one.

    One thing I got wrong and only caught by checking: the timer had never actually fired. Every test I had run was a manual invocation with sudo, so I had proven the command worked and proven nothing about the automation. Verifying through systemd is a different test from verifying the thing systemd runs.

    “If you fail to plan, you plan to fail” 

    Offsite, and why cadence beats discipline

    The office drive is 16 TB, refreshed manually, and honest about it: “roughly every two months” is what a task with no schedule turns into.

    The temptation is to fix that with resolve. Do it monthly. Put it in the calendar. That has never worked for anything and it will not work here, because the two-month drift is the measurement. A manual task on a long rhythm always slips, and the recovery point slips with it.

    The better answer is to stop asking the office drive to be the guarantee. Let it carry the bulk static data, where being weeks behind is genuinely fine, and give the changing subset an automated path that involves no human at all. Two drives rotating instead of one also removes the “the single unverified disk I needed did not spin up” scenario, which is a small change for a meaningful reduction in risk.

    However, with the AI RAM-ageddon spreading to NVME and spinning disks, the cost for a second 16 TB drive is too crazy for me to even consider. So its a risk I’m going to have to put up with.

    damn you AI!

    Cloud: smaller than it sounds

    My cloud leg is Synology Cloud Sync pushing documents and photos to OneDrive. That is it. It is a real offsite copy of the data I would grieve, and it is nowhere near the whole picture.

    Sync is not backup for the same reason replication is not backup, so it depends entirely on whether it runs one-way with destination deletes disabled, and on the version history behind it. Second, the entire Proxmox estate is not in it.

    That has a cheap answer that I have costed and not yet built. 242 GB on Backblaze B2 is about $1.60 a month, no minimum, free egress up to three times what you store. Twenty dollars a year to put every restore point in the cluster somewhere the house cannot reach. It is the single cheapest meaningful upgrade available to me and it is still sitting on the list.

    Two traps I found while costing it, which are worth knowing before you try:

    • Do not point archive tiers at a PBS datastore. Glacier Deep Archive would be 24 cents a month, but PBS churns constantly — garbage collection deletes chunks daily — and a 180-day minimum retention charges you for every deleted chunk for months it did not exist. Archive tiers are for write-once data
    • Do not rclone a chunk store to object storage. It is millions of small files. A naive sync means millions of API calls and never finishes cleanly. Use PBS remote sync to a second PBS, or snapshot the filesystem and back up the snapshot

    What I decided not to do

    PBS client-side encryption is off, deliberately. The datastore never leaves hardware I control, and I judged the NAS secure enough for my threat model. The consequence, which I wrote down rather than left implicit, is that backups are readable at rest — including a secrets manager’s encryption key. That is an accepted risk, not an oversight, and the distinction only exists because it is recorded somewhere I will find it again.

    I would rather have three things I have restore-tested than six I have merely configured. An untested backup is a hope with a cron entry.

    Where it actually stands

    The one-box risk is closed. The cluster has two independent PBS instances in different rooms on different power (and separate UPS’s), with a pull-based sync and a restore test behind it. Workstations back up nightly and have been restored from. Media has three local copies and a fourth at the office.

    The gap that remains is the same one I started with, just smaller and better understood: everything except documents and photos depends on a manual drive with a two-month rhythm. Twenty dollars a year fixes the important half of that.

    3-2-1 was never the goal. It is a way of asking a better question, which is: name every way this data can die, and for each one, point at the copy that survives it. Do that honestly and the number of copies stops mattering; the number of answers is the only thing that does.

  • I Built a Dashboard for the Things I Actually Check

    Five separate logins became one page. The interesting part was how often I turned out to be wrong.

    I check five things regularly: whether the homelab is healthy, my portfolio, my health trend, whatever trip is coming up, and my to-do list. Each one lived somewhere different. Each had its own login. None of them talked to each other. I have to go looking for information by opening a browser, going to my bookmarks page and finding the right tile to click:

    My Heimdall Dashboard of links to my self-hosted services.

    Information you have to go and fetch is information you check less often than you should.


    What It Looks Like Now

    One page. Five modules, each running in its own container, pulled together by a dashboard shell called Glance. The shell is configuration, not a product — it reads each service’s API and renders the numbers. That’s the whole trick (admittedly greatly helped by LLMs to design):

    The architecture for my data dashboard

    The health module was the biggest piece by some distance. I exported four years of Fitbit history and imported it alongside live Garmin data, so the trend line runs continuously from March 2022 to today instead of restarting when I changed watches. About 7 million data points. There’s a real gap of six days in January where neither device recorded anything, which is the week I switched. I’ve left it in. It’s honest.

    The finance module pulls positions from my broker and reconciles them against manually entered accounts for the things a broker doesn’t know about — Provident Fund, cash, the rest. Everything now sits behind a single login.


    Three of My Four API Guesses Were Wrong

    This is the part worth writing down.

    I planned the whole thing on paper first (with the help of AI, of course, including this post), including which endpoint each module would call and which fields it would read. When I came to build it, three of the four were wrong.

    The monitoring tool returned a flat structure where I’d assumed a nested one (I needed the LLM to explain this to me too!). The finance app had moved its performance endpoint to a new API version, and the old path returned a 404. The travel app used different field names to what I was expecting.

    None of these would have crashed anything. Each would have quietly rendered an error tile on a dashboard I’d then have trusted. The fix was boring and it worked: read a live response from every endpoint before writing a single line of template code.


    The Service I Decided Not to Build

    The last gap was authentication. Several services were reachable on my network with no login at all, and the obvious answer was to put single sign-on in front of everything.

    So I drafted the whole build. Container, secrets, rollout order, backups. Then I actually tested my assumptions and found that most of the services I’d listed as unprotected were, in fact, protected. The real list was less than half of what I’d written down (i.e. in my issue backlog in my Obsidian Vault). A lot of the services were things I wanted open so I could use them on the fly on an ipad or my phone.

    I closed the gap with tools already running instead. I stopped proxying my router’s admin interface entirely rather than putting a login in front of it, and turned off open registration on one app that had it. That took an afternoon and added nothing new to maintain.

    A new service is a permanent cost. Backups, updates, secrets to store, another thing that breaks at 2am. Since I already stayed up late last night to figure out why Jellyfin was transcoding when no one was using it, I figured “Simplex sigillum veri“.


    What I’d Tell Someone Starting

    Verify before you build. Every assumption I wrote in my .md vault was cheap to check and expensive to get wrong.

    Back up the databases properly. A snapshot of a running database is a snapshot of a database mid-write. Every module here dumps itself fifteen minutes before the nightly backup, and each dump refuses to complete unless it passes an integrity check. A backup you haven’t verified is a hope.

    And scope the problem before you scope the solution. I nearly built an identity provider to solve a problem that turned out to be four services and a router.

  • Home Network Topology 2026: The Full Map

    Between the 2024 network upgrade post and the Homelab 1.0 post, I’ve talked a lot about individual components — the 10G switches, the Proxmox cluster, the NAS stack. But I’ve never actually drawn the full picture of how everything connects. Partly because it was always evolving, and partly because I suspect I was slightly embarrassed about how complicated it had gotten.

    Well, here it is.

    Fair warning: it’s a lot.


    The Spine

    Everything in the house hangs off two devices: an Asus RT-BE88U running Merlin firmware as the main router, and a TP-Link TL-SX1008 8-port 10G switch that acts as the core distribution layer. The router connects to the switch via one of its 10G ports, and from the switch, a single 10G run goes to each room. This is the backbone that all four zones of the house branch off from.

    A quick correction from my 2024 post, where I said the house was running Cat 5E — that was wrong. It’s actually Cat 6 throughout. Apologies for the confusion. Cat 6 is rated for 10G up to 55 metres, so the 10G speeds across the house make complete sense, and I can stop sounding like I’m living dangerously.


    Media Console

    The media console area is the most “legacy” part of the network. It uses a 1G switch to service the TV, Xbox 360, Xbox One X, Nintendo Switch, Raspberry Pi 5 (running OpenElec), Denon AVR, and the HTPC. None of these devices need more than gigabit, so there’s no point in running 10G cabling to each of them. The 1G switch itself connects back to the main 10G switch, so aggregate throughput is fine.

    The one exception in this zone is the Proxmox test node, which gets a direct 10G connection from the main switch. I use this node to experiment with new configs before rolling anything out to the main cluster in the study room — having it on full-speed connectivity matters when I’m moving VM images around.


    Bedrooms

    Bedroom 1 is simple: a direct 10G connection straight to the gaming PC. No middlemen.

    Bedroom 2 has a second RT-BE88U running in AP mode, which provides Wi-Fi coverage to that end of the house. It also has the TV and Denon AVR wired into it directly. I previously ran a GT-AX11000 Pro in this role, but after the primary router upgrade, having two RT-BE88U units running Merlin in AiMesh mode has made the whole setup much more coherent.

    One thing I’m looking forward to exploring with the RT-BE88U is Guest Network Pro, which allows each wireless SSID to be assigned to its own VLAN. This means I could eventually segment the IoT and media devices in this zone — the TV, the Denon — onto their own isolated network, completely separate from the rest of the house. It’s the kind of proper network segmentation I’ve been putting off, and having hardware that natively supports it on both the primary router and the AP removes the main excuse I had for not doing it.


    Study Room: Where It Gets Complicated

    My homelab in a rack

    This is where most of my infrastructure lives, and the network reflects that.

    A TP-Link 5-port 10G switch sits at the top of the study room hierarchy, connected back to the main 10G switch. From there, things fan out:

    • DS920+ NAS and ZimaOS NAS each get a direct 10G connection from the 5-port switch. These are the primary working NAS devices, and they need fast access to the rest of the network.
    • An 8-port 10G switch provides 10G connections to the workstations: DS1621+ NAS, Unraid NAS, Windows 11 desktop, Mac Mini M4, and CachyOS desktop. Everything in this group transfers data at full 10G speeds, which makes a real difference when shuffling large files to and from the NAS.

    The interesting bit is the Proxmox cluster networking, which has two separate paths:

    The 8-port 2.5G switch handles the data network. Each of the three Proxmox nodes connects with two bonded 2.5G ports — so each node has an effective 5G data link for VM traffic and storage replication.

    The 8-port 1G switch handles the management network. Each Proxmox node has a dedicated 1G management interface, completely separate from the data path. The DS920+ gets two bonded 1G ports through this switch (for its management/backup traffic), and the ZimaOS NAS also has a 1G management link here. Keeping management traffic off the data network is something I should have done from the start — it makes the cluster much more stable and gives me a reliable out-of-band path when something inevitably breaks.


    What’s Next

    The topology is largely stable at this point, but there are a few things I’m considering:

    The 8-port 1G switch has a few spare ports that I’d eventually like to use for dedicated IPMI/iDRAC-style out-of-band management on the NAS devices — the Synology units support this to a degree, but I haven’t gotten around to configuring it properly.

    And with Cat 6 confirmed throughout the house, at least I can stop worrying about the cabling holding back my 10G links — that particular anxiety has been retired.

    If you’ve made it this far, congratulations — you now know more about my home network than most people probably want to. The diagram is linked above if you want to zoom in on any particular section.