The false comfort
My guests had four tiers of backup. Every container and VM in thefellowship, my three-node Proxmox cluster, went to a Proxmox Backup Server (in a VM on a NAS) every night. That copy was synced to a second PBS on a standalone box called gollum. A weekly copy went to a NAS share, and an office drive got a manual refresh now and then.
So when I asked myself a simple question, I expected a simple answer. If the house burned down tomorrow and I had new hardware, could I rebuild the lab from my notes and backups alone?
No. The guests would come back, but nothing they ran on would.
A guest backup is the payload. It assumes a platform underneath: hosts with the right storage names, bridges and VLANs, a router handing out the right addresses, switches trunking the right tags, a NAS exporting the right shares. None of that was in a backup. Most of it lived in my head, or in notes I’d written months earlier and never checked against the real thing.
The audit: one row per component
I started with a table. One row per thing that would have to exist again, and two columns: what would restore this, and is that true today.
| Component | Rebuild source before | Reality |
|---|---|---|
| 3 cluster hosts + gollum | notes | no backup of /etc or the cluster database at all |
| Primary PBS (a VM on the DS1621+) | nothing | its own config was never written down |
| Router (ASUS on Merlin) | old notes | no config export, six containers with no DHCP reservation |
| Core 10G switch | rebuild sheet | fine |
| Two small managed switches | nothing | VLAN layout existed only on the switches |
| Two Synology NAS | DSM auto-backup | backup jobs half-configured (more below) |
| Unraid cold box | array notes | no flash backup |
| Every guest | four tiers | never restore-tested |
The method matters more than the table. I didn’t write the rows from memory. I read every value off the live systems, over SSH and through each device’s web API, and compared it with the notes. That’s how I found the drift: guests still listed on a node I’d retired three weeks earlier, a reverse-proxy table that no longer matched the proxy, and IP addresses that had quietly changed.
Closing the gaps, layer by layer
Hosts: back up the platform, not just the payload
Proxmox doesn’t back up its own hosts. The cluster config lives in /etc/pve, which is a FUSE view of an SQLite database at /var/lib/pve-cluster/config.db. Lose a node without a copy and you’re rebuilding storage definitions, network bridges, firewall rules and backup jobs by hand.
The fix is a small script on each node, run by a systemd timer at 21:30. It takes a consistent dump of config.db with sqlite3 .backup, along with the package selection, disk layout, LVM and network state. Then it ships /etc, /root, /usr/local and that state folder to PBS as a host-type backup, in its own hosts namespace, kept for 7 days, 4 weeks and 6 months.
Two things cost me time:
/etc/pveis skipped unless you ask.proxmox-backup-clientdoesn’t cross into other filesystems, and pmxcfs is one. It needs--include-dev /etc/pve.- Exclude patterns are relative to each archive’s root, not the host’s filesystem.
--exclude /etc/lvm/archivematches nothing. Inside theetc.pxararchive, the path is/lvm/archive.
Network gear: export everything, and treat it as a secret
The router got a settings export plus a tarball of /jffs, where my firewall and DNS hook scripts live. I also added DHCP reservations for six containers that were running on dynamic leases while other config pointed at them by IP.
Each switch got a config export and a rebuild sheet written from what the switch actually reported: VLAN IDs, port modes, PVIDs and the management VLAN. The order of steps matters on these small switches. On the TP-Link, setting the management VLAN before moving the static IP strands you outside it.
One warning: these exports are not harmless. The TP-Link’s file is a 16-byte header followed by plain JSON, and it includes the admin account section. The D-Link’s binary export holds credentials too. They went into the same private folder as everything else that unlocks the lab.
Reading the TP-Link also showed that loop detection had never been turned on. Its sixth port feeds an unmanaged switch behind the TV, which is exactly where someone plugs both ends of a cable in. It’s on now, with the uplink port excluded so the switch can’t block itself.
NAS: config, keys, and a backup that was never scheduled
The Synology audit found four problems:
- The PBS VM on the DS1621+ didn’t autostart. After a power cut, the primary backup server would simply stay off.
- The DS1621+’s Hyper Backup job ran manually, last on 31 October. I replaced it with an encrypted daily job to the DS920+ with Smart Recycle retention.
- One share had 64 weekly snapshots kept forever. I cut it to 4.
- A share was still exported over NFS that nothing used any more.
I also exported each unit’s DSM configuration and the keys for the encrypted shares. Without the keys, an encrypted share on a rebuilt NAS is just noise.
Cold storage and the backup server itself
The Unraid box got a flash backup, the USB stick that holds its entire config and licence, both as a local zip and through Unraid Connect. Unraid Connect’s copy is not encrypted, so the local zip is the one I trust.
The primary PBS got a rebuild sheet: datastore, users, tokens, ACLs, jobs and the three places its fingerprint is pinned. Rebuilding it means a new VM, a re-seed from gollum’s copy, and updating that fingerprint on three consumers.
Docs generated from live state
The rebuild sheets aren’t prose. A collector script pulls host and router config into dated folders with secrets excluded, and the sheets are written from that output. When something changes, I re-run the script and diff. The difference between a note and a rebuild sheet is that the sheet was true on a date you can name.
The firewall change that broke a backup
The audit’s best lesson came from a change that should have been routine. I narrowed the DS920+ firewall so the Hyper Backup Vault port only accepted the DS1621+. The next backup failed.
The firewall was right. The traffic was wrong. On the DS920+’s side, the connection wasn’t arriving from the DS1621+ at all. It was arriving from nucbox, one of my Proxmox nodes. The DS1621+ ran Tailscale with accept-routes on, and nucbox advertises my LAN as a subnet route. So a NAS sitting on the same switch as its backup target was sending its backup out through the tailnet, into a container, and back onto the LAN. Conntrack on nucbox showed the flow.
tailscale set --accept-routes=false on the DS1621+ fixed it, and the rule now is that nothing on the LAN itself accepts the LAN route. The point is broader, though. I’d have sworn I knew how that backup travelled. I only found out when something cut the path.
Offsite, and the bootstrap problem
Everything that unlocks the lab now sits in one private OneDrive folder: the router export, the DSM configs, the switch exports, the share keys, the Hyper Backup key, the Unraid flash zip, and two zips of the rebuild notes and config snapshots.
The harder problem is circular. Two of my three password stores run on the cluster I’d be rebuilding. In a total loss, the first restore has to be the password manager, before I can log in to anything except PBS and the hypervisors. So the credentials for those two have to live outside the lab. The restore drill below turned up a twist: the self-hosted vault I’d listed as primary hasn’t had an item change since August. It isn’t the one I actually use every day, and my rebuild order said otherwise.
Backups that report their own failures
A backup that fails silently is worse than none, because you stop worrying. Every job here, including the new host backups, stamps a heartbeat on a small server in the lab when it finishes. Each sender has its own SSH key, locked to a forced command that can only write that sender’s stamp, and only from that sender’s addresses. A daily check reads the stamps against a maximum age and messages me only when something is late or failed. When everything is fine, it says nothing.
The first restore I ever tested
Backups are a hope until you’ve restored one, and I never had. So gollum now runs a drill. It restores a guest from its own copy of the backups onto a bridge with no physical ports, no host address and IPv6 disabled. Then it checks that the application works, not just that the container boots, records the numbers and destroys the copy.
Before any of that, I restored a harmless container and tried to get out. The router, the host, the internet and the PBS API were all unreachable. Good, because the restore had already gone wrong in a way I didn’t expect: pct restore --net0 is silently ignored. The container came back attached to my real LAN bridge with DHCP, exactly as it was backed up. It hadn’t been started yet, so nothing happened. If it had been my agent container, which holds SSH keys to every host and a Telegram bot, a careless drill would have put a second copy of it on the network. The drill script now moves the network to the isolated bridge with pct set before the first start, and refuses to start anything whose config still names another bridge.
The first full run, restoring the five guests I’d most hate to lose:
| Guest | Backup age | Restore | App healthy | Data check |
|---|---|---|---|---|
| Vaultwarden (passwords) | 18.6 h | 12 s | 21 s | SQLite integrity ok, 921 items, same as live |
| Obsidian LiveSync (CouchDB) | 18.6 h | 22 s | 30 s | 73 MB of vault shards |
| Infisical (service secrets, Docker) | 18.7 h | 119 s | 146 s | 29 secrets, 3 projects |
| Ghostfolio (portfolio, Postgres) | 18.7 h | 46 s | 61 s | 76 orders, 14 accounts |
| Hermes (my agent, 24 GB disk) | 18.7 h | 192 s | 205 s | 7 scheduled jobs, heartbeat job present |
All five passed. The slowest took three and a half minutes from “start restore” to a working app, and most of that was copying the disk. The backups were 18–19 hours old, because the restore came from the nightly copy. That’s the real recovery point, not the one I’d have guessed.
I also broke it on purpose. Pointing the probe at a wrong port made the drill fail, and the next heartbeat check reported it. A drill that can’t fail proves nothing.
Each check is about the data, not the port. For Vaultwarden, the drill stops the service so SQLite flushes its log, copies the database out and runs an integrity check. The restored copy held the same 921 items as the live one. For Infisical, it’s a row count from inside the Postgres container, which also proves Docker inside an unprivileged LXC survives a restore.
From now on, gollum drills two of those five on the first Sunday of every month, so each one gets tested every quarter, and the result goes to the same heartbeat.
What I’d tell anyone with a homelab
- List components, not backups. For each one, write down what would restore it. “Notes” doesn’t count unless they were checked against the live system recently.
- Back up the hosts. The hypervisor’s own config is the thing nobody backs up.
- Export network config and treat it as a secret.
- Keep the keys outside the thing they unlock: share keys, backup encryption keys, and the password manager’s way back in.
- Make every job report, and stay quiet when it’s fine.
- Restore something. On an isolated network, with an app-level check. You’ll find out things your backups never told you.