Homelab hardening, step by step: the commands

The exact audit I ran on my own homelab, in order, with every command you need to run it on yours.

In the first post I covered what to look for and in what order. This one is the toolkit. Every command below is one I ran, adjusted only to strip out my own addresses.

A few assumptions before we start:

Your client is a Linux desktop. Mine is the CachyOS machine I wrote about in Eight Months on CachyOS, and my shell there is fish. So every command you type on your own machine is fish syntax. If you use bash, the main differences are for x in a b; do ...; done instead of for x in a b; ...; end, and $(cmd) instead of (cmd).
Your servers are Debian or Ubuntu, or Proxmox, which is Debian underneath. Anything inside quotes after ssh root@host runs in the server’s shell, which is bash. That’s why those parts look different.
You have SSH keys set up to your servers, and one machine outside your network: a cheap VPS is ideal.

Give it an afternoon. Steps 1 to 3 are the ones that matter most.

Before you start: set up a way back

You’re about to change SSH and firewall settings on machines you reach over SSH. Get this wrong and you lock yourself out. Two habits stop that from happening.

Keep two terminals open. Make changes in one, test in the other, and don’t close the first until the second works.

Arm an automatic revert before every risky change. On the server:

systemd-run --on-active=300 --unit=ssh-rollback /bin/sh -c "rm -f /etc/ssh/sshd_config.d/10-hardening.conf; systemctl reload ssh"

That undoes the change in five minutes unless you cancel it. Check it’s armed, then cancel it once you’ve tested:

systemctl list-timers --all | grep ssh-rollback
systemctl stop ssh-rollback.timer

Use the same pattern for firewall changes. Point the command at whatever puts things back the way they were.

Step 1: see what the internet sees

List your forwards and UPnP mappings

Your router’s admin page will list port forwards (often under “Virtual Server” or “NAT”). Write each one down with what it points at. Check UPnP while you’re there: if it’s on, any device on your network can open ports for itself.

If your router lets you SSH in (Asuswrt-Merlin, OpenWrt, pfSense and others do), read the rules directly. They’re the truth, while the web UI is a summary:

ssh admin@192.168.1.1 'iptables -t nat -S | grep DNAT'

Any forward pointing at a machine that no longer exists should go today.

Scan from outside

From your VPS, scan your home IP:

nmap -Pn -sT --top-ports 1000 --open <your-home-ip>

My VPS didn’t have nmap and I didn’t want to install it, so I used nc from there instead. The VPS shell is bash:

for p in 22 80 443 8080 8443 32400; do nc -z -w3 <your-home-ip> $p && echo "open $p"; done

Add the ports from your forward list to that loop.

Check the scan really came from outside

This tripped me up. My client sends its internet traffic through a VPN, so I assumed a scan of my home IP from my desktop would look like an outside scan. It doesn’t: the router recognises its own address and answers from inside. The result looked plausible, and it was completely wrong.

Two checks tell you which side you’re on. First, where your traffic exits:

curl -s -4 https://ifconfig.co

Second, latency in the nmap output. “Host is up (0.00082s latency)” means the packets never left the building. A real outside scan to a home connection shows milliseconds, not microseconds.

Check your tunnel hostnames are actually gated

If you publish services through Cloudflare Tunnel or similar, check every hostname sends you to a login rather than straight to the app. On your client, in fish:

for h in app1 app2 app3
    printf '%-10s %s\n' $h (curl -s -o /dev/null -w '%{http_code} %{redirect_url}' https://$h.example.com/)
end

A 302 to your identity provider is what you want. A 200 means that app faces the internet with only its own login in front of it, so make sure that’s deliberate.

Step 2: map the inside

Discover hosts

Run one subnet at a time, including any VLANs your client can route to:

for net in 192.168.1.0/24 192.168.30.0/24
    echo "$net: "(nmap -sn $net -oG - | grep -c 'Status: Up')" hosts up"
end

Save the live addresses for the next step:

nmap -sn 192.168.1.0/24 -oG - | awk '/Up/{print $2}' > hosts.txt

Scan services politely

nmap -sT -sV --version-light -T3 --top-ports 1000 --open -iL hosts.txt -oA lanscan

-sT doesn’t need root, --version-light keeps the probes gentle, and -oA saves the results in three formats so you don’t have to scan again. Expect a few minutes for 50 hosts. Leave camera VLANs and switch management networks out of the scan, or scan them separately with fewer ports. Cheap embedded devices are the most likely thing to fall over.

Turn the output into a readable table

The raw nmap output is long. This prints one line per host with its open ports:

awk '/Nmap scan report/ {if (h) print h": "p; h=$NF; p=""} $2=="open" {p=p" "$1} END {print h": "p}' lanscan.nmap

Now read every line and ask the same two questions: should this be listening at all, and does it need a password?

Probe for services that don’t ask for a login

For anything that looks like a database, dashboard or API, try it without credentials:

for u in http://192.168.1.20:8086/query http://192.168.1.21:5984/_all_dbs http://192.168.1.22:8082/
    printf '%-42s %s\n' $u (curl -sk -o /dev/null -w '%{http_code}' --max-time 5 $u)
end

How to read the result:

401 or 403: it asked for credentials. Good.
200: it answered without asking. Open it in a browser and see what a stranger would see.
000: nothing answered, or a firewall dropped it.

This is the step where I found a dashboard with its login switched off, and a Python file server I’d started months ago and forgotten. It was still serving a folder to the whole network.

Step 3: lock down SSH everywhere

Audit first

See what each server is actually running with. Loop over them from your client:

for h in pve1 pve2 pve3 nas-box
    echo "== $h"
    ssh root@$h 'sshd -T | grep -E "^(permitrootlogin|passwordauthentication|maxauthtries) "'
end

permitrootlogin yes together with passwordauthentication yes is the combination to fix. Before you turn passwords off, check whether anything still uses them:

ssh root@pve1 'journalctl --since -30d -o cat | grep -oE "Accepted [a-z]+ for [a-z]+ from [0-9a-f.:]+" | sort | uniq -c'

If every line says publickey, nobody will notice the change. If you see password, find out who that is first.

Apply the drop-in under a rollback

Write the file locally once:

printf '%s\n' 'PermitRootLogin prohibit-password' 'PasswordAuthentication no' 'KbdInteractiveAuthentication no' 'MaxAuthTries 3' > 10-hardening.conf

(Fish has no heredocs, so printf with one argument per line does the job.) Then, for each server: copy it, arm the rollback, validate, and reload:

scp 10-hardening.conf root@pve1:/etc/ssh/sshd_config.d/
ssh root@pve1 'systemd-run --on-active=300 --unit=ssh-rollback /bin/sh -c "rm -f /etc/ssh/sshd_config.d/10-hardening.conf; systemctl reload ssh"; sshd -t && systemctl reload ssh'

sshd -t checks the config before the reload, so a typo can’t take sshd down.

Test properly

From your second terminal, check that key login works:

ssh -o BatchMode=yes root@pve1 hostname

Then check that password login is refused:

ssh -o PubkeyAuthentication=no -o PreferredAuthentications=password -o BatchMode=yes root@pve1 true

You want Permission denied (publickey): the server is no longer even offering passwords. If both checks pass, cancel the rollback timer.

One thing that will confuse you: Host key verification failed is not an authentication problem. It happens before any login attempt, and it means the client doesn’t recognise the server’s key. It has nothing to do with the change you just made.

fail2ban, then prove it counts

ssh root@pve1 'apt-get install -y fail2ban'

Give it a jail.local with your own machines in ignoreip, so a mistyped key doesn’t ban your own desktop or another cluster node:

[DEFAULT]
backend  = systemd
bantime  = 1h
findtime = 10m
maxretry = 5
ignoreip = 127.0.0.1/8 ::1 192.168.1.10 192.168.1.11 192.168.1.12

[sshd]
enabled = true

Now prove it sees failures. Newer OpenSSH versions log through a process called sshd-session rather than sshd. A jail that misses those lines sits there forever reporting zero, which looks exactly like a quiet network. From any machine that is not in ignoreip, make one failed login:

ssh -o BatchMode=yes -o PubkeyAuthentication=no nobody@pve1 true

Then check the counter went up:

ssh root@pve1 'fail2ban-client status sshd | grep "Total failed"'

If it’s still 0, the jail is blind. Fix that before you trust it.

Step 4: fence off what can’t log in

Some services have no authentication, or only weak authentication you can’t change. Don’t leave them open to the whole network. Allow only the machines that actually use them.

Find out who that is first. On the server running the service:

ss -tn state established '( sport = :8000 )'

Check its access log over a few days too. A client that connects once a week won’t show up in a single ss snapshot.

On a plain Linux server: ufw

ufw default deny incoming
ufw allow from 192.168.1.10 to any port 8000 proto tcp
ufw allow from 192.168.1.11 to any port 8000 proto tcp
ufw allow 22/tcp
ufw enable

On Proxmox: a guest firewall file

If the service runs in a Proxmox container or VM, use the built-in firewall instead. It’s enforced on the host, so a compromised guest can’t switch it off. Create /etc/pve/firewall/<vmid>.fw on the node:

[OPTIONS]
enable: 1
policy_in: DROP
policy_out: ACCEPT

[IPSET allowed_clients]
192.168.1.10
192.168.1.11

[RULES]
IN ACCEPT -source +guest/allowed_clients -p tcp -dport 8000 -log nolog
IN ACCEPT -source 192.168.1.0/24 -p tcp -dport 22 -log nolog

Then turn the firewall on for the guest’s network card. Copy your existing net0 line exactly and add firewall=1. If you leave out the hwaddr, the guest gets a new MAC address and probably a new IP. Arm a rollback first:

ssh root@pve1 'pct config 123 | grep ^net0'
ssh root@pve1 'systemd-run --on-active=300 --unit=fw-rollback pct set 123 -net0 name=eth0,bridge=vmbr0,hwaddr=BC:24:11:00:00:01,ip=dhcp,type=veth'
ssh root@pve1 'pct set 123 -net0 name=eth0,bridge=vmbr0,firewall=1,hwaddr=BC:24:11:00:00:01,ip=dhcp,type=veth'

If the guest uses DHCP, add dhcp: 1 under [OPTIONS]. Otherwise it can’t renew its lease.

Verify from both sides

Check from an allowed client and from one that should be blocked:

curl -s -o /dev/null -w '%{http_code}\n' --max-time 5 http://192.168.1.50:8000/
ssh root@some-other-box 'curl -s -o /dev/null -w "%{http_code}" --max-time 5 http://192.168.1.50:8000/'

You want a real status code from the first and 000 from the second. Only then cancel the rollback.

Step 5: patch, including the boxes you forget

Hosts

Preview before you install. -s simulates the upgrade:

for h in pve1 pve2 pve3
    echo "$h: "(ssh root@$h 'apt-get update -qq; apt-get -s dist-upgrade | grep -c ^Inst')" pending"
end

Every container on a Proxmox node

Nested quoting through fish, ssh and bash gets ugly fast. Put the loop in a small script and pipe it in instead. Save this as ct-check.sh:

for id in $(pct list | awk 'NR>1 && $2=="running" {print $1}'); do
  n=$(pct exec "$id" -- sh -c 'apt-get update -qq >/dev/null 2>&1; apt-get -s dist-upgrade | grep -c ^Inst')
  echo "CT $id: $n pending"
done

Then run it on each node:

ssh root@pve1 bash -s < ct-check.sh

Kernels and reboots

A new kernel does nothing until you reboot. Compare what’s running with what’s installed:

ssh root@pve1 'uname -r; ls /boot/vmlinuz-* | sort -V | tail -1; ls /var/run/reboot-required 2>/dev/null'

Services that didn’t restart themselves

Some packages update the binary but leave the old process running. I caught a Cloudflare tunnel connector doing exactly this: the package had upgraded, but the running process was still the old version. Look for processes whose program file has been replaced:

ssh root@host 'ls -l /proc/*/exe 2>/dev/null | grep deleted'

Anything listed is still running old code. Restart that service.

Firmware

There’s no command for this one. Log in to your router and NAS, check the update pages by hand, and put a recurring reminder in your calendar.

Step 6: segment, and restrict the overlay VPN

Check that a VLAN really can’t reach the main LAN

Rules on paper aren’t proof. From a device on the IoT or guest network, try to reach something on your main LAN:

nc -zv -w3 192.168.1.10 22

It should time out. If your router has an interface on every VLAN, it’s the easiest place to run this test from.

Tailscale access rules

The default policy allows everything to reach everything. Tag your internet-facing server and give it nothing, then add a tests block. Tailscale checks the tests when you submit the policy and rejects any change that breaks them:

{
  "hosts": {"nas-box": "100.101.102.103", "pve1": "100.101.102.104"},
  "tagOwners": {"tag:public": ["autogroup:admin"]},
  "grants": [
    {"src": ["autogroup:member"], "dst": ["*"], "ip": ["*"]}
  ],
  "tests": [
    {"src": "you@example.com", "accept": ["tag:public:22"]},
    {"src": "tag:public", "deny": ["nas-box:445", "pve1:22"]}
  ]
}

Members can still reach the public server, but the public server can’t reach anything. The tests prove both halves.

Step 7: write it down

Keep one note per audit. A table is enough:

Finding	Severity	Fix	Status
Root password SSH on 2 hosts	Medium	Drop-in + fail2ban	Fixed, tested
Dashboard with no login	Medium	Guest firewall allowlist	Fixed, auth pending
Plain HTTP service on LAN	Low	Accepted: LAN-only, no secrets in transit	Review next quarter

The last column is the useful one. “Accepted” with a reason stops you re-investigating the same thing next time.

Run it again

Repeat Steps 1 to 3 every quarter, and every time you add a new box. That’s the gap that got me: I’d done all of this once, then added two servers and never went back. Hardening doesn’t stay done on its own.