What I found when I finally counted the SSH keys I’d handed out to an AI agent.
I opened authorized_keys on my Proxmox cluster expecting a short list. What I got was seventeen non-blank lines, the same key pasted eleven times, and one line reading PASTE_YOUR_PUBKEY_HERE that had been sitting there long enough that I have no idea when it arrived.
Nothing was broken. Everything worked. That’s the problem with credential sprawl — it doesn’t announce itself, and nothing in your day prompts you to go and look.
What was actually in the file
Deduplication took seventeen lines down to eight distinct keys. Pruning grants that no longer had an owner took it to five. Adding one legitimate new key later brought it to six.
It is seven now. The seventh arrived after this audit and it is the most interesting line in the file, so I will come back to it.
Of those six, three were the cluster’s own node-to-node root keys, which have to be there. The other three were keys I’d generated so an AI coding agent could reach the cluster from three different machines: my desktop, a Windows workstation, and a laptop that travels.
Every one of them was unrestricted root on all three nodes. No source restriction, no forwarding restrictions, no expiry.
One of them also held root inside a container that stores an encryption key. I’d revoked a different key months earlier believing that closed the exposure. It hadn’t. The container was reachable through a door I hadn’t counted.
Count the logins before you touch anything
The instinct with a key you don’t recognise is to delete it. Resist that for one afternoon and pull thirty days of auth logs first (using AI, of course!), because “is this safe to remove” is not answerable without usage data.
The busiest of the three had 2,051 logins in the thirty days to 24 August — 1,618 from the LAN, 433 over the tailnet, and 439 on a single day. That is a live, daily dependency. Break it and I’d know within the hour.
That number is also worth reading twice. When I first ran this audit it was 919. It has more than doubled since, without me deciding anything, because the agent got more useful and I reached for it more. Credential sprawl is not the only thing that grows quietly.
The quietest had 77 logins across four scattered days, and had been dormant for over a week. That one looked like an obvious deletion candidate.
It has not been used since. The count is still exactly 77, the last login was on 5 August, and as I write this it has been silent for nineteen days.
It wasn’t.
The key named after a laptop that never used it
All 77 of those logins came from a single source address. That address belongs to one of the cluster nodes.
The key is named after the travelling laptop. The laptop had never used it. Not once in thirty days.
So a root key, named after one machine, was being used exclusively from a different machine, and I couldn’t say why. I spent an evening trying to close that gap and failed:
- The private key isn’t on that node’s filesystem. I (ok, Claude) searched every file under 20 KB containing
PRIVATE KEYacross the whole disk and fingerprinted each match. No hit. - Agent forwarding is disproved. If the laptop had hopped through that node carrying its key with
-A, the laptop would appear in the node’s own auth log for those dates. It doesn’t. In the relevant window the node accepted logins from exactly two sources, and neither was the laptop. - The one
ssh-agenton the box isn’t it either. It runs as a container’s mapped user rather than a host user, and it started four days after the last use of the key.
Which left me with a dormant unrestricted root key, in active-enough use that something clearly depends on it, and no identified caller.
I scoped it instead of deleting it. That is not the satisfying answer, and I want to be clear it’s a holding position rather than a fix — a root key with no known owner is not a good resting state. But deleting a dependency you can’t identify is how you discover what it was, usually at the worst possible moment, and scoping was non-breaking because the mystery caller already sits inside the range I was restricting to.
Find the caller, then delete it. That’s still on the list.
The standard I applied
Every key now carries this prefix:
from="192.168.1.0/24,100.64.0.0/10",restrict,pty ssh-ed25519 AAAA...restrict is the useful one and it’s newer than most of the advice you’ll find. It disables port forwarding, agent forwarding, X11 forwarding and user-rc in a single token, and it’s future-proof in a way that listing individual no-* options isn’t — when OpenSSH adds a new capability, restrict covers it by default and your hand-written list doesn’t.
It also disables PTY allocation, which breaks interactive shells, so pty goes back on the end.
The from= uses my LAN subnet plus the tailnet’s CGNAT range, matching an IP set the cluster firewall already uses.
The seventh key does it better
Since the audit I added one more key, for a scheduled job that checks every node and container for pending updates and reports to Telegram. It does not follow the standard above. It goes further:
command="/usr/local/sbin/pve-patrol",from="192.168.1.70",restrict ssh-ed25519 AAAA...Two differences matter. The from= is a single address rather than a range, because unlike my laptops this caller is a container that never moves. And command= pins the key to exactly one program. Present that key and the server runs the patrol script, whatever you actually asked for. There is no shell behind it. Note also that pty is absent, because nothing here needs an interactive terminal.
This is what I should have been doing all along for anything automated. A forced command is not a narrower version of root access, it is a different thing entirely: the key stops being a login and becomes a single button. It also quietly fixes the audit problem I complain about further down. I cannot tell you what any of my other keys did in a given session, but this one can only ever have done one thing.
The catch is that it only works where the job is a known, fixed task. An AI coding agent needs a real shell precisely because I cannot enumerate in advance what it will need to run. That is the honest limit of this approach, and it is why the other three keys still carry a shell.
Four things that cost me time
On a Proxmox cluster, /root/.ssh/authorized_keys is a symlink. It points at a single cluster-shared file. Editing it on any one node changes all three. I ran my edit three times before working that out, and then spent a while confused that nodes two and three reported “0 lines rewritten” while already showing the new options. Standalone hosts outside the cluster still need their own edit.
ssh -L cannot test whether you’ve successfully blocked forwarding. This one is genuinely misleading. A local forward binds a port on your machine and only contacts the server when something actually connects to it, so ExitOnForwardFailure never fires and a correctly-restricted server looks like it’s still allowing tunnels. You have to test with a remote forward, which needs a server-side bind and fails immediately:
ssh -o ExitOnForwardFailure=yes -R 19999:127.0.0.1:22 root@HOST "echo ALLOWED"
# expected: "Error: remote port forwarding failed for listen port 19999"If that prints ALLOWED, your restriction isn’t working, whatever -L told you.
Never pin exact IP addresses in from=. One of my machines moves between three addresses depending on how it’s connected. A DHCP change against a pinned address locks you out of a host silently, and you find out the next time you need it. Use ranges.
Arm a rollback on any host with only one way in. Before editing, schedule the restore, then cancel it from a new connection once you’ve proven the new one works:
cp -a /root/.ssh/authorized_keys /root/.ssh/authorized_keys.bak-DATE
nohup setsid bash -c "sleep 180; [ -f /tmp/keep-new-key ] || \
cp -a /root/.ssh/authorized_keys.bak-DATE /root/.ssh/authorized_keys" >/dev/null 2>&1 &
# apply the change, then from a brand new session:
ssh root@HOST "touch /tmp/keep-new-key"Three minutes is enough to test and not so long that you sit around waiting.
Ping is not identity
I nearly did the laptop’s key while the laptop was switched off.
That morning, both of its addresses answered ping while SSH timed out, which reads as “the machine is up, sshd is unhappy”. Later, once the laptop was genuinely powered on and reachable, those same addresses stopped answering ping. Something else had been holding the DHCP leases and answering on its behalf.
Confirm identity with a login that tells you the hostname. Never with a ping.
The related discovery was that my plan’s safety rule — do this while the laptop is on the home LAN, so there’s a fallback path — bought me nothing. ufw on that machine blocks inbound SSH on its LAN addresses, while Tailscale bypasses ufw entirely. The tailnet was the only way in whether it was home or not. The armed rollback was the actual safety net, and the LAN rule was comfort.
What this does not fix
Scoping controls where a key can be used from and what it can tunnel. It does nothing about what the key can do once it’s connected.
All of these are still root. And because everything logs in as root, there is no per-key audit trail at all — I can see that a key was used and from where, but every session looks identical once it lands. That’s why the count of 77 logins told me a machine and not a purpose.
Genuinely reducing blast radius means non-root users with a sudo allowlist on each host, which container management makes awkward enough that I haven’t done it. I looked at replacing shell keys with scoped API tokens and rejected it, because that removes shell access rather than narrowing it, and shell access is the thing I actually need.
So this is a partial fix, honestly labelled. It closes credential theft from outside the two ranges I trust. It does not close what happens after a legitimate connection.
Seven weeks later, it had already slipped
This draft sat unpublished for seven weeks. Before finally posting it, I took the advice in the last paragraph and read the files again.
The cluster’s shared file is exactly as I left it: seven keys, every agent key scoped, the patrol key pinned to its one script. The two files that aren’t shared were a different story.
The standalone backup server still had the dead keys. In July I pruned two grants belonging to machines that no longer exist (neither hostname resolves anywhere on my network), plus a stale key the server had issued to itself. That prune went into the cluster’s shared file. The standalone server keeps its own file, the prune never reached it, and all three were still there in October as unrestricted root. I’d even written a note flagging them in August and not acted on it. The warning about standalone hosts is in this very post, a few sections up.
The host I built after the audit has an unrestricted key. A fourth Proxmox machine joined in September, standalone, outside the cluster. Two of the three agent keys on it carry the standard prefix. The third, the one from my desktop, is bare: root from any address, with every kind of forwarding. Nothing broke, so nothing told me.
That second one is the real finding. Scoping the existing keys was a one-off job, and it was done properly. The standard did not survive the first new host, because it lived in my notes and in the files I had already edited, not in the steps I follow when I build a machine. An audit fixes a state. Only a habit fixes the next host.
The mystery caller behind the laptop’s key is still unidentified, too. The key is still scoped, and still on the list.
The actual lesson
The eleven duplicates weren’t a breach. They were the absence of an inventory.
Every one of those keys was created deliberately, for a reason I could have explained at the time, and then never revisited. Agent tooling makes credentials cheap to issue — a key here so it can reach the cluster, a key there so it can reach the laptop — and nothing in the workflow ever asks you to go back and count.
Set a reminder to read your own authorized_keys files. Not to fix anything. Just to read them, and see whether you can still name the owner of every line. Start with the machines you’ve added since the last time you looked, because that’s where the standard slips first.