A ThinkCentre in my homelab kept hard-freezing and left nothing behind. This is how I narrowed it down, and why I swapped it out before I had the answer.
The symptom
My Proxmox cluster, thefellowship, runs on three small machines. Thorin, a Lenovo ThinkCentre M90q Gen 3 with an i5-12500T, carried the media side of the house: Jellyfin, qBittorrent, the *arr stack, and Frigate, the camera NVR.
On 27 September it stopped. There was no reboot and no panic. The power LED was on, the fan was spinning, and both network interfaces were dead. It froze again twice on 5 October, at 12:17 and 18:33, and once more on 6 October at 09:46. The gaps went from 8 days to 6 hours to 15 hours. It was getting worse.
The cluster did its job each time. Two of three votes is still quorum, so ser9 and nucbox carried on. But nothing on Thorin came back by itself, and the fourth freeze left it dark for about ten hours before I noticed. The watchdog never fired because I don’t run Proxmox HA. The software watchdog is armed but nothing enforces it, so a frozen node stays frozen.
Wrong theory #1: the kernel
journalctl --list-boots told a tidy story. Nine boots on kernels 7.0.14-8 through -17, across 53 days, all ended cleanly. Then came -19 and -20: three unclean ends in nine days. That has the shape of a regression, and the likely suspect was the i915 iGPU driver. Frigate runs OpenVINO object detection on the UHD 770, and Jellyfin uses the same GPU for transcoding.
So I pinned the last good kernel:
proxmox-boot-tool kernel pin 7.0.14-17-pveIt froze 14.5 hours later.
Nine boots is a small sample. Kernel -19 had also run 4.5 days clean once, which I’d waved away because it didn’t fit.
I’d misread the GPU too. gt_cur_freq_mhz sat at 1333 of 1450 MHz, which looked like an iGPU pinned near its ceiling. That file is the requested clock. The actual clock, gt_act_freq_mhz, read 0, and RC6 residency was about 98%. The GPU was asleep most of the time.
Making the next freeze talk
When a machine dies silently, the first job is to make the next death leave evidence. On Thorin:
- journald
SyncIntervalSec=15s. The default is five minutes, and the first freeze lost the last two minutes of logs. kernel.panic=10andkernel.softlockup_panic=1. Any fault the kernel can see now reboots the box instead of hanging it.kernel.printk = 5 4 1 7. Sends warnings to the console, not just critical messages.- netconsole to ser9. It runs over the physical NIC, not the bridge, with a
socatlistener writing to a log file on the other node.
I left out kernel.hung_task_panic. Thorin had hard NFS mounts, so a NAS update that took longer than two minutes would have panicked the host and restarted every guest on it.
The next freeze produced nothing. netconsole stayed silent, and kernel.panic=10 didn’t reboot the box. That silence is a finding: whatever froze Thorin happened below the point where the kernel can report anything. The suspects moved from software to hardware: RAM, the power brick, the board, or the iGPU itself.
An A/B test with a twin
I had a second M90q Gen 3, Balin, built for another project. It’s the same model with the same CPU. The plan was to move Frigate across and watch both boxes for ten days, since the longest gap between freezes had been eight.
| Thorin (no Frigate) | Balin (Frigate) | What it means |
|---|---|---|
| stable | freezes | Frigate’s workload is the trigger |
| freezes | stable | Thorin itself is faulty |
| stable | stable | Frigate plus something specific to Thorin’s hardware |
| freezes | freezes | Something common to the platform |
The twins weren’t identical, and I wrote down how. Balin ended up on a newer BIOS. Its main bridge sat on a different NIC. It also lost netconsole, because netpoll won’t run on a bridge port. Each box was also a single sample.
The result after 10 days: neither box froze. Thorin was fine without Frigate and Balin was fine with it. That’s the third row, the least satisfying one.
Things that broke along the way
The 10G card that stopped listening
I flashed Balin’s BIOS through fwupd. The capsule was ignored twice until I enabled the Lenovo setting that allows OS-initiated firmware updates. After the flash, the Aquantia AQC113C 10G card linked at 10 Gb/s and transmitted happily, but its hardware receive counters stayed at zero. On the next reboot it vanished from the PCIe bus. I replaced it with a Realtek RTL8125B 2.5G card, which measured 2.35 Gbit/s to my desktop.
Link state lies. Check the RX counters in ethtool -S.
GPU nodes that renumber
A monitor plugged in for the BIOS setup made the GPU come up as card1 instead of card0. Frigate’s container config said card0, so it failed to start. Switching to /dev/dri/by-path/...-card fixed that and started a new problem: Frigate crash-looped with Context was not initialized. Once by-path exists inside the container, Intel’s compute runtime enumerates GPUs only from the by-path/*-render nodes. If you pass the by-path card node, pass the render node too.
Two shares, one name
Balin’s ProxDrive1 storage pointed at a different NAS from the cluster’s ProxDrive1. Same name, same folder layout, and something was syncing small files between them. The backup looked like it had landed, but only the log file had. I streamed the archive directly instead and checked the sha256 at both ends.
The swap
With the A/B test inconclusive, I wanted Thorin on the desk for memtest. Running the cluster on two nodes meant zero failure tolerance, so Balin took Thorin’s seat.
- Corosync on its own NIC. Cluster traffic runs on the 1G onboard port, and production runs on the 2.5G bridge. A dead 2.5G card no longer drops the node out of the cluster.
- Empty before joining.
pvecm addrefuses a node that has guests. I moved Frigate’s config out of/etc/pve, joined, and put it back. Frigate was down for about five minutes. - Four nodes for ten minutes, not longer. Four votes needs three for quorum, so it tolerates one failure, the same as three nodes, with more hardware that can fail.
- Offline migrations. Bind mounts need
shared=1beforepct migratewill move a container. qBittorrent took 33 seconds, the *arr stack 1 minute 18, and Jellyfin 2 minutes 5. - Separate before you delete. I removed Thorin’s corosync config before running
pvecm delnode, so it can’t rejoin if it boots by accident. A separated node keeps a full copy of the cluster’s secrets, though: API tokens, 2FA config, the CA key, the other nodes’ SSH keys, stored passwords. I scrubbed all of it and regenerated the certificates before Thorin left the rack.
Balin’s first reboot in the rack exposed a boot race. Proxmox starts guests before it activates NFS storage. Jellyfin bind-mounts subfolders of a share, and those don’t exist until the share is mounted, so it failed to start. The other containers bind the share root, so they started against an empty folder, which is worse because nothing reports it.
The fix is a systemd drop-in on pve-guests that runs pvesm status to activate storage and waits up to 60 seconds for the mount. Its - prefix means a NAS outage delays boot but never blocks it. I proved it with a real reboot, because nothing short of one tests boot ordering.
What’s still unknown
memtest86+ passed on Thorin, so RAM is unlikely. That leaves the power brick, the board or iGPU, or the combined workload.
Balin now carries exactly what Thorin did before all this: Frigate and Jellyfin sharing one iGPU, plus the CIFS share for downloads. If Balin starts freezing, the workload is the cause. If it doesn’t, the fault is Thorin’s hardware, and the brick is next on the desk.
I’ve already muddied that test. I unpinned the kernel on Balin today, so its next reboot moves to 7.0.14-23. A freeze after that can’t be pinned on the workload alone.
What I’d tell myself in September
- Silence is data. If netconsole and a panic timer both stay quiet, stop reading kernel changelogs and start suspecting hardware.
- Test the theory before trusting it. One pinned kernel disproved a 53-day correlation in 14 hours.
- Read actual values, not requested ones. That means
gt_act_freqovergt_cur_freq, and RX counters over link state. - Never pass GPU nodes into an LXC by number. Use by-path, and include the render node.
- Separating a node isn’t decommissioning it. It walks away with your cluster’s keys.
- Boot ordering only fails on boot. Reboot to prove it.