A few weeks ago I retired Thorin, a Lenovo ThinkCentre M90q Gen 3, from my Proxmox cluster. It had hard-frozen four times between 27 September and 6 October. Each time the power LED stayed on, the fan kept spinning, and nothing reached the logs. Balin, its twin, took its place. I wrote that up in Four freezes, no logs: retiring a Proxmox node without knowing why it died.
That post ended with the cause still unknown. memtest86+ had passed, so RAM was unlikely. What was left was the board, the iGPU, the power supply, or the workload. So I put Thorin on my desk with no guests and tried to make it freeze on purpose.
The setup on the desk
Everything ran as one transient systemd unit, so a single command started it and a single command stopped it:
- CPU: stress-ng, 12 workers doing matrix multiplication across all 12 threads.
- RAM: two stress-ng workers holding 8.6 GB with
--verify, which checks every write it reads back. - iGPU media: two ffmpeg jobs decoding 1080p H.264 on the UHD 770, scaling to 720p and re-encoding, the same path Frigate and Jellyfin used.
- iGPU compute: clpeak over OpenCL, standing in for Frigate’s OpenVINO detector.
- NVMe: fio doing 4K random reads and writes on the system disk, rate-limited so a day costs about 0.4 TB of wear instead of 15.
The point of the test was to catch a freeze, so the capture mattered more than the load. Telemetry went out every 10 seconds: package temperature, clocks, power, GPU frequency, NVMe and NIC temperatures. It went to a local file and to the kernel log, and netconsole carried the kernel log to another node. If Thorin froze, the last readings would already be off the box.
I also armed every tripwire I could. The Intel hardware watchdog was set to reset the board after 10 seconds without a heartbeat. Soft lockups, hard lockups and oopses were all set to panic, and a panic reboots after 10 seconds. The unit never restarts itself after a reboot. A reboot mid-run is the result.
24 hours, nothing
The run started at 20:01 on 28 October and finished at 20:01 the next day.
| Check | Result |
|---|---|
| Boot | Same boot as at the start. No crash records in pstore. |
| Kernel log | Nothing but my own telemetry. |
| stress-ng | 14 workers passed, 0 failed, with memory verification on. |
| ffmpeg and clpeak | No errors. |
| fio | 1.68 TB read, 841 GB written, 0 errors. |
| Temperatures | Package peaked at 80 °C and held 77 °C at about 30 W. NIC peaked at 66 °C. |
The GPU’s render and video engines sat at 100% busy for the whole day at 1450 MHz, their maximum. That’s more than Frigate ever asked of it.
Adding the network
Both NICs had died together in every freeze, and the first run left the network idle. So I added a second load: iperf3 with four streams to my desktop’s 10G port, in both directions at once. The 10G card I’d lost on Balin had failed on the receive side, so send-only wouldn’t have been a fair test.
Before that run I flashed Thorin’s BIOS from M4GKT37A to M4GKT3AA (1.55 to 1.58), to match Balin. Lenovo’s own update catalog still lists the older 39A as the newest. The readme inside the 3AA package is what confirms it’s current.
The second run lasted an hour, on top of the full CPU, GPU and disk load:
- 2.34 Gbit/s each way, about 1.96 TB through the Realtek 2.5G port.
- 0 retransmits sending, and 1 to 21 per 10-minute round receiving.
- 0 NIC errors. The NIC missed 29 incoming packets, which is a short receive buffer overflow at line rate with every core busy.
- NIC temperature peaked at 70 °C.
No freeze.
Meanwhile, the twin
Balin has quietly run its own test the whole time. It ran Frigate alone on 7.0.14-17, one of the kernels Thorin froze on, for about ten and a half days. Since 17 October it has carried Thorin’s entire old workload: Frigate and Jellyfin sharing one iGPU, plus the CIFS share for downloads. That’s 15 days and counting, on kernels -24 through -26. Every boot ended cleanly.
That points at Thorin rather than the workload. It doesn’t close the case, because Balin never ran the full workload on -17.
Why “couldn’t reproduce” isn’t “fixed”
Freezes that come at random need enough test time to show up. Thorin froze four times in about nine and a half days, but the gaps were shrinking: 8 days, then 6 hours, then 15 hours. If the fault were still there at the same rate, a 24-hour pass could happen by luck:
| Freeze rate assumed | Chance of 24 h with no freeze anyway |
|---|---|
| The last two gaps, about 2 a day | about 12% |
| The whole run of freezes, about 0.42 a day | about 64% |
At the average rate, a faulty Thorin passes this test roughly two times in three. It only rules out the fastest rate.
The test also didn’t match the conditions it froze under:
- Kernel. It froze on 7.0.14-17, -19 and -20. Both runs were on -26.
- BIOS. The freezes were on 37A. The second run was on 3AA, which changed more than the version number (below).
- Workload. The load was synthetic. The real one had network shares, and one of them had stalled for three minutes on the day of the second freeze.
A clean week would mean more. At 0.42 freezes a day, seven days without one happens by chance about 5% of the time.
Things I tripped on
I applied power tuning in the middle of the burn-in. I rolled out a powertop --auto-tune service to every node on the morning of the 29th, and Thorin was on the list. It ran for 13 minutes before I noticed and reverted it. The worst part isn’t the slower clocks: auto-tune turns off the NMI watchdog, which is what reports a hard lockup. For those 13 minutes, a lockup would have gone unreported. Nothing happened, but I flagged that window in the results. Check that nothing is mid-test before you tune a box.
The BIOS flash changed more than the version. Intel’s thermal framework device disappeared from the PCI bus after the flash. It was what had held Thorin’s sustained power limit at 30 W. It has stayed gone on every boot since, and the limit now reads 35 W, the same as Balin. My guess is that the flash reset a setting to its default. It also renumbered the GPU from card0 to card1, which blanked the GPU column in my telemetry until I switched the script to the /dev/dri/by-path name. That’s the second time renumbering has bitten me on these boxes.
Some test plumbing that isn’t in the docs.
- ffmpeg’s
-stream_loopwith hardware frames died at every loop point. A concat playlist of the same file loops cleanly. - clpeak exited with code 255 on the old Clover OpenCL driver, so I pointed it at Mesa’s rusticl instead.
- ffmpeg’s
-tcounts media time, not wall time. Usetimeout.
A card that vanished on every warm reboot. Thorin had an Intel I226-V 2.5G add-on card as a spare. It dropped off the PCI bus after every warm reboot, three times out of three, and only came back after a full power cycle. It wasn’t part of the freezes, but it’s out of the box now.
Where that leaves Thorin
Thorin survived 25 hours of everything I could load onto it, with nothing in the logs. Balin has run the same workload for more than two weeks without a hiccup. The honest summary is that the fault hasn’t come back and probably belongs to that one machine. That’s not the same as being gone.
For now Thorin stays on my desk as a box for test VMs. netconsole and the hardware watchdog are still armed. If it freezes again, the watchdog should reboot it within seconds instead of leaving it dark for ten hours, and whatever the kernel manages to say will already be on another machine.