Nazeem.Me

A blog about technology, football and all the other random stuff in my life

Tag: monitoring

  • I Built a Dashboard for the Things I Actually Check

    Five separate logins became one page. The interesting part was how often I turned out to be wrong.

    I check five things regularly: whether the homelab is healthy, my portfolio, my health trend, whatever trip is coming up, and my to-do list. Each one lived somewhere different. Each had its own login. None of them talked to each other. I have to go looking for information by opening a browser, going to my bookmarks page and finding the right tile to click:

    My Heimdall Dashboard of links to my self-hosted services.

    Information you have to go and fetch is information you check less often than you should.


    What It Looks Like Now

    One page. Five modules, each running in its own container, pulled together by a dashboard shell called Glance. The shell is configuration, not a product — it reads each service’s API and renders the numbers. That’s the whole trick (admittedly greatly helped by LLMs to design):

    The architecture for my data dashboard

    The health module was the biggest piece by some distance. I exported four years of Fitbit history and imported it alongside live Garmin data, so the trend line runs continuously from March 2022 to today instead of restarting when I changed watches. About 7 million data points. There’s a real gap of six days in January where neither device recorded anything, which is the week I switched. I’ve left it in. It’s honest.

    The finance module pulls positions from my broker and reconciles them against manually entered accounts for the things a broker doesn’t know about — Provident Fund, cash, the rest. Everything now sits behind a single login.


    Three of My Four API Guesses Were Wrong

    This is the part worth writing down.

    I planned the whole thing on paper first (with the help of AI, of course, including this post), including which endpoint each module would call and which fields it would read. When I came to build it, three of the four were wrong.

    The monitoring tool returned a flat structure where I’d assumed a nested one (I needed the LLM to explain this to me too!). The finance app had moved its performance endpoint to a new API version, and the old path returned a 404. The travel app used different field names to what I was expecting.

    None of these would have crashed anything. Each would have quietly rendered an error tile on a dashboard I’d then have trusted. The fix was boring and it worked: read a live response from every endpoint before writing a single line of template code.


    The Service I Decided Not to Build

    The last gap was authentication. Several services were reachable on my network with no login at all, and the obvious answer was to put single sign-on in front of everything.

    So I drafted the whole build. Container, secrets, rollout order, backups. Then I actually tested my assumptions and found that most of the services I’d listed as unprotected were, in fact, protected. The real list was less than half of what I’d written down (i.e. in my issue backlog in my Obsidian Vault). A lot of the services were things I wanted open so I could use them on the fly on an ipad or my phone.

    I closed the gap with tools already running instead. I stopped proxying my router’s admin interface entirely rather than putting a login in front of it, and turned off open registration on one app that had it. That took an afternoon and added nothing new to maintain.

    A new service is a permanent cost. Backups, updates, secrets to store, another thing that breaks at 2am. Since I already stayed up late last night to figure out why Jellyfin was transcoding when no one was using it, I figured “Simplex sigillum veri“.


    What I’d Tell Someone Starting

    Verify before you build. Every assumption I wrote in my .md vault was cheap to check and expensive to get wrong.

    Back up the databases properly. A snapshot of a running database is a snapshot of a database mid-write. Every module here dumps itself fifteen minutes before the nightly backup, and each dump refuses to complete unless it passes an integrity check. A backup you haven’t verified is a hope.

    And scope the problem before you scope the solution. I nearly built an identity provider to solve a problem that turned out to be four services and a router.

  • Homelab 1.0: Compute, Storage, and Power

    I’m back for my annual update on the state of my home network (state-of-the-network?)

    This year, I’ve gone in at the deep-end with a full 3-node Proxmox cluster for compute, and expanded from 2 to 3 storage servers.

    Compute

    Starting with Proxmox, I experimented with re-purposing an old i7 4770K build, but it would have been cost prohibitive to run it full time as it idle’d at 60 watts. That would cost me $15 per month, while giving me less performance than a modern solution.

    I picked up 2 mini pc’s with AMD H 255 processors (8 cores, 16 threads), 32GB RAM, 2.5Gbe networking and 1TB NVME storage. These were the Beelink Ser9 and the GMKTEC K12. I also picked up some used PC parts (i5 7500, 16GB RAM, NR200 Case) for $70 and cobbled together a third node for the Proxmox cluster.

    The 3 Proxmox nodes run the following containerised services:

    • pi-hole
    • pialert
    • openspeedtest
    • NUT (for monitoring UPS status)
    • heimdall dashboard
    • karakeep
    • beszel
    • netvisor
    • sterling-pdf
    • termix

    In addition, they also host 4 virtual machines (VMs):

    • Windows 11
    • Linux Mint
    • Home Assistant
    • Docker (for Portainer)

    I have configured high-availability (HA) across the 2 primary nodes, with the third node acting as the quorum breaker. These services help me monitor my home, network and systems, store and edit notes and test different operating systems.

    Storage

    For storage, I added a third Network Attached Storage (NAS) device – a Synology DS1621+, to complement my existing stack – a Synology DS920+ and an Unraid server. In total, across the 3 devices, I have 150TB of storage space. Data is replicated on all three. My critical data (~1TB) is also backed up automatically to the cloud. The media stack (photos and video) runs natively on my storage devices, as it avoids having to move data from my compute nodes to storage (even though I have a multi-Gig network).

    I plan to eventually move the media stack to the compute nodes, but the challenge will be the end-user user interface. I looked at Immich for the photo backup solution, but will continue to use Synology Photos as it provides a smoother user experience. I’m also stuck at mounting the storage devices to my compute nodes. This is entirely due to my lack of knowledge, for now.

    To save electricity costs, I aggressively spin down the drives on the Synology units, as well as turn-off the Unraid server entirely. It serves as my cold backup that I spin up once a month to back up data. By doing this, I effectively have a 3-2-1 backup strategy in play.

    Power

    All my network, storage and compute devices are connected to Uninterruptible Power Supplies (UPS), so that in a power outage, my systems have time to automatically power down. The NUT server will get a warning from one of the UPS, and trigger all devices to shut down if power is not restored in 5 minutes.

    Overall, across my 3 server nodes and 3 storage servers, I average at approximately 120W. If I include the network stack (2 x routers, 2 x 10G switched and 3 x 1G switches), the total average power consumption comes up to 200W, which equates to $39 per month, and accounts for 20% of my total electricity bill. I will need to find a way to reduce this running cost over the long term.