About the lab
Here there be servers. This is a homelab I built by hand, predominantly “pre-AI” (funny what a watershed that’s become). It’s also a direct result of blood, sweat, and tears (all three literal and a story for another time). It consists of one main server running Proxmox, a backup box running Proxmox Backup Server, an office PC running OPNsense, and a segmented home network.
For the past two-ish years my setup was rock solid as long as I didn’t touch it. I set up every container and service by hand, and it ran with relative reliability until it came time for a new piece of the puzzle. Cracks started showing and I found myself feeling like each new service required an increasing amount of effort to get working with everything else.
To learn new tools as well as make my homelab more resilient and flexible, I’ve made it my mission to refactor things one piece at a time. This involves things like: moving manual installs to Ansible as the opportunities present themselves, being more intentional and consistent with my documentation (as much as one can be), and closing the many, many gaps I find along the way. This blog is the record of that work and a testament to the countless hours I have invested in a poorly cooled stack of rust, scrap, and duct tape.
The map
Hardware
| Machine | What it is | Job |
|---|---|---|
| Main node | AMD Ryzen, 64 GB RAM, Intel Arc A380, ZFS pools on SSD and HDD | Runs every service, as LXC containers and VMs |
| Backup server | An OptiPlex 3050 (i5-7500) | Proxmox Backup Server |
| Router / firewall | An OptiPlex 7040 (6th-gen i5) running OPNsense | Routing, firewall, VLANs |
| Switch | Managed, PoE | Carries the VLANs |
| Wi-Fi | One Wi-Fi 7 access point | Hopefully self-explanatory… |
Network
The network is split into VLANs: trusted devices, the lab, IoT, work, the rest of the household, guests, and a DMZ. The firewall then decides what can talk to what.
I assign every service an internal name using a DNS rewrite, and the reverse proxy serves them over HTTPS with a wildcard certificate for an internal domain. This means I never type (or memorize) an IP address or click through a certificate warning.
There are no port forwards and no tunnels, so nothing is exposed to the internet. Away from home, I connect over Tailscale. While not in the spirit of “own everything” and FOSS, it is ridiculously easy to set up and use. Another added benefit is that it’s relatively painless (ymmv) to get other people on board. A few friends and family can reach select services through Tailscale sharing, and only the containers those services run on, not the rest of the network.
What runs on it
~thirty containers, in four groups:
- Media: Jellyfin (hardware transcoding on the A380), Audiobookshelf, an ebook library, and YouTube archiving.
- Home and life: Home Assistant (with MQTT and Zigbee), Actual Budget for budgeting, LubeLogger for vehicle maintenance, and Syncthing.
- Dev and AI: an AI stack for experimenting, a Discord bot I wrote, a Docker test host, and a sandbox VM for security learning.
- Infrastructure: DNS, the reverse proxy, the Tailscale subnet router, file shares, a Docker host for compose stacks, the Wi-Fi controller, latency and speed monitoring, and a dashboard.
How it’s run
- Code public, data private. This split may be a little unconventional, but I’m weird about privacy and security. Sue me. The Ansible roles and playbooks are in a public repo. The inventory, variables, and secrets live in a separate private repo, and a pre-commit check is in place to keep identifying details out of the public one.
- Unlike the USPS: names, not addresses. This is something I’ve been doing for a while, but haven’t been consistent about until recently. Everything points at names, so moving a service means changing one DNS entry and not the config of every service that points at it.
- Ansible where I’ve converted it (and the dark ages everywhere else). The media server, reverse proxy, AI stack, and a file share are managed by Ansible. The rest was set up by hand and each service gets converted the next time I touch it or it looks at me funny, whichever comes first. Here’s the full list.
- Backups loading… Every container and VM is backed up nightly to the backup server, with retention and weekly verification. The data they serve (file shares, my desktop recordings, the media server’s state) and the host’s own config aren’t covered yet, and nothing is off-site. And yes, I’m aware the backup server has exactly one disk. Fixing that is the next big project, and it’ll get a post when I feel things are complete.
How it got here
I estimate this whole thing started around the beginning of 2022. It has gone through many iterations since, and I have nuked everything to start from scratch on more than one occasion. Since then, my current lab (roughly v3 I’d say) has been a Ship of Theseus: replaced one harvested part or eBay purchase at a time. Like most long-lived systems, this lab has held up with varying degrees of grace and a number of shouted curses.
It may sound lame, but this homelab has been an important part of my life and something that has helped me grow and grown alongside me. We’ve been through 4 apartments together (so far), and I deeply believe that my life would be in a very different place had I not started this hobby.