Every app I have written about here, plus the one's I haven't the photo server, the password vault, the grocery bot, the home dashboards, all of it sits on top of the same boring foundation. So here is the foundation. Two servers in my house, tied together into one cluster, plus a third machine fifteen miles away at my mom's house that ended up doing a job I did not plan for when I dropped it off.

The two boxes at home

The cluster is called ProxCluster and it is two Proxmox nodes. proxserver1 is the workhorse. It has 48 cores and a pile of RAM, and it runs basically everything: the virtual machines, the containers, the storage pools, a couple dozen guests at this point. proxserver2 is the little one, 12 cores and 23 gigs, and it mostly stands by to take over if the big one needs to go down for maintenance.

There used to be a third node in the rack. It was a loud dual-socket 1U that pulls real power, and it now stays powered off as a cold spare. I turn it on when I actually need the extra capacity, and the rest of the time I would rather not pay to keep it spinning. That decision is what created the interesting problem in the first place.

The two-node problem

Two nodes is worse than one node in one specific way.

Proxmox uses a voting system to decide whether the cluster is healthy enough to keep running. It needs a majority of the votes, more than half. With two nodes and one vote each, if either node drops off you are down to one vote out of two. For you non-math majors out there, one is not a majority. So the surviving node does the responsible thing and freezes itself, because it has no way to know whether its partner actually died or whether the two of them just lost the network between them and are both still running. If they both kept going and both wrote to the same disks, you would corrupt everything. So it stops.

A two-node cluster does not survive losing a node. It needs a tie-breaker.

The fix is a third vote, and the nice part is that the third vote does not have to be a third full server. Proxmox lets you run a tiny thing called a QDevice. It gets exactly one vote, it takes part in no actual work, and it exists only to break ties. So I needed to park a third voter somewhere. And it happened that I had a machine about to leave the house anyway.

The box at my mom's house

I built a fourth machine, prox4, out of the guts of my old gaming PC. The original plan for it was simple and had nothing to do with the cluster. I wanted an offsite backup target. If my house floods or burns or gets broken into, I do not want every copy of my data to be in that one building. So the plan was to drive prox4 over to my mom's, plug it in, and let my cluster ship backups to it every night.

It runs Proxmox Backup Server. It has four 4-terabyte drives in a RAIDZ2 pool, which means two of the four can die before I lose anything. Every night my cluster sends its backups across an encrypted tunnel to that pool.

The part I actually like is that the exact same box is also the cluster's third vote. It is running the QDevice. So one cheap machine sitting in my mom's house is quietly doing two completely unrelated jobs at the same time. It is where my backups go, and it is the tie-breaker that keeps my cluster alive when a node dies back home. I did not set out to combine those two things. It just turned out that the box I needed offsite for backups was also exactly the box I needed offsite for a quorum vote, so it does both.

Terminal output of pvecm status showing the two home nodes and the offsite Qdevice, with three total votes and the cluster quorate
That bottom line just labeled Qdevice, with no address next to it, is the machine at my mom's. It has a vote and it is fifteen miles away.

Because that vote exists, either server in my house can fall over now and the cluster barely reacts. Two votes at home plus one vote across town is three total, and any two of them still make a majority. The thing that used to freeze my whole setup is just not a problem anymore.

The Tailscale glue

None of this works without Tailscale. The box at my mom's is on her network, behind her router, on a totally different subnet than mine. I do not have a static address over there, and I am not going to start forwarding ports on my mother's internet connection. Tailscale builds a direct encrypted link between my house and hers, and everything rides over it: the nightly backups, the quorum votes, and my own remote access when I need to log in and check on it. The round trip is around 55 milliseconds, which you would never notice.

One detail there matters more than it looks. The quorum vote cannot depend on anything that lives inside the cluster, because the entire point of it is to survive a cluster node going down. So Tailscale runs directly on the two Proxmox hosts themselves. Early on I had it running inside a virtual machine, which was tidier, until I realized that if the node hosting that VM died, the tunnel would die with it, and so would the offsite vote. Now the tunnel lives on the metal, underneath everything, so it stays up even when a node does not.

The backups themselves

The backups run at one in the morning, every night, on every VM and every container. Proxmox Backup Server deduplicates as it goes, so after the first big run each night only ships the blocks that actually changed since yesterday. The pool at my mom's is about 7 terabytes usable and it is sitting around 12 percent full right now, so there is a lot of room to grow. On top of the nightly job there is garbage collection on the weekend, a pruning schedule that keeps a week of dailies, a month of weeklies, and six months of monthlies, and a weekly verify pass that re-reads everything on disk to make sure none of it is quietly rotting.

The stuff I genuinely cannot replace gets a third copy beyond even that. The family photos and the home videos live on the NAS at home, they go to the box at my mom's every night, and they also get pushed encrypted up to cloud storage. Three copies, two of them off the main machine, one of them completely out of the house. That is the old 3-2-1 rule.

The upshot

None of this is fancy. It is two servers, a powered-off spare, and an old gaming PC in a my mom's basement. But I like how it turned out. The machine I sent offsite to hold my backups is also the machine that keeps my cluster from freezing, and a spare PC at my mom's house is now a real part of my disaster recovery plan. She thinks it is just a quiet little computer that hums in the basement. It is actually the reason my house could lose a server, or lose the entire rack, and I would barely feel it.