Building a break-glass network that survives its own failures

There's a particular kind of anxiety that comes with running your own infrastructure remotely. Everything's humming along, you're a few hundred kilometres away — maybe in another country — and a nagging question sits at the back of your mind: if the thing I use to reach my servers goes down, how do I reach my servers?

That circular dependency is the whole problem. Most of us start with a single remote-access path — a VPN server running on one of our own machines. It works beautifully right up until the machine hosting it is the thing that breaks. Then you're locked out of your own house with the key inside.

This is the story of how I dug my way out of that trap, and ended up with something I'm genuinely happy with: a layered, break-glass setup where no single failure can lock me out. I'll walk through the architecture, the access layers, and the firewall philosophy that ties it together.

None of the addresses, ports, or keys below are real. The shapes are.

The layout

Two physical sites, each with its own separate internet uplink and its own public IP.

Site A — home. This is where the main compute lives. Two servers: Caradhras, the primary that runs most of my services, and Edoras, an always-on secondary that handles the things that must never blink — mail, a reverse proxy, DNS filtering, and a VPN server. Both sit on the same LAN, call it 10.0.A.0/24.

Site B — another flat I am renting. This holds Rhun, a mirror backup server in a locked ventilated cabinet — off-site backup that isn't in the same building as the thing it's backing up, which is the entire point of off-site backup. Its own LAN, 10.0.B.0/24, its own router, its own uplink.

The two sites are stitched together with two separate tunnels between the routers — one GRE, one IPsec — and static routes on each side decide which one carries traffic. GRE is the preferred path; if it drops, the routes fail over to IPsec automatically, and the preference can be reversed whenever I want to steer traffic the other way. LAN routing works in both directions, so each site can reach the other's network as if it were local. Belt and suspenders on the inter-site link itself.

Both sites run the same make of prosumer router, which matters because it means one mental model, one config language, one set of habits across the whole estate.

Topology at a glance

The four ways in, at a glance:

1. OpenVPN on Edoras — everyday convenience (and DNS filtering); dies if Edoras dies.
2. WireGuard on Router A — break-glass at home; survives losing any server.
3. WireGuard on Router B — break-glass off-site; survives losing the entire home site.
4. IP-KVMs — console-level access when a server is up but won't boot.

The access layers

Here's the resilience thesis, stated plainly: I want more than one independent way in, and I want them to fail independently. If they all depend on the same box, I don't have redundancy — I have the illusion of it.

So the setup has three access layers, deliberately chosen to fail for different reasons.

Layer 1 — the everyday VPN (convenience)

Edoras, the always-on secondary at Site A, runs an OpenVPN server with a handful of client profiles. Beyond remote access, routing a phone through home means its traffic passes through my DNS filtering, which quietly eats ads and scam domains before they ever load. It's the convenient daily-driver path back into the network.

But — and this is the crux — this VPN runs on a server. If Edoras is down, this path is down. It's convenient, not resilient. Which is exactly why it can't be the only door.

Layer 2 — the break-glass VPN (resilience)

This is the layer I'm proudest of, and it's almost embarrassingly obvious in hindsight: put a VPN endpoint on the router itself.

The routers are the most reliable devices in the whole setup. They're purpose-built, they have no moving parts, they're already the thing terminating the internet connection, and they're up whenever the internet is up. If I can reach my public IP at all, the router answered — so a VPN that terminates on the router works in exactly the scenario where a server-hosted VPN doesn't.

I put a lightweight modern tunnel — the fast, boring, reliable kind — on each router, one per site, each on its own public IP. Two endpoints, not one. This is the part that makes it genuinely break-glass rather than just "a second VPN":

• If Caradhras or Edoras dies, Router A's tunnel still lets me in.
• If the entire home site has a problem, Router B's tunnel is a completely independent entry point on a different uplink in a different building.
• Either site is reachable even if the other is a smoking crater.

Because the two sites are already linked by the inter-site tunnels, once I'm into either site's break-glass VPN I can route across to the other site's LAN too. That's a convenience bonus — but I keep the framing honest with myself: the independence comes from each site having its own endpoint. Reaching Site B through Site A is a nice-to-have; reaching Site B directly is the guarantee. A break-glass tool that chains through the very thing that might be broken isn't break-glass.

Small but important detail: because these are static public IPs, the client configs just point at raw addresses. No dynamic-DNS dependency, nothing else to fail. Fewer moving parts is the entire aesthetic here.

Layer 3 — the console of last resort (hardware)

The final layer answers a nastier question: what if a server is up on the network but broken — hung at boot, wedged in the BIOS, a bad kernel, a borked network config? No amount of VPN helps if the OS itself won't talk.

For that, I added tiny IP-KVM devices — little RISC-V gadgets that plug into a server's HDMI and USB and give you its screen and keyboard over the network, exactly as if you were standing at the machine. They don't care what the server's OS is doing, because they're impersonating a monitor and a keyboard at the hardware level. Server won't boot? I can watch it fail to boot and fix it. Need to reinstall the OS remotely? They can mount an install image as a virtual USB drive. At home a single KVM is wired across both servers' console and power, so one cheap device covers Caradhras and Edoras alike; the backup site has its own.

They are, in effect, my hands and eyes at the physical console — reachable from the same VPNs. And they're the layer that made me think hardest about trust, which brings me to the firewall.

The firewall philosophy: guilty until proven innocent

These IP-KVM devices are wonderful and I do not trust them an inch.

They're inexpensive devices from a vendor with a documented history of security sloppiness — default credentials that aren't forced to change, services that phone home, DNS settings that quietly reset themselves on every boot. And I've wired them into the console and power of my servers. That's a lot of implicit trust to hand a cheap gadget that wants to talk to the internet.

So the governing principle became: the management devices get zero internet access. They can be reached; they cannot reach out.

Concretely, on each router I built an address group covering the block of IPs these management devices live in, and an egress rule that drops any connection they try to initiate toward the internet — while still permitting replies to connections I initiate toward them. In plain terms:

• I open a KVM's web UI from my VPN -> allowed (it's a reply to my inbound connection).
• The KVM tries to phone home, fetch an update, or re-assert some DNS server -> dropped.

The "allow established/related, then drop the rest" ordering is the standard trick, and it's what lets inbound management coexist with total outbound lockdown. The device is a roach motel: packets check in, they don't check out.

One habit worth stating out loud, because it's easy to skip: after writing an egress rule, check the packet counter. A drop rule bound to the wrong interface silently matches nothing — the block looks "on," but the device merrily updates itself. A rule that never sees traffic is worse than no rule, because it looks like protection. Confirm the counter climbs; then you know it's real.

Keeping time without reaching out

A neat side-effect of the total egress block: the management devices couldn't reach public time servers, so their clocks drifted and their logs were stamped with nonsense dates. The wrong fix is to poke a hole in the firewall for NTP. The right fix is to point them at an NTP source they're already allowed to reach — the local router. The router keeps good time; the devices sync from the router over the LAN; no egress required; logs get honest timestamps; the lockdown stays intact.

Small thing, but it's the whole philosophy in miniature: when isolation creates a problem, solve it inside the isolation rather than breaching it.

The foundations underneath

The access layers get the attention, but they sit on top of a few less glamorous things that do a lot of quiet work — the parts you only notice when they're missing.

Power, two deep. The home site rides on two UPS units in series, chosen to cover different failure durations rather than duplicate each other. First in line is a large portable power station — the kind of battery you'd take camping — sized to carry the whole rack (a ~400 W draw) for roughly two hours. That's the "the street lost power, ride out the outage" tier. Behind it sits a proper line-interactive UPS (an Eaton Ellipse PRO 1600) which only holds the load for about five minutes on its own — but brings the things the big battery doesn't: automatic voltage regulation, surge and line conditioning, and clean signalling for graceful shutdowns. Big-battery-then-conditioner in series means I get both long runtime and clean power, instead of picking one. It's the same "independence of failure" idea applied to electricity: two devices covering two different problems.

Two providers, two buildings. Because each site has its own uplink from its own ISP, I effectively have provider and geographic diversity for free. An outage or routing mess at one provider doesn't touch the other. Policy-based routing keeps the important traffic behaving sensibly across the estate.

Hardening at the edge. The routers do the unglamorous perimeter work: dynamic blocklists for known-bad address ranges, automated banning of hosts that come knocking too persistently (the classic fail2ban pattern), and firewall/routing rules aimed at cutting down the background noise of the internet before it reaches anything that matters.

Eyes on the physical layer. The environment is watched by CCTV with on-device AI doing real-time object detection, so "someone is physically near the gear" is a thing I find out about promptly rather than in hindsight. Remote access is only half of security; the other half is knowing what's happening at the actual location.

None of this is exotic on its own. The point is the stack — each layer assuming the one below it might fail, and covering for it.

What resilience actually bought me

Step back and look at the failure modes this setup now shrugs off:

• Main server (Caradhras) dies -> Router A's break-glass VPN still lets me in; the KVM lets me fix it at the console.
• Always-on secondary (Edoras) dies, taking the everyday VPN and DNS filtering with it -> the router VPNs are untouched; I get in and sort it out.
• Entire home site offline -> Router B's VPN is an independent door on a separate uplink at Site B.
• A server is up but won't boot -> the IP-KVM is my hands at the keyboard.
• A management device misbehaves or is compromised -> it's already walled off from the internet; the blast radius is contained by default.
• Inter-site GRE tunnel drops -> static routes fail the inter-site traffic over to the IPsec tunnel automatically.
• Mains power cuts out at home -> the big battery carries the rack for a couple of hours, with the line-interactive UPS conditioning the feed and handling a graceful shutdown if the outage outlasts it.

No single box, when it fails, takes down my ability to respond. That's the property I was chasing. Not "nothing ever breaks" — things break constantly — but "when something breaks, I can still get in to fix it."

Closing thought

The through-line of this whole build, if there is one, is independence of failure. It's not enough to have backups, or redundant paths, or isolation — they have to fail for different reasons. Two VPNs that both die when one server dies aren't two layers; they're one layer wearing a disguise. The work is in making the layers genuinely orthogonal: different hardware, different sites, different uplinks, different trust assumptions.

Get that right, and remote infrastructure stops being a source of low-grade dread and becomes something you can actually relax about — because you've made sure that no matter which piece fails, there's always another way back in.