Upgrading Production DNS by Never Touching the Live Node

DNS is the service where a homelab stops being a hobby. Nothing in the house works without it, nobody in the house cares why, and the failure is total rather than partial.

Mine runs as a pair: two Pi-hole instances, each with its own recursive resolver, sharing a floating address managed by VRRP. One node is master with a higher priority and preempts; the other is a warm backup. Clients only ever talk to the floating address.

Recently the backup needed a major-version upgrade — a new base image that also dropped the bundled recursive resolver, so that had to become its own container too. Enough moving parts to be genuinely risky on the service that everything depends on.

The procedure is defined by which node you touch

The entire safety of this operation comes from one precondition, checked before anything else:

Confirm the master holds the floating address, and that the node you are about to work on is the backup.

If that is true, the backup's DNS can blink, restart, break, or be rebuilt from scratch with zero client impact, because no client is talking to it. If it is false — if the backup silently took over at some point and you did not notice — then the identical sequence of commands is a full house outage.

Same commands. Same host. Same everything. The difference between routine maintenance and an incident is one query you run first. I now write that check as an explicit gate at the top of the runbook, with a "do not proceed if" attached, because the version in my head is the version I skip when I am confident.

The health check is a contract

The failover health check was one line: can something connect to the web interface port.

That turns an implementation detail — the port the admin interface happens to listen on — into a load-bearing contract between two containers that do not know about each other. The new image defaults to a different port. Had I let it, the health check would have failed, the backup would have declared itself unhealthy, and I would have spent the evening debugging VRRP rather than DNS.

So the new deployment explicitly pins the web server to the old port. The configuration line looks arbitrary and vestigial. It is the only reason failover works, and I wrote a paragraph next to it saying so, because the next person to tidy up an "arbitrary" port binding is me in eight months.

Two nodes are not two configurations

Having a redundant pair creates a new problem: the two must agree. Block lists, local records and settings drift if both are edited, and drift in DNS produces the worst class of bug — behaviour that depends on which node answered.

The answer was to stop treating them as peers for configuration. One node is the source of truth, and a sync service pushes a selected subset to the other. Not everything: gravity and specific settings, deliberately scoped, because a full replication would also copy per-node identity and undo the thing that makes them separate machines.

Redundancy is a property of the data plane. The control plane should still have exactly one writer. Two Pi-holes I can both edit is not high availability, it is two Pi-holes.

What made this safe was boring

A backup taken before anything else, into a directory on the host, with the old configuration preserved beside it. Images pinned by digest instead of a floating tag, so a rollback goes back to the same bytes. Auto-update explicitly disabled on the DNS containers, because a service the whole house depends on should not change while I am asleep because somebody upstream pushed a tag.

None of that is clever. All of it is why a major-version upgrade on production DNS was a normal evening.