The Outage Was Never the Thing I Was Debugging
A homelab-wide outage at 01:45. I blamed the reverse proxy, then the IPS, then the SSO layer. The root cause was a network port down at the switch — everything I suspected was a symptom.
Around 01:45 the homelab went down — not one service, all of them. This is the postmortem, including the part where my first fix didn't work and the reason it didn't.
What I suspected, in order
Traefik. Everything routes through it, so everything failing looks like the router failing. It was healthy.
CrowdSec. An IPS that had recently gotten more aggressive rules. A mass ban would look exactly like this. It hadn't banned anything relevant.
Authentik. Forward-auth sits in front of most internal services, so an SSO outage also produces total unavailability. Also healthy.
Three plausible causes, each of which would produce the observed symptom, all three innocent. That is the signature of looking at the wrong layer.
The failed workaround
Name resolution was clearly involved, so I moved to pin DNS to a known-good resolver instead of the floating virtual IP. It didn't take.
The reason is worth recording: I ran the redirect without root, so the systemd stub symlink stayed intact and my change was silently a no-op. Nothing errored. The system simply kept using the old path. I spent a while re-verifying a fix that had never been applied.
Second attempt, as root, pinned resolution to the real address of the primary DNS host and bypassed the keepalived VIP entirely. That restored service.
The actual root cause
A network port went down at the switch level. Not DNS, not Traefik, not CrowdSec, not Authentik. The DNS symptom was the loudest because name resolution fails fast and everything else fails behind it.
Later that day I confirmed the VIP and the primary DNS host shared the same MAC, which proved keepalived still owned the address correctly — the HA layer had been doing its job the whole time. It could not route around a dead physical link, which is not what it is for.
The follow-up that stung
One of the open items I wrote for myself was: did monitoring alert on this at all, and if not, add a check.
I have a dashboard, an uptime monitor, a metrics stack, a log aggregator and a push-notification service. I found out about a total outage by trying to use something. Coverage of services is not coverage of the layer underneath them.
What I keep from this
- The thing you suspect first is usually downstream of the thing that broke. Three healthy suspects in a row is information: stop investigating and start descending layers.
- Verify that your fix was applied before evaluating whether your fix worked. A silent no-op costs more than a loud failure.
- The postmortem keeps the failed attempt in it. Cleaning that out would have made a tidier document and a less useful one.
Written by
Adrian Romo
Senior Backend Engineer building scalable Python APIs, AWS Lambda architectures, voice systems, and enterprise integrations.
Related
Keep reading
The Alert That Named a Thing and Called It Evidence
Five high-severity alerts from my own monitoring. All five were the same defect: a record that names something being read as an observation of a property it never measured.
A Write Loop That Never Merges Anything
My homelab brain noticed problems and did nothing about them. Closing the loop meant four human gates and an executor whose defining feature is that it stops at a draft PR.
Teaching a System to Say “I Don’t Know”
The most useful thing my homelab brain does is refuse to answer. Grounded-or-blocked: every claim cites the fact that produced it, or it never reaches me.
Keep going
Where to next?
Browse more technical writing, see the engineering case studies, or reach out directly.