Tag · 4 posts
#incident
Everything I've written tagged Incident.
Seconds of Work, Hours of Residency
My morning briefing started failing. Ollama was up and returned HTTP 500, because a 21-second image render was still holding 6.6 GB of VRAM hours later.
The Alert That Named a Thing and Called It Evidence
Five high-severity alerts from my own monitoring. All five were the same defect: a record that names something being read as an observation of a property it never measured.
Upgrading Production DNS by Never Touching the Live Node
Two Pi-holes, a floating IP, and a major-version upgrade on the box that resolves everything in the house. The trick is that the safe procedure and the risky procedure look identical from the outside.
The Outage Was Never the Thing I Was Debugging
A homelab-wide outage at 01:45. I blamed the reverse proxy, then the IPS, then the SSO layer. The root cause was a network port down at the switch — everything I suspected was a symptom.