The Network Is the Layer I Documented Last

My homelab is documented to an unreasonable degree. Every host has a page. Every stateful service has a tested restore procedure. There is a relationship graph, a fact store, and an automated check that complains when documentation drifts from reality.

The network — the thing all of it runs on — has a config file whose contents are placeholders, and a warning at the top of it saying so.

What actually exists

The instrumentation is fine. There is an IPAM system running with its own database, workers and backups. There is a poller pulling metrics off the router and access points into Prometheus, dashboards in Grafana, blackbox probes, uptime monitoring and alert routing. A pair of DNS servers with automated failover. A reverse proxy whose routes are inventoried by a collector.

So the network is observed. It is not described. Those are different, and I had conflated them for a long time.

Why this layer, and not another

Because I document what breaks, and the network almost never breaks.

Application layers break weekly, in small recoverable ways, each one producing a fix and a note. That is a documentation flywheel: frequent, low-stakes failure generates writing. The network breaks roughly never — and when it does, it takes everything with it, which is precisely the moment you have no capacity to write anything down.

The result is an inverse relationship between how much a layer matters and how well I have described it. The most critical layer generates the least writing, because it is the most reliable. That is not a personal failing so much as a structural property of learning by incident, and it is worth knowing that your documentation is densest where your problems were smallest.

The most useful thing in that file is the warning

The network file is placeholders. What it also has, at the top, is an explicit banner saying every value is a placeholder and must be confirmed against the router and the hypervisor before anyone believes it.

That banner is doing more work than most of my real documentation.

A file full of plausible-looking values with no provenance is actively dangerous — someone reads it, believes a subnet, and plans a change around a number nobody ever verified. The same file that announces its own unreliability is safe, because it cannot be mistaken for evidence. It is a known gap instead of an unknown lie.

I have a whole system built around the principle that a claim which cannot cite its source should be blocked rather than hedged. That banner is the same rule, applied by hand, before I built the machinery. The cheapest honest thing a document can do is tell you not to trust it yet.

The outage that proves the point

The one time the network did break, it took the entire homelab down at quarter to two in the morning. I blamed the reverse proxy, then the intrusion prevention layer, then the single sign-on service. The root cause was a port down at the switch, and every single thing I suspected was a symptom of it.

I have never written a network page. I did write a postmortem for that night, and it is the most-read thing I have about my own network — which tells you exactly what the flywheel rewards.