Homelab Second Brain

Problem

I wanted a local model, on my own GPU, to answer questions about my own infrastructure — what depends on what, why a thing is configured the way it is, what changed recently. Plain retrieval over my documentation cost 30,000 to 50,000 tokens per question and had no notion of time: a decision written in June and an observation from this morning came back with equal authority.

My role

Designed and built the whole thing solo, over roughly eight weeks, as 220 merged pull requests.

Constraints

• Must run entirely local. A 24GB consumer GPU that is frequently busy with other work.
• Must never become another source of truth. Git and live runtime observations stay authoritative; anything derived has to be deletable and rebuildable.
• Must be honest. An operations assistant that confabulates is worse than no assistant, because you cannot tell a thin answer from a wrong one.

Architecture

A compiled layer over authoritative sources: a temporal fact store that closes facts rather than overwriting them, a relationship graph (940 nodes, 1,449 edges), generation-stamped summary capsules, and a vector index — assembled into a token-budgeted context packet.

Retrieval is staged and stops early: exact, then lexical BM25, then capsule, then graph, then semantic. Conflicts resolve by fixed precedence, runtime over fact over capsule, and a derived artifact whose generation lags its source is marked stale so the query path falls back to raw evidence.

On top sit the analysis layers: decay (which documented decisions are no longer true), correctness validation against live sources with a committed baseline and a CI regression gate, drift reconciliation between declared and observed state, blindspot detection, causal and multi-hop reasoning, and a gated detect-propose-approve-act loop whose executor stops at a draft pull request and never merges.

Throughout, one rule: grounded or blocked. Selection and ranking are deterministic and reproducible offline. A language model may phrase an already-selected result and may never add, drop, or reclassify one.

Outcome

Around a 99% reduction in tokens per query, validated in shadow mode against the previous approach before promotion. 31 command-line tools, 93 read-only tools exposed over Model Context Protocol, and 293 tests over the memory subsystem alone within a 116,000-line Python operations layer carrying 1,865 tests.

What I'd do differently

I built the fact store before I built the thing that audits whether the facts are still true. Those belong in the same release — a memory without self-audit is a confident cache. And I would have separated "what did I observe" from "what do I conclude" in the detector contract from the start; conflating them produced a batch of well-cited alerts that were all the same bug.