A Second Brain That Refuses to Guess

My homelab runs 14 hosts and around 110 containers. I wanted a local model on my own GPU to answer questions about it — "what depends on the DNS box", "why is this service configured this way" — without shipping any of it to a hosted API.

The naive version worked and was unusable. Plain RAG over my docs meant every question dragged 30,000 to 50,000 tokens of loosely-ranked chunks into context. On a 24GB card that is most of your budget spent before the model has thought about anything. Worse, the retrieved chunks had no notion of time: a decision record from June and a runtime observation from this morning came back with equal authority.

The actual problem

It wasn't retrieval quality. It was that I had no representation of what is true right now, separate from what was written down once.

Three specific gaps:

• No lexical stage. Embedding similarity alone is bad at exact identifiers. Ask about a specific service name and you get thematically-similar neighbours instead of the row that names it.
• No reranking. The top-k by cosine distance is not the top-k by usefulness.
• No temporality. Nothing distinguished "this is the current state" from "this was the state when someone wrote the doc."

What I built

A compiled memory layer. The important word is compiled — it is a derived, deletable, rebuildable view over sources that stay authoritative. It is explicitly not another database to keep in sync:

Git + runtime observations   <- authoritative
Temporal facts (SQLite)      <- current vs historical, close-don't-overwrite
Context graph                <- relationships (940 nodes / 1,449 edges)
Vector index                 <- semantic search
Hierarchical capsules        <- 400-1,200 token summaries, generation-stamped
2-5K context packet          <- what the model actually sees

Retrieval runs staged and stops early: exact match, then lexical (SQLite FTS5/BM25 — stdlib, no extra service to operate), then capsule, then graph, then semantic. Most operational questions never reach the semantic stage.

Conflict resolution is a fixed precedence: runtime beats fact beats capsule. If a derived artifact's generation stamp lags its source, it is marked stale and the query path falls back to raw evidence rather than serving a confident summary of a world that has moved on.

The reranker, and why the default is boring

Three backends. Deterministic scoring by relevance x priority x freshness is the default at roughly half a second, fully offline and reproducible. A listwise LLM pass costs 2-5 seconds. A real cross-encoder on the 3090 costs about 3 seconds warm, 11 cold.

The constraint that made this safe to ship: a reranker may only reorder candidates. It may never add or drop one. That is enforced in tests. Any backend failure degrades to the deterministic order instead of erroring. This means the interesting-but-flaky path can never take down the boring path.

The result

Roughly a 99% reduction in tokens per query, and answers that carry citations. The measurement that mattered more: I ran the compiled path in shadow mode against the old RAG path before promoting it, so "better" was a number and not a vibe.

What I'd do differently

I built the fact store before I built the thing that checks whether the facts are still true. Those should have shipped together — a memory with no self-audit is just a confident cache. That correction became its own project, and its own post.