Teaching a System to Say “I Don’t Know”

A retrieval system that always answers is worse than one that sometimes declines, because you cannot tell the two failure modes apart. A thin answer and a wrong answer look identical at the point of use.

So the operating rule across my homelab's memory layer is grounded-or-blocked: every conclusion must cite the specific fact, edge, or observation that put it there. An uncited conclusion is not softened with a hedge. It is dropped, and counted.

Where the rule shows up

Multi-hop synthesis. The reasoning layer chains decision records to entities to causal relations to graph topology. Ask what a given architectural decision governs, and it walks outward to dependents, then reports which of that blast radius has no documentation at all. Every hop cites the edge that produced it. A chain with one uncited hop is blocked entirely — not returned with a caveat. Only a fully grounded chain is allowed to be phrased by a language model, and the model's job is strictly phrasing. It never selects.

Self-audit. A decay pass re-checks every documented decision and runbook against the current fact store, classifying each as intact, contradicted, stale, or unverifiable. A contradiction has to name the fact that triggered it. If the verdict cannot cite, the verdict gets downgraded rather than surfaced. A separate correctness pass re-derives the world from live sources and re-checks every stored belief, with a committed baseline so that a belief which was true and has become contradicted fails a regression check in CI.

Blindspots. A set difference between the subjects the graph knows exist and the subjects the fact store actually covers, ranked by how central each is. This is the piece that lets the system say "I don't know this" instead of confidently serving thin context. It found, among other things, that my own NAS was a blindspot.

The part that surprised me

Making the rule strict increased the amount I trusted the output, obviously. What I didn't expect was that it also increased the amount of output. When blocked chains are counted rather than silently dropped, the block count becomes a work queue. "Seven conclusions blocked for want of a citation" is a better morning signal than seven plausible sentences.

Where the model is allowed to touch anything

Narrow, and by design:

• Selection, ranking, and classification are deterministic. Reproducible offline, no model in the loop.
• A model may phrase an already-selected result.
• A model may never add an item, drop an item, or change a verdict.

Every layer degrades to the deterministic path when the model is unavailable. My GPU is frequently busy generating video, and I would rather have a plain-language-poor briefing than no briefing.

The honest limitation

Grounded-or-blocked protects against fabrication. It does not protect against a well-cited claim built on a bad premise — where the citation is real but the thing it records was never an observation of the property being claimed. That failure took me five false alarms to see, and it is a different post.