Seconds of Work, Hours of Residency

Two mornings in a row, my local observer produced a partial briefing and recorded the language model as unavailable. Ollama was running. The service was healthy. Every check I had said the stack was fine.

It returned HTTP 500 because the model needed 14,191 MiB and the card had 124 MiB free.

One GPU, several tenants

A single 3090 hosts everything here: the local LLM that writes the briefing, a cross-encoder reranker, a resident worker for the content pipeline, and image generation for social posts. Most of the time this is fine, because most of them are small or bursty.

Two things were not fine.

The first was a night job that overran into the working day — a job queued by a scheduler inherits a fresh duration budget, not the remaining wall-clock window, so a task meant to finish before dawn ran four hours into the morning. I had already fixed that one, and verified it holding.

The second was the interesting one.

Twenty-one seconds of work, all day of residency

The content scheduler renders cards and carousels through a local diffusion server. The render takes about 21 seconds and uses about 6.6 GB. Then it finishes, logs a cheerful "0 models unloaded", and keeps the VRAM.

Seconds of work. Hours of residency. And because the job succeeded, nothing anywhere reported a problem — the only symptom was a different service, hours later, failing to find room.

There is a naming detail I like here, because it is the same defect I keep finding: the timer is described as planning the day's content, the service is described as publishing scheduled content, and what it actually does is generate on the GPU. Nothing in the name suggested it was a GPU tenant at all. If you had asked me which units competed for the card, I would not have listed it.

Fixing the second one: two changes, both about giving memory back

Neither of these is the night-job fix, which was already done. Both address the resident render server.

Release what you are done with. The scheduler now calls the diffusion server's free endpoint on exit, guarded on the queue being empty so it can never unload models out from under a job that is still running. It is non-fatal and preserves the original exit code — a publish must never fail because a memory release did.

Stop reserving what you never use. The observer was requesting a 32k context window for a prompt that measures about 7.6k tokens, which reserved a 5,440 MiB key-value cache it would never touch. Halving the context dropped the cache to 2,720 MiB and the total requirement from 14,191 MiB to 11,327 MiB — and moved the model from 37 of 41 layers on the GPU to all 41.

The second fix is the one worth internalising. I did not add memory or remove a tenant. I stopped reserving memory on the basis of a default I had never questioned. The worst realistic daytime case now fits with room to spare even if the first fix never fires.

The error message was lying, politely

Before any of this, the briefing recorded the model as "unreachable." It was not unreachable. It answered immediately, with a refusal.

Those are different problems with different fixes, and collapsing them into one word cost me the first morning entirely. "Unreachable" sends you to look at networking, ports, and whether the container is up. "Refused" sends you to look at what it was asked for. I changed the label the same week I changed the memory.

An error string is a diagnosis, and a wrong diagnosis in a log is worse than no diagnosis, because you believe it.