Michael Hospedales

Software engineer / Miami, FL

The notebook

Writing

Small thoughts, things I learned the hard way, and ideas that needed more room.

Follow via RSS — every new thought, field note, and essay in one feed.

A few places to start

Browse the notebook

Clear filters

10 pieces about Evaluation & measurement

Essay

The kernel kills your inference server first, and by default

An 87 GiB model got OOM-killed with nothing in its own log. The process that triggered it was a 27 MB dashboard service. The reason my inference server was the kernel first choice was not its size — it was a systemd user-manager default that scores every terminal-launched process to die before any system service. Then restarting it killed my editor.

Essay

Tokens per second told me the wrong model was faster

A 125B model decoding at 26.6 tok/s looks unremarkable beside a 31B at 29.3 — until you notice the 31B is running a drafter, so its number is a forward-pass rate multiplied by an acceptance length. Per forward pass the big model was 3x faster. Here is how to decompose the number, and what the bandwidth arithmetic says is still on the table.

Field note

A per-request speculative.type of "none" still drafted 198 tokens

llama-server accepts per-request speculative.n_max, p_min and type, which makes sweeping a drafter on one load look safe. It is not: the per-request value appends to the launch-time list rather than replacing it, so the echoed settings read speculative.types = none,draft-mtp and every row of a six-configuration sweep was the same configuration. The "off" arm reported 198 drafted tokens. A speculative-decoding baseline needs a server restart, and the echoed settings block is the only thing that reveals it.

Field note

pgrep -f matched my own shell, and nohup did not survive the timeout

Two process-management traps that both look like a crashed inference server. pgrep -f matches the command line of the process doing the grepping, so a stop script can kill its own shell and a wait loop can "confirm" a server stopped when it did not. And nohup guards against SIGHUP, not the SIGTERM a timing-out parent sends to its process group — so a 90-second model load inside a 2-minute command budget is killed at exactly two minutes and looks identical to a load failure. Have the child write its own PID, and use setsid.

Essay

My guardrail checked the config, not the corpus

I have swapped the local model under my newspaper 85 times in four months. That looked like a natural experiment, so I built an instrument to read it — including a gate that refuses comparisons contaminated by time. The gate passed. It was reading my switch log instead of my writing, and the two models it cleared had never once alternated.

Essay

My LLM judge was flipping a coin a third of the time

I used a language model to pick the better of two drafts, and trusted it for months. Then I gave it two drafts from an identical configuration and asked it to choose. Here is what measuring a judge’s noise floor costs, and why every A/B result before it was unreadable.

Essay

The model ran out of room to think, and every diagnostic guessed wrong

Three times this year a reasoning model spent its whole token budget thinking and handed back empty content under HTTP 200. Three times, the code reading the result invented a different explanation — a parser bug, then an unconstrained grammar. Neither was true, and one of them fired inside a safety check.

Field note

DefaultOOMScoreAdjust=200 in the systemd user manager kills terminal-launched servers first

A production llama-server process was killed by the kernel with nothing in its own log: no assert, no stack trace, just silence, noticed only because /metrics stopped responding. The killed process carried oom_score_adj:200. The cause was not the coding-agent session that launched it — it was a systemd user-manager default, DefaultOOMScoreAdjust=200, which scores every terminal-launched process to die before system services do, even ones many times smaller. That score cannot be lowered after launch by an unprivileged process, so the fix is to run the server as a systemd system service instead, which defaults to OOMScoreAdjust=0.

Field note

Detecting which inference engine owns a port: vLLM vs. llama.cpp

Detection that infers the engine from the port number breaks the moment either engine moves. Moving llama.cpp onto the port normally used by vLLM mislabeled it, and downstream code silently wrote null. /props is a clean discriminator: llama-server serves it, and the vLLM OpenAI-compatible server has no such route and returns 404. A second gotcha in the same detour: docker ps --filter publish= does not match a container using host networking.