Michael Hospedales

Software engineer / Miami, FL

The notebook

Writing

Small thoughts, things I learned the hard way, and ideas that needed more room.

Follow via RSS — every new thought, field note, and essay in one feed.

A few places to start

Browse the notebook

Clear filters

14 pieces about Linux & hardware

Essay

The cache hit that answered the wrong question

I turned prefix caching back on for a 125B hybrid model on a DGX Spark, with the fixes carried from five open pull requests. Hits landed, a number planted deep in the prompt came back exactly every time, and an 11,000-token prompt dropped from 5.99 s to 0.96 s. Then one request answered the question two other requests were asking. Proving that was not a leak took a better instrument than the one I started with.

Essay

I grafted a speculative decoding head into a 90 GB model file

The model card advertised a 4B multi-token-prediction head. No published GGUF contained it, and the architecture had no code path to run it. Both halves arrived within 36 hours from two different strangers, in incompatible forms — so I merged the head into the target file myself. It went from 27.75 to 43.30 tok/s, and three of my four mistakes along the way were about verification, not tensors.

Essay

The kernel kills your inference server first, and by default

An 87 GiB model got OOM-killed with nothing in its own log. The process that triggered it was a 27 MB dashboard service. The reason my inference server was the kernel first choice was not its size — it was a systemd user-manager default that scores every terminal-launched process to die before any system service. Then restarting it killed my editor.

Essay

Tokens per second told me the wrong model was faster

A 125B model decoding at 26.6 tok/s looks unremarkable beside a 31B at 29.3 — until you notice the 31B is running a drafter, so its number is a forward-pass rate multiplied by an acceptance length. Per forward pass the big model was 3x faster. Here is how to decompose the number, and what the bandwidth arithmetic says is still on the table.

Essay

I turned off concurrency and the server got faster

Four slots served my agentic pipeline at 1.2% prefix-cache reuse. One slot served the same pipeline at 86.7%, and saved 11.6 minutes of prefill in a 39-minute window. Slots schedule requests; they do not add compute — and on a single GPU they scatter the one thing a prefill-bound workload actually depends on.

Field note

pgrep -f matched my own shell, and nohup did not survive the timeout

Two process-management traps that both look like a crashed inference server. pgrep -f matches the command line of the process doing the grepping, so a stop script can kill its own shell and a wait loop can "confirm" a server stopped when it did not. And nohup guards against SIGHUP, not the SIGTERM a timing-out parent sends to its process group — so a 90-second model load inside a 2-minute command budget is killed at exactly two minutes and looks identical to a load failure. Have the child write its own PID, and use setsid.

Field note

DefaultOOMScoreAdjust=200 in the systemd user manager kills terminal-launched servers first

A production llama-server process was killed by the kernel with nothing in its own log: no assert, no stack trace, just silence, noticed only because /metrics stopped responding. The killed process carried oom_score_adj:200. The cause was not the coding-agent session that launched it — it was a systemd user-manager default, DefaultOOMScoreAdjust=200, which scores every terminal-launched process to die before system services do, even ones many times smaller. That score cannot be lowered after launch by an unprivileged process, so the fix is to run the server as a systemd system service instead, which defaults to OOMScoreAdjust=0.

Field note

Detecting which inference engine owns a port: vLLM vs. llama.cpp

Detection that infers the engine from the port number breaks the moment either engine moves. Moving llama.cpp onto the port normally used by vLLM mislabeled it, and downstream code silently wrote null. /props is a clean discriminator: llama-server serves it, and the vLLM OpenAI-compatible server has no such route and returns 404. A second gotcha in the same detour: docker ps --filter publish= does not match a container using host networking.

Field note

RSS is the wrong instrument for memory on a DGX Spark

The llama-server process shows about 29 GiB of RSS while about 87 GiB of weights for Qwen3.8-Flash-Next sit in CUDA buffers that RSS never counts, and nvidia-smi reports [N/A] for memory on GB10 because there is no separate GPU pool to report. MemAvailable before and after start is the honest measurement, and that holds for every model on a DGX Spark, not just this one. The idle headroom measured at launch was not production headroom either: the same configuration was later killed by the kernel under real load, with nothing in its own log, and working defaults moved to a smaller context size and ubatch afterward.

Field note

Every FP8 GEMM traps on DGX Spark (SM121)

vLLM guards its FP8 CUTLASS kernels on __CUDA_ARCH__ == 1200 and executes a deliberate trap instruction on anything else. DGX Spark is arch 1210, so every FP8 GEMM crashes. Two lines fix it.