The notebook

Writing

Small thoughts, things I learned the hard way, and ideas that needed more room.

Follow via RSS — every new thought, field note, and essay in one feed.

A few places to start

Browse the notebook

30 pieces

Thought

Why I build

I build to solve problems. I have creative interests outside of programming, like art and music. I used to make my own 3D models and assets for Quake and Unreal, back in the day…

Essay

I grafted a speculative decoding head into a 90 GB model file

The model card advertised a 4B multi-token-prediction head. No published GGUF contained it, and the architecture had no code path to run it. Both halves arrived within 36 hours from two different strangers, in incompatible forms — so I merged the head into the target file myself. It went from 27.75 to 43.30 tok/s, and three of my four mistakes along the way were about verification, not tensors.

Essay

The kernel kills your inference server first, and by default

An 87 GiB model got OOM-killed with nothing in its own log. The process that triggered it was a 27 MB dashboard service. The reason my inference server was the kernel first choice was not its size — it was a systemd user-manager default that scores every terminal-launched process to die before any system service. Then restarting it killed my editor.

Essay

Tokens per second told me the wrong model was faster

A 125B model decoding at 26.6 tok/s looks unremarkable beside a 31B at 29.3 — until you notice the 31B is running a drafter, so its number is a forward-pass rate multiplied by an acceptance length. Per forward pass the big model was 3x faster. Here is how to decompose the number, and what the bandwidth arithmetic says is still on the table.

Essay

I turned off concurrency and the server got faster

Four slots served my agentic pipeline at 1.2% prefix-cache reuse. One slot served the same pipeline at 86.7%, and saved 11.6 minutes of prefill in a 39-minute window. Slots schedule requests; they do not add compute — and on a single GPU they scatter the one thing a prefill-bound workload actually depends on.

Field note

Downloading one new file from a Hugging Face repo invalidates the path to all the others

Fetching a newly published mmproj moved refs/main to a fresh snapshot directory containing only that one file, while 93.6 GB of weights stayed in the previous snapshot. Any launcher resolving through refs/main — which is the recommended practice, since snapshot hashes change on every re-pull — then fails its own existence check on weights that are plainly on disk. The fix downloads nothing: a second hf download with the original include pattern relinks the existing blobs into the current snapshot.

Field note

Dropping --kv-unified silently turns a 262144-token server into a 65536-token one

llama-server divides the context size by the slot count unless --kv-unified is set: n_ctx_seq = n_ctx / n_seq_max. There is no error and no warning. The same command line that asks for --ctx-size 262144 --parallel 4 serves 65536 per sequence, and the only evidence is one line in the boot log. Long prompts then fail or shift in ways that look like a model problem rather than a configuration one.

Field note

GGML_ASSERT(layer.nextn.hc_head_norm) — two MTP pull requests disagree about one tensor name

A published Qwen3.8-Flash-Next MTP head loads against PR #27739 and aborts against PR #27836 with "MTP block missing nextn.hc_head_norm". The head is not damaged and no tensor is missing: the same three hyper-connection tensors are exported as top-level output_hc_norm/down/up by one PR and expected as blk.N.nextn.hc_head_norm/down/up by the other, because a standalone head file has no trunk to collide with and a grafted head does. The published merge script filters to blk.* and silently drops all three. Rename them instead of dropping them.

Field note

A per-request speculative.type of "none" still drafted 198 tokens

llama-server accepts per-request speculative.n_max, p_min and type, which makes sweeping a drafter on one load look safe. It is not: the per-request value appends to the launch-time list rather than replacing it, so the echoed settings read speculative.types = none,draft-mtp and every row of a six-configuration sweep was the same configuration. The "off" arm reported 198 drafted tokens. A speculative-decoding baseline needs a server restart, and the echoed settings block is the only thing that reveals it.

Field note

pgrep -f matched my own shell, and nohup did not survive the timeout

Two process-management traps that both look like a crashed inference server. pgrep -f matches the command line of the process doing the grepping, so a stop script can kill its own shell and a wait loop can "confirm" a server stopped when it did not. And nohup guards against SIGHUP, not the SIGTERM a timing-out parent sends to its process group — so a 90-second model load inside a 2-minute command budget is killed at exactly two minutes and looks identical to a load failure. Have the child write its own PID, and use setsid.

Essay

My guardrail checked the config, not the corpus

I have swapped the local model under my newspaper 85 times in four months. That looked like a natural experiment, so I built an instrument to read it — including a gate that refuses comparisons contaminated by time. The gate passed. It was reading my switch log instead of my writing, and the two models it cleared had never once alternated.

Essay

My LLM judge was flipping a coin a third of the time

I used a language model to pick the better of two drafts, and trusted it for months. Then I gave it two drafts from an identical configuration and asked it to choose. Here is what measuring a judge’s noise floor costs, and why every A/B result before it was unreadable.

Essay

The model ran out of room to think, and every diagnostic guessed wrong

Three times this year a reasoning model spent its whole token budget thinking and handed back empty content under HTTP 200. Three times, the code reading the result invented a different explanation — a parser bug, then an unconstrained grammar. Neither was true, and one of them fired inside a safety check.

Field note

DefaultOOMScoreAdjust=200 in the systemd user manager kills terminal-launched servers first

A production llama-server process was killed by the kernel with nothing in its own log: no assert, no stack trace, just silence, noticed only because /metrics stopped responding. The killed process carried oom_score_adj:200. The cause was not the coding-agent session that launched it — it was a systemd user-manager default, DefaultOOMScoreAdjust=200, which scores every terminal-launched process to die before system services do, even ones many times smaller. That score cannot be lowered after launch by an unprivileged process, so the fix is to run the server as a systemd system service instead, which defaults to OOMScoreAdjust=0.

Field note

Detecting which inference engine owns a port: vLLM vs. llama.cpp

Detection that infers the engine from the port number breaks the moment either engine moves. Moving llama.cpp onto the port normally used by vLLM mislabeled it, and downstream code silently wrote null. /props is a clean discriminator: llama-server serves it, and the vLLM OpenAI-compatible server has no such route and returns 404. A second gotcha in the same detour: docker ps --filter publish= does not match a container using host networking.

Field note

A quantized KV cache crashes qwen4exp on llama.cpp, in two different ways

A quantized KV cache (--cache-type-k q8_0) crashes qwen4exp twice, not once. The first crash is a Hadamard-rotation gate that qwen4exp does not implement; disabling that rotation produces a server that loads, saves 4 GiB, and matches f16 on every quality test run against it, then fails a second, different assert once real concurrent load arrives across all four slots. That second assert looked quantization-specific at first. It is not: the identical assert was later confirmed on f16 too, with no quantization involved, so the real cause is a multi-slot desync in llama_memory_hybrid_idx that q8_0 merely reaches sooner. Verdict: f16, run with --parallel 1 until upstream fixes the desync.

Field note

The Qwen3.8-Flash-Next MTP head was missing from the August 26 GGUFs

On August 26, the converter disabled MTP export and the architecture lacked its inference path. This historical field note now links to the August 28 follow-up, which documents working speculative decoding using a third-party head, a draft PR, and a local GGUF graft.

Field note

qwen4exp is not qwen3next: the August 26 llama.cpp loading failure

In my August 26 setup, llama.cpp master could not load the qwen4exp architecture. Building PR #27742 ran the 125B model at 22-27 tok/s without speculative decoding. This dated investigation now links to the August 28 experiment that added a working drafter through a local GGUF graft.

Field note

RSS is the wrong instrument for memory on a DGX Spark

The llama-server process shows about 29 GiB of RSS while about 87 GiB of weights for Qwen3.8-Flash-Next sit in CUDA buffers that RSS never counts, and nvidia-smi reports [N/A] for memory on GB10 because there is no separate GPU pool to report. MemAvailable before and after start is the honest measurement, and that holds for every model on a DGX Spark, not just this one. The idle headroom measured at launch was not production headroom either: the same configuration was later killed by the kernel under real load, with nothing in its own log, and working defaults moved to a smaller context size and ubatch afterward.

Essay

A page transition that breathes

How this site went from a heavy ink-wipe page transition to a 300ms breath through paper — WebGL bokashi gradation, mathematical opacity guarantees, and the fail-safes that keep navigation working when the shader does not.

Field note

Every FP8 GEMM traps on DGX Spark (SM121)

vLLM guards its FP8 CUTLASS kernels on __CUDA_ARCH__ == 1200 and executes a deliberate trap instruction on anything else. DGX Spark is arch 1210, so every FP8 GEMM crashes. Two lines fix it.