Question the measurement
My LLM judge was flipping a coin a third of the time
What happened when I tested the judge I had trusted for months.
The notebook
Small thoughts, things I learned the hard way, and ideas that needed more room.
Follow via RSS — every new thought, field note, and essay in one feed.
Question the measurement
What happened when I tested the judge I had trusted for months.
Follow a fix
An exact failure, the hardware behind it, and the two-line fix.
Look at the craft
The engineering behind a quieter page transition on this site.
30 pieces
I build to solve problems. I have creative interests outside of programming, like art and music. I used to make my own 3D models and assets for Quake and Unreal, back in the day…
The model card advertised a 4B multi-token-prediction head. No published GGUF contained it, and the architecture had no code path to run it. Both halves arrived within 36 hours from two different strangers, in incompatible forms — so I merged the head into the target file myself. It went from 27.75 to 43.30 tok/s, and three of my four mistakes along the way were about verification, not tensors.
An 87 GiB model got OOM-killed with nothing in its own log. The process that triggered it was a 27 MB dashboard service. The reason my inference server was the kernel first choice was not its size — it was a systemd user-manager default that scores every terminal-launched process to die before any system service. Then restarting it killed my editor.
A 125B model decoding at 26.6 tok/s looks unremarkable beside a 31B at 29.3 — until you notice the 31B is running a drafter, so its number is a forward-pass rate multiplied by an acceptance length. Per forward pass the big model was 3x faster. Here is how to decompose the number, and what the bandwidth arithmetic says is still on the table.
Four slots served my agentic pipeline at 1.2% prefix-cache reuse. One slot served the same pipeline at 86.7%, and saved 11.6 minutes of prefill in a 39-minute window. Slots schedule requests; they do not add compute — and on a single GPU they scatter the one thing a prefill-bound workload actually depends on.
Fetching a newly published mmproj moved refs/main to a fresh snapshot directory containing only that one file, while 93.6 GB of weights stayed in the previous snapshot. Any launcher resolving through refs/main — which is the recommended practice, since snapshot hashes change on every re-pull — then fails its own existence check on weights that are plainly on disk. The fix downloads nothing: a second hf download with the original include pattern relinks the existing blobs into the current snapshot.
llama-server divides the context size by the slot count unless --kv-unified is set: n_ctx_seq = n_ctx / n_seq_max. There is no error and no warning. The same command line that asks for --ctx-size 262144 --parallel 4 serves 65536 per sequence, and the only evidence is one line in the boot log. Long prompts then fail or shift in ways that look like a model problem rather than a configuration one.
A published Qwen3.8-Flash-Next MTP head loads against PR #27739 and aborts against PR #27836 with "MTP block missing nextn.hc_head_norm". The head is not damaged and no tensor is missing: the same three hyper-connection tensors are exported as top-level output_hc_norm/down/up by one PR and expected as blk.N.nextn.hc_head_norm/down/up by the other, because a standalone head file has no trunk to collide with and a grafted head does. The published merge script filters to blk.* and silently drops all three. Rename them instead of dropping them.
llama-server accepts per-request speculative.n_max, p_min and type, which makes sweeping a drafter on one load look safe. It is not: the per-request value appends to the launch-time list rather than replacing it, so the echoed settings read speculative.types = none,draft-mtp and every row of a six-configuration sweep was the same configuration. The "off" arm reported 198 drafted tokens. A speculative-decoding baseline needs a server restart, and the echoed settings block is the only thing that reveals it.
Two process-management traps that both look like a crashed inference server. pgrep -f matches the command line of the process doing the grepping, so a stop script can kill its own shell and a wait loop can "confirm" a server stopped when it did not. And nohup guards against SIGHUP, not the SIGTERM a timing-out parent sends to its process group — so a 90-second model load inside a 2-minute command budget is killed at exactly two minutes and looks identical to a load failure. Have the child write its own PID, and use setsid.
I have swapped the local model under my newspaper 85 times in four months. That looked like a natural experiment, so I built an instrument to read it — including a gate that refuses comparisons contaminated by time. The gate passed. It was reading my switch log instead of my writing, and the two models it cleared had never once alternated.
I used a language model to pick the better of two drafts, and trusted it for months. Then I gave it two drafts from an identical configuration and asked it to choose. Here is what measuring a judge’s noise floor costs, and why every A/B result before it was unreadable.
Three times this year a reasoning model spent its whole token budget thinking and handed back empty content under HTTP 200. Three times, the code reading the result invented a different explanation — a parser bug, then an unconstrained grammar. Neither was true, and one of them fired inside a safety check.
A production llama-server process was killed by the kernel with nothing in its own log: no assert, no stack trace, just silence, noticed only because /metrics stopped responding. The killed process carried oom_score_adj:200. The cause was not the coding-agent session that launched it — it was a systemd user-manager default, DefaultOOMScoreAdjust=200, which scores every terminal-launched process to die before system services do, even ones many times smaller. That score cannot be lowered after launch by an unprivileged process, so the fix is to run the server as a systemd system service instead, which defaults to OOMScoreAdjust=0.
Detection that infers the engine from the port number breaks the moment either engine moves. Moving llama.cpp onto the port normally used by vLLM mislabeled it, and downstream code silently wrote null. /props is a clean discriminator: llama-server serves it, and the vLLM OpenAI-compatible server has no such route and returns 404. A second gotcha in the same detour: docker ps --filter publish= does not match a container using host networking.
An engine-swap safety check asked a reasoning model for a JSON verdict inside a 40-token budget. Reasoning consumed the whole budget, content came back empty, and the probe scored that as a schema that does not bind. The grammar was fine. A three-arm test isolates budget from grammar to show why.
A quantized KV cache (--cache-type-k q8_0) crashes qwen4exp twice, not once. The first crash is a Hadamard-rotation gate that qwen4exp does not implement; disabling that rotation produces a server that loads, saves 4 GiB, and matches f16 on every quality test run against it, then fails a second, different assert once real concurrent load arrives across all four slots. That second assert looked quantization-specific at first. It is not: the identical assert was later confirmed on f16 too, with no quantization involved, so the real cause is a multi-slot desync in llama_memory_hybrid_idx that q8_0 merely reaches sooner. Verdict: f16, run with --parallel 1 until upstream fixes the desync.
On August 26, the converter disabled MTP export and the architecture lacked its inference path. This historical field note now links to the August 28 follow-up, which documents working speculative decoding using a third-party head, a draft PR, and a local GGUF graft.
In my August 26 setup, llama.cpp master could not load the qwen4exp architecture. Building PR #27742 ran the 125B model at 22-27 tok/s without speculative decoding. This dated investigation now links to the August 28 experiment that added a working drafter through a local GGUF graft.
The llama-server process shows about 29 GiB of RSS while about 87 GiB of weights for Qwen3.8-Flash-Next sit in CUDA buffers that RSS never counts, and nvidia-smi reports [N/A] for memory on GB10 because there is no separate GPU pool to report. MemAvailable before and after start is the honest measurement, and that holds for every model on a DGX Spark, not just this one. The idle headroom measured at launch was not production headroom either: the same configuration was later killed by the kernel under real load, with nothing in its own log, and working defaults moved to a smaller context size and ubatch afterward.
Muse-Glimmer delimits reasoning by channel markers rather than a token pair, so vLLM cannot resolve reasoning token ids, so reasoning_config.enabled stays false, so the budget cap never engages — on a model whose chat template reasons by default.
Every completion reports reasoning_content: "" and reasoning_tokens: 0, even when a hundred tokens of reasoning were generated. The parser returns the reasoning when called directly. The defect is somewhere between the two, and I have not found it.
How this site went from a heavy ink-wipe page transition to a 300ms breath through paper — WebGL bokashi gradation, mathematical opacity guarantees, and the fail-safes that keep navigation working when the shader does not.
A KV-cache batching optimization calls cuMemcpyBatchAsync with a signature the CUDA 13.0 aarch64 driver API does not have. The build stops before anything runs.
One model needs --moe-backend cutlass and is slow under marlin. Another needs VLLM_SCALED_MM_BACKEND=marlin and crashes without it. NVIDIA officially recommends marlin. All three statements are true.
FlashInfer ships a precompiled fused-MoE library built for SM120. Its TMA descriptors do not initialize on SM121, so concurrent MoE batches crash after hours of clean operation. vLLM cannot patch it.
vLLM guards its FP8 CUTLASS kernels on __CUDA_ARCH__ == 1200 and executes a deliberate trap instruction on anything else. DGX Spark is arch 1210, so every FP8 GEMM crashes. Two lines fix it.
Mistral NVFP4 traps with an illegal instruction at around ten hours. Qwen3.6 int4 deadlocks silently at around seven, still answering health checks at zero tokens per second. Whether these share a root cause is still open.
vLLM builds a Lark grammar enumerating every registered tool as an alternative. Past roughly 28 tools the guidance parser gives up, and the model generates tokens that never reach the client.
A change to how multimodal processors handle token inputs crashes Pixtral-architecture models during dummy-input construction, before the server ever accepts a request.